E2E & Stress Testing using Protractor

· 5 minute read
Protractor + Bash

Protractor is a testing framework for web applications that allows developers to write a collection of E2E test suites. E2E tests are typically used to simulate a user’s experience whilst interacting with the application, and can often take a long time to run. This is due to the design of the E2E tests placing emphasis on thoroughly replicating the various sequences that a user may follow when interacting with the application, thus typically making assertions over a larger amount of the codebase than unit level testing. Given the long execution time of E2E tests it is often recommended to test as much as possible at the unit level, and to rely on E2E tests purely to simulate real user scenarios. Google themselves reinforce this belief by suggesting a 70/20/10 split: 70% unit tests, 20% integration tests, and 10% end-to-end tests. Following this advice can help to drastically reduce the overall run-time of your testing phase and speed up the continuous delivery process, however there are still optimisations to be made to the remaining E2E tests that can reduce the run-time even further.

Parallelisation

A well-written test specification makes assertions that are independent of the application’s existing state by ensuring that its setup and teardown processes define the context that the specification’s assertions are to be run within. Tests that are intertwined with one-another often lead to maintenance issues such as a reliance on execution order; if the execution order changes as a result of refactoring then the previously passing tests will now begin to fail for reasons other than the test’s assertions not being fulfilled.

Protractor builds upon this principle by encouraging you to make your tests independent at the file level in their official Style Guide. In addition to the benefit of clear & concise test specifications in each file, Protractor facilitates a technique known as “test sharding” that relies on your tests being independent at the file level. This technique involves modifying your Protractor’s configuration to define a maximum number of browser instances that Protractor can instantiate to run tests in, as shown below.

// protractor.conf.js
exports.config = {
    capabilities: {
        browserName: 'chrome',
        shardTestFiles: true, (1)
        maxInstances: 10 (2)
    },
    // remainder omitted
};
  1. To enable the sharding of tests at the file level, we must configure the shardTestFiles flag within the capabilities to return true.
  2. The maxInstances number represents the maximum amount of browser windows that Protractor is allowed to create. For example, with maxInstances set to 10, a test suite of 200 tests would result in Protractor creating 10 Chrome instances with each instance running 20 tests, whereas without sharding this would be one Chrome instance running all 200 tests.

With Protractor now configured correctly, each browser will place a lock on any file that they have used during the execution phase to ensure that the tests are not duplicated. This is necessary as the browsers are not capable of communicating with one another and thus cannot share the list of specifications they have already processed.

Stress Testing

With the E2E tests running in parallel, an opportunity presents itself to run a large amount of parallel tests at once over a prolonged period. Doing so can place a web application under significant stress, emulating the environment that it will be working in when targeted users are interacting with it in production. This technique is referred to as “Stress Testing” and its inclusion within a deployment pipeline can help to identify any performance issues before the product is released, analysing parts of the application that are particularly unresponsive when placed under significant stress.

With this goal in mind we can write two simple bash scripts to instantiate the E2E tests. These bash scripts are ideal for continuous integration solutions such as Jenkins which provide build steps to run arbitrary shell scripts. The first script will fork child process to run npm run e2e in the current directory N amount of times. For example, calling ./e2e.sh 5 will create a child process that runs npm run e2e 5 times before terminating.

#!/bin/bash
N=$1 (1)

for i in `seq 1 $N`
do
    npm run e2e (2)
done
  1. The N variable represents the amount of times to run the E2E tests. It is read as a the first program argument.
  2. The reference to the e2e npm script can be changed if you have a different task registred to run your E2E tests.

As we now have a bash script that will create a child process to run the E2E tests a certain amount of times, we can now build upon this by creating another bash script that will create M amount of child processes at once. For example, calling ./e2e-parallel.sh 2 10 will create 2 child processes, both of which run npm run e2e 10 times.

#!/bin/bash
M=$1 (1)
N=$2 (2)

if [ -z $M ] || [ -z $N ]
then
    echo "You must specify both parameters"
    exit 1
fi

for i in `seq 1 $M`
do
    nohup ./e2e.sh $N &> ./out/e2e-tests-$i.out & (3)
done
  1. The M variable represents the amount of child processes to be forked.
  2. The N variable represents the amount of times that each child process should run npm run e2e.
  3. The output of each child process is written to a file under the ./out directory.

With the scripts above configured correctly you can simulate an environment that places your web application under a significant amount of load, at which point you may wish to run a further set of tests to ensure that the application is still responding despite being swarmed with requests from the child processes.