End-to-end tests with docker-selenium

I spent quite a bit of time at work last year thinking about end-to-end test infrastructures. The main idea behind end-to-end tests is to use browser automation to check whether a web page has rendered correctly, and behave correctly under user interactions.

18 months ago, I first set up a simple architecture using mocha and webdriver.io. Mocha is the main test runner, and as it invokes webdriver.io, the latter will communicate with a selenium server running in the background and send along commands to the browser. What this looks like is, when you run a command like

:; mocha test/e2e/index.js

it will open up a browser of your choice, interact with the browser as instructed, such as scrolling, clicking, getting element text content, assert any result, close the browser window and finish the test.

6 months ago, I updated to using the wdio test runner instead, which works with mocha, so the majority of the test code continued to work. It is a bit nicer to use as it provides a global handle to call browser commands from, and thus avoids a bit of boilerplate code to set that up (as much as I usually dislike anything global).

Up until now, the way our tests are set up requires a selenium server to be run in the background, and we use selenium-standalone for that, which is quite pleasant. This however becomes an annoyance sometimes for a couple reasons. The selenium version and browser drivers versions were not tracked in our source code, so sometimes the tests could be broken all of a sudden if a bug is introduced in the latest version, or an outdated version is installed. It is also kind of cumbersome to have a separate terminal window open to run this process, or use a process manager to keep it running in the background, especially when many are used to running test with a simple npm test command.

Now comes the exciting part – the reason for this blog post in the first place. I put together a relatively simple script that could eliminate the need for this background selenium process using docker-selenium. The idea is that a selenium docker container is spun up before each test run so the test will connect to it via a port (default to 4444), and then removed after the test has finished (either successfully or otherwise). The script looks something like this:

#!/usr/bin/env bash
# bin/e2e-docker.sh
docker run -d --name chrome -p 4444:4444 -v /dev/shm:/dev/shm selenium/standalone-chrome:2.52.0
# run actual test command in a subshell to be able to rm docker container afterwards
(
    # passing arguments along to the test process
    npm run test:e2e -- "$@"
)
docker stop chrome 1> /dev/null && docker rm chrome 1> /dev/null

# save exit code of subshell
testresult=$?
exit $testresult

The chrome docker container exposed port 4444, which is the default port for the wdio test runner to connect to. The above script could be invoked via npm run scripts described in package.json as

"scripts": {
    "test:e2e": "wdio test/e2e/wdio.conf.js",
    "test": "bin/e2e-docker.sh"
}

Now we can simply run our end-to-end tests with a simple npm test command. I thought of using the pretest and posttest to set up and tear down the docker container, but posttest will not be executed if the test fails, thus leaving the container running after the test has finished. Pulling the setup and teardown steps into a bash script also allowed me to do a bit more complex setup, which will be described below.

While removing the need to run selenium server in the background, this approach requires that you have docker running on your host machine. If you’re on Windows or OS X and use docker-machine to run the docker server, the docker container lives at a different IP address than localhost. To get around that, we can tell the wdio test runner what host it should connect to.

:; npm test -- --host $(docker-machine ip)

With this approach of using docker-selenium, I can now track the version of selenium drivers in my source code. Furthermore, the browsers are now opened and run inside the docker container instead of the host machine, which I find to be a bit faster and less disruptive to the developer experience. However, there are times during the test writing process where you might want to see what is actually going on for each test command. Fortunately, docker-selenium provides debug images that has a running VNC server that we can connect to in order to visually see the browser window. Keeping the same npm run scripts, I could enable debug mode by running npm test -- --debug with the following modification to the bin/e2e-docker.sh script:

#!/usr/bin/env bash

CONTAINER_NAME="chrome"
# if a debug flag is passed in, use the debug image and open vnc screen sharing
if [[ $@ == *"--debug"* ]]; then
    # parse the script arguments for the docker server IP address
    ip=$(grep -Eo '([0-9]{1,3}.){3}[0-9]{1,3}' <<< "$@")
    docker run -d --name $CONTAINER_NAME -p 4444:4444 -p 5900:5900 -v /dev/shm:/dev/shm selenium/standalone-chrome-debug:2.52.0
    sleep 2 # wait a bit for container to start
    open vnc://"$ip":5900
else
    docker run -d --name $CONTAINER_NAME -p 4444:4444 -v /dev/shm:/dev/shm selenium/standalone-chrome:2.52.0
fi 
# run actual test command in a subshell to be able to rm docker container afterwards 
(
    npm run test:e2e -- "$@" 2> /dev/null
)

# save exit code of subshell
testresult=$?

docker stop $CONTAINER_NAME 1> /dev/null && docker rm $CONTAINER_NAME 1> /dev/null

exit $testresult

A couple of changes have been made to this script. First, the container name was abstracted to the variable CONTAINER_NAME to keep it DRY. If you run this script in a Jenkins environment, this could potentially be parameterized with the build number as well. Second, an if/else block is used to detect whether a --debug flag was used, and decide which docker image to use accordingly. If debug mode is on, it will also try to open a VNC viewer (I have only tested this on OS X) so you can see the browser window actions (the password is secret).

Notice that I keep the wdio test/e2e/wdio.conf.js command behind its own npm run script instead of putting it in the bin/e2e-docker.sh script. This is so that I could still use the old method of running a local selenium server manually and run the test through there. While more cumbersome, it is sometimes helpful to be able to pause the browser and interact directly with it, such as inspecting the DOM elements. Writing end-to-end tests could be rather tricky, and having the ability to inspect could save a lot of time debugging.

comments powered by Disqus