Skip to main content

How to Prepare Your Tests for Continuous Integration

Intro​

If you want your Selenium tests to automatically run and report failures to you and your team, you'll want to wire them up to a Continuous Integration (CI) server.

In order to make the most of the Continuous Integration setup, you'll need to make sure your test suite outputs a machine readable test report. But what does this entail? How do you set it up? And will it work when running things in parallel?

A Solution​

CI servers work with a standardized report format called JUnit XML which is a standard offering across testing frameworks. If it's not something available off the shelf for your framework, odds are there's a plug-in to enable it.

The report will include test results (e.g., failure messages and stack traces) and test metrics (e.g., start time, end time, number of tests, etc.). Your CI server (once configured to do so) will consume this report after each test run and tell you when things pass and fail. It will also offer trend data over time to help determine the stability of your test suite.

Let's go through an example of how to configure a test suite in order to output a JUnit XML report.

Example​

Each example has a simple test (it loads the homepage of the-internet and checks the title), and the configuration needed to write a JUnit XML report every time the tests run:

  • Python: pytest writes JUnit XML out of the box with --junitxml.
  • C#: the JunitXml.TestLogger package adds a junit logger to dotnet test.
  • Java: Maven Surefire (which runs the tests in mvn test) writes JUnit XML reports by default.
  • JavaScript: Mocha comes with an xunit reporter, which writes the same XML format.
  • Ruby: the RSpec JUnit Formatter gem adds a JUnit XML formatter to RSpec.

If you run your tests in parallel, make sure each process writes its own report file (most CI servers can read several of them), or use a tool that merges them.

Expected Behavior​

  • The tests run
  • A JUnit XML report with the results is written to disk
  • Your CI server reads the report and shows the results

Summary​

Now you're ready to plug your Selenium tests into a CI server and let them report the results.

Happy Testing!

Authors​

Dave Piacente
Dave Piacente
Original creator of Elemental Selenium
Diego Molina
Diego Molina
Elemental Selenium maintainer