How to Automatically Retry Failed Tests
Intro
You want reliable tests. But no matter how hard you try, you're going to run into intermittent test failures. These can be frustrating time wasters that distract you and your team and begin to erode the trust placed in the automated tests. Once that begins, it's a slippery slope.
A Solution
Rather than rerun your entire test suite to suss out transient failures -- you can retry just the tests that fail. Most test runners can do this for you, either out of the box or with a well-known plugin.
Let's continue with an example.
Example
To have a test that fails some of the time, we use the notification message example from the-internet. Each time the page loads, it shows either Action successful or Action unsuccesful, please try again (most of the time, the latter). Our test loads the page and checks that the message is the successful one, so it's flaky on purpose.
Then we tell the test runner to retry the test when it fails, up to 9 more times (the page fails more often than not, so we need a few attempts). When a retry passes, the test passes, and the report tells us it needed retries -- which is a good hint that the test (or the application) has a transient issue worth looking into.
Expected Behavior
- The test runs and, most of the time, fails
- The failed test gets rerun, until it passes or runs out of retries
- The report shows that the test needed retries
Summary
Retrying failed tests is a useful safety net, but use it sparingly. Keep an eye on the tests that need retries, and fix the ones that do so often, since they may be hiding a real issue.
The original version of this tip was inspired by rspec-rerun and this write-up.
Happy Testing!
Authors
