Retrying Test Actions

The Problem

We all write tests with the best intention of having them pass, and when they fail, to reveal legitimate issues in the application we're testing. But that's not always easy to accomplish. Especially when your application is dependent upon third-party service providers (e.g. payment providers, social networks, etc).

A Solution

Rather than have your tests fail intermittently for reasons that don't actually reflect an issue in your application (or that are really difficult to track down), we'd like to trigger a context specific retry for a specific set of actions (rather than the whole test) that will back-off after a few attempts.

If it finds what it needs, the tests passes. If it doesn't, the test fails -- and you know you have a legitimate problem on your hands.

An Example

The example application is a page that displays a notification message at the top of the page. It will either say Action successful or Action unsuccessful, please try again. You can see it here.

Our goal in this example is verify an Action successful message.

The beginning of our test script starts off simply with accessing our libraries (Selenium and RSpec) and creating our setup, teardown, and run actions.

require 'selenium-webdriver'
require 'rspec-expectations'

def setup
  @driver = Selenium::WebDriver.for :firefox

def teardown

def run

A first pass at this example would look something like this:

run {
  @driver.get 'http://the-internet.herokuapp.com/notification_message'
  notification_message = @driver.find_element(id: 'flash').text
  notification_message.should =~ /Action successful/

The problem with this approach is that it will fail intermittently because the page's notification message could very well be something else. But the next time you load it, it could be the correct thing.

Perhaps something like this would be better suited:

def get_notification_message
  @notification_message = @driver.find_element(id: 'flash').text

def retry_if_notification_message_contains(fail_message)
  count = 0
  until !@notification_message.include? fail_message || count == 3
    count =+ 1

run {
  retry_if_notification_message_contains 'please try again' do
    @driver.get 'http://the-internet.herokuapp.com/notification_message'
  @notification_message.should =~ /Action successful/

In this approach we have abstracted out the text grab of the notification message into a method which sets what it finds into an instance variable. And we've added a new method called retry_if_notification_message_contains that accepts an argument of fail_message. In it we accept a yield in 2 places and have constructed a simple loop that runs 3 times or until the notification message does not match the fail_message, whichever comes first. And the yields will execute whatever code is passed both before and during the loop.

In the run block we pass our navigation and text grab actions as a block into retry_if_notification_message_contains, passing the failure message of please try again as the argument for fail_message to receive.

And we top everything off with an assertion.

Expected Behavior

  • Load the page
  • Grab the notification message text
  • Compare it to the failure message
  • If it matches, retry
  • If 3 retries attempted, end loop, and fail test
  • If it doesn't match, pass the test


With this approach there may be a concern that we could be obfuscating a real issue that could bubble up to our users. That's why it is important to use this approach sparingly, and on issues you more or less understand (e.g. a third-party test environment having one-off service blips, or expensive tests that involve sharing something to a third-party and checking to see that it appeared).

Hopefully this example will help you build a more resilient and relevant set of tests.

Happy Testing!

Back to the archives