Retrying Test Actions
Intro
We all write tests with the best intention of having them pass and, when they fail, to reveal legitimate issues in the application we're testing. But that's not always easy to accomplish. Especially when your application is dependent upon third-party service providers (e.g. payment providers, social networks, etc.).
A Solution
Rather than have your tests fail intermittently for reasons that don't actually reflect an issue in your application (or that are really difficult to track down), we'd like to trigger a context specific retry for a specific set of actions (rather than the whole test) that will back-off after a few attempts.
If it finds what it needs, the tests passes. If it doesn't, the test fails -- and you know you have a legitimate problem on your hands.
Example
NOTE: We are using a notification message example found on the-internet. It displays a notification message at the top of the page that will either say Action successful or Action unsuccesful, please try again.
Our goal in this example is to verify an Action successful message.
A first pass would be to load the page, grab the notification message, and assert on it. The problem with this approach is that it will fail intermittently, because the message could very well be the unsuccessful one. But the next time you load the page, it could be the correct thing.
So instead, we add a small helper that receives the failure message and the actions to perform (loading the page and grabbing the notification message). It performs the actions, and repeats them while the message contains the failure message, up to a maximum number of attempts. Then the test asserts on the last message it got.
The page shows the unsuccessful message most of the time, so we allow up to 10 attempts. With only 3 attempts the test would still fail roughly a quarter of the time.
Expected Behavior
- Load the page
- Grab the notification message text
- Compare it to the failure message
- If it matches, retry
- If 10 attempts are reached, end the loop and fail the test
- If it doesn't match, pass the test
Summary
With this approach there may be a concern that we could be obfuscating a real issue that could bubble up to our users. That's why it is important to use this approach sparingly, and on issues you more or less understand (e.g., a third-party test environment having one-off service blips, or expensive tests that involve sharing something to a third-party and checking to see that it appeared).
Hopefully this example will help you build a more resilient and relevant set of tests.
Happy Testing!
Authors
