How to Opt-out of A/B Tests

Intro

Occasionally when running tests you may see unexpected behavior due to A/B testing (a.k.a. split testing) of the application you're working with. In order to keep your tests running without issue we need a clean way to opt-out of these split tests.

A Solution

First, here is a primer on Split Testing.

Split Testing

Split testing is a simple way to experiment with an application's features to see which changes lead to higher user engagement. A simple example would be testing variations of an e-mail landing page to see if more people sign up. In such a split test there would be the control (how the application looks and behaves now) and variants (e.g., 2 or 3 changes that could include changing text or images on the page, element positioning, color of the submit button, etc.). Once the variants are configured, they are put into rotation, and the experiment starts. During this experiment each user will see a different version of the feature and their engagement with it will be tracked. Split tests live for the length of the experiment or until a winner is found (e.g., tracking indicates that a variant converted higher than the control). If no winner is found, new variants may be created and another experiment tried. If a winner is found, then the winning variant becomes the new control and the feature gets updated accordingly.

Thankfully there are some standard opt-out mechanisms built into A/B testing platforms. They tend to come in the form of an appended URL or forging a cookie.

Let's start with an example for each approach with a popular A/B testing platform, Optimizely.

Example

Our example page is from the-internet and can be seen here. There are three different versions of the page that are available. On each page the heading text will vary:

• Control: A/B Test Control
• Variation 1: A/B Test Variation 1
• Opt-out: No A/B Test

We have three tests:

• Forge a cookie on the target page: we load the page, confirm that we are in one of the A/B test groups by checking the heading text, add the opt-out cookie (optimizelyOptOut with the value true), refresh the page, and confirm that we are no longer in the A/B test group.
• Forge a cookie before visiting the target page: we load the main page of the site first, add the opt-out cookie, and then visit the split test page and perform our check.
• Append a URL parameter: we opt out without forging a cookie, by appending ?optimizely_opt_out=true to the URL. Keep in mind that this approach triggers a JavaScript alert, so we have to wait for it and dismiss it before performing our check.

Expected Behavior

• Open the browser
• Opt-out of the split tests (either by cookie or appended URL)
• Visit the split testing page
• Grab the header text
• Confirm that the session is opted out of the split test
• Close the browser

Summary

Hopefully this tip was helpful in guiding you in different methods of how to opt out of split tests.

Happy Testing!

Authors

Dave Piacente

Original creator of Elemental Selenium

Diego Molina

Elemental Selenium maintainer