Traffic Allocation in A/B Testing

📘

This article helps you understand:

Why splitting Audience Traffic evenly is a best practice
Why it's tempting to limit a Variant to a smaller proportion of Traffic
How Minimum Sample Size is affected when Audience Allocation is unevenly split

The basic principle behind A/B testing or experimentation is that a certain proportion of the incoming traffic sees each Variant. The exact proportion is determined by the traffic allocation, as described in Applying Basic Audience Allocation.

There can be pressure to minimize any negative effects of a proposed change by limiting one of the Variants to a small proportion of traffic, such as 10%. In this case, the run time for statistical significance will be affected because the smallest volume Variant still has to reach the minimum sample size.

👍

Best Practice Tip

It's a best practice is to split audience traffic evenly among all the Variants. This ensures the duration of the test won't overextend.

If you do need to limit the size of any of the variants, you should account for this when working out the overall Minimum Sample Size for the test. For example, if the sample size calculator suggests 20,000 for each Variant and the desire is to limit the control (or test) Variant to 10%, then the overall Sample Size will be 200,000 to ensure that the Minimum Sample Size is met for the variant with the smaller traffic allocation.

Malcolm Lightbody

Product Manager - Analytics