A/B Test Sample Size: Why 100 Conversions Proves Nothing

The test finished, the new variant won by 12 percent, and the team shipped it. The test had 100 conversions per variant, and at that size a 12 percent difference is noise wearing a winner's badge. The math of sample size is unforgiving, and most stores run tests too small to detect the effects they are looking for.

A test that is too small does not give a wrong answer. It gives an answer that feels right and means nothing, which is worse, because a wrong answer gets questioned and a meaningless one gets shipped.

The math, without the jargon

A test's power is its ability to detect a real difference, and the standard target is 80 percent power at 95 percent confidence. For a store converting at 2 percent, detecting a 10 percent relative lift, from 2 percent to 2.2 percent, needs roughly 30,000 visitors per variant. Detecting a 20 percent lift needs roughly 8,000 per variant. A test with 100 conversions per variant, which is about 5,000 visitors at 2 percent, can only reliably detect lifts of 30 percent or more.

Most real changes are smaller than 30 percent. A better product image, a clearer shipping message, a shorter form, these move conversion by 5 to 15 percent when they move it at all. A test sized for a 30 percent lift will miss them all, and the team will conclude that nothing works, when the truth is that the test could not see what was working.

The minimum detectable effect is the number to set before the test starts. Pick the smallest lift that would matter to the business, usually 10 percent relative, and size the test for that. If the traffic cannot reach that size in a reasonable time, the test should not run. A test that cannot detect the effect it is looking for is not a test, it is a coin flip with a dashboard.

The traps that shrink a test after it starts

Peeking is the first trap. Checking the result daily and stopping the moment the variant pulls ahead guarantees false winners, because early results swing wildly and the first swing up looks like a win. Decide the sample size in advance, run the test to that size, and read the result once. A test with a fixed finish line is a test. A test that stops when it likes the answer is a story.

Short tests are the second trap. A test that runs for two days measures two days, not the store. Weekday traffic converts differently from weekend traffic, and a test that covers only weekdays has measured the weekday customer. Run every test for at least one full business cycle, seven days at minimum, fourteen if the traffic is uneven, and do not stop early because the result looks good.

Segmenting after the fact is the third trap. A test that loses overall but wins on mobile is not a mobile win, it is a slice of the data that happened to look good. Every slice you check after seeing the result multiplies the chance of a false positive. Decide the segments before the test, or do not segment at all.

What to do instead of small tests

Run fewer, bigger tests. One test a month sized to detect a 10 percent lift beats four tests a week that detect nothing, because the one test produces an answer and the four produce noise. And test big changes, not button colors. A new value proposition, a different offer, a restructured product page, these are changes large enough for a small store to detect, and they are the changes worth knowing about anyway.

Frequently asked questions

How many conversions do I need for an A/B test?

For a store converting at 2 percent, detecting a 10 percent relative lift needs roughly 30,000 visitors per variant. One hundred conversions per variant can only detect lifts of about 30 percent or more.

Can I stop a test early if one variant is winning?

No. Stopping when the result looks good guarantees false winners, because early results swing wildly. Decide the sample size in advance and run to it.

How long should a test run?

At least one full business cycle, seven days minimum, fourteen if traffic is uneven. A two-day test measures two days, not the store.

What if my store does not get enough traffic for a proper test?

Test bigger changes, which are easier to detect, and run fewer tests. A small store should test value propositions and offers, not button colors, because the big changes are the ones its traffic can actually measure.