Statistically Significant and Commercially Worthless: The A/B Test Trap

The test finished, the result is significant at 95 percent confidence, and the winning variant lifts conversion by 0.2 percent. The team ships it, celebrates, and moves on. The lift is worth less than the meeting that announced it.

Statistical significance is a statement about chance, not about value. A test with enough traffic can find a tiny difference and certify it as real, and the certificate is not the point. The point is whether the difference is worth the cost of shipping, maintaining, and explaining the change.

What p less than 0.05 actually says

A p-value below 0.05 means that if the two variants were truly identical, you would see a difference this big less than 5 percent of the time by chance alone. That is all it means. It does not mean the difference is large, it does not mean the difference is valuable, and it does not mean the difference will hold next month.

The trap is sample size. With enough traffic, almost any difference becomes statistically significant, because the test has enough data to detect tiny effects. A store with a million sessions a month can certify a 0.1 percent lift as real, and the certification is technically correct and commercially useless. The test answered the question it was asked, and the question was the wrong one.

The minimum detectable effect, decided before the test

The minimum detectable effect is the smallest lift your test can reliably find, and it is set by your traffic and your baseline conversion rate. A store with 1,000 conversions a month cannot reliably detect a 3 percent lift, because the noise in the data is bigger than the signal. It can detect a 15 percent lift, and that is the honest scope of what the test can tell you.

Decide the minimum effect that would change your decision before the test starts. If a 2 percent lift would not change what you ship, then a test that can only detect 10 percent lifts is the wrong tool, and the right answer is to test something bigger or to accept the uncertainty. Running the test anyway and celebrating a significant 0.2 percent lift is the same as deciding nothing and calling it data.

The check that stops bad test calls

Before you ship a significant result, ask three questions. How big is the lift in money, not in percent? What does it cost to ship and maintain the change? And would you make the same call if the confidence were 80 percent instead of 95? If the money is small, the cost is real, and the answer changes with the confidence level, the result is not a decision. It is a number looking for one.

Revenue per session is the metric that keeps this honest, because a variant can lift conversion while lowering average order value, and the business is worse off. A test that moves conversion and revenue in opposite directions has not found a winner. It has found a trade, and the trade needs a judgment, not a p-value.

Frequently asked questions

What does statistical significance mean in an A/B test?

It means the observed difference is unlikely to be due to chance alone, usually expressed as p less than 0.05, or 95 percent confidence. It says nothing about whether the difference matters to the business.

What is practical significance?

Whether the size of the lift is worth acting on. A 0.2 percent conversion lift can be statistically significant with enough traffic, and still be worth less than the cost of shipping the change. Practical significance is a business judgment, not a statistical one.

What is the minimum detectable effect?

The smallest lift your test can reliably detect given your traffic and baseline conversion rate. If your minimum detectable effect is 10 percent and the real lift is 3 percent, the test will never find it, no matter how long it runs.

How long should an A/B test run?

Long enough to cover at least one full business cycle, usually two weeks, and long enough to reach the sample size your minimum detectable effect requires. Stopping early because the result looks good is peeking, and peeking inflates false positives.