Peeking at A/B Tests: Why Checking Daily Breaks Your P-Value

A p-value of 0.05 means a 5% false positive rate if you check once, at the planned end. Check every day and stop when it first dips below 0.05, and the real false positive rate climbs past 20%. The test did not get better. You gave chance more opportunities to look significant. Decide the sample size and the runtime before the test starts, then wait.

The test has been running for three days. The dashboard shows a p-value of 0.04, the variant is winning by 12%, and someone in the meeting says to ship it. The test was planned for two weeks. Nobody remembers why two weeks was chosen, and the number on the screen feels like an answer.

It is not an answer. It is a peek, and peeking breaks the math. A p-value of 0.05 promises a 5% false positive rate on one condition: you check once, at the planned end, with the planned sample size. Check every day and stop the first time it dips below 0.05, and the real false positive rate is not 5%. It is over 20%, and it climbs with every additional peek.

Why the false positive rate explodes

The p-value is a probability under a specific procedure. Run the test to the planned sample size, check once, and a 5% threshold gives a 5% false positive rate. That promise is attached to the procedure, not to the number. Change the procedure, checking daily and stopping at the first significant result, and the promise no longer holds.

The intuition is simple. Each check is another chance for random noise to look like a real effect. A variant that is truly no better than the control will, by chance, show a p-value below 0.05 on some days and above it on others, as the data accumulates. If you stop at the first day it dips below the line, you are selecting the lucky day, not measuring a real effect. The more days you check, the more lucky days there are to catch.

The numbers are worse than most teams expect. With daily peeking over a two-week test, the false positive rate can exceed 20%, four times the nominal 5%. With hourly peeking, it climbs further. The dashboard that shows a live p-value is showing a number whose meaning depends on a procedure nobody is following.

The sample size you actually need

The fix starts before the test, with a sample size calculation. Decide the minimum effect you care about, the baseline conversion rate, and the power you want, usually 80%, and the calculation tells you how many conversions per variant you need. A common rule of thumb, and a weak one, is 100 conversions per variant. For a small effect on a low baseline rate, the real number is often in the thousands.

The sample size is the commitment. It is the number of conversions, not the number of days, and not the number of visitors. A test that reaches its visitor target but not its conversion target has not finished, because the power calculation was done in conversions. Running a test for two weeks because two weeks felt right is not a sample size. It is a calendar.

Pre-register the decision rule. Write down, before the test starts, the sample size, the runtime, and the threshold for shipping. Then wait. The waiting is the hard part, and it is the whole discipline. A team that cannot wait for the sample size will ship noise and call it a win, and the next test will contradict it.

If you must peek, peek properly

There are methods that allow interim looks: sequential testing, alpha spending, and Bayesian approaches that update continuously. These work because they adjust the threshold for each look, making it harder to declare a win early. A sequential test that allows five looks does not use 0.05 at each look. It uses a smaller threshold early, and the full threshold only at the end.

The practical version for most teams is simpler: use a tool that does sequential testing for you, or do not peek at significance at all until the sample size is reached. Check that the test is running, that the data is flowing, that the variant is not broken. Do not check whether it is winning. The winning question has a scheduled answer date, and the date is the sample size.

The cost of peeking is not abstract. A false positive ships a change that does not help, and the next test, or the next quarter's numbers, will show the improvement was never real. The team loses trust in testing, and the next real win gets ignored because the last three wins evaporated. The discipline of waiting is what keeps testing credible.

Frequently asked questions

What is peeking in A/B testing?

Peeking means checking the significance of a running test before it reaches its planned sample size, and stopping early when the p-value first dips below the threshold. Each peek is another chance for random noise to look significant, so the real false positive rate climbs far above the nominal 5%.

How bad is the false positive rate if I check daily?

With daily peeking over a two-week test, the false positive rate can exceed 20%, four times the nominal 5%. The more often you check, the worse it gets. The p-value only means what it says if you check once, at the planned end, with the planned sample size.

What is the minimum sample size for an A/B test?

It depends on the baseline conversion rate and the effect you want to detect. A common rule of thumb is 100 conversions per variant, but for a small effect on a low baseline the real number is often in the thousands. Calculate it from the baseline rate and the minimum detectable effect before the test starts.

Can I use a sequential testing method instead of waiting?

Yes. Sequential testing and alpha spending adjust the significance threshold for each interim look, so early peeks need stronger evidence. Bayesian methods update continuously without a fixed threshold. Either approach works if the tool implements it correctly. Checking a standard p-value daily is not a sequential method.