How long should you run a Shopify A/B test before calling it?

October 10, 2026

Calling a test too early is the most common way to ship a losing variant. How long Shopify tests actually need, what sample size matters, and the peeking problem that quietly ruins results.

The minimum is one full business cycle

Run every test for at least one full week, and ideally two. Shopify traffic is not uniform across the week: weekday shoppers behave differently from weekend shoppers, and payday cycles shift conversion rates. A test that runs Monday through Wednesday only measures weekday behavior, and the winner it crowns may lose on the weekend. One full week is the floor, not the target.

Two full weeks is better for most stores because it smooths out one-off events: a newsletter send, a sale, a traffic spike from social. If your store has strong seasonality or runs frequent promotions, avoid testing during the promotion entirely. Promotional traffic converts differently, and a test run during a sale measures the sale, not the variant.

Sample size beats calendar time

Time alone does not make a test valid. What matters is how many visitors and conversions each variant saw. A common rule of thumb is at least a few hundred conversions per variant before judging, though the exact number depends on your baseline conversion rate and the size of the difference you are trying to detect. Small differences need larger samples; a variant that lifts conversion by 2 percent needs far more traffic to prove than one that lifts it by 20 percent.

Low-traffic stores face a hard truth: if you get fifty orders a week, a test of a subtle change may need two months to reach significance, and by then the store has changed anyway. In that situation, test bigger changes with bigger expected effects, or test on higher-traffic pages like the homepage or collection pages rather than deep product pages. Do not run a test you cannot finish.

The peeking problem

Peeking means checking results mid-test and stopping early when one variant looks ahead. It feels efficient and it is statistically broken. Every time you peek, you give random noise another chance to look like a real winner. Peek five times during a test and the chance of a false positive climbs far above the 5 percent your tool reports.

The fix is discipline: decide the run length or the sample size before the test starts, and do not look at the results until it is reached. If your testing tool supports sequential testing methods that account for peeking, use them. Otherwise, set a calendar reminder for the end date and stay out of the dashboard until then. The test you do not peek at is the test you can trust.

When to stop early anyway

There are legitimate reasons to end a test before its planned finish. If a variant is clearly broken, throwing errors, or tanking conversion by a large margin, stop it. Protecting revenue beats statistical purity. Similarly, if an external event like a site outage or a pricing error contaminates one variant's data, the test is compromised and should be restarted clean rather than continued.

What does not count as a reason: boredom, a stakeholder asking for results, or a variant being ahead by a little after three days. Those are the exact moments the peeking rule exists for. Write the stop conditions down before launch so the decision is made by the plan, not by impatience.

Calling it right

When the planned duration or sample size is reached, look at the primary metric you chose before the test, not whichever metric looks best afterward. If the winner is not statistically significant, the honest answer is "no detectable difference," not a coin flip dressed up as a decision. Shipping the non-significant winner is harmless but teaches nothing; the learning is the point of testing.

Document the result either way: what you tested, how long it ran, the sample size, and what you decided. A log of past tests prevents retesting the same ideas and builds an institutional memory of what actually moves conversion on your store. The stores that test well are not the ones with the fanciest tools. They are the ones that finish their tests.

Frequently asked questions

How long should a Shopify A/B test run?

At minimum one full business cycle, usually two to four weeks for most stores. Tests need to cover weekday and weekend patterns plus enough visitors for statistical confidence. Ending early because the results look good is the most common way to ship a losing variant.

What is statistical significance in A/B testing?

It is the confidence that the difference between variants is real and not random noise. Most teams use 95 percent confidence as the bar. Reaching it requires enough visitors and conversions, which is why small stores need longer test durations than large ones.

Should I stop a test early if one variant is clearly winning?

No. Early leads are often noise, and stopping at the peak of a random fluctuation is how false winners get shipped. Pre-commit to a duration or sample size before the test starts, and let it run. The discipline matters more than any single test result.