Simpson's Paradox: When Your Winning Variant Loses in Every Segment

Simpson's paradox is when a test's overall result reverses once you break the data into subgroups. You ship variant B because it beat variant A by 8 percent in aggregate. Three weeks later someone slices it by traffic source and finds variant A actually won among paid traffic, organic traffic, and returning visitors, individually, every single segment. B only won in aggregate because the segment mix shifted between the two groups. The classic case is UC Berkeley's graduate admissions in 1973: about 44 percent of male applicants were admitted versus about 35 percent of female applicants, which looked like bias against women, but when researchers split the data by department, women were admitted at a higher rate than men in four of six departments. The aggregate was a weighted average, and the weights were the problem. The defense is to pre-register three or four segment cuts before the test launches, new versus returning, mobile versus desktop, and the top two traffic sources by volume, and to run the segment breakdown before you declare a winner, not after.

The winner that loses everywhere

Here is a result that should be impossible but is not: a variant that wins on the topline, ships, and then underperforms, because it was actually worse for every single segment of users, and only looked better because the traffic mix between the two arms was not the same. That is Simpson's paradox, and it is hiding in more test reports than anyone wants to admit. The mechanism is the mix. Suppose you run a test on a new checkout flow. Variant B looks worse: 6.4 percent conversion versus 6.8 percent for the control. The standard read is that B lost, ship the control, move on. But your randomization split traffic by user, and during the test window a large paid campaign drove a flood of low-intent mobile traffic that, by chance or by timing, landed disproportionately in the B bucket. Segment by device, and B wins on desktop, 9.3 percent versus 9.0 percent, and wins on mobile, 4.6 percent versus 4.4 percent. It wins in every segment. It loses on the blend, because B was handed twice the share of mobile traffic, and mobile converts at half the desktop rate regardless of variant. The blended comparison is contaminated by a covariate, device mix, that the randomization failed to balance. Reading the aggregate, you ship the worse variant and kill the better one.

The 50-year-old example that explains your test report

The cleanest illustration is one of the most famous statistics cases ever documented. In 1973, UC Berkeley's graduate admissions looked discriminatory: about 44 percent of male applicants were admitted versus about 35 percent of female applicants. Aggregate says bias against women. But when researchers split the data by department, in four of six departments women were admitted at a higher rate than men, as Bickel, Hammel, and O'Connell showed in Science in 1975. The reversal happens because the groups are not evenly sized or evenly mixed across the subgroups. A growth team reports that overall signup-to-paid conversion improved from one quarter to the next, from 4.6 percent to 5.6 percent, but the low-converting paid channels fell from 70 percent of traffic to 35 percent while the high-converting organic and referral channels grew from 30 percent to 65 percent. The blend got better because the weights moved toward the channels that convert well, even as each channel individually deteriorated. The most dangerous number is not a wrong one. It is a true aggregate that points the wrong way. The math is correct, and the conclusion is the exact opposite of reality.

The defense, and what it costs

The defense is to pre-register your segment cuts before launch, not after you see a result you do not like. Pick three or four cuts, typically new versus returning, mobile versus desktop, and the top two traffic sources by volume, and run the segment breakdown before you declare a winner. The diagnostic is one extra cut: pull the result by your major segments and confirm the direction holds. If the aggregate points one way and every segment points the other, believe the segments, and do not ship on the aggregate, because the variant is losing everywhere that a real user actually lives. Splitting costs sample size: Splitting an aggregate test into two segments roughly doubles the sample size required to detect the same effect size at the same statistical power, because each segment now needs its own adequately powered cell. That is the price of the diagnostic, and it is worth paying, because a test that is underpowered per segment cannot tell you whether the paradox is present. And the default test report cannot show you this: it shows the pooled number, and the pooled number is exactly the thing the paradox corrupts. Hero sections and conversion-critical form flows are the highest-risk surfaces, because they interact heavily with traffic source and intent, and novelty effects and selection bias compound the paradox, because new visitors behave differently from returning ones, and both get folded into the same top-line number by default.

Frequently asked questions

What is Simpson's paradox in A/B testing?

It is when a trend that holds in every subgroup of your data reverses or disappears once you pool the subgroups together. A variant can win overall purely because it happened to get more of the traffic segment that already converts better, independent of whether the variant itself did anything. The math is not a bug: it is a consequence of unequal group sizes interacting with unequal baseline conversion rates. A blended conversion rate averages populations with structurally different intent, so a single number can mask one segment winning and another losing.

How do I detect Simpson's paradox before I ship?

Run your segment breakdown before you declare a winner, not after. Pre-register three or four segment cuts before the test launches, typically new versus returning, mobile versus desktop, and the top two traffic sources by volume. Then pull the result by those segments and confirm the direction holds. If the aggregate points one way and every segment points the other, believe the segments. The diagnostic takes one extra cut, and it is the difference between shipping a winner and shipping a loser.

Does splitting by segment change the sample size I need?

Yes. A test needs roughly twice the sample size to detect effects reliably once split into two traffic-source segments instead of analyzed in aggregate, because each segment now needs its own adequately powered cell. That is the cost of the diagnostic, and it is worth paying, because a test that is underpowered per segment cannot tell you whether the paradox is present. If you cannot afford the sample size for the segments, do not ship on the aggregate alone.

What causes the mix shift that creates the paradox?

Traffic composition is never static during a live test window. A mid-test campaign launch, a bot wave, a bot-filtering rule that fired differently between arms, or imperfect randomization can load one arm with easier-to-convert users. Hero sections and conversion-critical form flows are the highest-risk surfaces, because they interact heavily with traffic source and intent. Checkout-stage tests convert at a 32 percent median versus 25 percent for cart-stage tests, so pooling the two stages into one number already mixes populations with different baselines.