Cohort vs Blended Conversion Rate: The Number That Lies to You

Blended conversion rate mixes every traffic source into one number, so a change in traffic mix looks like a change in site performance. Cohort conversion rate splits the number by source, device, or new versus returning, and the split shows what actually changed. If the blended rate moved and the cohorts did not, the traffic changed, not the site.

Conversion rate went up 20% this month. The team celebrates. Then someone splits the number by traffic source and the celebration stops: new visitor conversion is flat, returning visitor conversion is flat, and the blended rate rose because the mix shifted toward returning visitors, who always convert higher. Nothing on the site improved. The traffic changed.

That is the blended rate's trick. It is one number that hides two numbers, and the hidden one is usually the important one. A blended conversion rate is a weighted average of cohorts, and when the weights move, the average moves, even if every cohort stays exactly the same.

How the blend lies

Take a store with two cohorts. New visitors convert at 1%, returning visitors at 4%. In January the mix is 80% new and 20% returning, so the blended rate is 1.6%. In February the mix shifts to 60% new and 40% returning, because a retention campaign brought old customers back, and the blended rate is 2.2%. The site did not change. No cohort improved. The blended rate rose 37%, and the dashboard calls it a win.

The same lie runs in reverse. A big paid campaign brings a flood of cold traffic, the new visitor share jumps, and the blended rate falls, even though the site converts every cohort exactly as well as before. The team blames the site, redesigns the checkout, and the rate does not recover, because the site was never the problem. The traffic mix was.

The mechanism is Simpson's paradox, and it is not exotic. It is the ordinary consequence of averaging groups with different base rates when the group sizes change. Every blended metric has this failure mode, and conversion rate is the one merchants watch most closely, which makes it the one that misleads most often.

Splitting the number properly

The split that matters first is new versus returning. These two cohorts convert at wildly different rates on almost every store, and their mix changes constantly with campaigns, seasons and retention work. A store that does not track them separately is a store that cannot tell a site improvement from a traffic shift.

The second split is by source. Organic search, paid search, social, email, direct: each converts differently, and each moves independently. A blended rate that mixes a 0.5% social cohort with a 4% email cohort is a number that describes neither. Report the cohorts, and report the mix alongside them, so a reader can see whether the rate moved or the weights moved.

The third split is by device, because mobile and desktop convert differently and the mobile share keeps growing. A store whose mobile share rose from 60% to 70% will see its blended rate fall even if mobile and desktop conversion both improved, because mobile converts lower and now weighs more. The dashboard will call it a decline. The cohorts will call it progress.

What to report instead

Report the cohorts and the mix, every time. A weekly report that says blended conversion is 2.1%, new is 1.2%, returning is 4.3%, and the new share is 65%, is a report that cannot mislead. A report that says only 2.1% is a report that will mislead the first time the mix moves, and the mix moves every week.

Judge tests by cohort, not by blend. An A/B test that improves the blended rate by bringing in more returning visitors has not improved the site. A test that improves new visitor conversion by 10% and leaves returning flat has improved the site, even if the blended rate barely moves because the mix shifted the other way. The test's success criterion should be written in cohort terms before the test starts.

The same logic applies to revenue per session, average order value and every other blended metric. The blend is a summary, not a diagnosis. The cohorts are the diagnosis, and the mix is the context. A merchant who reads the blend alone is reading the weather report without knowing which city it is for.

Frequently asked questions

What is a cohort in conversion rate analysis?

A cohort is a group of visitors defined by a shared trait: new versus returning, traffic source, device, or any other split that matters. Cohort conversion rate is the conversion rate within one group. Blended conversion rate is the weighted average across all groups, and it hides what the cohorts are doing.

Why did my conversion rate go up when nothing changed on the site?

Probably the traffic mix changed. If the share of returning visitors rose, or the share of high-converting email traffic rose, the blended rate rises even though no cohort improved. Split the rate by new versus returning and by source to see whether the site changed or the traffic changed.

Is Simpson's paradox common in ecommerce analytics?

Yes, and conversion rate is the most common place it shows up. Any time you average groups with different base rates and the group sizes change, the average can move in the opposite direction from every group. It happens whenever a campaign shifts the traffic mix, which is most weeks.

Should I stop reporting blended conversion rate?

No, but never report it alone. Report the blended rate alongside the cohort rates and the mix, so a reader can see whether the rate moved or the weights moved. The blend is a summary. The cohorts are the diagnosis.