Novelty Effect: Why Your Winning Variant Loses in Week Three

The variant that wins in week one usually loses in week three. A new button color, a new banner, a new layout gets clicked because it is new, not because it is better, and the lift fades as shoppers get used to it. Tests that call a winner in the first week are measuring novelty, not preference.

The effect is easy to miss, because the first week of data looks like a clean win. The variant is ahead by a wide margin, the sample is large enough to call it, and the team wants to ship it. Two weeks later the margin is gone, and nobody remembers that the test ever had a first week.

How novelty inflates the first week

New elements draw attention. Returning visitors notice the change and click out of curiosity, not because the new version is better at its job. The effect is strongest on high-traffic pages, where most visitors are returning, and weakest on new visitors, who never saw the old version and have nothing to compare it with.

The inflation is not small. A variant can lead by twenty percent in week one and settle at zero by week three, and the whole lift was curiosity. The test that stops at week one ships a change that does nothing, and the team learns the wrong lesson about what their shoppers want.

The test design that separates novelty from preference

Run at least two full weeks, and segment new vs returning visitors. A real win holds flat across both segments. A novelty win shows up in returning visitors and fades week over week, while new visitors, who have no old version to compare against, show little or no lift from the start.

Watch the lift decay, not just the average. A variant that wins by fifteen percent in week one, eight in week two, and two in week three is telling you the story in the slope. A variant that holds at eight percent across all three weeks is a real improvement. The slope is the test result. The average is the trap.

When to ignore the first week entirely

Tests on homepages and navigation, where every visitor sees the change, are the most novelty-prone. So are tests that change something visual, a color, a banner, a layout, rather than something functional. For those tests, the first week is not data. It is the adjustment period, and the honest read starts in week two, on new visitors only.

Frequently asked questions

How long should an A/B test run?

At least two full weeks, to cover weekday and weekend traffic and to let the novelty effect fade. High-traffic stores can reach significance faster, but the calendar matters as much as the sample: a test that runs only on weekdays measures weekdays, not shoppers.

What is the novelty effect in testing?

It is the lift a change gets simply because it is new. Returning visitors notice it and click out of curiosity, and the lift fades as they get used to it. A novelty win looks like a real win in week one and disappears by week three.

Can I trust a test that wins fast?

Not on its own. A fast win in the first days is often novelty, especially on high-traffic pages and for visual changes. Let it run a second week and check the new-visitor segment. If the win holds there, it is real.

Should I segment new vs returning visitors?

Yes, for any test that changes something visible. New visitors have no old version to compare against, so their response is the cleanest read on whether the change is actually better. Returning visitors carry the novelty effect, and their numbers flatter the variant.