A simple experiment log that keeps storefront tests from repeating themselves

Most stores have run the same test twice and argued about a result nobody wrote down. A lightweight experiment log fixes that: what to record, where to keep it, and the five fields that make an old result usable a year later.

The problem is memory, not testing

Ask a store team what happened when they tested free shipping thresholds last spring and you will get three versions of the answer, each confident, none documented. The test ran, someone saw a dashboard, a decision happened for reasons that felt obvious at the time, and the reasoning evaporated. Six months later a redesign quietly reverts the winner and nobody can prove it mattered.

The cost is not the lost document. It is the repeated work: tests rerun because the old result cannot be trusted, and arguments settled by seniority instead of evidence. A log fixes this for the price of ten disciplined minutes per test.

The five fields that matter

First, the hypothesis in one sentence, written before the test starts, with the metric it should move and by roughly how much. Second, the exact change, with screenshots or a link to the variant, because memories of what the variant actually was are the first thing to blur. Third, the run dates and the traffic split, so a later reader knows the context, including whether a sale period contaminated it.

Fourth, the result, with the actual numbers, not a verdict. Conversion moved from 2.41 to 2.63 percent on 18,000 sessions ages far better than it worked. Fifth, the decision and the reasoning: shipped, reverted, or parked, and why. That fifth field is the one everyone skips and the only one future teams truly argue about.

Where the log should live

One shared document or spreadsheet, in the place your team already looks for answers, beats a dedicated tool nobody opens. The log's enemy is not ugliness; it is a second location. If tests are planned in one system and logged in another, the log dies within a quarter.

Give every experiment a short name and number, and reference them in the pull requests, tickets, or theme changes that implement variants. That cross reference is what lets a future developer, staring at an odd line of code, find the experiment that justifies it before deleting it.

Keeping it alive

Review the log quarterly, briefly. Mark stale results whose context has changed: a winner from before a rebrand, a pricing change, or a traffic source shift may no longer hold, and the log should say so rather than letting an old verdict govern forever. Old results are evidence about a context, not eternal laws.

Start with the next test, not a historical reconstruction. Backfilling years of undocumented experiments produces fiction. One honest entry per test from today forward gives you a trustworthy log within a season, and turns next year's redesign arguments from memory contests into lookups.

Frequently asked questions

What tool should we use for an experiment log?

Whatever your team already opens daily: a shared document, a spreadsheet, or your project tracker. The best log is the one that gets updated, and a simple table with the five core fields beats a specialized platform that nobody maintains.

How detailed should each entry be?

One screenful at most. Hypothesis, exact change with a screenshot or link, dates and split, the numeric result, and the decision with its reasoning. If writing the entry takes more than ten minutes, the format is too heavy and people will stop.

Should failed tests go in the log?

Especially failed tests. A negative result recorded properly prevents the same idea from being retested every time a new team member proposes it. The log's most valuable entries are usually the dead ends, clearly marked with their numbers.

How long do experiment results stay valid?

Treat results as valid until the context materially changes: rebrand, pricing overhaul, new dominant traffic source, or a different audience. Note those events in the log so old verdicts can be re judged rather than blindly obeyed.