September 28, 2026
Carefully, because search is harder to test than pages. The unit is the query, not the visitor, and queries follow a power law: a handful of terms drive most searches while the long tail never reaches significance. Test the head terms with a proper split, use before-and-after analysis for the tail, and measure with search-specific metrics like null-result rate and search-to-cart rate, not page conversion.
A page test splits visitors and compares conversion. A search test splits queries, and queries are not interchangeable. "Red dress" and "x7q blue widget" are different experiments with different volumes, and randomizing at the visitor level mixes them together. Most stores do not have the search volume to power a test on anything but their top 10 to 20 queries.
The other problem is that search quality is multidimensional. A change can improve result relevance, reduce null results, and change the sort order all at once. If you measure only conversion, you cannot tell which mechanism moved. Search tests need search metrics first and revenue metrics second.
Null-result rate is the headline metric: the share of searches that return zero results. It is the clearest signal of a broken search experience, and it is easy to move with synonyms, typo tolerance, and stemming. A store with a 15 percent null rate has a search problem no ranking tweak will fix.
Search-to-cart rate and search revenue share come next: of the sessions that used search, how many added to cart, and what share of revenue touched a search. These connect search quality to money without pretending the test is a page test.
Refinement rate is the diagnostic: how often shoppers search, then search again. High refinement means the first results missed. It is the metric that tells you relevance is off even when the null rate looks fine.
For the top queries, run a real split. Route a share of sessions searching for a specific term to variant B and compare null rate, refinement, and add-to-cart for that query cohort. Keep the test at the query level: the same shopper can be in control for "red dress" and variant for "blue shoes" without contamination, because the queries are independent.
Size the test on the query's volume, not the store's traffic. A query with 200 searches a month cannot power a test of a 5 percent change. Either test bigger changes on head terms or accept longer runtimes. This is the same power math as page testing, applied to a smaller denominator.
For everything outside the head, a controlled split is not feasible. Use before-and-after analysis with a stable baseline: measure the tail metrics for two to four weeks, ship the change to all tail queries, and measure again. Control for seasonality and promotions by comparing against a holdout set of queries you deliberately do not touch.
This is weaker evidence than a split, and you should say so. But a before-and-after with a holdout beats the alternative, which is shipping search changes on vibes. The discipline is in the measurement window and the holdout, not in pretending it is an A/B test.
Synonym handling and typo tolerance move the null rate fastest and need no ranking changes. Then test the ranking of the top results for head queries: the first three results drive most of the clicks, so reordering them is the highest-leverage change. Test one mechanism at a time. A test that changes synonyms, ranking, and the results layout simultaneously teaches you nothing when it wins and nothing when it loses.
Search traffic is fragmented across hundreds of different queries, so each visitor is effectively running a different search. A layout change averages across all of them and hides the effect on the handful of head terms that drive most revenue. That is why search testing leans on segmenting by query and on before and after comparisons for the long tail.
Search to purchase conversion for the top queries is the headline, supported by click through rate on results, zero results rate, and refinement behavior. Revenue per search session is a good single summary. Raw clicks alone mislead, since a change can generate more clicking by making results worse, not better.
Long enough to collect solid sessions on your head terms, which for most stores means several weeks rather than days, because individual query volume is thin. Include at least one full weekly cycle, and avoid judging across a sale event, when query mix changes completely and the comparison stops being about the search design.
Zero results handling and the top ten queries experience, in that order. Zero results is a pure loss page, and small fixes there, suggestions, spelling tolerance, curated fallbacks, pay off immediately. Then test ranking and result presentation on the head terms before experimenting with the long tail, where data will always be thin.