Holdout test
Jump to a section
In short
An experiment that leaves a random group of shoppers without a change, so the difference against everyone else shows what the change did.
What it means
A holdout test splits shoppers, sessions or regions at random, applies a change to one group and leaves the other, the holdout, as it was. Because both groups live through the same weeks, the difference between them is the change's effect, with seasonality and campaigns cancelled out. Decide the length and the metric before you start, and don't stop early because the numbers look good. Evan Miller showed that checking after every visitor turns a test set at a 5% significance level into a real false-positive rate of 26.1%.[Evan Miller, Apr 2010] (opens the source)
On your store
You turn on a coupon-extension rule for most sessions and keep a random slice without it. After four weeks you compare revenue per session in the two groups, not this month against last month.
Why it matters
A before-and-after comparison mixes the change with everything else that happened that month. A holdout gives a result your finance team can check. Most changes don't do what was expected: about a third of well-designed Microsoft experiments meant to improve a key metric actually improved it.[Microsoft (Kohavi et al.), Sep 2009] (opens the source)
Sources
- 1Evan Miller, “How not to run an A/B test”, Apr 2010. evanmiller.org (opens the source)
- 2Microsoft (Kohavi et al.), “Online Experimentation at Microsoft”, Sep 2009. ai.stanford.edu (opens the source)
Read next
Related terms
- IncrementalityThe sales a channel or partner actually caused, as opposed to the sales it was credited with that would have happened anyway.
- Coupon-extension trafficThe store sessions in which a coupon or cashback extension was active, measured apart so you can see what those shoppers buy and what they pay.
