Your coupon control worked. How do you know?

You switched on a coupon control and the numbers rose. How to design a holdout that answers finance: sticky groups, one metric, fair size, stopping rules.

Author:Benson
MeasurementUpdated 6 October 202611 min read
Jump to a section

In short

A holdout keeps a random slice of eligible shoppers on the old experience while everyone else gets your coupon control, so both groups shop in the same weeks. Decide the eligible sessions, one primary metric and the stop date before launch, keep each shopper in one group, run whole weeks outside sales, and read the range around the result, not just the number.

For store owners

You made the sale. A coupon extension took a cut.

Add Benson free to see what coupon extensions took from your orders over two weeks. Set rules to hide pop-ups and control which codes work in extension sessions.

We'll email you about getting started. Unsubscribe any time.

See which of your codes coupon extensions hold

Three weeks ago you switched on a rule that hides coupon pop-ups on your product and cart pages. Since then, revenue per session is up and discount spend is down. You put the chart in the monthly report.

Finance replies with one line: how do you know it was the rule?

You look again. Those three weeks also held a payday, an email campaign, a restock of your best seller and the first cold weekend of the autumn. Any of them could have moved the line. You don't have an answer, and the change you were proud of is now a claim you can't defend.

This guide is about getting that answer before anyone asks. The tool is a holdout: a slice of eligible shoppers, picked at random, who keep the old experience while everyone else gets the change.

A holdout experimentShoppers with an extension are split at random. Most get Benson; a fixed slice is held out and sees the store as before. Both groups shop in the same weeks, so the difference between them is the effect of Benson.Extension sessionssplit at randomWith Bensonmost of the trafficHoldouta fixed slice, unchangedΔ
Same weeks, same shoppers, one difference. What's left is the effect.

Why does a rising line prove so little?#

A before-and-after comparison holds everything that changed in that window, not just your rule. Campaign mix, paydays, weather, stock and the traffic you bought all move at once, and the shoppers in the "after" period are different people.

Merchants run into this all the time, and not only with extensions.

There's a second, quieter problem. Most changes don't do what you hoped. When Microsoft looked back at well-designed experiments built to improve a key metric, the share that actually improved it was small.

About a third

of well-designed Microsoft experiments meant to improve a key metric actually improved it.

Microsoft (Kohavi et al.), Sep 2009(opens the source)

If good ideas from experienced teams fail that often, a rising line after your change is weak evidence on its own. Some of those "wins" are the season. Some are noise.

Where do false wins come from?#

Four traps cover most of them.

Peeking. You check the numbers every day and stop the moment the change looks like a winner. Evan Miller worked through the worst case, checking after every visitor: a test set at the usual 5% significance level ends up with a real false-positive rate of 26.1%. [Evan Miller, Apr 2010] (opens the source) Early results swing, and if you keep looking you will eventually catch a high.

Short tests. Shoppers behave differently on a Tuesday than on a Saturday. Microsoft's experimentation team treated a week as the minimum to see day-of-week effects, and warned that early results are often misleading. [Microsoft (Kohavi et al.), KDD 2012, Aug 2012] (opens the source)

Novelty. Some changes look better at first and fade as returning shoppers get used to them. The same Microsoft paper found most suspected novelty effects were statistical artefacts rather than real, which is another reason to judge a test on its full run, not its first days. [Microsoft (Kohavi et al.), KDD 2012, Aug 2012] (opens the source)

Too little traffic. This is the one small stores hit hardest. Detecting a modest lift takes far more sessions than most people expect.

We checked that arithmetic, and it holds. It doesn't mean small stores shouldn't test. It means they should test changes big enough to matter, and be honest when the answer is "we can't tell yet".

Why is finance asking now?#

Because budgets are being judged on evidence, and most marketing can't supply it. In Gartner's 2024 survey, only 52% of senior marketing leaders said they could prove marketing's value and get credit for it. [Gartner, Sep 2024] (opens the source)

The people signing off spend want experiments, not dashboards.

60%

of US senior decision-makers trust independent incrementality testing most among measurement methods, ahead of media mix modelling and in-platform reports.

EMARKETER (Haus survey), May 2026(opens the source)

That survey was run by Haus, which sells incrementality testing, so read it as a direction, not a verdict. The same survey found 78% of them believe at least a tenth of marketing spend is wasted for lack of measurement. [EMARKETER (Haus survey), May 2026] (opens the source)

Bad measurement cuts both ways. It can keep you paying for something that does nothing. In a Haus case study, a brand switched off its affiliate loyalty partners in the US for seven weeks while seven other countries carried on, and revenue didn't change, even though Google Analytics showed the affiliate share falling. [Haus, May 2023] (opens the source) A large field experiment at eBay found its brand keyword search ads had no measurable short-term benefit. [NBER (Blake, Nosko and Tadelis), May 2014] (opens the source) It can also do the opposite: a dashboard that credits the wrong channel can persuade you to cut something that was working.

Then there are your affiliate partners. If you hide a partner's pop-up or narrow which codes work in its sessions, the partner sees its numbers fall and will ask why. Almost 80% of US brand programmes in the PMA's 2026 survey still use last-click attribution, [Performance Marketing Association, Jul 2026] (opens the source) which means the partner's report and yours will tell different stories. A clean holdout gives both sides the same evidence to read. The partnership review guide covers that conversation.

How do you design a holdout that survives the question?#

You don't need a data team. You need to make a few decisions before launch and stick to them.

Write the question down first#

Before you switch anything on, write one paragraph: the change, who is eligible, the primary metric, the smallest effect worth acting on, the stop date, and what you'll do if the result is flat. This is pre-registration, and it's what stops you choosing the story after you've seen the numbers.

Only count shoppers the change can reach#

Define eligibility by what the control can touch. For a pop-up rule, that's extension sessions on the pages where the pop-up appears. For checkout rules, it's extension sessions that reach the code field. For your own offer widget, it's the traffic that can actually see it.

Counting "all sessions" because it's easy buries the effect under shoppers who never saw a difference. Installation isn't eligibility either: an installed extension can stay quiet. The visibility versus installation guide explains why.

Randomise, and keep each shopper in one group#

Assign eligible shoppers at random, and make it sticky. A shopper who lands in the control on Monday should still be in the control on Thursday. The usual method is to hash a stable id, so the same shopper always falls into the same group without storing a list.

Your consent settings decide how stable that id can be. Without analytics consent you may only have a session id, so a returning shopper can be counted twice. That's fine if both groups are treated the same way and you know it's happening.

Pick one primary metric#

Revenue per eligible session is usually the right one: it captures conversion, order value and discount together. Conversion rate and discount depth are useful supporting numbers. Pop-ups hidden or codes refused show reach, not results.

That's the trap with your own offer widget: it can win on sign-ups or codes applied while losing on margin. Choose the money metric up front, and don't swap it mid-test because a different number looks better.

Size it honestly#

The holdout still sees the old experience. If you're hiding pop-ups, the holdout still gets them, so a big holdout costs you while it runs. A small one makes the test slower, because the smaller group sets the pace.

Run whole weeks, and avoid your sales#

Run at least one full week, ideally two or more, and stop on the same weekday you started. Don't start during a sale: the result describes the sale, not your rule. The code freeze guide explains how to use ordinary weeks before peak instead.

Decide your stopping rule before you start#

Stop on the date you wrote down, not when the chart looks good. If you must look early, look for harm: a clear, sustained loss is a reason to stop. A promising early win is not. If you need to check often, use a method built for repeated looks rather than a standard test read daily.

Read the range, not the number#

A result is a point estimate plus a confidence interval: the range of effects still consistent with your data. If the whole range sits above zero, you have a lift. If it crosses zero, you don't yet know.

Know what to do with a null result#

A flat result is still an answer. A narrow range around zero tells you the effect is small; decide on cost and simplicity. A wide range tells you the test was underpowered; run longer, narrow the cohort, or test something bigger. Either way, write the decision down with an owner and a review date.

Your DIY options, and their catches#

  • A general A/B testing tool plus analytics. Testing tools can split visitors, and GA4 can read the result, though Google is clear that GA4 interprets experiments rather than running the assignment. The catch: most tools can't tell an extension session from any other, so your test dilutes across everyone. The extension impact guide shows how to tag sessions yourself.
  • Switching a channel off by region. This is how the Haus test worked. The catch: it needs several comparable regions and modelling to build the comparison, which suits bigger brands.
  • Before and after, carefully. Cheap and quick, and fine as a smoke test. The catch: everything in this article.

How does Benson run holdouts?#

Benson identifies extension sessions in your storefront, so the eligible group is built in. On Pro, you can put a holdout behind Block (pop-ups hidden on your storefront pages) or behind strict checkout rules (only the codes you allow work in coupon-extension sessions).

Your first experiment: baseline, one rule, a holdout, then the result.

Here's how it works:

  • Random and sticky. Extension shoppers are split by a hash of their anonymous visitor id when they've consented to analytics, otherwise their session. The holdout is one in ten by default; you can choose one in twenty, one in five or half.
  • A floor before any readout. Until a holdout has seven days and 500 sessions in each group, it shows "Not enough data yet". That's a floor, not a power calculation: a small store may need far longer for a clear answer.
  • A range, not a headline. The Value page shows revenue per session for protected and held-out shoppers, with a range. Insights tells you when the difference is clear, and tells you plainly if protected sessions earned less than the holdout.
  • Sales kept out. Mark a sale period and Benson holds back holdout suggestions while it runs.
  • Your own offer. On Pro, A/B experiments can show the widget to some visitors and not others. Those give no verdict before seven days and enough sessions and conversions in every group, hide the verdict if the split looks skewed, can stop a test that trips a guardrail, and end as inconclusive after 60 days.
Sample store
A holdout readout: revenue per session for protected and held-out shoppers, with a range. Sample data.

No method makes a low-traffic test conclusive faster, so Benson shows the range and waits for the data. On Shopify, no app can change what an extension shows inside checkout, so strict checkout rules work on which codes are accepted there.

The first experiment playbook walks through the whole loop in order. To start by seeing what's in your sessions, add Benson from the Shopify App Store for Shopify, or see Benson on your store for other platforms.

Questions merchants ask

How big should my holdout be?
Big enough to finish the test in a reasonable time, small enough that the shoppers still seeing the old experience don't cost you much. One in ten is a common starting point. If your traffic is low, a bigger holdout gets you an answer sooner.
How long should a holdout run?
At least one full week, and usually several, ending on the same weekday it started. Write the stop date down before launch. Stop early only for clear harm, never for an early win.
My result crossed zero. Did the change fail?
Not necessarily. It means the test can't tell a lift from no change at this sample size. A narrow range around zero says the effect is small; a wide one says you need more data or a bigger change to test.
Can a small store run a meaningful holdout?
Yes, for changes with a large effect. Small stores can't detect small lifts in any reasonable time, so test the changes most likely to matter and be honest when the range is still wide.
Should I keep a holdout running forever?
Not usually. Once you have a clear answer and a decision, end it and act. Start a fresh holdout when you change a pop-up rule, your checkout rules or the offer, or when your traffic changes a lot.

Sources#

Find out what's leaking. Free forever, no card.

Free forever up to 25,000 sessions a month. Every Pro feature free for your first 14 days. No card. We'll email you about getting started. Unsubscribe any time.