Measure coupon extension impact with holdouts

Coupon and cashback extensions are easy to have an opinion about and hard to measure. Some brands are sure they cost a fortune; others are sure they barely matter. Most are guessing, because the obvious ways to check give the wrong answer.

This guide walks through why the obvious method fails, which numbers actually matter, how to get a first read with Google Analytics 4 and your dataLayer, and the one method that gives an answer you can take to a finance meeting.

Why before/after lies#

The instinctive test goes like this: switch something on (hide the pop-ups, stop working with an affiliate, add your own discount widget), wait a fortnight, and compare the fortnight before with the fortnight after.

It feels rigorous. It isn't, for three reasons.

Seasonality moves faster than you think. Payday, the weather, a bank holiday, the start of a sale season: any of them can swing conversion rate by more than the effect you're trying to see. If your change happened to go live the week before Black Friday, the "after" period wins, and the change gets the credit.

Promotions overlap. Stores run several offers at once. A new email campaign, a price change, or an influencer post in the same fortnight will leak into the comparison, and there's no way to separate them afterwards.

The traffic changes. Paid campaigns get paused, a new affiliate starts sending traffic, an extension changes how it behaves. The shoppers in the "after" period aren't the same people as the "before" period, so you're not comparing like with like.

None of this means before/after numbers are useless. They're a fine smoke test. But if the answer matters, and it usually does when affiliate commission or margin is on the line, you need a comparison where the only difference between the two groups is the thing you changed.

What to measure#

Start by splitting your sessions into two cohorts: sessions where a coupon or cashback extension is present, and everyone else. Then compare the cohorts on five numbers.

1. Extension share. The percentage of sessions with at least one coupon extension. This tells you how big the issue is before you spend any more time on it. Look at it by extension, device and traffic source; the split is rarely even.
2. Conversion rate. Orders divided by sessions, per cohort. Extensions tend to appear late in the funnel, so a higher conversion rate in extension sessions doesn't automatically mean they help. These are often shoppers who were already at the checkout.
3. Average order value. Revenue divided by orders, per cohort, before discounts. A cohort that converts well but at a lower basket value can still be a net loss.
4. Discount depth. Total discount divided by gross revenue, per cohort. This is where extensions usually show up most clearly: a code that was meant for a small group gets applied to everyone who has the extension.
5. Margin per session. Revenue after discounts and affiliate commission, per session. It's the number that pulls the other four together, and the one that tells you whether the cohort is paying its way.

A useful habit: write down what you expect each number to look like before you look. It stops the numbers you find from quietly becoming the numbers you expected.

Reading the results#

Once you have both cohorts side by side, the numbers usually fall into one of three patterns.

• Small share, small gaps. Extensions touch a sliver of your sessions and those sessions look like everyone else's. You can stop worrying, keep an eye on it monthly, and spend your time elsewhere.
• Healthy conversion, deep discounts. Extension sessions convert well but carry much deeper discounts, and often a private or single-use code shows up in the list. These are shoppers who were going to buy anyway; the extension is handing them a discount you didn't intend them to have. It's a common pattern, and the one where hiding pop-ups and tightening code rules pays.
• Genuine lift. Extension sessions convert noticeably better at a similar discount depth, and the difference holds across devices and sources. The extension may really be persuading undecided shoppers. That's worth knowing, and worth testing properly before you touch it: you might do better offering the same deal to every shopper yourself, without the commission.

Whichever pattern you see, treat it as a hypothesis. The cohorts tell you where the difference is. Only an experiment tells you what causes it.

DIY with GA4 and your dataLayer#

You can get a first read without buying anything, as long as something on your site can tell you when an extension is present.

Benson's storefront script pushes a dataLayer event the first time it sees an extension in a session. Its name and fields are stable, so you can build on them in Google Tag Manager:

// Pushed by Benson once per session, per extension, when detection is positive
window.dataLayer.push({
  event: "benson_ext_detected",
  benson_actor_class: "extension",
  benson_actor_id: "honey",   // which extension
  benson_confidence: 0.95,
});

// Also available: benson_ext_blocked, benson_widget_shown, benson_code_applied

With that in place:

1. In Google Tag Manager, create a Custom Event trigger for benson_ext_detected, and Data Layer Variables for benson_actor_id and benson_confidence.
2. Send a GA4 event from that trigger (for example extension_detected) with actor_id as an event parameter. Register it as a custom dimension in GA4 so you can report on it.
3. Build two segments in a GA4 exploration: sessions that include extension_detected, and sessions that don't.
4. Compare the cohorts on sessions, purchases, purchase revenue, and your discount field (if you send coupon value with purchases). Export to a spreadsheet to calculate discount depth and margin per session.

If you're not using Benson, you can do the same with any reliable signal. Just be careful: a home-made check that looks for one extension's markup will miss the others and break when that extension updates.

A few traps to avoid:

• Consent. If your GA4 setup respects consent (it should), shoppers who decline analytics drop out of both cohorts. That's fine for comparing rates, but don't read the absolute counts as your whole audience.
• Late detection. Some extensions only appear on the cart or checkout. Count the session as an extension session wherever in the visit the extension first appears.
• Small numbers. A week of a few hundred extension sessions can swing wildly. Look at weeks, not days, and be wary of any difference smaller than the week-to-week wobble.

This gives you a solid descriptive picture: who the extension shoppers are and how they differ. What it can't tell you is what would happen if you changed something, because extension shoppers are different people from everyone else to begin with. For that, you need a holdout.

Holdouts: the only honest answer#

A holdout is the simplest possible experiment. Take the sessions you want to change (say, sessions with a coupon extension) and split them at random. Most get the change. A fixed slice, the holdout, doesn't. Then compare the two groups over the same weeks.

Because the split is random, both groups have the same mix of devices, traffic sources, seasons and promotions. The only systematic difference between them is the change you made. So when the group with the change earns more per session than the holdout, the difference is caused by the change, not by the calendar.

Three things make a holdout trustworthy:

1. Randomise per session, and keep it sticky. A shopper who lands in the holdout should stay there for the whole visit, or you'll smear the effect across both groups.
2. Decide the measure up front. Pick one primary metric (revenue or margin per session is usually right) before you start. Checking five metrics and reporting the one that moved is how false wins happen.
3. Report a range, not a point. A single number hides how sure you can be. A confidence interval shows it: if the whole interval sits above zero, the effect is real; if it straddles zero, you don't know yet.

How big should the holdout be? Bigger holdouts reach an answer faster but cost you the effect on those sessions while the test runs. Holding out somewhere between a tenth and a fifth of the sessions is a sensible default for most stores. Run it for at least two full weeks so every weekday appears in both groups.

A holdout also answers the question the rest of this guide can't: if an extension genuinely brings sales you wouldn't otherwise have had, the holdout will show it, and you'll know to keep working with it, or to offer the same deal to every shopper yourself.

How Benson does it automatically#

Benson runs this whole process for you, on your own store:

• Detection in real shopper sessions, with signatures for PayPal Honey, Capital One Shopping, Rakuten, RetailMeNot, Karma and others, updated without redeploying your site.
• Cohorts built in: extension share, conversion rate, average order value and discount depth for extension sessions and everyone else, joined to real orders once your store's orders are connected.
• Holdouts when you want proof. Start one when you hide extension pop-ups or apply a checkout code policy, or A/B test your own widget on Scale. The split is random and sticky per visitor, and the result comes with its confidence interval.
• A dataLayer bridge that pushes the events above to Google Tag Manager and GA4, so your own reporting stays in step.

It takes a couple of minutes to install, and you can start with a free trial. The playbooks turn this into step-by-step plans, and pricing shows what each plan includes.

See it on your own store#

Type your store's address to see the Benson dashboard with your own name and icon on it, and how your first ten minutes go. Install it, and your real extension share starts arriving the same day.

See Benson on your store