Run your first experiment

Change one thing, keep a control group, and let your own numbers decide.

Read the transcript1:02

Why test before you block

  1. You could block every extension on day one. It's a guess, but a confident one.

  2. Some of them really do bring you sales. Others take commission on orders you'd have had anyway. Your own numbers can tell you which.

  3. Run your first experiment.

Baseline, pick, rule

  1. Start from a baseline: a week or two in observe mode.

  2. Pick one thing to test. The extension you see most, or the page where pop-ups show up most.

  3. Write a rule for just that, and switch it on.

Keep a holdout

  1. Now keep a holdout: a small share of extension visitors who still get the pop-ups, as usual. They're your control group. They don't know. It's fine.

  2. Leave it running, and resist the urge to fiddle. Benson tells you in Insights when the difference is clear.

Read the result

  1. Then read the result: revenue per session for both groups, with a range. Actual evidence. Imagine that.

  2. Keep what worked, and pick your next test.

  3. Find this playbook in Benson.

At a glance

Change one thing, keep a control group, and let your own numbers decide.

A measured difference from your own shoppers, with a range, and a list of what to test next.

Jump to the steps
The result to look for

Revenue per extension session, protected against the holdout

8 practical steps

Who it's for

Stores with a week or two of observing behind them that want proof before they commit.

The problem

You've seen the extensions on your store and you want them gone. The trouble is that gone is a guess. Some extensions bring shoppers who wouldn't otherwise buy. Others take commission on orders that were coming anyway.

Switch everything off and sales dip, and was it the change, the weather or the week? Without a control group you can't say, and neither can anyone you report to.

Blocking everything is a guess: some extensions bring you sales, others take commission on orders you'd have had anyway. A holdout keeps a slice of extension shoppers unchanged, so the difference you see comes from your change, not the season. One test at a time builds a store setup you can defend.

“Is this a pigeon?” meme: the “Brand with no baseline” points at a butterfly labelled “Blocking every extension, everywhere” and asks “Is this a strategy?”

How to approach it

  1. Start from a baseline

    A week or two of observe mode tells you what normal looks like on your store.

  2. Change one thing

    One extension, or one page. A rule that changes several things at once gives you a result you can't explain.

  3. Keep a holdout, then wait

    A small share of extension visitors carry on as before. Comparing them with everyone else is the experiment. Leave it alone until Insights says the difference is clear.

The steps

Work through the plan

0 of 8 completed

Progress stays on this device · Local marks are a reading aid; they do not verify your store or change Benson.

  1. To do

    Step 1: Start from a baseline

    Observe mode records what Benson would hide, without hiding anything. It's free on every plan.

    Benson checks this in your account

  2. To do

    Step 2: Pick one thing to test

    Start with the extension you see most, or the page where pop-ups appear most. One change at a time keeps the result clear.

    You decide when this is complete

  3. To do

    Step 3: Write a rule for just that

    A rule can name one extension, or only act on some pages, countries, devices or traffic sources.

    Benson checks this in your account

  4. To do

    Step 4: Switch it on

    In Block mode, your rules hide the pop-ups they match on your storefront.

    Benson checks this in your account

  5. To do

    Step 5: Keep a holdout

    Leave a small share of extension visitors unprotected. Comparing the two groups is the experiment.

    Benson checks this in your account

  6. To do

    Step 6: Leave it running

    Don't change the rule while the test runs. Benson tells you in Insights when the difference is clear.

    You decide when this is complete

  7. To do

    Step 7: Read the result

    Value shows revenue per session for protected and unprotected shoppers, with a range.

    Benson checks this in your account

  8. To do

    Step 8: Plan your next test

    Keep what worked, then pick the next idea: another extension, another page, or your own offer.

    You decide when this is complete

Common pitfalls

  • Peeking and stopping early

    Early results swing. Stopping the moment it looks good locks in noise. Wait for Benson to say the difference is clear.

  • Changing the rule mid-test

    Every edit restarts the question. If you have a better idea, write it down for the next test.

  • Testing during a sale

    A sale changes how everyone shops, so the result describes the sale, not your rule.

  • Making the holdout too big

    The holdout still sees pop-ups. Keep it small: enough to compare, not so much that it costs you.

Expanding brain meme: “Ignore coupon extensions”, then “Block every extension, everywhere”, then “Block the one you see most, on product pages”, and the glowing brain: “Test it against a holdout and let your own numbers decide”.

How you'll know it worked

A measured difference from your own shoppers, with a range, and a list of what to test next.

What to watch: Revenue per extension session, protected against the holdout

Want the method behind the numbers? Read how to measure extension impact.

Do this in Benson

Every store gets every Pro feature free for its first 14 days, no card. Steps that need Pro say so.

Start free

Find out what's leaking. Free forever, no card.

Free forever up to 25,000 sessions a month. Every Pro feature free for your first 14 days. No card. We'll email you about getting started. Unsubscribe any time.