Sample Ratio Mismatch (SRM): Is Your A/B Test Broken?

What SRM is

When you start an A/B test, you choose how to split visitors: usually 50% original, 50% variant. A random split is never perfect. With 20,000 visitors you won't get exactly 10,000 and 10,000; you might get 10,060 and 9,940. That's normal.

Sample ratio mismatch is when the gap is too big to be chance. For example:

A 51.5/48.5 split doesn't look dramatic. But with 20,000 visitors, a gap this large would happen by chance in only about 2 in 100,000 tests (p ≈ 0.00002). Something is going on.

Why SRM invalidates results

A/B testing works because the only difference between the two groups is the change you made. Random assignment makes the groups alike in everything else: devices, traffic sources, buying intent, returning customers.

SRM is a sign that this is no longer true. Visitors are going missing, or being added, in one group, and almost never at random. Some typical examples:

• The variant has a JavaScript error on older iPhones, so some of those visitors are never counted. The variant group now has fewer iPhone users.
• The variant redirects to another URL. Visitors who leave during the redirect are counted in the original group but not in the variant.
• A bot filter removes "suspicious" sessions. The variant causes more clicking, which looks bot-like, so real visitors get filtered out of the variant only. Microsoft describes exactly this case on MSN, where a more engaging variant confused the bot detection (Microsoft Research).

In each case, the groups are no longer comparable. The missing visitors are a specific type of person, and they could easily be the ones who buy more or less. Lukas Vermeer, who led experimentation at Booking.com, gives an example where a couple of thousand missing users shift a conversion rate by about 18% on their own (Lukas Vermeer, SRM FAQ). That's bigger than most real test effects.

So the rule is: if a test has SRM, don't use its result. Not the winner, not the loser, not the segments.

How to check for SRM

The check is a chi-square goodness-of-fit test: it compares the visitors you got in each group with the visitors you expected, and gives a p-value for "how likely is a gap this big if the split works?"

In Google Sheets or Excel

Put the observed visitors in one column and the expected visitors in the next:

Then:

=CHISQ.TEST(A2:A3, B2:B3)

This returns the p-value (about 0.000022 here). For an uneven planned split, such as 20% variant and 80% original, use the planned shares instead of 0.5. For three or more versions, add rows; the formula is the same.

Which threshold to use

Use a strict threshold. You'll check every test, often several times, so a loose bar like 0.05 would raise false alarms in one in twenty healthy tests.

• Microsoft uses p < 0.0005 (Microsoft Research).
• Lukas Vermeer's former SRM checker extension flagged at p < 0.01 (SRM FAQ).
• Varly flags an "Uneven split" at p < 0.001.

Any of these is reasonable. We suggest p < 0.001: strict enough to avoid false alarms, sensitive enough to catch real problems on shop-sized tests.

What counts as "too uneven"?

It depends on the number of visitors. Small tests can wobble a lot by chance; large tests can't. For a planned 50/50 split, this is where p drops below 0.001:

So "51/49 is fine" is true at 10,000 visitors and false at 100,000. Always run the test; don't eyeball it.

More examples

A "watch" result is not proof of a problem. Keep the test running and check again; a real cause usually gets clearer as visitors grow, and a chance wobble fades.

When to check

• After the first few days. Most SRM causes are bugs that show up immediately. A dry run (splitting visitors but showing everyone the original) is a good way to check the split before anyone sees a change.
• Before you read any result. Make it the first line of every test report.
• Per segment, if you use segments. A problem on mobile Safari can be hidden in the total. If the overall split is fine but you read mobile results, check the mobile split too.

Common causes in online shops

Microsoft's research groups SRM causes by where they happen: when visitors are assigned, while the test runs, when data is processed, and during analysis (Fabijan et al., KDD 2019). For shops, these are the ones we'd check first.

1. The variant breaks or loads slower. A JavaScript error, a slow image or a layout shift means some visitors leave before they're counted. Vermeer's FAQ describes a real 50/50 test with 9,463 vs 7,681 users, traced to the variant loading slower and affecting tracking (SRM FAQ).

2. Redirects (split URL tests). If the variant sends visitors to another URL, some leave during the redirect, especially on slow mobile connections. The redirect can also drop tracking parameters. Count visitors at assignment, not on the landing page, and keep redirects fast.

3. Caching. If your CDN or a WordPress caching plugin stores the page with the variant applied, everyone gets the same version for a while. Server-side tests on cached pages need the cache to vary by test group.

4. Bots and filtering. Bots may hit one version more (for example, if the variant adds links that crawlers follow), or your bot filter may remove real visitors from one group, as in the MSN example.

5. Consent banners (especially in the EU). Most testing tools only include visitors who accept cookies. If a tool assigns a version before consent but counts visitors only after, and the variant changes how likely people are to accept (for example, it moves content behind the banner, or adds a pop-up that competes with it), one group loses more visitors. Assign and count at the same moment, after consent. If your banner or consent settings change mid-test, note the date and check the split before and after.

6. Ad blockers and privacy browsers. If the variant loads an extra script or image from a domain that blockers stop, visitors with blockers may see a broken variant or not be counted.

7. Targeting that's checked at the wrong time. "New visitors only" or "mobile only" rules that are evaluated after assignment, or differently per version, can push people out of one group.

8. Changing the split during the test. Going from 10% to 50% of traffic mid-test, or pausing one version for a day, mixes visitors from different periods. Keep the allocation fixed from start to end.

9. Internal traffic and QA. Your team, forcing themselves into the variant with preview links, can add up on a small shop. Exclude internal traffic, or keep preview visits out of the results.

10. In-app browsers. Links opened in Instagram, Facebook or TikTok open in a built-in browser that may block or reset storage. If a test changes what these visitors see, they can drop out unevenly.

What to do when you find SRM

1. Stop trusting the result. Pause the test, or at least make sure nobody acts on it.
2. Look for the pattern. Split the visitor counts by device, browser, traffic source, new vs returning and by day. SRM is often concentrated in one segment (one browser, one day after a release). That's your lead.
3. Look at the variant. Open it on the devices in that segment. Watch a few session recordings of each version. Check the browser console for errors. Compare page speed between versions.
4. Check what changed. A theme update, new consent settings, a caching rule, a campaign that sends traffic to a URL only one version handles.
5. Fix and restart. Start the test fresh, so old (broken) data doesn't mix with new data. Then check the split again after a few days.

Don't try to rescue the result by reweighting or by throwing away the "extra" visitors. You don't know which visitors are missing, so you can't put them back. Vermeer notes that statistical bounds for missing visitors exist, but are usually so wide that they add little value (SRM FAQ).

Preventing SRM

• Run a dry run first: split visitors, show everyone the original, and check the split after a few days.
• QA the variant on real devices, including an older iPhone and an in-app browser.
• Keep variants fast. Don't load large extra files only in the variant.
• Don't change allocation, targeting or the variant mid-test. If you must, restart.
• Assign and count in the same place, after consent.
• Make the SRM check automatic and show it on top of every result. A warning that lives in a separate tool tends to be ignored.

In Varly, every test result has a split check that runs this chi-square test and shows "Uneven split" below p = 0.001. The verdict then says that visitors are not split as planned, instead of naming a winner. See Read your test results and Preview and check a test.

FAQ

Is a small imbalance always a problem? No. Small gaps are normal, especially in small tests. Only a gap that's too big for chance (p < 0.001) is SRM. Use the test, not your gut.

Can SRM happen with a 50/50 split even if my tool is fine? Yes. Most causes are outside the tool: page speed, redirects, caching, consent, bots. The tool's random assignment can be perfect while the counting is not.

My test has SRM but the variant clearly won. Can I ship it? You don't know whether it won. The missing visitors might be exactly the ones who would have changed the picture. Fix the cause and rerun the test.

Does SRM matter for Bayesian results too? Yes. Any method that compares two groups assumes the groups are alike. SRM breaks that assumption whatever statistics you use.

Where can I learn more? Microsoft's article on diagnosing SRM and the KDD 2019 paper by Fabijan, Vermeer and colleagues behind it, Lukas Vermeer's SRM FAQ, and Ron Kohavi, Diane Tang and Ya Xu's book Trustworthy Online Controlled Experiments (Cambridge University Press, 2020), which treats SRM as one of the first checks of every test. For the bigger picture of reading results, see statistical significance in A/B testing and our A/B testing guide.

---

Get a split check on every test. Varly runs the SRM check automatically, next to a plain-words verdict, and lets you start with a dry run. Varly is in private beta. Join the waitlist.