A/B Testing Mistakes: Why Your Results Look Strange

The winner that disappeared

The classic story: the variant won with "97% significance" and +18% conversion. You built it into the site. Three months later, conversion is about where it was.

Nothing magical happened. Usually, one of these did:

1. The winner was real, but smaller than it looked

This is the most common reason, and the least understood. When a test is only just big enough (or too small), the results that cross the "significant" line are the ones where chance pushed the difference up. So significant results from small tests exaggerate the true effect. Statisticians call it the winner's curse or a "type M" (magnitude) error (Gelman and Carlin, 2014).

We simulated this. A shop runs a test at a 2% conversion rate with 21,106 visitors per version, which is the right size to detect +20%. Then we varied the true effect:

(50,000 simulated tests per row. Method, rerun on 7 November 2026: each version's orders drawn from a binomial distribution, a "win" is a variant that is higher with p < 0.05 in a two-sided two-proportion z-test, and the lift shown is the variant's rate ÷ the original's rate − 1, averaged over the winners only.)

So if the change truly adds +5%, most tests miss it, and the few that "win" report about +18%. After launch, you get the real +5%, which is hard to see in your normal numbers. The win didn't disappear. It was never +18%.

What to do: plan tests with enough power, look at the confidence interval rather than the point estimate, and use the lower end of the interval in forecasts. See statistical significance explained.

2. The winner was false

When most ideas don't work, a good share of significant results are false positives. Ron Kohavi, Alex Deng and Lukas Vermeer calculate that when about 10% of ideas truly work (the rate they cite for Booking.com, Google Ads and Netflix), about 22% of significant results at p < 0.05 are false, even with properly powered tests. With 20% power, more than half are (A/B Testing Intuition Busters, KDD 2022).

What to do: for important or surprising wins, run the test again. A real effect will usually show up again; a fluke usually won't.

3. You compare against the wrong "after"

After launch, you compare this month to last month. But last month had different ads, a different season and different stock. A before/after comparison can easily hide or invent a 10% change. The A/B test was the fair comparison; your dashboard isn't.

Peeking

Peeking means checking results during the test and stopping when they look good. It's the single biggest source of false winners in small teams.

Each check is another chance for noise to cross the line. Evan Miller showed that repeated checking can push a 5% false positive rate to 26.1% (Evan Miller). In our own simulation of 50,000 tests with two identical versions (2% conversion rate, 1,000 visitors per version per day, 28 days):

• Look once, at the end: about 5% false winners.
• Look weekly and stop at the first significant result: about 13%.
• Look daily and stop at the first significant result: about 28%.
• Look daily and stop at the first "95% chance to beat" (Bayesian): about 23%.

(Method, rerun on 7 November 2026: daily orders per version drawn from a binomial distribution, a two-sided two-proportion z-test at p < 0.05 at each look, and for the Bayesian line a flat Beta(1, 1) prior with the variant counted as a false winner once its chance to beat the original reaches 95%. Repeated runs with different random seeds land between 27% and 28% for daily looks.)

How to spot it afterwards: the test was stopped before its planned end, or there was no planned end. The "win" came early and was large.

What to do: decide the duration (in full weeks) before you start, and only check during the test for problems, not for winners. Our sample size calculator helps you set it.

Novelty and primacy effects

Some changes work differently in week one than in week four.

• Novelty effect: returning visitors notice something new and click it out of curiosity. A new menu, a badge, an animated banner. The effect fades as the novelty wears off.
• Primacy effect (change aversion): returning visitors are used to the old layout and struggle at first. The variant looks worse early and improves as people get used to it.

How to spot it: plot the daily difference between the versions. A strong start that drifts toward zero suggests novelty. A weak start that improves suggests primacy. Split by new vs returning visitors: if the effect is mainly in returning visitors and fades, novelty is likely.

How much it matters for shops: less than for apps, because many shop visitors are new. But for shops with many repeat customers (groceries, pet food, cosmetics), it can matter a lot.

What to do: run at least two full weeks, preferably more for changes to navigation or layout. Judge the effect on new visitors, who see both versions with fresh eyes, as a sanity check.

Seasonality, sales and Black Friday

Visitors during a sale aren't your normal visitors. They come with a discount in mind, compare prices, and buy faster. A change that works during Black Friday (urgency messages, for example) may do nothing in March, and the other way around.

Typical problems:

• The test runs half in a sale and half outside it. The variant's effect is a blend of two different shops.
• The test runs entirely in a sale, and you assume the result holds all year.
• A payday, holiday or weather change (garden furniture, anyone?) shifts behavior during the test.

What to do:

• Don't start or end tests across big events (Black Friday, Christmas, summer sale). Pause your testing calendar around them, or run tests specifically designed for that period.
• Write events into your test log: sales, newsletters, TV mentions, stock problems. When a result looks strange, check the log first.
• Run full weeks, so weekday patterns are balanced.

Mixed traffic (paid campaigns mid-test)

Your traffic mix changes during the test: a new Meta campaign starts, an influencer mentions you, a newsletter goes out. Both versions get the extra visitors equally (the split is random), so the comparison stays fair. But the result now describes a different audience.

Example: a product page variant with more detailed specs wins clearly. During the test, a big Google Shopping campaign brought in many comparison shoppers who love specs. After the test, most traffic comes from Instagram, where people buy on look and feel. The win shrinks.

What to do:

• Note campaign starts and stops in your test log.
• Look at the result by traffic source after the test, as a check rather than a decision. If the effect only exists in one source, and that source is temporary, be careful.
• Never change the traffic split mid-test. Going from 10% to 50% of visitors halfway through mixes periods with different visitors and can produce misleading totals (a version of Simpson's paradox). If you must change it, restart the test.

Too many metrics

You defined one goal (orders), but the report shows twelve: add to cart, bounce rate, time on page, revenue, three devices, four traffic sources. One of them is "significant". You report that one.

With 20 comparisons at 95% confidence, you expect about one to be significant by chance alone. This is sometimes called the "garden of forking paths": the more ways you look, the more you find.

What to do:

• Pick one primary metric before the test. Only that metric decides the outcome.
• Add a few guardrail metrics (orders, revenue per visitor) to make sure you don't break anything. They don't decide a win; they can veto one.
• Treat segment results as ideas for the next test, not as findings.

Broken tracking and SRM

Before you believe any result, check that the test worked.

Sample ratio mismatch (SRM): visitors aren't split as planned. Planned 50/50, got 52/48 on 20,000 visitors. That's too uneven for chance, and it means something treats the versions differently: a redirect, an error, caching, a bot filter, a consent banner. Microsoft reported SRM in about 6% of its experiments (Microsoft Research). If your test has SRM, its result is unusable. See our guide to sample ratio mismatch.

Other tracking problems:

• The goal isn't recorded in one version (the variant moved the button, and the click tracking points at the old one).
• Orders aren't linked to visitors, so one version looks like it converts at 0%.
• The variant flickers: visitors see the original for a moment, then the change. That can hurt the variant and blur the result. See the A/B test flicker effect.
• The variant breaks on some devices. Watch a few session recordings of each version on mobile.

Kohavi's advice for surprising results applies here: "any figure that looks interesting or different is usually wrong" (Twyman's law, quoted in the KDD 2022 paper). A +60% lift is far more often a bug than a breakthrough.

Running overlapping tests

"Can I run two tests at the same time?" Usually yes.

When two tests run on the same site and visitors are split independently for each, a visitor can be in both. People worry the tests will "interact". In practice, harmful interactions are rare. Microsoft's experimentation team looked for interactions between concurrent A/B tests across four products. In three they found none; in the fourth, the rate was about 0.002% of test-pair metrics, and none of those showed two significant effects pointing in opposite directions (Microsoft Research, A/B Interactions: A Call to Relax).

Where overlap does cause trouble:

• Two tests change the same element. Test A changes the add-to-cart button text; test B changes its color and position. Combinations can look odd or break.
• One test changes the path of another. Test A changes the cart; test B measures cart-to-checkout clicks on the cart page.
• Small shops. Running two tests halves nothing (each gets all visitors), but small shops have less room for weird combinations to average out.

What to do: avoid overlapping tests on the same page element or the same step. Tests on different pages (homepage and product page, for example) can usually run side by side.

A diagnostic flowchart

When a result looks strange, go through these questions in order. Stop at the first "no".

1. Is the split OK? Run the SRM check. If not → result unusable; find the cause and rerun (SRM guide).
2. Does the goal get recorded in every version, on every device? If not → fix tracking; rerun.
3. Did the test run its planned duration, in full weeks, without being stopped early? If not → treat the result as unproven; rerun with a fixed duration.
4. Was the test big enough for the effect you saw? If the planned MDE was +20% and you saw +18%, the true effect is probably smaller. Look at the interval's lower end.
5. Was the period normal? Check your log for sales, campaigns, holidays, stock problems. If not → rerun in a normal period, or limit your conclusion to that kind of period.
6. Did you decide on one primary metric before the test? If the "win" is in a secondary metric or a segment → it's an idea for a new test, not a result.
7. Does the effect hold for new visitors and over time? If it fades or only shows in returning visitors → possible novelty effect; run longer.
8. Did another test change the same element or step? If yes → check combinations, or rerun without the overlap.
9. Is the result surprisingly large? If yes → look for a bug first, then repeat the test before you act.

If everything checks out and the result still looks odd, you might simply have learned something unexpected. Write it down, with your evidence, and test the follow-up idea.

FAQ

Why did my A/B test winner not increase revenue? Usually because the measured win was inflated (small test), false (peeking, too many metrics), or specific to the test period. Also check that your primary metric was close to revenue: an add-to-cart win doesn't always become more orders.

How common are false positives in A/B testing? At p < 0.05, a change that does nothing has a 5% chance of "winning" if you look once. With peeking, much more. And because most ideas don't work, Kohavi and co-authors estimate that a fifth or more of significant results can be false, even in well-run programs.

Should I rerun winning tests? For important decisions or surprising results, yes. A successful rerun is the best evidence you can get.

How long should I run a test to avoid novelty effects? At least two full weeks; three or four for changes to navigation or layout that returning visitors will notice.

For the bigger picture, see our A/B testing guide for ecommerce. To write tests that are easier to read afterwards, start from a clear hypothesis.

---

Catch broken tests before you trust them. Varly gives no verdict before the planned duration ends, runs a split check on every test and links each version to its session recordings. Request access.