Statistical Significance in A/B Testing, Explained Simply

What "significant" really means

Every A/B test shows a difference. Even if you show the exact same page to two groups, one group will buy a bit more, just by chance. Significance testing asks one narrow question:

If the change made no difference at all, how often would we see a gap at least this big, just by chance?

That "how often" is the p-value. A p-value of 0.03 means: if the variant truly did nothing, a difference this large (or larger) would turn up in about 3% of tests. That's unusual, so we treat it as evidence that something real is going on. By convention, below 0.05 is called "statistically significant".

What the p-value is not:

• It is not the chance that the variant is better. "p = 0.03, so we're 97% sure" is the single most common mistake in CRO reports. Ron Kohavi, Alex Deng and Lukas Vermeer show that even testing vendors and well-known A/B testing books have published this wrong definition (A/B Testing Intuition Busters, KDD 2022).
• It is not the chance that your result is a false positive. That depends on how often your ideas work in the first place (more below).
• It is not a measure of size. A tiny, useless lift can be very significant on a huge site. A big lift can be "not significant" on a small one.

A worked example

A shop tests a new product page layout. After the planned four weeks:

The variant converts 15% better (2.30 / 2.00 = 1.15). A standard two-proportion z-test gives p ≈ 0.039. Below 0.05, so "significant at 95% confidence".

Now the same shop, but the variant got 440 orders (2.20%, +10%). Then p ≈ 0.16. Not significant. That doesn't prove the variant is useless; it means this test can't tell a +10% effect apart from chance with these numbers. The test was planned for bigger effects. (Our sample size calculator shows that a +10% effect at 2% needs about 81,000 visitors per version.)

Confidence levels: 90, 95, 99

The confidence level is the bar you set before the test. It controls the false positive rate: how often a change that does nothing would still be called a winner.

Which one to use:

• 95% is the default nearly everywhere. Use it unless you have a reason not to.
• 90% shortens tests by about a fifth. It's acceptable for low-risk, easily reversed changes on smaller shops, as long as everyone knows that one in ten do-nothing changes will look like winners.
• 99% is for changes that are expensive to build or hard to undo, or for results that look too good to be true.

One more setting matters: power, the chance to detect an effect that is really there. At 80% power you miss one in five real effects of the size you planned for. Going to 90% power costs about a third more visitors (28,255 instead of 21,106 in the table's example).

Confidence intervals are more useful than one number

A single p-value answers "is there probably an effect?" A confidence interval answers the question you actually care about: "how big could the effect be?"

For our first example (2.00% vs 2.30%, 20,000 visitors each), the 95% confidence interval for the difference is:

• +0.02 to +0.58 percentage points, or
• roughly +1% to +29% relative to the original.

Read it like this: the data are consistent with a tiny gain and with a very large gain. The best single estimate is +15%, but you should not put "+15% conversion" in a forecast. A more honest line is "probably positive, somewhere between barely and a lot".

For the second example (+10%, p ≈ 0.16), the interval runs from about −4% to +24%. It includes zero, which is why it isn't significant. It also tells you something useful: a large negative effect is unlikely. If the variant is cheaper to run, or better for the brand, keeping it may still be reasonable.

Three rules for reading intervals:

1. If the interval includes 0%, the test hasn't shown a direction.
2. The width tells you how much you learned. Wide interval = small test, little information.
3. Use the lower end for business cases. If a +1% gain wouldn't pay for the work, a "significant" result may still not be worth shipping.

Bayesian vs frequentist in plain English

There are two main schools of statistics behind A/B testing tools. Both use the same data; they answer slightly different questions.

Frequentist (p-values, confidence intervals): "Assuming the change does nothing, how surprising is this data?" You set a bar in advance and decide yes or no at the end.

Bayesian (chance to beat, credible intervals): "Given this data and what we assumed beforehand, how likely is it that the variant is better?" The answer is a probability, such as "92% chance to beat original", which is what most people want to hear.

Bayesian results need a prior: what you believe before the test. Most testing tools use a flat or very weak prior ("we assume nothing"). That makes the result easy to compute, but it has a consequence few people mention:

With a flat prior, chance to beat original ≈ 1 − (two-sided p-value ÷ 2).

So in our examples:

(The exact number differs a little per tool: some calculate on the absolute difference, others on the relative change.)

This means a "95% chance to beat" rule is looser than "p < 0.05". It roughly matches p = 0.10. In practice: under a 95% chance-to-beat rule, a change that does nothing has about a 5% chance of being called a winner in the variant's favor. Under p < 0.05 two-sided, that chance is about 2.5% (the other 2.5% are false "losers"). Neither is wrong; they're different bars. Just know which one you're using, and don't assume Bayesian is "faster" for free. It is faster partly because it accepts more risk.

Where Bayesian really helps:

• Communication. "There's about a 97% chance the new layout is better, and the most likely gain is +15%" is easier for a client than "p = 0.039".
• Expected loss. Some tools show how much you'd lose, on average, if you picked the variant and it turned out worse. That's a good business framing.
• Real priors. Large companies with thousands of past tests can use priors based on history. Kohavi and co-authors describe how Microsoft's experimentation platform does this with historical data (KDD 2022). Most shops don't have enough past tests to do that well.

In Varly, results are shown as Bayesian "chance to beat original" by default, and you can switch to frequentist p-values under Settings → A/B testing. Both are computed on the same data; see Read your test results.

Peeking and stopping early

"Peeking" means checking results during the test and stopping when they look significant. It's the fastest way to fool yourself.

The reason is simple: every check is another chance for random noise to cross the line. Evan Miller showed that checking repeatedly can push a 5% false positive rate to 26.1% (Evan Miller).

We checked this with our own simulation: 50,000 A/A tests (two identical versions, so every winner is false), 2% conversion rate, 1,000 visitors per version per day, 28 days.

The last row matters. A common claim is that Bayesian tests "can be checked any time". You can look at a Bayesian result any time; it's still a correct summary of the data so far. But if you stop whenever it crosses a threshold, you get many false winners, just like with p-values.

What works:

• Fix the duration in advance (full weeks; see how long to run an A/B test) and decide only at the end.
• During the test, check only for broken things: no data, uneven split, errors. See sample ratio mismatch.
• If you need to stop early for real business reasons, use a tool built for sequential testing, which adjusts the bar for repeated looks. Optimizely's Stats Engine is one example built on that idea (Optimizely).

Testing revenue, not just conversions

Most significance calculators handle one kind of metric: a yes/no outcome per visitor (bought or didn't, added to cart or didn't). Revenue is different. Most visitors spend €0, a few spend €40, and one spends €900. That spread makes revenue per visitor much noisier.

What this means in practice:

• Revenue tests need more visitors, often several times more than conversion tests, because one large order can swing the average.
• Don't run a conversion-rate calculator on revenue numbers. The formula assumes yes/no data. Revenue needs a test for averages (such as a t-test on revenue per visitor), and ideally a rule for outliers decided before the test (for example, cap orders above the 99th percentile).
• Watch for the trade-off. A free-shipping threshold might lift conversion rate and lower average order value. Report both, and decide on the metric you chose before the test.

A practical approach for most shops: make conversion rate (or add-to-cart rate) the primary metric that decides the test, and look at revenue per visitor and average order value as secondary checks to make sure you didn't win clicks and lose money.

5 ways significance fools you

1. Most ideas don't work, so many "winners" are false

At Bing about 15% of ideas improved the key metric; at Booking.com, Google Ads and Netflix about 10%; at Microsoft overall about a third (Kohavi, Deng and Vermeer, KDD 2022, citing several sources). When only 1 in 10 ideas truly works, the same paper calculates that about 22% of significant results at p < 0.05 and 80% power are false positives. With low power it gets worse: at 20% power, more than half are false. A shop that runs underpowered tests on small tweaks will find many "winners" that never show up in revenue.

2. Too many metrics and segments

Check 20 metrics or 20 segments at 95% confidence and, on average, one will be "significant" by chance. Pick one primary metric before the test. Treat everything else (mobile only, new visitors, a secondary goal) as ideas for the next test, not as proof.

3. Too many variants

Three variants against one original means three chances to get lucky. Raise the bar (for example, Bonferroni: divide 0.05 by the number of variants) or test fewer variants.

4. Significant but tiny

On big traffic, a +0.5% effect can be highly significant. Whether it's worth building is a business question. Always look at the interval and the money, not just the verdict.

5. Too good to be true

A +60% lift from a headline change is far more likely to be a tracking bug, a bot, or a broken variant than a miracle. Kohavi calls this Twyman's law: "any figure that looks interesting or different is usually wrong." Check the split, the tracking and the recordings before you celebrate, and repeat the test if the decision is important. Our article on why A/B test results look strange has a checklist.

How to read a result, step by step

1. Did the test run its planned duration, in full weeks? If not, don't decide yet.
2. Is the split OK? Visitors should be split as planned. If not, the result is not usable. See SRM.
3. What does the primary metric say? Significant (or past your chance-to-beat bar) or not.
4. What's the interval? Is even the lower end worth having?
5. What happened to guardrails? Orders, revenue per visitor, returns.
6. Does it pass the smell test? Big surprises need a second look.

Then write it down: what you tested, what you expected, what happened, and what you'll do next. Our A/B testing guide covers reporting.

FAQ

What does "95% significance" mean in my testing tool? Usually "p < 0.05": if the change did nothing, a result this extreme would happen less than 5% of the time. It does not mean a 95% chance the variant is better. Some tools also use "95%" for a Bayesian chance to beat, which is a looser bar (see above). Check your tool's documentation.

My test isn't significant. Did it fail? No. "No clear difference" is a valid result: the change didn't matter enough for your test to see it. Look at the interval. If it's narrow around zero, the change probably doesn't matter. If it's wide, the test was too small to tell.

Can a result become significant if I run longer? Yes, if there's a real effect, more data narrows the interval. But only extend a test if you planned for that (or use a sequential method). Extending "until it's significant" is peeking in reverse.

Is Bayesian better than frequentist? Neither is better in general. Bayesian is easier to explain; frequentist is the classic standard with well-known error rates. What matters more: a fixed duration, one primary metric, enough visitors, and honest reporting.

What's a good p-value? There's no good or bad p-value, only a threshold you chose before the test. Below 0.05 is the common bar. For surprising results or expensive decisions, ask for more (0.01 or lower) or a second test.

---

See every result in plain words. Varly shows each test with a verdict, a chance to beat (or p-value), a likely range and a split check, and gives no verdict before the planned duration ends. Varly is in private beta. Join the waitlist.