A/B Test Sample Size Calculator (MDE and Duration)

How to use the calculator

Enter your visitors, your conversion rate and the smallest change you care about. The calculator tells you how many visitors each version needs and how many weeks that takes at your traffic.

Three tips before you type anything:

1. Use the visitors of the page you will test, not of the whole site. If you test the product page template, count the visitors who see a product page. If only mobile visitors take part, count only them.
2. Use the conversion rate of that same group. If 2% of product page visitors place an order, enter 2%, even if your site-wide rate is different.
3. Pick the smallest change you would act on, not the change you hope for. More on that below.

The result is a starting point for a decision, not a promise. A test that is "possible in 5 weeks" can still end with no clear difference. That is normal and still useful: it tells you the change does not matter much to your visitors.

What sample size means

Every A/B test shows some difference between the versions, even when the change does nothing. Visitors are different from each other, and on any given day a few more buyers can land in one group by pure chance. The question a test answers is: is this difference bigger than chance would usually produce?

The more visitors you have, the smaller the differences you can separate from chance. Sample size is simply the number of visitors needed to separate a difference of the size you care about from noise, with the level of certainty you chose.

Two settings define that certainty:

• Confidence (95%). If the change truly does nothing, there is at most a 5% chance the test still shows a "winner". This is the false positive rate, also written as alpha = 0.05.
• Power (80%). If the change really works as well as your MDE, the test has an 80% chance of showing it. The other 20% of the time you miss a real effect. This is the false negative rate.

These two numbers are industry defaults, not laws of nature. They are what most calculators use, including the ones from Evan Miller, Optimizely and VWO, so the results are comparable.

Minimum detectable effect, in plain English

The minimum detectable effect is the smallest improvement the test must be able to prove. If the true effect is smaller than your MDE, the test will usually end with "no clear difference".

Think of it as the resolution of your test. A test planned for a +20% MDE is like a camera that sees big objects sharply and small ones as a blur.

How to choose it:

• Start from money, not statistics. Ask: "What is the smallest lift that would be worth building and keeping?" If a change costs a developer two days and adds maintenance, a +2% lift may not be worth it on a small shop. On a large shop it might be worth a lot.
• Then check what your traffic allows. The calculator shows the smallest effect you can detect in, say, four weeks. If that number is +40%, small tweaks are not testable on that page. You need a bigger change.
• Be realistic about what changes do. Large, real lifts are rare. Ron Kohavi, who led experimentation at Microsoft and Airbnb, warns that a result that looks unusually good is more often a mistake than a breakthrough ("Twyman's law"), and he and his co-authors note they have "never seen" anything close to a 300% conversion lift in tens of thousands of tests (Kohavi, Deng and Vermeer, A/B Testing Intuition Busters, KDD 2022). Planning for a +50% effect on a headline change is wishful thinking.

How MDE changes the sample size

This table shows the visitors needed per version for different conversion rates and MDEs (relative change, 95% confidence, 80% power).

Two patterns stand out:

1. Halving the MDE roughly quadruples the visitors. At a 2% conversion rate, +20% needs about 21,000 visitors per version; +10% needs about 81,000.
2. Higher conversion rates need fewer visitors. That is why testing on "added to cart" (often 5–10% of product page visitors) is so much faster than testing on "placed an order" (often 1–3%).

Relative vs absolute MDE

This trips up a lot of people, and it can make your plan wrong by a factor of 25.

• Relative MDE is a percentage of your current rate. +10% relative on a 2% conversion rate means going from 2.0% to 2.2%.
• Absolute MDE is a difference in percentage points. +1 point absolute on 2% means going from 2.0% to 3.0%, which is a +50% relative change.

Our calculator, and most CRO tools, use relative MDE. When you read someone else's plan or report, check which one they mean.

Worked examples

All examples are hypothetical shops. Duration = total visitors needed ÷ visitors per week, rounded up to full weeks, with a minimum of two weeks.

Example 1: a product page, testing orders

A shop gets 10,000 visitors a week on its product pages. 2% of them place an order. The team wants to test a new delivery-information block.

What this means: on this page, the test can only reliably catch effects of about +15% or bigger within two months. In four weeks, the smallest effect it can detect is about +21%. If the team believes the delivery block will move orders by maybe 5%, an order-based A/B test is the wrong tool here.

Example 2: same page, testing add-to-cart instead

Same page, same 10,000 visitors a week, but now the goal is added to cart, which 8% of visitors do.

• +10% MDE: 18,869 per version, 37,738 in total: 3.8 → 4 weeks.

The same test becomes feasible in a month. The trade-off: more add-to-carts don't always mean more orders. Keep orders as a secondary metric and check that they don't drop. Our guide on why A/B test results look strange explains why a "micro-conversion" win sometimes doesn't reach the bank.

Example 3: a small shop

A shop with 2,500 visitors a week on the page and a 1.5% order rate.

• +20% MDE needs 28,301 per version: 22.6 → 23 weeks. Not testable.
• In 8 weeks it can detect about +35%. In 4 weeks, about +51%.

That shop should not run small A/B tests on orders. It can still improve: see A/B testing with low traffic for what to do instead.

Example 4: adding a second variant

Back to Example 1 (10,000 a week, 2%, +20%), but now with two variants plus the original. With more variants there are more chances for one to look good by luck, so the bar for a winner goes up. A common way is the Bonferroni correction: split the 5% error margin over the variants (2.5% each).

• Visitors per version: 25,560 (instead of 21,106).
• Three versions: 76,680 in total: 7.7 → 8 weeks (instead of 5).

Each extra variant costs you twice: more groups to fill, and more visitors in each group. On a small or mid-size shop, one strong variant is almost always the better plan.

Why run full weeks (and why 2–8 weeks is the usual range)

Full weeks, because people shop differently on a Monday morning than on a Saturday night. Traffic sources change across the week (newsletters, weekend ads), and so do buyers. If you stop after 10 days, some weekdays count twice and others once, and the mix can push the result either way. Rounding up to whole weeks evens this out.

At least two weeks, so every weekday appears at least twice and one odd day (a campaign, an outage, a TV mention) has less weight.

At most about eight weeks, for practical reasons:

• Visitors get re-counted. Many testing tools remember a visitor's version in the browser. People switch devices and clear cookies, and Safari caps cookies set by scripts at seven days (WebKit, ITP 2.1), so a returning visitor can end up counted as a new visitor in another group. The longer the test, the more of this mixing you get.
• The world changes. Sales, seasons, new products and price changes pile up. After two months, you are no longer testing one change in one context.
• Opportunity cost. A page locked in one test for months can't be used for the next idea.

These are rules of thumb, not statistical laws. Some teams accept 10–12 weeks for an important change. But if your calculator says "26 weeks", the honest answer is: this test is not feasible.

Why you shouldn't stop early

It's tempting to check results every day and stop when the variant is ahead with "95% significance". Don't.

Every time you check and are ready to stop, you give chance another opportunity to cross the line. Evan Miller showed that checking repeatedly and stopping at the first significant result can turn a 5% false positive rate into 26.1% (Evan Miller, How Not To Run an A/B Test).

We ran our own simulation to show the size of the problem for a shop. We simulated 50,000 A/A tests (both versions identical, so every "winner" is false): 2% conversion rate, 1,000 visitors per version per day, 28 days.

So with daily peeking, more than one in four "winners" of a do-nothing change would be fake. Bayesian results don't escape this either: in the same simulation, stopping the first day "chance to beat original" reached 95% produced a false winner in about 23% of tests. Our article on statistical significance explains why.

What to do instead:

• Fix the sample size and duration before the test starts.
• Look at the data during the test only to check that it works (both versions get visitors, conversions are recorded, the split is even). Our guide on sample ratio mismatch shows how.
• Make the decision at the planned end.

If you must be able to stop early, use a method designed for it (sequential testing), not a normal calculator and daily peeking.

What to do if the answer is "months"

A long duration is useful information. It means: this page and this metric can't prove an effect of this size. You have five honest options:

1. Test a bolder change. A full redesign of the product page information, a new offer or a different checkout path can move behavior by +20% or more. Changing a button color will not.
2. Test on more traffic. Run the test on a whole template (all product pages) instead of one page, or on the whole site for site-wide changes such as a delivery promise in the header.
3. Measure an earlier step. Add to cart, reaching the cart, or clicking "checkout" happen more often than orders, so they need fewer visitors (see Example 2).
4. Don't test, just fix it. A broken filter on mobile or a confusing delivery cost is not a hypothesis. Fix it and watch your numbers.
5. Research instead. Heatmaps, session recordings and on-site surveys tell you why people don't buy, with far less traffic than a test needs.

Our low-traffic guide walks through these options in order. To rank which ideas are worth a test at all, see how to prioritize A/B test ideas.

The math (optional)

For those who want to check the numbers. We use the standard formula for comparing two conversion rates (normal approximation):

n per version = (z₁₋α/₂ + z₁₋β)² × [p₁(1 − p₁) + p₂(1 − p₂)] / (p₂ − p₁)²

• p₁ = current conversion rate, for example 0.02
• p₂ = conversion rate you want to detect = p₁ × (1 + MDE), for example 0.022
• z₁₋α/₂ = 1.96 for 95% confidence, two-sided
• z₁₋β = 0.8416 for 80% power

Worked through for 2% and +10%: (1.96 + 0.8416)² = 7.849; p₁(1 − p₁) + p₂(1 − p₂) = 0.0196 + 0.021516 = 0.041116; (p₂ − p₁)² = 0.002² = 0.000004. So n = 7.849 × 0.041116 / 0.000004 ≈ 80,679 per version.

Quick rule of thumb. For 95% confidence and 80% power, n ≈ 16 × p(1 − p) / δ², where δ is the absolute difference you want to detect. Evan Miller recommends it in How Not To Run an A/B Test, and Kohavi, Deng and Vermeer use it in their KDD 2022 paper (3.7% conversion, +10% relative: about 41,642 users per variant). It gives slightly lower numbers than the full formula (78,400 instead of 80,679 for our 2% example), which is fine for a quick check.

With more variants, we divide alpha by the number of variants (Bonferroni) and recalculate, as in Example 4.

Different calculators use slightly different versions of the formula (pooled or unpooled variance, with or without a continuity correction). Expect differences of a few percent between tools. That doesn't change any decision.

FAQ

What if I don't know my conversion rate? Use the last four full weeks of data for the page you will test. If you have no data at all, run your analytics for a few weeks first. A guess of 3% when the truth is 1% makes your plan three times too optimistic.

Should I use sessions or users? Use the unit your testing tool assigns and counts. Most client-side tools assign a browser (a "visitor") and count each visitor once. If your conversion rate is per session but your tool counts visitors, your plan will be a bit off. Be consistent.

Can I test revenue per visitor instead of conversion rate? Yes, but revenue is much noisier than "did they buy or not", because a few large orders swing the average. Revenue tests usually need several times more visitors than conversion tests. If your tool decides on conversion rate, look at revenue per visitor as a secondary check.

Is 90% confidence OK for a small shop? It makes tests shorter, but one in ten do-nothing changes will then look like a winner. If you lower the bar, also lower your expectations: treat a 90% result as "probably not harmful" rather than "proven to win".

My test reached the sample size in 9 days. Can I stop? Run until the end of the second full week. The sample size is a minimum, not a finish line.

What does Varly do? In Varly (currently in private beta), the launch checklist runs this same calculation on your page's real traffic and conversion rate, plans in full weeks (minimum two), and warns you when a test can't finish within eight weeks. You can still launch it, and Varly offers ways to get a clearer answer: run up to 12 weeks, test a bolder change, or accept a lower certainty level. See Launch, pause and end a test.

---

Plan tests on your real numbers. Varly combines heatmaps, recordings, surveys and A/B testing in one script, and checks whether every idea has enough traffic before you build it. Varly is in private beta. Join the waitlist.