Ecommerce A/B Testing: The Complete Guide

What an A/B test can and can't tell you

In an A/B test, you split visitors at random into two groups. Group A sees your current page (the control or original); group B sees a changed version (the variant). After a planned period, you compare how many visitors in each group did what you care about, for example placing an order.

Because the split is random, the two groups are alike in everything except the change: same mix of devices, traffic sources, weekdays, returning customers. So if group B orders more, and the difference is bigger than chance would explain, the change caused it. That's why A/B testing is the standard for decisions in companies like Microsoft, Amazon and Booking.com: it measures cause, not just correlation.

What an A/B test can tell you:

• Whether a specific change increased (or decreased) a specific metric, for the visitors in the test, during the test period.
• Roughly how big the effect was, as a range.

What it can't tell you:

• Why the change worked. For that you need research: surveys, recordings, user tests.
• What to test. A test only answers the question you ask. Good questions come from research. Our ecommerce CRO guide explains where they come from.
• Whether the effect lasts forever. Seasons, customers and competitors change.
• Anything about small effects on small traffic. A shop with 5,000 visitors a month can't prove a 5% improvement, whatever tool it uses.

Do you have enough traffic?

This is the first question, not the last. The traffic you need depends on:

• Visitors who see the tested page (not your whole site)
• The conversion rate of those visitors on your chosen metric
• The smallest effect you want to detect (minimum detectable effect, MDE)

A quick reference, for a page where 2% of visitors place an order:

Most single changes move conversion by a few percent, not by 30%. So:

• Under ~20,000 visitors a month on the page, focus on fixing problems and research, and test only bold changes on earlier steps like add-to-cart. See A/B testing with low traffic.
• Between ~20,000 and ~100,000, you can test meaningful changes on templates (all product pages, the cart) with a 3–8 week horizon.
• Above ~100,000, you can start testing smaller refinements and run several tests in parallel.

Ron Kohavi, Alex Deng and Lukas Vermeer summarize the general guidance: A/B tests are useful for detecting effects of reasonable size when you have "at least, thousands of active users, preferably tens of thousands" (A/B Testing Intuition Busters, KDD 2022).

"Can I test this?" in six questions

1. Is it a bug or an obvious problem? → Fix it. Don't test.
2. Would you ship it even if the test showed no difference? (Legal text, accessibility, a broken filter) → Ship it and monitor.
3. Do you have evidence that it's a problem? → No: do research first. Yes: continue.
4. Can the test finish in 2–8 weeks with a realistic effect, using our sample size calculator? → No: make the change bolder, test on a whole template, or use an earlier metric. Still no: ship carefully and compare before/after.
5. Can you build it without breaking anything (prices, checkout, legal)? → No: rethink the variant.
6. Is nothing else changing on that page in the coming weeks (redesign, big sale)? → No: plan it for later.

If you get to the end, you have a testable idea.

Write a hypothesis

A hypothesis turns an idea into a test. It says what you saw, what you'll change, what you expect, and how you'll know. A simple template:

Because we saw [evidence], we believe that [change] for [audience] will [effect]. We'll know when [metric] changes by at least [X%] after [N] full weeks.

The "because we saw" part matters most. "Let's test a bigger button" is a guess. "Because 60% of mobile visitors never scroll to the add-to-cart button, we believe a sticky button on mobile will increase add-to-cart rate" is a hypothesis.

Our A/B test hypothesis template has 20 ecommerce examples by page. For a ranked list of ideas, see how to prioritize A/B test ideas and ecommerce A/B test ideas.

Choose one primary metric (plus guardrails)

Decide before the test which metric decides the outcome. Only one.

Primary metric options for shops:

Guardrail metrics are one or two metrics that must not get worse. They don't decide a win; they can veto one. Typical guardrails: orders (when your primary is add-to-cart), average order value, returns, page speed.

Why only one primary metric? If you look at ten metrics, one will probably look "significant" by chance. Picking the winner afterwards from the list is a reliable way to fool yourself. Our article on statistical significance explains why.

A note on checkout: many testing and analytics tools don't run on checkout pages (on Shopify, for example, the checkout isn't open to ordinary testing scripts, and privacy-minded tools often skip checkout on purpose). You can still test what happens before checkout and measure orders as the outcome, as long as orders are linked to the test visitors.

Sample size, MDE and duration in plain English

Three numbers define your test plan:

• Sample size: how many visitors each version needs.
• Minimum detectable effect (MDE): the smallest improvement the test can reliably prove. Planning for +10% means a true +5% effect will usually come out as "no clear difference".
• Duration: sample size × number of versions ÷ visitors per week, rounded up to full weeks, with a minimum of two weeks.

Example: product pages get 14,000 visitors a week; 3.5% place an order. You plan for +12%.

• Visitors needed: 31,787 per version (63,574 in total).
• 63,574 ÷ 14,000 = 4.5 weeks → 5 full weeks.

If you planned for +6% instead, you'd need 4 times as many visitors: about 18 weeks. Not feasible. Halving the MDE roughly quadruples the sample size.

Why full weeks: people shop differently on Mondays and Saturdays, and campaigns follow weekly rhythms. Why a maximum of about 8 weeks: the longer the test, the more visitors switch devices or clear cookies (and get counted twice), and the more your shop and season change around it. More in our sample size, MDE and duration guide.

More variants cost more. Each extra variant needs its own group of visitors, and the bar for a winner rises (you get more chances to be fooled by luck). On most shops, one strong variant beats three weak ones.

Build and QA the variant (incl. flicker)

Build the variant with the simplest tool that does the job:

• A visual editor for text, images, hiding or moving elements, and simple layout changes.
• Custom code (CSS/JavaScript) for anything more complex.
• A split URL test when the variant is a completely different page (a new landing page, a new product page template): visitors in the variant group are sent to another URL.
• Server-side changes for prices, shipping rules, search results or anything that the browser shouldn't decide.

QA before launch. Most broken tests are broken on day one. Check:

• The variant on desktop, tablet and mobile, in at least Chrome and Safari (iPhone).
• Every interactive element still works: variant pickers, add to cart, the mini-cart, pop-ups, cookie banner.
• The variant on several pages of the template (a long product name, a sold-out product, a product with one photo).
• The primary metric is recorded in both versions (do a test order or an add-to-cart in each).
• No console errors, no layout shifts, no slower loading.
• No flicker: visitors shouldn't see the original flash before the variant appears. Load the testing script high in the <head>, not through a tag manager. See the A/B test flicker effect.

Consider a dry run. Split visitors for a few days without showing the change. If the split comes out even and data arrives, you know the setup works before any customer sees the variant.

Running the test: what not to touch

Once the test is live, your job is mostly to leave it alone.

Don't:

• Stop early because it looks good (or bad). Early results swing wildly. In our simulation of 50,000 tests where both versions were identical, checking daily and stopping at the first "significant" result produced a false winner in about 28% of tests, versus about 5% when you only look at the end. Evan Miller documented the same effect (Evan Miller).
• Change the variant mid-test. If you must fix a bug, restart.
• Change the traffic split (for example from 10% to 50%). It mixes periods and can distort totals.
• Launch a big sale or a site redesign during the test without noting it.
• Run another test that changes the same element or the same step.

Do:

• Check health, not results, in the first days: data arriving, both versions getting visitors, no errors.
• Check the split. If visitors aren't split as planned (sample ratio mismatch), something is broken. See below.
• Keep a log of anything that happens: campaigns, newsletters, stock problems, outages.

Some exceptions are fine: if the variant clearly breaks something (orders drop to zero, a bug blocks checkout), stop it. That's quality control, not peeking.

Reading results: significance, confidence, Bayesian vs frequentist

At the planned end, read the results in this order:

1. Is the split OK? If the planned split was 50/50 but you got 51.5/48.5 on 20,000 visitors, that's too uneven to be chance. The result can't be trusted, however good it looks. See sample ratio mismatch.

2. What does the primary metric say? Testing tools give a verdict in one of two ways:

• Frequentist (p-value): "If the change did nothing, how surprising is this result?" Below p = 0.05 is called significant at 95% confidence. It does not mean "95% sure the variant is better", a mistake even vendors make, as Kohavi and co-authors document (KDD 2022).
• Bayesian ("chance to beat original"): "Given the data, how likely is it that the variant is better?" Easier to explain to a client. With the flat prior most tools use, a 95% chance to beat is roughly the same evidence as p = 0.10 rather than 0.05, so it's a slightly looser bar.

Both lead to the same decision in most tests. What matters more is that you set the bar before the test and wait for the planned duration.

3. How big is the effect, as a range? Look at the confidence (or credible) interval. "+11%, likely between +3% and +19%" is far more useful than "+11%". Use the lower end for business cases.

4. What happened to the guardrails? If add-to-cart went up but orders went down, it's not a win.

5. Segments are for ideas, not decisions. "It won on mobile" can be chance, especially with small numbers. Use segment results to plan the next test.

Our statistical significance guide explains all of this in more depth.

When results look odd (SRM, novelty, seasonality)

Some results deserve suspicion:

• Surprisingly large wins. Kohavi calls it Twyman's law: "any figure that looks interesting or different is usually wrong." Look for a bug before you celebrate.
• Uneven split (SRM). Microsoft reported that about 6% of its experiments had a sample ratio mismatch (Microsoft Research). Common shop causes: redirects, caching, consent banners, a variant that breaks on some devices.
• Novelty effects. Returning visitors click something new out of curiosity; the effect fades after a week or two.
• Seasonality and campaigns. A test that ran across Black Friday or a big paid campaign measured a different audience than your normal one.
• Winners that disappear after launch. Small tests that "win" tend to overstate the effect. A test planned for +20% that catches a real +5% effect will, on average, report something like +18% when it wins.

We cover each of these, with a diagnostic checklist, in why your A/B test results look strange.

How often tests win (and why that's fine)

Most A/B tests don't produce a winner. Kohavi, Deng and Vermeer collected published success rates (the share of ideas that truly improved the target metric):

Source: Kohavi, Deng and Vermeer, KDD 2022, Table 2, citing several earlier publications. Definitions differ per company, so treat these as ballpark figures. In a 2017 Harvard Business Review article, Kohavi and Stefan Thomke wrote that at Google and Bing "only about 10% to 20% of experiments generate positive results" (Kohavi and Thomke, HBR).

These are companies with large research teams and mature products. A young shop with obvious problems may do better at first. But expect many tests to end with "no clear difference".

Why that's fine:

• A loss is a cheap lesson. You found out in four weeks, with half your traffic, that an idea doesn't work. Without the test, you'd have shipped it to everyone.
• "No difference" saves you work. If the expensive new product page performs the same as the old one, you can skip the redesign.
• Wins compound. A few real wins of +5–10% a year on key templates add up.
• Low win rates mean many "winners" are false. The same paper calculates that when 10% of ideas truly work, about 22% of significant results at p < 0.05 and 80% power are false positives. That's why important wins deserve a second test.

The real failure is a test that teaches you nothing: no hypothesis, no planned duration, no documentation.

Reporting and documenting

Write up every test, including losers. A test report should answer, in this order:

1. What did we test and why? The hypothesis and the evidence.
2. What happened? Primary metric with the range, guardrails, the split check, the duration.
3. What do we conclude? Win, loss or no clear difference, in plain words.
4. What will we do now? Ship, don't ship, test a follow-up, or research more.
5. What did we learn about our customers? One or two sentences. This is the part people will search for in a year.

Keep all reports in one place with tags (page, element, audience, outcome), so you can find "all tests on the product page" or "everything about delivery". Our A/B test report template has a one-page version and a full version.

A worked example: from hypothesis to report

The evidence. A hypothetical home-goods shop runs a one-question survey on product pages: "Is anything stopping you from buying today?" Of 240 answers, 31% mention delivery: cost, time, or both. Recordings show mobile visitors scrolling to the footer and back, looking for shipping information.

The hypothesis. Because 31% of survey answers mention delivery and recordings show visitors searching for shipping info, we believe that showing the delivery cost and expected delivery date directly under the price, for all product page visitors, will increase orders. We'll know when orders per visitor on product pages rise by at least 12% after 5 full weeks, without average order value dropping.

The plan. Product pages get 14,000 visitors a week; 3.5% order. One variant. At +12% MDE, 95% confidence and 80% power: 31,787 visitors per version, 5 full weeks. Primary metric: orders per visitor. Guardrail: average order value.

QA and launch. The variant is built in a visual editor, checked on three devices and on products with long names and no stock. A three-day dry run shows an even split and orders arriving in both groups. Then the test goes live.

During the test. The team checks health twice a week. No peeking at the winner. A newsletter goes out in week 3; it's noted in the log.

The result after 5 weeks.

• Split check: 35,120 vs 34,880 is well within chance (p ≈ 0.36). OK.
• Effect: +11.4%, 95% interval about +3% to +19%.
• p ≈ 0.005; chance to beat original about 99.6%.
• Average order value: about the same in both groups.

The report, in plain words. "Showing delivery cost and date under the price increased orders per visitor by about 11% (likely between +3% and +19%). The split was even, the test ran its planned 5 weeks, and order value didn't change. We'll build it into the theme. Learning: delivery uncertainty is a real barrier on product pages. Next: test the same message in the cart, and a clearer returns message."

Note the forecast: the team uses roughly +3% (the low end) for the business case, not +11%.

If the result had been "+5.7%, likely between −2% and +14%, p ≈ 0.16", the report would say: "No clear difference. The change didn't hurt and may help a little, but the test couldn't prove it. We'll keep the original for now, and test a bolder version that also shows delivery information in the cart." That's a fine outcome too.

Client-side vs server-side vs native platform tests

There are three main ways to run A/B tests on a shop.

Client-side is where most shops start. It's quick to set up and doesn't need a developer for every test. The downsides are flicker and the weight of the script. See the A/B test flicker effect.

Server-side is cleaner and more powerful, but every test is a development task. It's the right choice for price tests, shipping rules and big template changes.

Native platform testing is growing. Shopify, for example, offers Rollouts, which can schedule changes to the theme, checkout and accounts pages, discounts or catalogs, and run an Experiment that compares a treatment against a control (Shopify Help Center). Check what it measures and which changes it supports for your plan. Our Shopify A/B testing guide goes deeper.

Since Google Optimize shut down on September 30, 2023, there's no free, general-purpose testing tool from Google; most shops now use a dedicated tool. If you're comparing tools, see our overview of VWO alternatives.

In Varly, A/B tests are client-side, built with a visual editor or code, or as split URL tests, and every test shows a traffic plan, a dry-run option, a split check and a plain-words verdict. Results are Bayesian by default, with frequentist available. See Create an A/B test and Read your test results.

A/B, A/B/n, multivariate and split URL: which test type when

• A/B test (one variant against the original): the default. Fastest to reach a result, easiest to read.
• A/B/n test (several variants against the original): useful when you have two or three genuinely different solutions to the same problem. Each variant needs its own group, and the bar for a winner should go up with every extra variant (a common method, Bonferroni, divides the 5% error margin by the number of variants). In our calculator examples, going from one variant to two stretched a 5-week test to 8 weeks.
• Multivariate test (several elements changed in all combinations, for example 2 headlines × 2 images = 4 versions): tells you which element mattered and whether they work together. It needs a lot of traffic. Most small and mid-size shops should test the combination they believe in as one variant instead.
• Split URL test (the variant is a different URL): the cleanest way to test a completely new page or template that's hard to build with a visual editor. Watch out for slow redirects, which can cause an uneven split.

A/B testing and cookie consent in the EU

In the EU and UK, most client-side testing tools store which version a visitor saw in the browser, which normally requires consent. In practice, that means:

• Only visitors who accept cookies take part. Your test population is smaller than your traffic. If 60% of visitors accept, a page with 20,000 visitors a month gives you about 12,000 test visitors. Plan your sample size with that number.
• Your results describe people who accept cookies. They may differ a little from people who decline (for example, more returning customers). That's usually acceptable, but worth knowing.
• Visitors who decline see your normal site. That's the safe default.
• Consent and assignment should happen in the right order. If a tool assigns a version before consent but only counts visitors after, and the variant affects how likely people are to accept, you can get an uneven split. See sample ratio mismatch.

Some vendors claim their testing needs no consent. Regulators see it differently in many cases; check with your privacy advisor. Our GDPR guide for heatmaps, session recordings and A/B testing covers this in detail.

Building a testing program

One test is a project. A testing program is a habit. The basics:

• A research rhythm. Every month: look at analytics, heatmaps, recordings and survey answers. Add findings to the backlog.
• A backlog with scores and a traffic check for each idea.
• A test calendar. Which test runs on which template, when. Avoid overlaps on the same element and pause around big sales.
• A fixed review meeting. Every two weeks: what's running, what's finished, what's next.
• A searchable library of every test, with tags and learnings.
• One owner. Someone who keeps the process running, even when it's busy.

Start with one test at a time. Add parallel tests on different templates only when the process works.

FAQ

How long should an A/B test run? Until it reaches the planned sample size, rounded up to full weeks, with a minimum of two weeks. If that's more than about eight weeks, change the plan.

What's a good conversion lift from an A/B test? There's no standard. Many real effects are a few percent. Big lifts happen mostly when you fix a real problem (unclear delivery costs, a broken step). Treat very large lifts with suspicion until you've checked the setup.

Can I A/B test with a small shop? Yes, but only bold changes, on templates, measured on earlier steps like add-to-cart. Below about 10,000 visitors a month on a page, research and fixing usually beat testing.

Should I test on mobile and desktop separately? Test on all devices together unless your change is device-specific. Look at device segments afterwards for ideas, not for the decision.

Can I run multiple tests at the same time? Usually, yes, as long as they don't change the same element or step. Microsoft's experimentation team found harmful interactions between concurrent tests to be rare (Microsoft Research).

Do A/B tests work with cookie consent? Most tools only include visitors who accept cookies, so your test population is smaller than your traffic, and possibly different (people who accept may behave differently). Plan your sample size with the visitors who will actually take part.

---

Run your first test on solid ground. Varly puts heatmaps, recordings, surveys and A/B tests in one script, checks the traffic before you build, and gives no verdict before the planned duration ends. Varly is in private beta, with planned pricing at launch of $99 per website per month. Join the waitlist.