A/B Testing With Low Traffic: What Small Shops Can Do

How much traffic do you really need?

You've probably read rules like "you need 1,000 conversions a month" or "at least 5,000 visitors a week". They're not wrong, but they hide the real question. A test needs enough visitors to tell the effect you care about apart from chance. Big effects need few visitors; small effects need a lot.

This table shows the smallest effect a page can reliably detect, given its monthly visitors. "Order rate 2%" is a typical product page that converts 2% of visitors into orders; "add-to-cart 8%" is the same page measured on add-to-cart.

(Monthly visitors are converted to weeks as monthly × 12 ÷ 52. Effects are relative: +20% on 2% means 2.0% → 2.4%.)

How to read it: a shop with 20,000 visitors a month on its product pages can, in four weeks, prove an effect on orders of about +31% or bigger. Most single changes don't do that. On add-to-cart, the same shop can detect +14%, which is a lot more realistic for a solid change.

Ron Kohavi, Alex Deng and Lukas Vermeer, who between them have run experimentation at Microsoft, Airbnb and Booking.com, put the general guidance like this: A/B tests are useful for effects of reasonable size when you have "at least, thousands of active users, preferably tens of thousands" (A/B Testing Intuition Busters, KDD 2022). In the same paper they show what happens below that: a published test with about 80 users per variant had about 3% power to detect a 10% change, which makes the result close to meaningless.

Two things that make your real traffic smaller

• Only the tested page counts. Your site may get 50,000 visits a month, but if the test runs on the cart page, only the visitors who reach the cart take part.
• Consent. In the EU, many testing tools only include visitors who accept cookies. If 60% of visitors accept, a page with 20,000 visitors gives you about 12,000 test visitors. Use the number that will actually take part.

Put your own numbers into our sample size and duration calculator: enter your visitors, conversion rate and the smallest change you care about.

The ladder: what to do instead (or before)

Think of the options as a ladder. Start at the bottom; climb only as far as your traffic allows.

1. Just fix it
2. Research instead of testing
3. Test bigger changes
4. Test whole page templates
5. Measure earlier steps
6. Before/after comparisons, carefully

Option 1: Just fix it

A lot of what small shops "want to test" isn't a question. It's a bug or a clear usability problem:

• The size filter doesn't work on mobile.
• Delivery costs only appear at the last checkout step.
• The "add to cart" button is below three screens of text on a phone.
• A discount code field that sends people off to search for codes.
• Product photos that don't zoom, on a product where details matter.

You don't need a test to know these hurt. Fix them, note the date, and keep an eye on your numbers. Baymard Institute's checkout research is a good source of "known problems": in its survey of US shoppers who abandoned a purchase, 40% named extra costs (shipping, taxes, fees) as the reason, and 18% a forced account creation (Baymard Institute, updated October 2026). If your shop has one of those, that's a fix, not a test.

A good rule: if you'd ship the change even if a test showed "no difference", don't test it. Testing is for decisions where you could be wrong in either direction.

Option 2: Research instead of testing

Research tools need far less traffic than a test, because they answer why rather than how much.

• Heatmaps show where people click, how far they scroll, and which elements nobody notices. A few hundred visits to a page is usually enough to see the big patterns. See our guide on how to analyze heatmaps.
• Session recordings show real visits. Filter on visits that reached the cart but didn't buy, or on rage clicks (repeated clicks on the same spot), and watch 10–20. You'll find bugs and confusion that no report shows.
• On-site surveys ask visitors directly. One open question on the cart page ("Is anything stopping you from ordering today?") can tell you more in two weeks than a test could in three months. See our list of on-site survey questions.
• Funnel reports show where people drop off: product page → cart → checkout. Fix the biggest leak first.

Research doesn't prove that a change will help. But it tells you what's wrong, so the changes you make (or test) are much more likely to matter. For a small shop, that's the best use of time.

Option 3: Test bigger changes

If you test, make the change big enough to be worth proving. That doesn't mean random redesigns. It means changes that alter what visitors know, feel or have to do:

Bold changes have a downside: if they win, you won't know exactly which part did it. For a small shop, that's fine. You want more orders, not a research paper. If you want to understand which part worked, test the pieces later, when you have more traffic.

Also: test one variant, not three. Every extra variant needs more visitors and a stricter bar. In our calculator example, going from one to two variants on a page with 10,000 visitors a week stretched a 5-week test to 8 weeks.

Option 4: Test whole page templates

Don't test one product page. Test all product pages at once, using the same change on the template. Same for category pages, or the cart.

Most shops' traffic is spread over many pages of the same type. A single product page might get 800 visits a month; all product pages together might get 15,000. The change (for example, a delivery-information block under the price) applies to all of them, and so does the test.

Site-wide elements work too: the header, the menu, a free-shipping bar. Those reach every visitor.

Option 5: Measure earlier steps

Orders are rare; earlier steps are not. Common choices:

• Added to cart (often several times more frequent than orders)
• Reached the cart or clicked "checkout"
• Clicked a key element, such as the size guide

Look at the table at the top: the same page that can only detect +31% on orders can detect +14% on add-to-cart.

The catch: a change can lift add-to-carts without lifting orders. A pushy "Only 2 left!" might get more items into carts that are then abandoned. So:

• Use the earlier step as the primary metric (it decides the test).
• Keep orders as a secondary metric, and don't ship if orders clearly drop.
• Choose steps close to the purchase where possible. "Clicked checkout" says more about buying than "clicked a product photo".

Option 6: Before/after comparisons, carefully

Sometimes you can't split traffic at all: a new checkout provider, a new theme, a new shipping policy. Then the only option is to compare before and after. This is the weakest form of evidence, because everything else changes too: season, ads, competitors, the weather.

You can make it less bad:

• Compare like with like. Same number of full weeks before and after, no big sales in either period, same days of the week.
• Use a comparison group if you have one. Changed the product page for one category only? Compare how that category moved against the categories you didn't change. If both went up by 10%, your change probably did nothing.
• Look at year-over-year too. If conversion rose 8% after your change but also rose 8% in the same weeks last year, it's probably the season.
• Write down your prediction before the change. "We expect add-to-cart on mobile to rise from 6% to 7%." It stops you from finding a story afterwards.
• Watch the direction, not the decimal. A before/after can tell you "clearly better", "clearly worse" or "can't tell". Don't report "+4.3%".

What not to do

Don't lower the bar until things "win". Going from 95% to 80% confidence makes tests faster, but at 80% one in five do-nothing changes will look like winners. If you do it, call results "probably not harmful" rather than "proven".

Don't assume Bayesian is a shortcut. Bayesian "chance to beat" numbers are easier to read, and some vendors present them as faster. Part of that speed comes from a looser bar: with the usual flat prior, a 95% chance to beat is roughly the same evidence as a p-value of 0.10, not 0.05. That's a trade-off, not a free lunch. Our article on statistical significance explains it.

Don't stop when it looks good. Small tests swing wildly in the first days. In our simulation of identical versions, checking daily and stopping at the first "significant" result produced a false winner in about 28% of tests. Fix the duration up front.

Don't trust segment results. "It won on mobile returning visitors!" A small test split into small segments is mostly noise.

Don't run tests that can't finish. A 6-month test isn't a test; by the time it ends, your shop, your ads and your season have changed.

A plan for shops under 30,000 visits a month

A practical 12-week plan, assuming a few hours a week:

Weeks 1–2: measure and fix

• Check your analytics: conversion rate, add-to-cart rate, top pages, mobile vs desktop.
• Walk through your own shop on a phone, from ad to order confirmation. Write down every annoyance.
• Fix bugs and obvious problems (delivery costs, broken filters, slow pages).

Weeks 3–6: research

• Set up heatmaps on your homepage, a top category page and the product page template.
• Watch 20 session recordings of visitors who added to cart but didn't buy.
• Run one survey question on the product page or cart. Aim for 50–100 answers.
• Group what you find into 5–10 problems, with evidence for each.

Weeks 7–8: pick and build

• Turn the problems into ideas and rank them (see how to prioritize A/B test ideas). Ask for each one: is this a fix or a test? Can it finish with our traffic?
• Choose one bold change on a template, with add-to-cart as the primary metric and orders as a guardrail.

Weeks 9–12: test (or ship and watch)

• Run the test for its planned full weeks. Don't stop early.
• If the page can't support a test, ship the change and do a careful before/after.
• Write down what you learned, whatever the outcome, and go back to research.

Repeat. After a year you'll have fixed a dozen real problems and run a handful of meaningful tests, which beats twenty inconclusive button tests.

For the full process, see our ecommerce CRO guide and the A/B testing guide.

FAQ

Is A/B testing worth it for a small shop? Sometimes. If your tested page gets fewer than about 10,000 visitors a month, only very large effects are provable on orders. Start with fixes and research. Test bold changes on templates and earlier steps when you can.

What's the minimum traffic for an A/B test? It depends on the effect size. As a rough guide: with 20,000 monthly visitors on a page converting at 2%, four weeks can detect about +31% on orders. Use the calculator with your own numbers.

Can I run an A/B test for a few days to get a quick answer? No. Short tests on small traffic are mostly noise, and you'll miss weekday patterns. Run at least two full weeks.

Should I use multi-armed bandits on low traffic? Bandits shift traffic toward the leading variant during the test. They're useful for short-lived decisions (like which banner to show during a sale), but they don't create information out of nothing: with low traffic they'll often lock in on noise. For learning what works, a fixed test is clearer.

Isn't qualitative research just opinions? Not if you collect it systematically. Twenty recordings showing the same confusion, or 30% of survey answers naming delivery costs, is evidence. It just isn't a precise measurement of the effect size.

---

Research and test in one place. Varly combines heatmaps, recordings, surveys and A/B testing in one script, and shows for every idea whether your page has enough traffic to test it. If it doesn't, it tells you. Varly is in private beta. Join the waitlist.