A/B Test Hypothesis Template (+20 Ecommerce Examples)
A simple A/B test hypothesis template for online shops, with 20 examples by page, good vs bad hypotheses, and how to add metrics, guardrails and sample size.
By Joost BakkerPublished 12 min read
On this page
What a good hypothesis has
A hypothesis is a short, testable statement about cause and effect. In CRO, it does three jobs:
- It forces you to say why. If you can't name the evidence, you probably shouldn't test it yet.
- It defines success before the test. That stops you from searching the results for any metric that looks good afterwards.
- It makes the result useful either way. If the test wins, you learn the evidence was right. If it loses, you learn something about your customers, instead of just "the green button didn't work".
The structure we use builds on Craig Sullivan's Hypothesis Kit, which many CRO teams use in some form (Growth Method on the Hypothesis Kit). The core idea: start from data, state the change and the expected outcome, and define how you'll measure it.
A good hypothesis is:
- Based on evidence: a survey answer, recordings, a heatmap, analytics, a user test, customer service tickets.
- Specific: a concrete change on a concrete page, for a defined group of visitors.
- Measurable: one primary metric, with the size of effect you plan for.
- Testable with your traffic: the test can finish within a reasonable time (usually 2–8 weeks).
- Falsifiable: there's a clear outcome that would prove it wrong.
The template
Copy this into your test plan, backlog or project tool:
Because we saw [evidence: data, quotes, recordings, with numbers] we believe that [the change] for [audience: all visitors, mobile, new visitors, a traffic source] will [expected effect: what behavior changes, and why] We'll know when [primary metric] changes by at least [X% relative] after [N full weeks], without [guardrail metric] getting worse.
A filled-in example:
Because we saw that 31% of answers to our product page survey ("Is anything stopping you from buying today?") mention delivery costs or delivery time, and session recordings show mobile visitors scrolling up and down between price and footer, we believe that showing the delivery cost and expected delivery date directly under the price for all product page visitors will increase orders, because visitors no longer need to look for this information or leave to find it. We'll know when orders per visitor on product pages rise by at least 12% after 5 full weeks, without average order value dropping.
The numbers in "we'll know when" come from a sample size calculation. In this hypothetical shop, product pages get 14,000 visitors a week and 3.5% of them place an order. To detect +12% at 95% confidence (two-sided) and 80% power, the test needs about 31,787 visitors per version, which is about 4.5 weeks of traffic, so 5 full weeks. Use our sample size calculator for your own numbers.
A shorter version for your backlog
For the backlog, before you've done the sample size math, a three-line version is enough:
- If we see… 31% of survey answers mention delivery costs or time.
- …then we change… show delivery cost and date under the price on product pages.
- …because we expect… more visitors to order, since they don't have to look for it.
You'll add the metric, threshold and duration when the idea makes it to the top of the backlog. (Varly's ideas use exactly these three fields; see Where test ideas come from.)
20 examples by page
These are example hypotheses for a typical online shop. The evidence in each is hypothetical, but it's the kind of finding you get from heatmaps, recordings, surveys and analytics. Replace it with your own data; never copy a hypothesis without the evidence.
Homepage
1. Clear value proposition Because we saw that 40% of new visitors leave the homepage without clicking anything, and survey answers include "not sure what you sell", we believe that a hero section stating what we sell, for whom, and our main benefit, for new visitors, will increase the share who click through to a category. We'll know when category click-through from the homepage rises by at least 10%.
2. Shortcuts to top categories Because we saw in the click map that 55% of homepage clicks go to the menu, and only 6% to the hero banner, we believe that replacing the rotating banner with tiles for the four best-selling categories will increase visits to category pages and orders per visitor.
3. Delivery promise above the fold Because we saw that "delivery" is the most common word in our cart-page survey answers, we believe that a delivery promise ("Ordered before 22:00, delivered tomorrow") in the header bar, site-wide, will increase orders per visitor.
Category pages and search
4. Working mobile filters Because we saw rage clicks on the "Size" filter in 4% of mobile category sessions, and recordings show the filter panel closing after each choice, we believe that a filter panel that stays open until "Show results" is tapped, for mobile visitors, will increase product page visits from category pages. (If the filter is simply broken, fix it instead.)
5. Review stars on product tiles Because we saw that 22% of survey answers mention reviews or "not sure about quality", and product pages with reviews convert better in analytics, we believe that showing star ratings on category tiles will increase clicks to products and add-to-cart rate.
6. Better zero-result searches Because we saw that 12% of site searches return no results (top terms are spelling variations and brand names), we believe that showing suggestions and popular products on the no-results page will reduce search exits.
7. Default sort order Because we saw that most category visitors never scroll past the first two rows, and the default "newest" sort puts low-stock items first, we believe that sorting by "best-selling" by default will increase add-to-cart rate from category pages.
Product pages
8. Delivery information next to price (The worked example above.)
9. Sticky add-to-cart on mobile Because we saw in the scroll map that 60% of mobile product page visitors never reach the add-to-cart button below the product description, we believe that a sticky add-to-cart bar on mobile will increase add-to-cart rate, without lowering orders.
10. Size guide with real measurements Because we saw that "size" is the top reason for returns and the top survey answer on product pages, we believe that a size guide with garment measurements and "fits like" notes from reviews will increase orders and reduce size-related returns.
11. Answer the top questions Because we saw that 30% of customer service emails ask the same three questions (material, washing, delivery to islands), we believe that answering them in a short FAQ block under the add-to-cart button will increase orders per visitor.
12. Product photos in use Because we saw that visitors who open the photo gallery convert more than those who don't, and the most-viewed photo is the only "in use" shot, we believe that putting an in-use photo first will increase add-to-cart rate.
13. Clearer stock and variant states Because we saw dead clicks on grayed-out size buttons, and recordings show visitors not understanding which sizes are sold out, we believe that clearly labeling unavailable sizes ("sold out, notify me") will increase add-to-cart rate for products with partial stock.
Cart
14. Show the total early Because we saw that 45% of cart visitors leave without starting checkout, and survey answers mention unexpected shipping costs, we believe that showing the shipping cost and total in the cart (based on the visitor's country) will increase checkout starts.
15. Make guest checkout visible Because we saw in user tests that 3 of 5 people thought they had to create an account, we believe that a clear "Continue as guest" option on the cart page will increase orders.
16. Free-shipping progress Because we saw that 18% of orders are just below the free-shipping threshold, we believe that a progress message in the cart ("Add €8 for free shipping") will increase average order value without lowering conversion rate.
17. Fewer distractions Because we saw that 9% of cart visitors click the discount code field and then leave the site (recordings show them searching for codes), we believe that collapsing the field behind a small "Have a code?" link will increase checkout starts.
Site-wide
18. Trust in the header Because we saw that new visitors from paid social convert at half the rate of returning visitors, and survey answers ask "is this shop legit?", we believe that showing review score, return policy and payment logos in the header, for new visitors, will increase orders.
19. Faster mobile pages Because we saw that mobile product pages take over 4 seconds to show the main image, and mobile conversion is half of desktop, we believe that loading the main image first (and deferring other scripts) will increase mobile orders. (Often better shipped as a fix and measured before/after.)
20. Exit survey insights turned into a promise Because we saw that "I want to compare first" is the top exit survey answer, we believe that a price-promise or easy-returns message near the price will reduce exits and increase orders.
Not sure where to find this kind of evidence? Start with heatmaps, recordings and a single survey question. See our guide to analyzing session recordings.
Bad vs good hypotheses
| Bad | Why it's bad | Better |
|---|---|---|
| "A green button will convert better." | No evidence, no reason, no metric. | "Because we saw in the heatmap that the add-to-cart button gets fewer clicks than the wishlist icon next to it, we believe that making it the only high-contrast element will increase add-to-cart rate." |
| "Let's test a new homepage." | Not a hypothesis, just a project. | Pick the specific problem the new homepage solves, and state the expected behavior change. |
| "Adding reviews will increase conversion by 30%." | A promise, not a hypothesis; no evidence; unrealistic precision. | "Because 22% of survey answers mention quality doubts, we believe showing reviews will increase add-to-cart rate. We'll know when it rises by at least 10%." |
| "Competitor X has a sticky bar, so we should too." | Copying, not evidence. Competitors test badly too. | Check your own scroll data first. If visitors don't reach the button, you have evidence. |
| "Improve the user experience of the cart." | Vague; can't fail. | Name the change, the audience, and the metric. |
| "We'll measure conversion, bounce rate, time on site and revenue." | Too many metrics; one will look good by chance. | One primary metric, one or two guardrails. |
Adding metric, guardrails and sample size
Before a hypothesis becomes a test, add these four things:
1. One primary metric. The metric that decides the test. Choose the one closest to your goal that you can still measure with enough visitors. Orders per visitor is ideal; add-to-cart rate is a common fallback for smaller shops. Our article on A/B testing with low traffic explains the trade-off.
2. Guardrail metrics (one or two). Metrics that must not get worse: orders (when the primary is add-to-cart), average order value, returns. They don't decide a win, but they can veto one.
3. Minimum detectable effect and duration. The smallest effect worth detecting, and how many full weeks that takes with your traffic. If it's more than about 8 weeks, change the plan: bolder change, bigger page group, earlier metric, or ship without a test.
4. Audience and pages. Where the test runs (one page, all product pages, the whole site) and who takes part (everyone, mobile only, new visitors only). The hypothesis should match: if your evidence is about mobile, test on mobile.
A complete test brief then looks like this:
| Field | Example |
|---|---|
| Hypothesis | Because we saw… we believe… for… will… |
| Evidence links | Survey results, recording playlist, heatmap |
| Page / template | All product pages |
| Audience | All visitors who accept cookies |
| Primary metric | Orders per visitor |
| Guardrails | Average order value |
| MDE | +12% (relative) |
| Visitors needed | 31,787 per version |
| Planned duration | 5 full weeks |
| Variants | 1 (plus original) |
| Decision rule | Ship if significant at 95% after 5 weeks; otherwise keep original and log the learning |
To decide which hypothesis to test first, use a scoring framework: see PXL vs ICE vs PIE. After the test, write it up with our A/B test report template.
What to write after "We'll know when": the smallest effect your traffic can prove
A product page with 5,000 visitors a week and a 3.5% order rate can only prove an effect of about +22% on orders in a 4-week test, but about +13% on add-to-cart. So the "at least X%" in your hypothesis isn't a wish. Your traffic sets it, and it differs a lot per metric.
We computed it for the four metrics the 20 examples above use:
| Metric (examples that use it) | Baseline rate (assumed) | 1,000 visitors/week | 2,500/week | 5,000/week | 10,000/week | 25,000/week |
|---|---|---|---|---|---|---|
| Orders per visitor, product pages (3, 8, 10, 11, 18, 20) | 3.5% | +52% | +32% | +22% | +15% | +9.5% |
| Add-to-cart rate, product pages (5, 7, 9, 12, 13) | 9% | +30% | +18.5% | +13% | +9.1% | +5.7% |
| Click-through from homepage to a category (1, 2) | 35% | +12% | +7.7% | +5.4% | +3.8% | +2.4% |
| Checkout starts from the cart (14, 17) | 55% | +8.0% | +5.1% | +3.6% | +2.5% | +1.6% |
Each cell is the smallest relative effect a 4-week test can detect, for visitors per week to the page you test.
How we calculated it (7 November 2026): two-sided test at 95% confidence, 80% power, one variant, 50/50 split, 4 full weeks, so visitors per version = 2 × weekly visitors. We solved n = (1.96 + 0.84)² × [p₁(1 − p₁) + p₂(1 − p₂)] ÷ (p₂ − p₁)² for p₂, where p₁ is the baseline rate. The baseline rates are illustrative. With your own rates, use the sample size calculator.
How to use it:
- Metrics many visitors reach can prove small changes. Clicks and checkout starts happen often, so a few percent is detectable. Orders happen rarely, so on most product pages you need an effect of +15% or more.
- Use the traffic of the page you test. A cart gets far fewer visitors than your product pages. If your cart gets 1,500 visitors a week, read between the 1,000 and 2,500 columns, not the column for your product pages.
- If the number is bigger than you can honestly expect, change the plan, not the number. Measure one step earlier (add-to-cart instead of orders, with orders as a guardrail), run on all product pages instead of one, or combine several changes that address the same problem. Writing "+10%" on a page that can only prove +30% just books a "no clear difference" in advance.
- An earlier metric is not a free win. A +5% click-through effect on the homepage says nothing about orders on its own. Keep orders as a guardrail. Our article on A/B testing with low traffic explains the trade-off.
The delivery example above passes this check because product pages get 14,000 visitors a week: in 5 weeks, that's enough to prove about +11–12% on orders.
Using AI to draft hypotheses (and checking them)
AI assistants are good at turning messy research into neat hypotheses. Give one your survey answers, a summary of what you saw in recordings and your analytics numbers, and ask for hypotheses in the template above. It will produce ten in a minute.
The risks:
- Invented evidence. The AI may write "Because we saw that 35% of visitors…" with a number that's not in your data. Check every number against the source.
- Generic best practice. Without your data, AI falls back on "add trust badges" and "use urgency". These are guesses dressed up as hypotheses.
- Untestable effects. AI doesn't know your traffic. It may propose a +5% effect on a page that needs +30% to be testable.
- Overconfidence. "Will increase conversion by 25%" is not something anyone can know in advance.
A quick checklist for AI-drafted hypotheses:
- Every "because we saw" points to a real finding you can link to.
- The change is specific enough to build.
- The audience matches the evidence (mobile evidence → mobile test).
- The metric is one you can measure, with a sample size check done.
- Ideas based only on "best practice" are labeled as such, and ranked lower.
For more on what AI does well and badly in CRO, see our guide on AI for conversion optimization.
Copy the template
Here's the complete template again, ready to paste into a doc, spreadsheet or Notion page:
HYPOTHESIS
Because we saw: [evidence, with numbers and links]
We believe that: [change]
For: [audience]
Will: [expected behavior change, and why]
We'll know when: [primary metric] changes by at least [X%] after [N] full weeks
Without: [guardrail metric] getting worse
TEST PLAN
Page / template:
Audience:
Primary metric:
Guardrail metrics:
Baseline rate:
Minimum detectable effect:
Visitors needed per version:
Planned duration (full weeks):
Decision rule:
RESULT (after the test)
Outcome: win / loss / no clear difference
Result with interval:
What we learned:
Next step:
FAQ
What's the difference between a hypothesis and an idea? An idea is "let's add reviews to category pages". A hypothesis adds why (evidence), what you expect, and how you'll know. Every hypothesis starts as an idea; not every idea deserves to become one.
Do I need a hypothesis for every test? Yes. Even a quick test benefits from a one-line "because we saw…, we expect…". It's what turns a result into a lesson.
What if my hypothesis is wrong? That's the point of testing. A clearly wrong hypothesis is a useful result: it tells you the problem you saw wasn't the problem that mattered, or that your solution didn't fix it. Write down why you think it failed.
Should a hypothesis predict the exact effect size? No. State the minimum effect you plan for (your MDE), not a prediction like "+23%". The test tells you the actual size, with an interval.
Can one hypothesis cover several changes? Yes, if they all address the same problem (for example, several changes that make delivery costs clearer). You'll learn whether the package works, not which part did it. On small shops, that's often the smart choice.
Keep hypotheses next to the evidence. In Varly, every idea is written as "if we see… then we change… because we expect…", linked to the heatmaps, recordings and survey answers behind it, with a traffic check before you build. Request access.
