A/B Test Report Template (Copyable, Incl. Inconclusive Tests)

A/B TEST REPORT — [Test name]                       [Test ID] · [Date]

1. DECISION
   [Ship the variant / Keep the original / Rerun with changes / Stop]
   One sentence why: [...]

2. RESULT (primary metric: [e.g. orders per visitor])
   Original: [x.xx%]  ([orders] / [visitors])
   Variant:  [x.xx%]  ([orders] / [visitors])
   Change:   [+/-xx%], likely between [low %] and [high %] (95% range)
   Confidence: [chance to beat original xx%] or [p-value 0.0xx]
   Split check (SRM): [OK / problem]
   Guardrails: [e.g. revenue per order, add-to-cart, mobile]: [no change / changed]

3. REVENUE IMPACT (per month, if shipped to everyone in the test audience)
   Low:      [€...]   (bottom of the range)
   Expected: [€...]   (measured change)
   High:     [€...]   (top of the range)
   Assumptions: [visitors per month], [average order value], [audience]

4. ORIGINAL PLAN (as agreed before launch)
   Hypothesis: Because we saw [evidence], we believe [change] for
               [audience] will [effect]. We'll know when [metric] [threshold].
   Page(s) / audience: [...]
   Primary metric: [...]   Secondary metrics: [...]
   Smallest effect we planned to detect: [+xx%]
   Planned sample: [n per version]   Planned duration: [x weeks]
   Actual: [n per version], [x weeks], [start date – end date]
   Deviations: [none / what changed and why]

5. LEARNINGS
   - [What this tells us about our customers]
   - [What surprised us]
   - [Segment notes, marked as "to verify", not as results]

6. NEXT STEPS
   - [Action] — [owner] — [date]
   - [Follow-up test idea] — [added to backlog with score]

Links: [test in tool] · [screenshots] · [hypothesis doc] · [recordings/heatmaps]

A/B TEST REPORT — Delivery date under price (PDP)        T-014 · March 3

1. DECISION
   Ship the variant on all product pages.
   Why: more visitors ordered, and the whole likely range is positive.

2. RESULT (primary metric: orders per visitor)
   Original: 2.50%  (800 / 32,000)
   Variant:  2.85%  (912 / 32,000)
   Change:   +14%, likely between +4% and +24% (95% range)
   Confidence: p-value 0.006 · chance to beat original 99.7%
   Split check (SRM): OK
   Guardrails: add-to-cart up; refund rate in shop data unchanged

3. REVENUE IMPACT (per month, all product-page visitors)
   Low:      +40 orders  ≈ €2,800
   Expected: +140 orders ≈ €9,800
   High:     +240 orders ≈ €16,800
   Assumptions: 40,000 product-page visitors/month who accept cookies,
   average order value €70 (unchanged), effect holds after launch.

4. ORIGINAL PLAN
   Hypothesis: Because 31% of exit-survey answers mention delivery time,
   we believe showing the delivery date under the price for all visitors
   will increase orders. We'll know when orders per visitor rise by 15%+.
   Primary metric: placed an order. Secondary: added to cart.
   Smallest effect planned: +15%. Planned sample: ~29,200 per version.
   Planned duration: 4 weeks. Actual: 32,000 per version, 4 weeks.
   Deviations: none.

5. LEARNINGS
   - Delivery certainty matters to our visitors at the moment they
     look at the price, not only in the checkout.
   - The effect looked larger on mobile (to verify in a follow-up test;
     segment too small to conclude).

6. NEXT STEPS
   - Build delivery date into the theme — dev team — March 17
   - Test the delivery date in the cart and on category cards — backlog

Reporting losers and "no difference"

Most tests don't produce a clear winner. That's normal, and a report that hides it destroys trust faster than a losing test. Report them in the same format, with the same care.

A worked example: no clear difference

Hypothetical example. A shop tests a shorter product description on product pages.

1. DECISION
   Keep the original. Don't ship the shorter description.
   Why: no clear difference, and the change costs copywriting time.

2. RESULT (orders per visitor)
   Original: 2.50%  (750 / 30,000)
   Variant:  2.57%  (770 / 30,000)
   Change:   +2.7%, likely between −7.4% and +12.7% (95% range)
   Confidence: chance to beat original 70%
   Split check (SRM): OK

3. REVENUE IMPACT (per month)
   Low: −€5,180 · Expected: +€1,890 · High: +€8,890
   (40,000 visitors/month, €70 average order value)
   → The range runs from a loss to a gain. We can't tell them apart.

(Low: 40,000 × 2.50% × −7.4% = −74 orders × €70. Expected: 40,000 × 2.50% × 2.7% = 27 orders. High: 40,000 × 2.50% × 12.7% = 127 orders.)

How to write about these results

• Say what you can rule out. "If the shorter description helps, it helps by less than about 13%." That's useful: it tells you not to expect big gains from copy length on this page.
• Don't call it a failure. You learned that description length isn't what's holding visitors back. That saves you from rewriting 400 product descriptions.
• Don't say "the variant performed slightly better." With a range that includes zero, you don't know that.
• Don't dig through segments until one "wins." If you see something interesting in a segment, write it down as a new hypothesis and test it properly.
• Losers are the cheapest lessons. A variant that lost cost you a few weeks of a small share of sales. Shipping it without a test would have cost you that loss for good.

Wording that helps

---

Business impact without over-promising

Revenue projections are where test reports lose credibility. Five rules:

1. Use the range. Always show low, expected and high, never only the expected value.
2. Use the right audience. Multiply by the visitors who'll actually see the change: the page or template, and only visitors comparable to the test audience. If your tool only tests visitors who accept cookies, say so.
3. Keep average order value fixed unless you measured it. If the test decided on conversion rate, don't assume order value changed. Check order value in your shop data and mention it as a guardrail.
4. Be careful with yearly numbers. Effects often fade (novelty wears off, season changes, the site changes around it). If you show a yearly figure, show it as "12 × monthly, if the effect holds" and use the low end.
5. Don't add up test results. Five tests that each "add €10,000 a month" don't add €50,000. They overlap, some effects fade, and some are noise.

Many testing tools show an expected change and a 95% range on the results page. In Varly, for example, the results card shows the expected change with a line like "Likely between −2.0% and +22.0% (95% range)," a chance to beat the original or a p-value, and a split check (Varly help: read your test results). Those three numbers are exactly what section 2 of the template needs.

---

Building a test library clients can search

One report per test is good. A library of all tests is better: it stops you from testing the same idea twice and shows clients the value of the program over time.

One row per test. Copy this header into a spreadsheet:

test_id, name, client, site, page_type, element, hypothesis, evidence_type,
change_size, audience, primary_metric, start_date, end_date, visitors_per_version,
result (win/loss/no difference/invalid), change_pct, range_low_pct, range_high_pct,
decision, shipped (yes/no), learning, follow_up_test_id, report_link

Tags that make it searchable:

• Page type: homepage, category, product, cart, checkout, site-wide.
• Element: price block, delivery info, images, reviews, navigation, search, filters.
• Evidence type: heatmap, recordings, survey, analytics, customer service, best practice.
• Change size: big, medium, small.
• Result: win, loss, no difference, invalid (SRM or tracking problem).

Habits that keep it useful:

• Write the row the day the test ends, not at the end of the quarter.
• Link every test to the hypothesis and to the follow-up test.
• Review the library every quarter: which page types and evidence types produce winners for this client? Feed that into prioritization. See PXL vs ICE vs PIE.
• Give clients read access, so they can browse past tests themselves.

---

The agency CRO stack: tools, costs per client, access

If you run tests for several clients, the report is only one part. You also need to manage tools, costs and access per client.

Tools and cost per client

Typical setups (prices from our CRO tool pricing tracker, checked on vendor sites):

Add your hours: setup, consent integration, QA and reporting. Three tools per client means three installs and three consent checks per client.

Access

What clients usually need:

• Read access to their own data only. Never to other clients.
• Results in plain language, with the numbers behind them.
• Share links to a specific heatmap, recording moment or test result for meetings.

In Varly's agency setup, for example, each client's sites and data are kept separate; you can give a colleague access to All clients or Only these clients, invite client staff with a Viewer role for their own sites, share links to heatmaps and recording moments, and download heatmap element data as CSV (Varly help: agency onboarding, client access). Notifications can go to Slack or a webhook when a test result is ready (Varly help: notifications).

Onboarding checklist per client

1. Add the client and their site(s); one site per domain or country.
2. Give access: your team (all clients or only this one), client staff as viewers.
3. Install the snippet or plugin; turn tracking on.
4. Connect the cookie banner; check that nothing loads before consent.
5. Exclude checkout and account pages; check masking.
6. Connect orders, so tests can decide on sales.
7. Run a first research round (heatmaps, recordings, one survey) for 2–4 weeks.
8. Build the backlog, score it, and agree on the first test and its plan.
9. Agree on the report format (this template) and the meeting rhythm.

When Varly is not the right choice for an agency

• You need fully white-label reports. Varly's monthly client reports (in the client's login, as a link or as a PDF) show your agency's name, but also say "Prepared with Varly".
• Your clients need mobile-app tests or US data residency. Varly doesn't offer these.
• You want revenue per visitor to decide winners. Varly's verdict compares conversion rates.

---

FAQ

What should an A/B test report include? The decision, the result with a range, the revenue impact (low/expected/high), the original plan, learnings and next steps. Add screenshots, setup and data checks in a longer version.

How do I present an inconclusive A/B test to a client? Lead with the decision ("keep the original"), show the range, explain what you can rule out, and link to the next test. Use neutral wording: "no clear difference," not "failed."

Should I report segment results? Yes, but label them as exploratory and turn interesting ones into new tests. Don't pick a winner from a segment.

How do I calculate revenue impact from an A/B test? Visitors per month × baseline conversion rate × relative change × average order value. Do it three times: with the low end of the range, the measured change and the high end.

How long should a test report be? One page for readers. The full version can be longer, but the first page should stand on its own.

Related: the A/B test hypothesis template, how to prioritize test ideas and A/B testing for ecommerce.

---

Varly shows every test result in plain words, with the range and the split check, and keeps each client's tests in one place. Request access, or for agencies, see Varly for agencies.