Read your test results

What is counted

• Visitors: unique visitors who saw the test. Each visitor counts once, in the version they saw first. Visitors outside Who takes part are not counted.
• Orders (or Added to cart, Reached page, Clicked, Converted, depending on the goal): how many of those visitors reached the primary goal after they first saw the test. An order or a page visit can be anywhere on your site; a click counts on the test page. Returning customers count too: a visitor who already ordered before the test counts again when they order after seeing it. Each visitor counts once, however often they reach the goal.
• Conversion: visitors who reached the goal divided by visitors.

Varly compares the share of visitors who reach the goal. Order value is not part of the verdict.

The line under the title shows the Primary goal, for example "Placed an order" or "Added to cart". You choose it when you launch the test. See Launch, pause, resume and end a test.

Only the time the test runs counts. If you started with a dry run and then clicked Go live: show the new version to visitors, the count starts again at that moment: the results only include visitors and orders from the live period.

At the top of the card you see the day the test is on (for example "Day 9 of 21"), the planned end date and a progress bar. Paused days don't count: while a test is paused the line starts with "Paused" and the day count stands still, and after you resume, the planned end moves back by the length of the pause. A test in a dry run shows "dry run: visitors are split but the page is unchanged". Below that, Who takes part shows which visitors are in the test, for example "All visitors" or "Mobile only · new visitors · 50% of visitors". The results only apply to that group.

The verdict

Varly sums up the result in one sentence:

• Collecting data: there aren't enough visitors in the test yet. Until then the card only shows the visitors and conversions per version; the chance, expected change and split check appear once there is enough data.
• Keep running: the planned duration isn't over. Varly gives no verdict before then, even if one version looks better now.
• The variant wins: the proof is strong enough after the full planned duration.
• The variant converts worse: keep the original.
• No clear difference: keep the original, or test a bolder change.
• Visitors are not split between the versions as planned: see the split check below.

Bayesian or frequentist

Varly can calculate results in two ways. Choose the method under Settings → Experiments → Statistics for A/B tests (the link is under How is this calculated? at the bottom of the Results card). Bayesian (recommended) is the default.

Chance to beat original (Bayesian). This is the chance that the variant really converts better than the original, based on the data so far. 92% means: if you picked the variant now, you would be right about 9 times in 10. Varly calls a winner from 95%, or higher when there are more variants.

p-value (frequentist). This is the chance of seeing a difference at least this big if the variant made no difference at all. The lower it is, the stronger the proof. Below 0.05, Varly shows Significant ("Significant at 95.0% confidence"). With more variants, the bar is stricter. Varly calculates the p-value with a two-sided test that compares the two conversion rates.

Both methods usually lead to the same decision. Bayesian is easier to explain to a client, because "a 97% chance it's better" is what most people want to know.

Expected change and the interval

Expected change is the relative difference in conversion. +10% means that if the original converts at 3.0%, the variant converts at about 3.3%.

Below it you see the 95% range, for example "Likely between −2.0% and +22.0% (95% range)". The true effect most likely lies within this range. In Bayesian results this is a credible interval: a 95% chance that the true change lies within it. In frequentist results it is a confidence interval, which follows the same idea in classic statistics. If the range includes 0%, the variant could still be better or worse. The more visitors, the narrower the range.

The split check (sample ratio mismatch)

Split check answers one question: Are visitors split as planned? Varly splits visitors evenly. Small differences are normal. A difference so big that chance can't explain it (less than a 1 in 1,000 chance) shows Uneven split, and the verdict says that visitors are not split as planned.

When that happens, the results can't be trusted, so don't use them. Something is treating the versions differently. For example, the variant might break part of the page so that some visitors aren't counted. Check the test in the preview on every device, and watch a few Recordings of each version (the link appears next to each version when recordings are on).

When to stop

• Wait for the planned duration. Results go up and down in the first days. If you stop as soon as a version looks good, you end up with many false winners. That's why Varly gives no verdict earlier.
• After the planned duration, read the verdict and click End test. Record the outcome, your decision and what you learned. From then on, the Results card shows the frozen numbers, with the line "Frozen when the test ended". To share the result, open the results report.
• If the variant wins, have the change built into your site. Varly stops showing the variant when the test ends.
• If there is no difference, that is a result too: the change doesn't matter enough to your visitors. Test something bolder, or move on to the next idea.

Secondary goals

If you added secondary goals at launch, they appear under Secondary goals below the main numbers: for each version how many visitors reached the goal, the conversion, the change against the original and the chance to beat it (or the p-value).

They are for context only and don't decide the test. Check them to see whether the change helped or hurt something else. For example: the variant gets more add-to-carts (primary goal), but the orders (secondary goal) stay the same. Don't pick a winner on a secondary goal that happens to look good: with several goals, one of them looks better by chance more often.

Results by segment

Below the Results card, Results by segment shows the same numbers for smaller groups of visitors. It appears once every version has 100 visitors; before that, a short line says so. Switch between three views:

• Device: Desktop, Mobile and Tablet.
• Visitors: New visitors and Returning visitors.
• Source: where the visit came from, such as Direct, Organic search, Paid ads, Social or Email.

For each segment and version you see the visitors, how many reached the primary goal, the conversion, the change against the original and the chance to beat it (or the p-value). Each visitor counts in one segment: the one they were in when they first saw the test. So a visitor who first saw the test on their phone counts under Mobile, even if they ordered later on a computer. Unknown means Varly couldn't tell, for example because no page view was recorded in that visit.

A typical finding: "the variant does better on desktop but worse on mobile". That's a good reason to look closer, for example in the preview on a phone.

That's why Varly never marks a winner in a segment. Varly also only compares a version with the original in a segment when both have at least 100 visitors in it and at least 10 conversions together. Otherwise the row is grey and shows the raw numbers, but no change and no chance (or p-value). When no version can be compared, the segment says Too few to compare.

If you see a clear difference in one segment, test it properly: launch a new test for just that group: choose it under Who takes part when you launch, for example only mobile visitors. See Launch, pause, resume and end a test.

More than one variant

With several variants, the table gets a Chance to beat original (or p-value) column, and the best version is marked "best so far". The bar for a winner is stricter, and the text under the table shows the bar. A multivariate test also shows Effect per part. See A/B/n and multivariate tests.

Common problems

• "No orders are linked to this test yet, and orders are the primary goal, so Varly can't pick a winner." Connect your shop under Install → 5. Connect your orders. See Orders and revenue. For your next test, you can also pick another primary goal, such as Added to cart.
• A secondary goal stays at 0. For a custom event, check that the name matches what your site sends. See Send your own events with Varly.track.
• The numbers differ from your shop's dashboard. A test only counts visitors who accepted cookies and saw the test during its live period, and orders that came after their first visit to it. See Why revenue numbers differ.
• The chance jumps from 80% to 40%. That is normal early on, when there are only a few orders. It settles down as more data comes in.
• The numbers dropped after going live. That is expected: going live starts the count again, so the dry run visitors are no longer included.