From first insight to first A/B test result in three weeks

Varly never judges a test before two full weeks, so every weekday counts at least twice. Add one week of research, and three weeks is the fastest honest path to a first result.

Week 1, day 1: set everything up

1. Add the snippet from Install to your site. See Install the snippet.
2. Click Turn tracking on under 3. Turn tracking on on the Install page. Then check under Settings → Tracking & privacy that the Consent banner setting matches your cookie banner. See Cookie consent.
3. Connect your shop under Install → 5. Connect your orders. Tests pick their winner on orders. See Orders and revenue.
4. Check Settings → Session recordings. New sites record All visits with Balanced text hiding. Change How many visits to record or Hide text in recordings if your privacy policy asks for it. See Set up session recordings.
5. Add an AI key in the AI key card of Organization settings (not Site settings). See Add your own AI key.
6. Open the link under Exclude your own visits on this device in each browser you use, so your own clicks stay out of the data.

Week 1, days 2 to 6: let research come in

Start one survey, such as Why are you leaving? or Is anything unclear about this product?, so you also hear why visitors hesitate. See Create a survey.

Then look at Heatmaps for your busiest pages and watch a few Recordings. Find out why visitors don't buy walks you through this.

Week 1, day 7: choose and build the test

1. Open Optimize → Ideas and click Find new ideas. Add your own findings with New idea.
2. Open Optimize → Prioritization. Pick an idea with a high Score and a Test duration of about 2 weeks. Skip ideas marked Not enough traffic: the page has too little traffic to show that effect within 8 weeks.
3. Open the idea. Under Can we test this?, choose One new version (one variant against the original). You can describe the change under Instructions for the new version (optional). Click Create test.
4. The AI builds the variant, and Varly checks it on desktop, tablet and mobile. This usually takes one to three minutes. See Create an A/B test.

Week 1, day 7: check and launch

1. Compare Original and variant on Desktop, Tablet and Mobile, and read What changes. If you see Check before launch, go through every point.
2. Open the link under Preview on your site on your phone and computer. A bar at the bottom switches between the versions. Nothing is measured. See Preview and check a test.
3. Under Launch, pick the Smallest effect the test can detect: the smallest change the test must be able to prove. Smaller needs more visitors, and Varly shows how many weeks that takes.
4. Set Planned duration (weeks). It can't be shorter than the traffic needs.
5. Under How to start, choose Dry run first (recommended) or Live. In a dry run, visitors are split but the page doesn't change, so you can check the split first. Plan a few extra days for it. When the split looks good, click Go live: show the new version to visitors on the test page. The count then starts again, so the two weeks start from that moment.
6. Tick the box to confirm you checked every version, then click Launch experiment. See Launch a test.

Weeks 2 and 3: let it run

Look at the test now and then, but don't act on early numbers. Until the planned duration is over, the result says Keep running, even if one version looks better. Check two things:

• Split check should say OK. Uneven split means visitors aren't split as planned (sample ratio mismatch), so the result can't be trusted. Check your setup.
• The message "No orders are linked to this test yet" should not appear. If it does, see Why are orders not linked?.

To see how visitors use each version, click Recordings next to it, or pick the test on a heatmap and choose Compare: Test versions.

End of week 3: read the result

After the planned duration, Varly gives a verdict:

• The variant wins: with Bayesian statistics, at least 95% chance it converts better. Make the change on your site.
• No clear difference: keep the original, or test a bolder change.
• The variant converts worse: keep the original. You still learned what not to do.

Click End test to finish. Then open the idea and use Move to: → Done. See Read your test results.

Common problems

• Every idea says "Not enough traffic". Test a page with more traffic, such as a group of product pages, or a bolder change with a bigger Smallest effect the test can detect.
• "Building a test needs an AI key for your organization." Click Add it in Organization settings; an owner adds the key there. See Add your own AI key.
• "The variant could not be built". Read the reason shown and click Try again. See Create an A/B test.