Create an A/B test

An A/B test shows the original page to half of your visitors and a changed version (the variant) to the other half. Then it compares how many visitors in each group place an order. You don't need a developer or a visual editor: the AI builds the variant from the idea.

Create the test

1. Go to Optimize → Ideas and open the idea.
2. Check Can we test this?. It tells you whether the page gets enough visitors, and roughly how many weeks a test needs. See Prioritization.
3. Choose the test type: One new version (an A/B test with one variant), Several versions (two or three variants) or Combinations (a multivariate test). For the last two, see A/B/n and multivariate tests.
4. Optionally, fill in Instructions for the new version (optional), for example "Put the delivery date under the price". If you leave it empty, the AI picks the clearest way to build the idea.
5. Click Create test.

To make the changes yourself instead, click Create test and edit it myself under Or make the changes yourself. See Without an AI key below.

Varly takes you to the new experiment, which shows Building your A/B test… and these steps: Waiting in line, Opening the page, Designing the variant with AI, Checking on desktop, Checking on tablet and Checking on mobile. This usually takes one to three minutes, and the page updates by itself. The idea moves to In test.

Without an AI key

Every idea can still become a test. Under Can we test this?, click Create test and edit it myself. Varly creates the test on the idea's page and moves the idea to In test. On the test page, click Open the visual editor, select a part of the page and choose Edit directly to change the text, a link, an image or the style. Then click Build the new version: Varly checks it on desktop, tablet and mobile, without AI. See Build a test by selecting parts of your live page.

Which page the test runs on

• Idea on a group of similar pages, for example Product pages (/products/…): Varly builds the variant on the most visited product page in the last 30 days, and the test runs on all product pages, because similar pages count as one. The header shows, for example, "Runs on product pages (/products/…). Built and previewed on /products/blue-shirt."
• Idea on a single page, for example your homepage or a landing page: the test runs on that page only.
• Idea without a page (Whole site): Varly builds and previews the variant on your homepage, and the test runs on every page of your site. The header shows "Runs on the whole site."

Tests never run on pages listed under Never track these pages in Settings → Tracking & privacy.

What the AI can change

The AI changes the page that is already there. It can rewrite text, change styles and attributes, add new content such as a delivery note, and hide or remove elements. Each version changes at most 50 elements. After building, Varly applies the changes in a real browser on desktop, tablet and mobile, and notes anything that did not work.

Review what was built

When the status is Ready to review, the experiment page shows:

• Original and variant: screenshots of the top part of the page. Use Desktop, Tablet and Mobile to switch devices. With more variants, click a variant's name to see it.
• What changes: a plain-language summary. Open the technical changes line to see each change in detail.
• Check before launch: warnings, for example a change that could not be applied, or a script error on the page after the change. Read these carefully.
• Preview on your site: a link to see the test on your real site. See Preview and check a test.

Goal and traffic split

• Goal: you choose what counts as a conversion when you launch the test, for example Placed an order or Added to cart. See Launch, pause, resume and end a test.
• Traffic split: visitors are split evenly over all versions: 50/50 for an A/B test, and equal shares when there are more variants. Each visitor keeps seeing the same version.

Not happy with the variant?

On the experiment page, at the bottom of What changes, open Rebuild with AI, or test several versions or combinations of parts. Change the instructions or the test type and click Rebuild. The AI builds the test again. Your current version stays until the new one is built: if building fails, it comes back, with a note that says why. You can't rebuild a running test: click Pause first.

Without an AI key, Varly can only rebuild a test whose versions consist of your own direct edits. Otherwise it says "Rebuilding with AI needs an AI key", and nothing changes. To adjust the version yourself, use Open the visual editor and Edit directly.

Archive or delete a test

Open the test and use the buttons at the top right. Varly asks you to confirm first.

• Archive moves the test out of the experiments list. Everything is kept, including the results. You can archive any test that isn't running or building; pause or end a running test first. To see archived tests, click Archived above the list, open one and click Restore from archive.
• Delete removes the test for good, with its changes and screenshots. You only see it for a test that never ran (marked changes, ready to review or build failed), so no results are lost. If it came from an idea, the idea goes back to Shortlist.

Only Owners and Editors can archive or delete tests.

Common problems

• There is no Create test button, only Create test and edit it myself. Your organization has no AI key. Make the changes yourself (see Without an AI key), or ask an Owner to add a key.
• "The new version could not be built", with a reason such as "Varly could not open https://… : the page was not found (404)" or "None of the changes could be applied to the page." The message names the page and the cause: not found, a login or block, a server error, or a site that can't be reached. Check that the page opens normally in a private browser window, then click Try again. If the address is wrong, create a new test with the right page and delete the old one.
• The test is built on the wrong page. Change the Page of the idea, click Save and create the test again.
• Some changes are missing in the screenshots. Look at Check before launch: a change is skipped when the AI's target element can't be found, or when more than one element matches. Rebuild with instructions that name the element more precisely.