Keep tests apart

Why

Two tests on the same page for the same visitors can change each other’s results. A new headline in one test and a new button in the other can work well together, or get in each other’s way. Keeping the tests apart prevents that: no visitor sees both.

Keep tests apart

You can do it in three places:

• In the launch checklist. Under Who takes part, Other tests on this page lists the running, paused and ready tests on the same page. Tick Keep apart from a test. Varly shows the share of visitors this test gets, and the plan counts with it.
• In the calendar. When two tests overlap, the yellow box at the top lists them with a Keep these tests apart button. On a phone, the button sits under the test in the list.
• More than two. Add a third test the same way: tick it in its launch checklist, or keep it apart from one of the tests in the group. Keep at most five tests apart at a time.

How visitors are split

• Fair: with two tests, each gets half of the visitors on the page; with three, each gets a third.
• Stable: the same visitor always lands in the same test, on every visit.
• Valid results per test: within each test, visitors are still split evenly over the original and the variants. Which test a visitor lands in has nothing to do with which version they see, so the comparison inside each test stays fair.
• When a test joins: it first gets the share of a test that ended. If there isn’t one, every test gives up an equal part of its share. Most visitors stay in the test they were in.
• Paused tests keep their share. While a test is paused, its visitors see your normal page; they don’t move to the other tests. When you resume, the same visitors come back.
• Ended tests free their share. The other tests keep theirs. The next test you keep apart from them gets the free share.

The test page shows it under Results, for example "Kept apart from “New headline”: a visitor takes part in only one of these tests. This test gets 50% of the visitors who match."

Stop keeping tests apart

On the test page, click Stop keeping apart. Visitors can then take part in these tests at the same time again. Nobody leaves a test: the test you take out gets all visitors again. When only one test is left, it runs for everyone.

Changing how running tests are kept apart changes who sees them, so it needs permission to launch tests.

Good to know

• Server-side tests can’t be kept apart: your server decides who takes part there.
• Two groups of tests can’t be merged in one click. Stop keeping one test apart first.
• Who takes part still counts for every test: a visitor who doesn’t match a test’s rules doesn’t see it, and doesn’t see another test in its place either.
• Keeping tests apart gives each test fewer visitors, so each takes longer. When the tests are not on the same page, or are for different visitors (for example mobile and desktop), you don’t need it.