Launch, pause, resume and end a test

Nothing goes live without your click. The checklist makes you look at the numbers before you start, so you don't launch a test that can never give a clear answer.

The launch checklist

The Launch card sits below the screenshots and has four numbered steps:

1. What counts as a conversion. Choose the Primary goal: the one goal that decides the winner. Placed an order is the default. Add up to three Secondary goals if you want to see what else changed. See Choose what the test measures below.
2. Who takes part. By default everyone on the test page takes part. You can limit the test to some Devices, to new or returning Visitors, to some Traffic sources, or to a Share of visitors in the test (10%, 25%, 50%, 75% or 100%). See Choose who takes part below.
3. How long it runs. Varly shows the page's sessions per week, and how many of them reach the primary goal (for orders: the conversion rate). Choose the Smallest effect the test can detect, from +5% to +50%: the smallest improvement the test must be able to prove. The hint uses your own rate, for example "+10% turns 0.30% into 0.33%". The default comes from the idea's impact. The smaller the effect, the more visitors you need. Varly then says, for example: "Needs 9,800 visitors per version (95% confidence, and an 80% chance to spot the effect if it is real): about 3 weeks." With more versions, a winner needs stronger proof, so you need more visitors. Planned duration (weeks) starts at that number, and you can choose up to 12. Varly never judges a test before 2 full weeks, so every weekday counts at least twice, and it gives no verdict before the planned duration has passed.
4. Start. Choose Dry run first (recommended) or Live (see below). Tick the box "I checked the original and the variant on desktop, tablet and mobile, including the preview link" (see Preview and check a test), then click Launch experiment.

Tests with custom code

A test with your own JavaScript or CSS shows Uses custom code at the top. Above the checklist, Varly lists what still has to happen before you can launch:

• A person approved the code. Code that an AI assistant saved through a chat is a draft. Read it in the Custom code card and click I read this code, approve it.
• The check passed for this exact code. Click Save and check in the Custom code card. Varly runs the code on desktop, tablet and mobile and looks for errors, missing elements and security settings that block it. After every change to the code, the check runs again.

When both are done, the card says "Custom code: approved and checked on desktop, tablet and mobile without errors." The same rules apply when an AI assistant launches the test, and when you resume a paused test after changing its code. See Write your own JavaScript and CSS.

Not much traffic yet

You can always launch, also on a small shop. When the page doesn't get enough visitors to prove the effect within 8 weeks, step 3 says so in an orange box: "With this traffic, the test is unlikely to give a reliable result within 8 weeks." The button then reads Launch anyway. The results will be less certain, and Varly may end the test without naming a winner.

The box also lists ways to get a clearer answer. Click one to apply it:

• Run it longer: when up to 12 weeks would be enough, Varly offers that number of weeks. Tests longer than 8 weeks are less precise, because visitors clear cookies and seasons change along the way.
• Test a bolder change: Varly offers the smallest effect that does fit within 8 weeks.
• Use Added to cart, or a click, as the primary goal: shown when the goal is Placed an order. Those happen far more often than orders, so you need fewer visitors. In a split URL test Varly suggests a visit to a next page, such as the cart, instead of a click.
• Let more visitors take part: when you limited Who takes part in step 2.

Choose what the test measures

Under What counts as a conversion you choose the Primary goal. A visitor counts as converted when they reach the goal after their first visit to the test. The verdict is based on this goal only.

• Placed an order: the visitor placed an order. Needs your orders connected.
• Added to cart: the visitor added a product to the cart. On Shopify, Varly counts this by itself for the standard add-to-cart button of your theme. On other platforms, your site sends it. See Send your own events with Varly.track.
• Visited a page: the visitor reached a page, for example a thank-you page or a sign-up page. Pick it from the list, or type a path. End the path with * to count every page that starts with it, for example /blog/*. The test page itself can't be the goal, because everyone in the test sees it.
• Clicked an element: the visitor clicked a button or link on the test page. Pick one of the most-clicked elements from the list, or enter its CSS selector. Not available in a split URL test, where each version is a different page; there the variant pages can't be the goal either.
• Custom event: an event your site sends with Varly.track, for example newsletter_signup.

A goal that happens more often than an order, such as Added to cart, needs fewer visitors, so a small shop gets an answer sooner. The plan below updates when you change the primary goal.

Secondary goals are optional, up to three. The results show them below the main table, but they don't decide the test. Use them to check that a change that brings more add-to-carts doesn't cost orders, for example.

Choose who takes part

Under Who takes part you choose which visitors are in the test. Everyone else sees your normal page and is not counted in the results.

• Devices: Desktop, Mobile and Tablet. At least one stays ticked. Useful when the change is only about the mobile layout, for example.
• Visitors: New and returning, New visitors only or Returning visitors only. A visitor is returning if this browser visited your site before the current visit.
• Traffic source: Direct, Organic search, Paid ads, Social, Email, Referral and Other campaigns. When all are ticked, every source takes part. Varly uses the source of the first page in the visit.
• Share of visitors in the test: 10%, 25%, 50%, 75% or 100% of the visitors who match the choices above. The same visitor is always in or out, on every visit. Within the test, visitors are still split evenly over the versions.

A smaller group makes the test take longer: fewer visitors take part each week, so it takes more weeks to collect enough of them. The plan under the choices updates as you change them.

After the launch, the Results card shows your choice next to Who takes part, for example "Mobile only · new visitors · 50% of visitors". To change it, click Pause, open Change the plan, then resume below the test, choose again and click Resume with this plan.

Dry run or live

• Dry run: visitors are split over the versions and counted, but nobody sees a change yet. After a few days, check that Split check in the results says OK. That tells you the split works before any visitor sees a change.
• Live: visitors are split evenly over the original and the variants, and the variant groups see the changes. In a split URL test, the variant groups are sent to their variant page.

Each test has its own choice. A test in a dry run stays a dry run, even when another test on the same site is live.

Go from dry run to live

1. Open the test. While the dry run is running, the Results card shows the button Go live: show the new version to visitors.
2. Click it. The test keeps the same plan and now shows the new version to visitors.

Going live starts the count again: the day count goes back to day 1, and the results only include visitors from the live period. Visitors from the dry run are left out, so they can't blur the outcome.

Who can launch

Owners and Editors can launch, pause and end tests. Viewers can see tests and results, but can't change them. See Team roles.

An AI assistant connected to Varly can put a test forward for launch. You then see a message on the experiment page, for example "Claude asks you to launch this test for" followed by the client's name. An assistant can only launch a test itself if its connection is allowed to, and only after you say yes in the chat. The experiment page then shows who launched it, and from which app.

What visitors see

Once a test is live:

• Visitors who take part are split evenly over all versions. Each visitor keeps the same version on every visit in the same browser. Visitors outside Who takes part see your normal site.
• Only visitors who accepted cookies in your cookie banner take part. Everyone else sees your normal site.
• The test runs only on its own page, or on all pages of its kind (for example all product pages). A test built from an idea without a page runs on the whole site. It never runs on pages under Never track these pages.
• Custom code runs after the other changes, at the moment you chose. If it fails, the page keeps working: the code stops, or the visitor sees the original, as you chose under If the code fails.
• Parts of the page that load a little later are changed too. Varly waits up to 2.5 seconds for them and hides only the parts it is about to change until then, so visitors don't see the original flash by. If a part never appears, the visitor sees the original.
• It can take a minute or two after you launch before the test reaches your site.

Pause, resume or end a test

You find the buttons at the bottom of the Results card while the test is running. Pause asks you to confirm first; End test opens a short form.

• Pause stops the test for a while. Every visitor sees the original until you resume, and the status becomes Paused. Paused days don't count toward the planned duration: the day count stands still and the planned end moves back by the length of the pause.

• Resume test starts a paused test again with the same goals, visitors, duration and mode (dry run or live). To change any of those first, open Change the plan, then resume below the test. Changing the plan halfway mixes two plans in one result, so only do this when you must.

• End test stops the test for good. You fill in three things, then click End test and save result:

  • Outcome: Won, Lost, Inconclusive or Stopped early. Varly suggests one based on the results; you confirm it.
  • Decision: Ship it, Don’t ship or Iterate (test a new version of the idea).
  • What did we learn?: one to three sentences. With Ship it you can also fill in Live on the site since.

  The status becomes Completed, the idea moves to Done, and the numbers are frozen: they stay readable for good, also after Varly deletes old visit data. A completed test can't be started again. You can also end a paused test. To change the outcome, decision or learning later, click Edit result under Result and decision. See All tests and the test overview.

Ending a test doesn't change your site. If the variant won, have the change built into your site, because Varly stops showing it once the test ends.

To tidy up the list afterwards, archive the test.

Emergency stop: switch tracking off

To stop everything on a site at once, go to Settings → Tracking & privacy, untick Tracking on and click Save. The snippet then does nothing at all: no tests, no surveys, no measurement. This takes effect within a minute or two, and the menu shows Paused: not collecting data. Tests keep their status in Varly, but visitors only see your normal site. Tick Tracking on again to continue. The Turn tracking off and Turn tracking on button under Install → 3. Turn tracking on does the same in one click.

Common problems

• The Launch card is missing. You have the Viewer role, or the test is still building, already running, paused, completed or archived. A paused test continues with Resume test.
• "Tick this box first". Tick the checkbox above the button: it confirms you checked every version on every device.
• "Plan at least 2 and at most 12 full weeks". Enter a whole number of weeks between 2 and 12.
• "Varly never measures … it is under Never track these pages". The goal page is never measured, for example your cart or checkout. Pick another goal, such as Added to cart.
• The Launch experiment button stays gray after you changed a goal. The primary goal isn't complete yet. The red text under the goal says what is missing. Low traffic never blocks the button.
• The plan needs many more weeks than expected. Check Who takes part: a few devices, one traffic source or a small share leaves fewer visitors per week. Let more visitors take part, or pick a bigger effect.
• The test is running, but you can't see it on your site. Check that it isn't still a dry run. See Why is my test not showing?.
• "The custom code changed since the last check." Click Save and check or Run the check in the Custom code card and wait until it is done.
• "The code of Variant B was written in an AI chat." Read the code in the Custom code card and click I read this code, approve it.

If your shop uses Google Analytics 4, you can also see which version each visitor saw in your own GA4 reports. See See your A/B tests in Google Analytics 4.