Ship a winning test

A winning test only pays off once the change is a permanent part of your site. Varly helps with every step after "Variant B wins", but it never edits your shop or your code itself: your developer makes the change, and you tell Varly when it's done.

Where a winner stands

Ship the winner shows four steps:

1. Winner: the test ended with the decision Ship it.
2. Showing it through Varly (optional): Varly shows the winning version to everyone until it's on your site.
3. With your developer: you sent them the handoff.
4. Added to your site: your developer made the change a permanent part of your site.

You set each step yourself. Not adding it keeps the original instead; Add it after all undoes that. The same status shows in Tests, on the calendar, in the results report and in the weekly email.

Who adds it and Due date are for the person who makes the change and when it should be done. Cost of waiting is the expected gain per month from the test, divided by 4.3 weeks: what each week without the change costs, as an estimate. It shows until the change is on your site.

Show the winner to everyone until it's added

Your developer may need a few weeks. Meanwhile Varly can show the winning version to every visitor, with the same snippet that ran the test:

1. Open the test. Under Ship the winner, find Show the winner through Varly until it's added to your site.
2. Leave Keep 5% of visitors on the original off to show the winner to everyone, or tick it to keep comparing the two while you wait. The 5% group is a rough check, not a new test result.
3. Click Show the winner to everyone.

It works on the same pages and for the same visitors as the test (for example mobile only), but for all of them, not only the share that took part. It doesn't count as a test: it has no results or verdict, and it shows as Showing it through Varly in Tests and on the calendar. Click Stop showing it at any time.

Good to know:

• Only people who can launch tests can start it. See Who can launch tests.
• The test must have run live. After a dry run, visitors never saw the winner, so Varly can't show it.
• If A/B testing was off for the site, starting it turns it on again; the message says so.
• After 30 days Varly reminds you: "Still showing the winner through Varly. Has your developer added it to your site?" You see it on the test, and in Slack or a webhook if you turned on Follow-ups on a winning test under Notifications.

Hand off to your developer

Click Hand off to your developer. The handoff has everything your developer needs:

• What changed and why: the hypothesis, the result with how sure Varly was, and the expected gain per month (an estimate).
• Before and after: the screenshots from the build, per device.
• The changes: for each change, the element in words and its CSS selector, with ready-to-paste CSS or HTML. Custom code from the test is shown as it was. For a split URL test it says which page stays.
• Where to make the change: the usual steps for your site's platform (Shopify, WordPress also with WooCommerce, Magento, Framer, Webflow, or a custom site). With an AI key you can click Write implementation notes with Oran for notes on your platform. Oran writes them only when you click, and it can't see your files, so it says where a change usually lives instead of naming files. Check its notes against your own code.
• Acceptance criteria: checkboxes for what must look and work like the winning version on desktop, tablet and mobile, and the tracking that must keep working, such as the click goal of the test.

Send it in one of three ways:

• Copy as Markdown: paste it into Jira, Linear, GitHub or Trello. Headings and checkboxes stay.
• Download PDF: opens your browser's print window. Choose Save as PDF.
• Get a share link: a link to the handoff that works without a Varly account, for 30 days. Your developer can read and print it but can't change anything. Stop link makes it stop working straight away.

Then click Mark as sent to your developer, so everyone sees where it stands.

Mark it as added to your site

When your developer has made the change a permanent part of your site, click Mark as added to your site.

If Varly is still showing the winner, it first asks: Stop showing the winner through Varly now? We recommend Yes, stop showing it. Your site has the change itself now; if Varly keeps applying it too, the change is applied twice, which can look broken. Choose Keep showing it if you want to check your site first, and stop it later. Varly never stops it on its own.

Marked it too early? Click Not added after all.

Did the win hold?

Four weeks after you mark it as added to your site, Varly compares how often visitors on the test page reached the test's primary goal since then with how the original did during the test. It uses only the data Varly already collects; it doesn't open your site. You see the result on the test and in the weekly email, and in Slack or a webhook if you turned that on.

This is a before/after comparison, so it is weaker evidence than the test itself: seasons, campaigns and other changes to your site also count. With fewer than 200 visitors in either period, Varly shows the numbers but doesn't compare them.

Common problems

• There's no Ship the winner card. It shows after you end a test with the decision Ship it. To change the decision, use Decision and learning on the Results tab.
• I can't change the decision. Varly is still showing the winner. Click Stop showing it first.
• Show the winner to everyone isn't there. You need the right to launch tests, and the test must have run live.
• No cost of waiting. It needs a result with a range. Without orders as the goal, it's shown in conversions instead of money.
• Write implementation notes with Oran isn't there. Add an AI key under Organization settings. See Use your own AI key.