Test plans and results reports

Clients and managers want the same two documents for every test: the plan before it starts, and the result after it ends. Varly writes both from what it already knows about the test, so you only check and send them.

The test plan

The test plan shows, in this order: a short summary, why you test this (with the research), the hypothesis, the original and the variant with screenshots, what you measure and who takes part, how long the test needs, the decision rule agreed up front, the risks, and the decision makers.

How long and how many visitors uses the same sum as the launch checklist, with the visitors of the last 4 weeks. When the page has too little traffic, the plan says so honestly and shows which bigger change could be proven instead.

The results report

The results report starts with the decision: the Outcome and the Decision you recorded when you ended the test. Then:

• a short summary, and The result in plain words, for example "Orders: +5.4%, most likely between +1.8% and +9.1%, 97% chance it’s better", with a bar that shows the range;
• Revenue impact per month as Low, Expected and High, when orders are the primary goal and your orders are connected. It uses the visitors in the test, the conversion of the original and the average order value of the orders in the test. It is an estimate: seasons, novelty and building costs are not included;
• the numbers per version, screenshots, Can we trust this?, What we said before launch, What we learned, Next steps and How this was calculated.

All numbers come from the results Varly froze when the test ended. They never come from AI.

Edit the text

Click Edit under a text block, change it and click Save. "Written by Varly from the numbers" means you see Varly's standard text. Empty a block and save it to go back to Varly's text. What we learned is the same learning you wrote when you ended the test.

With an AI key, the summary has Draft with AI. The AI only writes words around Varly's numbers: if its draft contains a number that is not in the results, Varly doesn't use it. A draft is marked Draft by AI — review before sharing until you click Review and edit and Save. You can't share the report while a draft isn't reviewed. See Use your own AI key.

Share for review

1. Under Share for review, enter the Name and Email of a decision maker and click Share.
2. Varly emails them a personal link. The link also appears on screen with Copy, so you can send it yourself.
3. They open the report without an account and click Approve, Request changes (with a short explanation) or Comment.
4. You get an email for each response. The report shows each person's status, such as Not opened yet, Viewed, Approved or Changes requested, and all responses under Decision makers.

Each link works for 30 days and only for this one report. Stop this link makes it stop working straight away.

Approving never starts or changes a test. After approval, someone in your team who can launch tests still launches it. The Launch card shows who approved the plan.

Versions

When you share, Varly saves the report as it is at that moment: version 1. People with a link always see that version, even if you change the text or the test later. If you change the report and share it again, the next person gets version 2. Click Send the newest version next to an earlier reviewer to give them a new link to the latest version; their old link stops working.

Save as PDF

Click Download PDF in the report, or on the shared page. Your browser's print window opens: choose Save as PDF. The PDF is A4, without menus, with the report name at the top and page numbers at the bottom. Turn on "Background graphics" in the print window if colors are missing.

Common problems

• Results report says it is ready when the test has ended. End the test first: the report is built from the frozen results.
• The reviewer says the link doesn't work. It may have expired after 30 days, or you stopped it. Click Send a new link next to their name.
• No revenue impact. It needs orders as the primary goal and connected orders. See Orders and revenue.
• The reviewer got no email. Check their spam folder, or copy the link and send it yourself.