A/B/n and multivariate tests

The three test types

On the idea page you pick one of three tabs:

• One new version (an A/B test): the original and one variant. This is the right choice for most tests.
• Several versions (an A/B/n test): the original and two or three variants (B, C and D), each with its own approach. Use it when you have several clearly different ways to solve the same problem.
• Combinations (a multivariate test): you pick two or three parts of the page, such as the headline and the add-to-cart button, and give each part one or two alternatives. Varly tests every combination. Use it when the page gets a lot of traffic and you want to know which part makes the difference.

Set up an A/B/n test

1. On the idea page, choose Several versions.
2. Under Variants next to the original, write optional instructions for each variant. If you leave a field empty, the AI makes that variant clearly different from the others.
3. Click + Add variant for a third variant, or × to remove one.
4. Click Create test.

The AI builds the variants one after another. Each new variant avoids the approaches of the ones before it.

Set up a multivariate test

1. On the idea page, choose Combinations.
2. Under Parts to vary, name each part, for example "Headline", and write one or two alternatives, for example "lead with free delivery". Use + Another alternative and + Add part to add more.
3. Check the count below the form, for example "9 combinations including the original".
4. Click Create test.

How combinations are built

Each part has the original plus its alternatives. Varly multiplies those numbers to get every combination:

Parts	Combinations
2 parts with 1 alternative each	2 × 2 = 4
2 parts with 2 alternatives each	3 × 3 = 9
3 parts with 1 alternative each	2 × 2 × 2 = 8
3 parts, one of them with 2 alternatives	3 × 2 × 2 = 12, too many

A test can have at most 9 combinations, including the original. Above that, there are rarely enough visitors to give each one a fair share.

The AI builds each alternative on its own, without touching the other parts, and Varly then combines them. If two parts happen to change the same element, Varly keeps only the first change and adds a warning under Check before launch.

Why more variants need more traffic

There are two reasons:

1. Visitors are shared. Varly splits visitors evenly over all versions. With 4 versions, each one gets a quarter of your traffic instead of half, so each takes longer to collect enough visitors.
2. A winner needs stronger proof. Varly makes the bar higher for every extra variant (explained below).

The launch checklist takes both into account. It shows the number of visitors needed "per version" and the total number of weeks. A test that takes 3 weeks as a plain A/B test can easily need more than 8 weeks as a multivariate test; the checklist then warns you that the result will be less certain.

The stricter bar, in plain words

Every variant you add is another chance for a lucky result. Suppose each variant has a 5% chance of looking like a winner when it really isn't better. With three variants, the chance that at least one of them looks like a winner by luck is about 14%.

To prevent that, Varly divides the 5% margin by the number of variants. This is called the Bonferroni correction.

Variants next to the original	Bayesian: chance needed	Frequentist: p-value needed
1	95%	below 0.05
2	97.5%	below 0.025
3	98.3%	below 0.017
8 (a 9-combination multivariate test)	99.4%	below 0.006

This way, more variants do not mean more false winners. The results page shows the bar for your test.

How to read the result

The results table has one row per version, with a Chance to beat original (or p-value) column. The best version is marked "best so far". A multivariate test also shows Effect per part: each alternative compared with the original across all combinations. This tells you which part of the page moves conversion, even when the individual combinations don't have enough visitors yet. See Read your test results.

In the preview, the bar has a button for every version. For a multivariate test it has a dropdown for each part, so you can put together any combination.