Shopify A/B Testing and CRO in 2026 (incl. Rollouts)
What Shopify Rollouts can and can't test, when you need an app, what you can track in checkout, and how to set up heatmaps, surveys and tests step by step.
By Joost BakkerPublished 15 min read
On this page
- What Shopify offers natively in 2026
- What Rollouts can and can't test
- When you need an app (or another tool)
- Heatmaps and recordings on Shopify
- What you can and can't track in checkout
- Surveys on the order status page
- Speed: apps, scripts and your theme
- Setup step by step (with Varly as the example)
- Decision table
- FAQ
What Shopify offers natively in 2026
Shopify's built-in CRO tools come down to two things:
- Rollouts. Under Markets → Rollouts. A rollout can schedule changes, publish them to a share of visitors, run them for a set period, or test them as an experiment (Shopify changelog, June 5, 2026).
- Shopify Analytics. Your store reports show conversion rate, add-to-cart and reached-checkout rates. These are your baseline numbers before any test.
There are three rollout types (Shopify Help Center):
| Type | What it does | Plan |
|---|---|---|
| Launch | Applies changes permanently to a share of visitors | Basic or higher |
| Event | Publishes changes for a set period, then rolls them back (for example, a sale) | Basic or higher |
| Experiment | Tests a treatment against a control on a share of visitors | Grow or higher |
Shopify's Grow plan costs $79 a month billed yearly, or $105 billed monthly (shopify.com/pricing).
What Rollouts can and can't test
What it can change
From Shopify's rollout types and requirements pages:
| Resource | Change |
|---|---|
| Online store theme | Edit the main theme, or replace it with another theme |
| Checkout and accounts | Replace the configuration |
| Catalogs | Set a catalog to Active or Draft (this controls prices and availability) |
| Discounts | Activate or deactivate discounts (up to 250 per rollout) |
Because catalogs control prices, Shopify notes that experiments "can show different prices or products on your online store than on other sales channels." Product ads, feeds and third-party apps "might not display the same offer." If you test prices this way, check that your ads and feeds don't promise a price the visitor won't see.
What it can't do
- No Liquid template changes inside a rollout.
- No vintage themes.
- Online store only for experiments. Headless storefronts and POS aren't included in experiment results.
- No element-level editing. You compare whole themes or configurations. To test one headline, you duplicate the theme, change the headline, and test theme A against theme B.
What it measures
Experiment results depend on what you test (Shopify Help Center):
- Theme experiments: conversion rate over time, bounce rate, reached-checkout rate and add-to-cart rate.
- Checkout and accounts experiments: checkout conversion rate.
- You can't customize these metrics.
Defaults: a 50/50 traffic split and an end date 90 days after the start, both editable before you save (rollout types).
One gap: Shopify's help pages we checked don't explain how significance is calculated or when a winner is declared. That doesn't mean the results are wrong. It does mean you should decide your sample size and test length before you start, and not stop the test the moment one version looks better. Use our sample size calculator to plan.
When you need an app (or another tool)
Rollouts is a good fit for "new theme vs. old theme," checkout configurations and discount or catalog tests. You'll want something else when:
| You want to… | Use |
|---|---|
| Test one element (headline, image, button, layout block) without duplicating the theme | A testing app or tool that edits the live page |
| Test across many product pages at once with one change | A testing tool with page targeting |
| Pick your own goals (add-to-cart, a click, a page visit) | A testing tool with custom goals |
| Run price tests with a dedicated workflow | A price-testing app (or catalogs in Rollouts) |
| Test shipping rates | An app such as Intelligems (shipping tests need Shopify Advanced or Plus) |
| Find out what to test | Research tools: heatmaps, recordings, surveys |
Two Shopify-specific testing apps with public prices (checked November 5, 2026):
- Intelligems: content, theme, template and split URL tests on every plan; price testing as a separate module; shipping tests need Shopify Advanced or Plus; checkout tests need Plus. "Build Your Own" from $69/month at the lowest order tier (intelligems.io/pricing).
- Shoplift: theme and template testing, Core plan $74/month billed annually or $99 monthly; 14-day trial (shoplift.ai/pricing).
General testing tools (Convert, Kameleoon, Wingify, Optimizely, Varly) also work on Shopify through a script in your theme. We compare them in VWO alternatives.
Traffic first. Before you buy any tool, check whether your store gets enough visitors to finish a test. A product-page template with 5,000 visitors a week can only detect big changes. See A/B testing with low traffic and the traffic tags in 60 ecommerce A/B test ideas.
Heatmaps and recordings on Shopify
Any heatmap tool that runs a script on your storefront works on Shopify: Microsoft Clarity (free), Contentsquare (formerly Hotjar), Mouseflow, Lucky Orange, heatmap.com and others. We compare them in Hotjar alternatives for ecommerce.
Shopify-specific tips:
- Group your product pages. All your products use the same template. A heatmap of "all product pages" tells you more than one heatmap per product.
- Watch the cart drawer. Many Shopify themes open the cart as a slide-out drawer. Screenshot-based heatmaps don't capture clicks inside drawers and pop-ups well. If that matters, use a tool with interactive or dynamic heatmaps.
- Segment by device. Mobile and desktop heatmaps of the same page often look very different, so look at them separately. Read how to analyze heatmaps.
- Connect orders so you can filter recordings to "visitors who bought" and "visitors who didn't." That's where the useful differences show up.
What you can and can't track in checkout
Shopify has moved checkout customization to a locked-down model:
checkout.liquidis unsupported for the Information, Shipping and Payment steps (shopify.dev).- Thank you and Order status pages:
checkout.liquidand additional scripts were sunset on August 28, 2025. Script tags on those pages were sunset for Plus stores on August 28, 2025, and for other stores on August 26, 2026 (same source). - Tracking now runs through web pixels. Custom pixels you add yourself run in a sandboxed iframe that can't access the main page. App pixels run in an even stricter sandbox and can't read or write the page itself (shopify.dev, web pixels).
What that means for CRO tools:
- Heatmaps and recordings of the checkout generally aren't possible with a theme script, because the checkout doesn't load your theme.
- Conversion tracking works through pixels (for analytics) or through order webhooks (for tools that link orders to visits).
- Checkout tests run through Rollouts (checkout configurations) or apps that support checkout tests on Plus.
Is that a loss? Partly. You can't see where people struggle in the address form. But checkout pages hold names, addresses and payment details, so many privacy-minded tools skip them anyway. Varly, for example, never tracks pages starting with /checkout or /cart by default (Varly help).
Surveys on the order status page
A post-purchase survey ("How did you hear about us?", "What almost stopped you from buying?") is one of the most useful things you can run. On Shopify it has a catch: the Thank you and Order status pages belong to the checkout, and theme scripts don't run there. Since the 2025–2026 sunsets, survey apps need to use Shopify's checkout and customer account extensions to show anything on those pages (shopify.dev, script tag deprecation).
What to do:
- For post-purchase surveys on Shopify, use a survey app that is built on Shopify's checkout extensions. Check that the app says so explicitly.
- For on-site surveys (exit intent, product page, cart page), any survey tool that runs in your theme works.
- Keep it short. One question, optionally a follow-up. See 80 on-site survey questions.
Varly's on-site surveys run on your storefront pages. Its "show on the thank-you page" option works where the thank-you page is part of your own site (as on WooCommerce). On Shopify, use a checkout-extension app for that one survey.
Speed: apps, scripts and your theme
- Every app that adds a storefront script costs speed. Remove apps you no longer use, and check your theme code for leftover snippets from old apps.
- The script-tag deadline will remove some leftovers for you. Apps can't create or update script tags since October 1, 2026, and Shopify stops injecting script tags on storefronts on March 1, 2027. The replacements are app embed blocks (which you switch on in the theme editor) and web pixels (Shopify developer changelog). Code you paste into
theme.liquidyourself is not a script tag. - Watch anti-flicker snippets. Many testing tools hide the page briefly so visitors don't see the original flash before the variant. Shopify's own guidance: anti-flicker snippets add "an artificial delay in the critical rendering path," so turn them off when no tests are running (shopify.dev).
- Measure before and after. Run PageSpeed Insights on a product page before you add a tool, and again after. We publish our own measurements in the script weight study, and explain flicker in the A/B test flicker effect.
Setup step by step (with Varly as the example)
Here is how the setup works in Varly, the tool we build, from its Install page. Other tools follow similar steps.
1. Pick your platform
Open your site in Varly and click Install. Under 1., pick Shopify. The page then only shows the Shopify steps.
2. Add the snippet to your theme
Varly doesn't use a Shopify app. You paste one line of code into your theme (Varly help: Install on Shopify):
- In Varly, copy the snippet under 2. Add the snippet.
- In your Shopify admin, go to Online Store → Themes.
- Next to your live theme, open the menu (three dots) and choose Edit code.
- In the Layout folder, open
theme.liquid. - Paste the snippet high up in the
<head>, right below the opening<head>tag. The higher it is, the sooner A/B test changes appear. - Click Save.
The snippet is about 22 kB gzipped and loads asynchronously. Recording, survey and form analytics code only load when needed.
Heads-up: the snippet lives in the theme. If you switch to a new theme or install a fresh copy, paste it again.
3. Turn tracking on
In Varly, switch Tracking on (step 3 on the Install page, or Settings → Tracking & privacy). Varly only measures after a visitor accepts cookies, and never in the checkout.
Then set your cookie banner under Settings → Tracking & privacy → Consent banner. If you use Shopify's own banner (Settings → Customer privacy in Shopify), pick Shopify's own cookie banner; Varly then measures when the visitor allows analytics (Varly help: Cookie consent).
4. Check that data comes in
Open your store in a normal browser window, accept cookies and click through a few pages. Step 4 on the Install page shows how many page views and clicks came in during the last 24 hours. Testing on your .myshopify.com address? Add it under Settings → Tracking & privacy → Domains first. Still nothing? See Check that Varly is working.
5. Connect your orders
The snippet never reads orders. You connect them with Shopify webhooks, without installing an app (Varly help: Connect Shopify orders):
- In Varly, copy your order address from Install → 5. Connect your orders.
- In Shopify, go to Settings → Notifications → Webhooks and create a webhook for Order creation, format JSON, with that address.
- Create two more with the same address: Order cancellation and Refund create. Canceled and refunded orders then lower the revenue in Varly too.
- Copy the signing key Shopify shows below the webhook list ("Your webhooks will be signed with…") and paste it into Webhook signing key in Varly. Click Connect Shopify.
How orders are linked to visits: for visitors who accepted cookies, the snippet adds a hidden visitor ID and session ID to the cart. Shopify keeps them with the order, so Varly knows which visit led to the sale. Customers don't see them. Test orders from Shopify's test mode are skipped, and only orders placed after you connect are counted.
On Shopify, Varly also counts Added to cart by itself for your theme's standard add-to-cart button, so you can use it as a test goal (Varly help: custom events).
6. Choose where to read results
The last step asks where you want to read test results: Only in Varly, or In Varly and Google Analytics 4, so GA4 also learns which version each visitor saw (Varly help: GA4). Either way, Varly's results page is where it picks a winner.
What you get on Shopify
With the snippet and orders connected: heatmaps with order filters, recordings, funnels, on-site surveys, click impact (orders and revenue of visitors who clicked an element vs. those who didn't) and A/B tests on your storefront pages, including split URL tests. Pricing: €99 per website per month.
When Varly is not the right choice on Shopify
- You want to test checkout, prices or shipping rates. Varly never changes checkout and doesn't change prices in your store. Use Rollouts or a price-testing app.
- You want a one-click Shopify App Store install. Varly is installed by pasting a snippet and creating webhooks.
- Your theme tests are whole-theme swaps. Rollouts does that natively on Grow.
- You want revenue per visitor to decide the winner. Varly's verdict compares conversion rates; order value isn't part of it.
- You need post-purchase surveys on the Shopify thank-you page. Use a checkout-extension survey app.
Decision table
| Your situation | Native Rollouts | Shopify testing app | All-in-one tool (research + tests) |
|---|---|---|---|
| Testing a new theme against the current one | Best fit | Works | Works (split URL or theme preview) |
| Testing checkout configurations | Best fit (Grow+) | Plus-only in some apps | Not possible |
| Price or shipping tests | Catalogs in Rollouts | Best fit (e.g. Intelligems) | Not possible |
| Testing one element on all product pages | Clumsy (duplicate theme) | Works | Best fit |
| Finding out what to test (heatmaps, recordings, surveys) | Not included | Mostly not included | Best fit |
| Store on Basic plan | Experiments not available | Works | Works |
| Very low traffic (under ~7,000 visitors a week to the page) | Test only big changes | Same | Research first, test big changes |
FAQ
Does Shopify have A/B testing built in? Yes. Rollouts lets you run experiments on themes, checkout configurations, catalogs and discounts. Experiments need the Grow plan or higher.
Can I A/B test prices on Shopify? With Rollouts, catalog changes control prices, and Shopify says experiments can show different prices on your online store than on other channels. Price-testing apps such as Intelligems offer a dedicated workflow. Check the legal side in your market first.
Can heatmap tools record the Shopify checkout? Generally not with a theme script, because the checkout doesn't use your theme. Some vendors offer checkout capture for Plus stores through their own app (FullStory documents this, for example: FullStory help); check with your vendor.
Will the script-tag deprecation break my tools?
It affects apps that inject scripts through Shopify's ScriptTag API. Ask your app vendors whether they've moved to app embed blocks or web pixels. Code you pasted into theme.liquid yourself isn't affected.
Do I need consent for heatmaps on Shopify? In the EU, generally yes. Connect your tool to Shopify's cookie banner or your consent platform. See our GDPR guide for CRO.
For test ideas, see 60 ecommerce A/B test ideas.
Varly gives Shopify stores heatmaps, recordings, surveys, click revenue and A/B tests in one script. Request access.
