Skip to content
Log inRequest access

Magento / Adobe Commerce A/B Testing and Heatmaps (2026)

Your A/B testing and heatmap options on Magento and Adobe Commerce in 2026: native experimentation, full page cache pitfalls, Hyvä vs Luma and order tracking.

Joost BakkerBy Published 10 min read

Three stacked layers
On this page
  1. What Adobe Commerce offers natively
  2. Third-party options
  3. Full page cache and Varnish: how tests can break
  4. Hyvä and Luma differences
  5. Tracking orders
  6. Heatmaps and recordings
  7. Setup checklist (with Varly as the example)
  8. FAQ

What Adobe Commerce offers natively

Adobe Commerce as a Cloud Service (Adobe's newer SaaS version) positions built-in experimentation as a feature: "set up and run A/B tests directly in Adobe Commerce" without developers or third-party tools (Experience League).

Adobe's storefront documentation describes how it works (Adobe Commerce storefront docs):

  • An Experimentation plugin that "integrates seamlessly with the AEM boilerplate code," so experiments load early.
  • Metadata markers and rules decide which version of a page visitors see. You can target by device type and by campaign links.
  • Each variant's conversions are tracked with "real user metrics."
  • Consent and data handling "depend on your plugin configuration and your organization's policies." Check them with your developer before launch.

What this means for you: this applies to storefronts built on Adobe's newer AEM-based boilerplate. If your shop runs a classic Magento theme (Luma or Hyvä) on Magento Open Source or Adobe Commerce on your own servers or Adobe's cloud infrastructure, you won't have this plugin, and you'll use a third-party tool.

Third-party options

Type Examples How it works Good for
SaaS testing tools Convert, Kameleoon, Wingify (VWO + AB Tasty), Optimizely A script in the <head> changes the page in the browser Most stores; marketers can build tests
Magento extensions Various A/B testing extensions on the marketplace Variants managed inside Magento admin Simple content tests; check compatibility with your version and theme
Server-side / feature flags GrowthBook, Optimizely, Wingify feature experimentation Your code decides the variant on the server Developer-led teams; tests on search, pricing logic, checkout
All-in-one research + testing Varly, Crazy Egg One script for heatmaps, recordings and tests Teams that want research and tests together

Public prices (checked November 5, 2026): Convert from $299/month billed annually; Kameleoon from $495/month; GrowthBook free to start. Wingify and Optimizely are quote-only; as estimates from Vendr's buyer data, a typical VWO contract is about $1,400 a month (Vendr) and Optimizely starts from about $3,300 a month for one product (Vendr). Details in VWO alternatives.

Before buying an extension, check: does it support your exact Magento version, does it work with Hyvä (many Luma-era extensions need a Hyvä compatibility module), and does it cooperate with full page cache?

Full page cache and Varnish: how tests can break

Magento's full page cache (built-in, Varnish on your own servers, or Fastly on Adobe's cloud, per Adobe's docs) stores the HTML of pages and serves it to everyone. Personal content is handled separately:

  • Private content loads in the browser. Magento's customer-data library stores private data in the browser and loads private blocks with an extra request after the page renders (Adobe developer docs).
  • One non-cacheable block makes the whole page uncacheable. Marking a block as non-cacheable in layout XML "prevents the full page cache from caching that page" (same source).

What that means for testing:

  1. Client-side tests are the safe default. Everyone gets the same cached HTML; the testing script picks a version in the browser and changes the page. The cache never needs to know about the test.
  2. Watch elements that load late. The mini cart, customer name, customer-group prices and some product blocks arrive after the page has rendered. If your test changes one of them, the tool must wait for it, or the change silently doesn't happen. Ask your tool how it handles that.
  3. Server-side tests need cache work. If a Magento extension picks the variant in PHP, the cache will store whichever version it rendered first and serve that to everyone. Fixes include making the cache vary on a test cookie or marking tested blocks private. That's developer work; budget for it.
  4. Never mark a tested block non-cacheable to "fix" a test. You'll turn off full page cache for that page and slow it down for every visitor.
  5. Flush the cache after installing a script in the HTML head (System → Cache Management). Otherwise cached pages keep serving the old HTML without your script.

Flicker (the original showing before the variant) is a bigger risk on slower stores. See the A/B test flicker effect.

Hyvä and Luma differences

Many Magento stores have moved from the default Luma theme to Hyvä. Hyvä keeps Magento's layout XML and templates but drops Luma's UI components, Knockout, RequireJS and LESS, using Tailwind CSS and Alpine.js instead (Hyvä docs). Since November 10, 2025, Hyvä Theme is free and open source; Hyvä UI is sold separately (Hyvä announcement).

For CRO, that matters in three ways:

  • Selectors change. A test that targets .product-info-main .action.tocart on Luma won't find the same element on Hyvä. Rebuild tests and heatmap page rules after a theme migration.
  • Tailwind classes are generic. Classes like flex or text-sm appear on hundreds of elements. Ask your developer to add stable IDs or data- attributes to elements you test often (add-to-cart button, price, USP block).
  • Alpine.js shows and hides parts in the browser. Menus, tabs and drawers change state without a page load. Screenshot-based heatmaps capture the default state; recordings show the rest.

Don't compare before/after a theme migration as if it were a test. Season, campaigns and traffic mix change too. If you want to know whether Hyvä converts better, run a split URL test between the two themes on a staging-quality copy, or at least compare against the same period last year.

Tracking orders

To decide tests on orders, each order has to be linked to the visit and the version the visitor saw. On Magento that usually means:

  1. Read the visitor's ID on the site from the testing tool's script, before checkout (for example on the cart page).
  2. Save it with the order (a hidden field or the order's extra data).
  3. Send the order from your server to the testing tool after it's placed, with that ID.

Server-side sending is more reliable than a script on the success page. It isn't blocked by ad blockers, and you can resend an order after a refund or cancellation.

Check that refunds and cancellations are sent too, that test orders are excluded, and that you send the same order ID each time so retries don't double-count.

Heatmaps and recordings

Any script-based heatmap tool works on Magento: Microsoft Clarity (free), Contentsquare (formerly Hotjar), Mouseflow, Lucky Orange and others. Compared in Hotjar alternatives.

Magento-specific tips:

  • Group pages by type. Use URL rules or page groups for product pages and category pages; Magento URLs often don't share a common folder, so you may need patterns or rules.
  • Exclude checkout and account pages (/checkout, /customer/account) from recordings.
  • Multi-store setups: one store view per language or country usually means one site per domain in your tool, so data doesn't mix.

Setup checklist (with Varly as the example)

Varly, the tool we build, works on Magento through a snippet and its order API. This is what setup looks like (Varly help: other platforms).

  1. Pick your platform. In Varly, open Install and pick Magento / Adobe Commerce under 1.
  2. Add the snippet. In the Magento admin, go to Content → Design → Configuration, edit the store view you want to measure, open HTML Head and paste the snippet at the top of Scripts and Style Sheets. Click Save Configuration, then clear the cache under System → Cache Management. Repeat for each store view.
  3. Turn tracking on on the Install page or under Settings → Tracking & privacy. Varly measures only after a visitor accepts cookies, and never in the checkout. Pick your cookie banner under Consent banner (Varly help: consent).
  4. Check that data comes in. Visit your store, accept cookies, click around. The Install page shows page views and clicks of the last 24 hours. Add any extra domains (staging, other store views) under Domains.
  5. Connect your orders with the order API (Varly help: order API). An Owner creates an API key under Install → Connect your orders. Your developer then:
    • reads the visitor and session ID with Varly.ids() on a page before checkout (such as the cart) and saves them with the order;
    • sends each order from the server to Varly's order endpoint with the API key, including order_id, revenue, currency and the two IDs;
    • sends the order again with the same order_id and created_at and the new total after a refund or cancellation (0 if canceled). Varly counts each order_id once, so retries are safe. Orders without the IDs still count in totals, but not in tests or click impact.
  6. Choose where to read results: only in Varly, or also in Google Analytics 4 (Varly help: GA4).

Also check Never track these pages under Settings → Tracking & privacy: by default it contains /checkout and /cart. Add /customer/account and any localized paths (/en/checkout). Varly waits up to 2.5 seconds for parts of the page that load late, hiding only the parts it's about to change, so blocks loaded after full page cache can still be tested (Varly help: launch a test).

Pricing: €99 per website per month.

When Varly is not the right choice on Magento

  • You don't have a developer for the order API. Without it, Varly can't link orders to visits, so tests can only use goals such as add-to-cart (sent with Varly.track), clicks or page visits.
  • You're on Adobe's newer AEM-based storefront and the native Experimentation plugin covers your needs.
  • You need server-side tests on search, pricing logic or checkout.
  • You need custom JavaScript in variants. Varly's editor changes text, links, images and styles, not your own scripts.
  • You need US data residency. Varly stores data in the EU only.

FAQ

Does Magento Open Source have A/B testing built in? No. Use a SaaS testing tool or an extension.

Does Adobe Commerce have native A/B testing? Adobe Commerce as a Cloud Service promotes built-in storefront experimentation through an Experimentation plugin for its AEM-based storefront boilerplate. Check with Adobe or your partner whether it's available for your setup.

Will full page cache break my A/B tests? Not for tools that change the page in the browser. It can break tests that choose the variant on the server, unless the cache is set up to vary.

Can I still use Google Optimize on Magento? No. Google Optimize stopped working on September 30, 2023 (Google Analytics Help). Many older Magento guides still describe it.

What changes for testing when I move to Hyvä? Your element selectors. Rebuild tests and page rules after the migration, and add stable IDs to elements you test often.

Varly brings heatmaps, recordings, surveys and A/B tests to Magento stores with one snippet and an order API. Request access. Agencies: see Varly for agencies.

Related articles

All articles on this topic