The A/B Test Flicker Effect: Causes, Measuring and Fixes

What flicker is and why it ruins tests

Most A/B testing tools for shops work client-side. Your server sends the normal page, and a script in the visitor's browser changes it: new text, a new block, a different layout. If the browser shows the page before the script has done its work, the visitor sees the original for a moment, and then it jumps to the variant.

On a fast laptop that may be 100 milliseconds and nobody notices. On a mid-range phone on mobile data, it can be a second or more, and very visible.

Why that matters for your results:

• The variant isn't what you designed. Visitors in the variant group see the original, then a jump. That's a different experience from the one you meant to test.
• The jump itself changes behavior. Content that moves can make people misclick, lose their place, or trust the page less. It also adds layout shift, one of Google's Core Web Vitals.
• It biases toward "no difference" or against the variant. If the variant loses or ties, you won't know whether the idea was bad or the flicker was.
• It can break the split. If the variant is slower or glitchier, some visitors may leave before they're counted in one group only. That creates a sample ratio mismatch. See SRM: is your A/B test broken?.

Causes

1. The script loads late

The most common cause. The testing script can only change the page after it has downloaded and run. Things that delay it:

• Loading it through a tag manager. Google Tag Manager is itself a script that has to load first, and then loads yours. Simo Ahava measured this in 2020 with Google Optimize: pages were hidden on average 581 ms when the snippet was in the page, versus 964 ms when loaded via Tag Manager (Simo Ahava). VWO's help center also points to asynchronous tag managers as a cause of flicker (VWO help).
• Placing it low in the page, for example at the end of the <body>.
• Script-delay features in performance plugins, which hold back JavaScript until the visitor scrolls or clicks.
• Waiting for something else first, such as the cookie banner or another script.

2. The script is big

A testing script that includes the code for every test, every audience rule and every tracking feature takes longer to download and run, especially on slow phones. Scripts often grow with every test you add.

3. The element you change appears late

Some parts of a page are built by JavaScript after the page loads: product recommendations, reviews widgets, prices in a single-page app, lazy-loaded sections. The testing tool can't change something that isn't there yet. If it changes it the moment it appears, there can still be a flash.

4. The change itself is heavy

Changing text is quick. Loading a new image, a web font or a large block of HTML takes longer, and the visitor may see the original while it loads.

5. Consent waits

In the EU, many testing tools only run for visitors who accept cookies. On the very first page, before the visitor has clicked "accept", they see the original. What happens after the click depends on the tool: some change the page immediately (a visible switch), others wait until the next page. Check which one yours does; neither is wrong, but you should know.

How to measure it

You don't need special tools to see flicker. Measuring it takes ten minutes.

Method 1: Chrome DevTools filmstrip (5 minutes)

1. Open the test page in Chrome, in an incognito window, and force yourself into the variant (most tools have a preview or "force variant" link).
2. Open DevTools → Performance. Tick Screenshots.
3. Set CPU to 4× slowdown and Network to a slow mobile profile. Real phones on mobile data are slower than your laptop.
4. Click the reload-and-record button.
5. Step through the screenshots. Note the first frame where the original element is visible and the first frame where the variant is visible. The gap is your flicker.

Do it three times; the numbers vary.

Method 2: Film a real phone

Screen-record a mid-range Android phone loading the variant on mobile data. Play it back frame by frame. Crude, but convincing when you need to show a colleague or client.

Method 3: Measure real visitors

For a number based on real traffic, Simo Ahava describes a method that records, for each visitor, when the original element appears, when it becomes visible on screen, and when the variant is applied, using the browser's MutationObserver and IntersectionObserver (Simo Ahava, Simple Way To Measure A/B Test Flicker Impact). The flicker is the time between "original visible" and "variant applied". This needs a developer, but tells you how bad it is across real devices.

Varly measures this for you, on tests made in its visual editor and for visitors who accepted cookies: right before the snippet changes part of the page, it checks whether the visitor's browser had already shown that part, and for how long at most. The test's results then show what share of visitors saw the original for 100 ms or more. Browsers can't reliably report a flash of one or two frames, so treat anything under about 50 ms as "too short to measure", with any method.

Also measure the cost of your fix

If you use an anti-flicker snippet, measure First Contentful Paint and Largest Contentful Paint with the snippet on and off (DevTools Performance or Lighthouse, same throttling). Shopify recommends exactly this comparison, plus checking your real-user data (Shopify).

Fixes

Put the script in the <head>, as high as possible, directly in the page

Not through a tag manager, not at the bottom of the body. The earlier it starts, the earlier it can change the page. This is the single most effective fix and costs nothing.

Keep the script small

Ask your vendor how big the script is (compressed) and whether it grows with each test. Remove old, finished tests from your tool. Remove testing scripts from tools you no longer use.

Synchronous vs asynchronous loading

• Synchronous (the browser waits for the script before showing anything): no flicker, but if the script's server is slow, the whole page waits. A slow or unreachable testing server can make your shop blank.
• Asynchronous (the browser keeps going while the script loads): your page is never held hostage, but the script has to work fast to avoid flicker.

Several big tools still recommend synchronous loading, and most asynchronous ones hide the whole page while they load (see the table below). If your tool recommends synchronous loading, ask what happens when its server is slow.

Hide only what you change

Instead of hiding the whole page, a tool can hide just the elements it's about to change, apply the change, and show them again, with a maximum wait in case something goes wrong. Visitors see the rest of the page immediately, and the changed element appears once, in its final form.

That's the approach Varly takes: the snippet loads asynchronously, hides only the parts of the page it is about to change, and shows them again as soon as the change is made, after 2.5 seconds at most. See Why is my test not showing? and Install the snippet.

If you write custom JavaScript for a variant, do the same yourself: hide the target element with CSS first, make the change, then show it. VWO gives the same advice for custom code (VWO help).

Prefer CSS for visual changes

Changing colors, sizes, order or visibility with CSS is applied instantly once the stylesheet is in place. Heavy DOM rewrites take longer.

Test server-side when the change is big

In server-side testing, your server (or CDN edge) decides the version and sends the finished page. There's nothing to swap in the browser, so no flicker and no anti-flicker cost. Shopify puts it simply: "Server-side A/B testing has no anti-flicker performance cost" (Shopify). The trade-off: it needs developer work for every test, and caching must be set up so each visitor gets the right version.

Avoid testing things that render very late

If the element only appears two seconds after load (a reviews widget, say), testing changes to it client-side will be messy. Change how it's loaded first, or test it server-side.

Anti-flicker snippets: pros and cons

An anti-flicker snippet is a small piece of code placed above the testing script. It hides the whole page (by setting its opacity to zero, or by covering it with a white layer) until the testing script says "ready", or until a timeout runs out. Google Optimize, the free tool Google shut down on September 30, 2023, popularized it; its snippet waited up to 4 seconds by default (Simo Ahava). Many tools still offer one; VWO's asynchronous code, for example, hides the page body by default while it loads, with configurable time limits (VWO/Wingify help).

Pros

• No flash of original content.
• Simple to add.

Cons

• Everyone waits, including visitors who aren't in any test, and including pages where no test runs.
• A blank page is worse than a flicker if the testing script is slow. The visitor stares at nothing until the timeout.
• It hurts your speed metrics. Shopify calls the snippet "synchronous JavaScript that blocks rendering" and says the cost is often over 2 seconds of First Contentful Paint (Shopify).
• It's often forgotten. The snippet stays in the theme long after the last test ended.

If you use one:

• Set the shortest timeout that works for your tool (well under the old 4-second default).
• Load it only on pages with a running test, if your tool or theme allows.
• Remove or switch it off when no tests are running. Shopify suggests a theme setting that wraps the testing code in a condition, so you can turn it off without editing code (Shopify).

We tested: what visitors see when the testing script is slow (November 2026)

When the testing script's server stopped answering, a page with a synchronous testing tag in the <head> stayed completely blank for the full 30 seconds we waited. With an anti-flicker snippet, the blank screen ended at the snippet's timeout (0.75 or 2 seconds), and then visitors saw the original page. Without any page hiding, the text appeared at once.

Method. On November 7, 2026 we served a simple test page (a headline and a paragraph) from our own computer, plus a stand-in "testing script": three lines of our own code that change the headline. A small local server sent that script after a delay we chose: 150 ms (a normal day), 3 seconds (a slow server or bad mobile connection) or never (the server hangs). We loaded the script in four ways, modeled on the documented default snippets in the table further down: synchronous in the <head>; asynchronous with the whole body hidden for at most 2 seconds (like VWO's asynchronous SmartCode); asynchronous with a white cover over the page for at most 750 ms (like Kameleoon's anti-flicker tag); and asynchronous without hiding anything. No vendor code was loaded. Headless Chromium 156 (Playwright 1.64), no throttling, 3 runs per setup, median. For the synchronous setup we recorded only when the page first appeared; the change is applied before that, so there is no flash by design.

When the script arrived in 150 ms, all four setups were fine: the page and the variant appeared within 0.2 seconds, and only the setup without hiding showed the original for about 0.15 seconds.

The surprise: Chrome's own speed metrics missed the white cover. With the white-cover snippet, Chrome reported First Contentful Paint and Largest Contentful Paint at 12 ms, while a screenshot at 400 ms showed a completely white screen. The text was painted, just underneath the cover. The body-hiding snippet did move both metrics, to 2.0 seconds. So if your tool hides the page with a cover, a good Lighthouse or PageSpeed score doesn't prove visitors aren't waiting. Use the filmstrip method above, which shows what's actually on screen.

What this means:

• A synchronous tag is a single point of failure. It has no timeout of its own: if the vendor's CDN hangs, your shop stays blank until the browser gives up.
• An anti-flicker timeout is a promise of a blank screen. Every slow load costs the full timeout, and then visitors see the original anyway, followed by the switch.
• Without page hiding, a slow script means flicker, not a blank page. That's the trade-off of hiding only what you change: it protects your speed, but it needs a small script that loads early.

How popular testing tools load by default (checked 7 November 2026)

What each vendor's documentation recommends today, and what that means while its script is still loading:

Two notes. Most of these tools can be set up differently (VWO and Convert offer both modes), so check the snippet on your own site. And Varly's approach has the weakness shown in the last row of our test: until its script has loaded, nothing is hidden, so on a very slow connection a visitor can still see the original before the change. That's why the placement advice in Fixes matters for every tool, including ours.

Platform notes

Shopify

• Paste the testing script into theme.liquid, high in the <head> (directly after <head> if your tool allows). See Install Varly on Shopify for an example.
• Use the theme-setting approach from Shopify's docs to switch testing code and any anti-flicker snippet off when no tests run.
• If your testing tool is an app, open your page source and check where its script ends up. The later it loads, the more flicker you'll see.
• For theme-level tests, Shopify's own Rollouts compares a changed theme against the live one natively (Shopify Help Center). More in our Shopify A/B testing guide.

WordPress (with or without WooCommerce)

• Add the script to the <head> with a plugin or your theme's header, not through a tag manager. See Install Varly on WordPress.
• Exclude the testing script from "delay JavaScript", "defer" and "combine/minify JavaScript" features in performance plugins such as WP Rocket or LiteSpeed Cache. Delayed testing scripts are a classic cause of heavy flicker.
• Full-page caching is fine for client-side tests (the page is the same for everyone; the change happens in the browser). It's a problem for server-side tests unless the cache varies by test group. More in our WooCommerce A/B testing guide.

For a broader look at how much JavaScript CRO tools add to a page, see our CRO script weight study.

Checklist

• Testing script is in the <head>, high up, directly in the theme (not via a tag manager).
• Script is excluded from JavaScript delay/defer/combine features.
• Old tests and unused testing tools are removed.
• Variant changes are mostly CSS or small DOM edits; custom code hides the element before changing it.
• Flicker measured in DevTools with 4× CPU slowdown and a slow mobile network profile, on the variant.
• If an anti-flicker snippet is used: short timeout, only on test pages, switched off when no tests run, FCP/LCP checked with and without it, plus a filmstrip (a white cover doesn't show up in FCP/LCP).
• Large or late-rendering changes are tested server-side, or not at all.
• You checked what the tool does on the first page before cookie consent.
• After launch, a few session recordings of the variant on mobile confirm that it looks right.

FAQ

Is a little flicker OK? A flash under about a tenth of a second is hard to notice. Anything visible (a few hundred milliseconds or more) is worth fixing, especially above the fold.

Does flicker affect SEO? Indirectly. Layout shifts and delayed rendering show up in Core Web Vitals, which Google uses as a page experience signal. Whole-page anti-flicker snippets can hurt First Contentful Paint and Largest Contentful Paint.

Should I just use server-side testing? For big or late-loading changes, it's the cleanest option. For most small and mid-size shops, a well-placed client-side tool that hides only the changed elements is good enough and far less work.

Does an A/A test show flicker? No. In an A/A test nothing changes, so nothing flickers. It does help to check the split and the tracking. Test flicker on the real variant.

For how flicker fits into running reliable tests, see our ecommerce A/B testing guide.

---

Test without hiding your whole page. Varly's single script loads asynchronously and hides only the elements a test changes, for at most 2.5 seconds. Request access.