How to Prioritize A/B Test Ideas: PXL vs ICE vs PIE

Why you need a backlog (not a to-do list)

A to-do list is "things we should do". A test backlog is "things we believe might improve the shop, and how strongly we believe it". The difference matters because:

• You can only run a few tests at a time. A shop with average traffic can usually run one or two tests per page type per month. Picking the wrong ones costs months.
• Most ideas don't work. At large companies with mature testing programs, only about 8–33% of ideas improve the metric they target, depending on the company (Kohavi, Deng and Vermeer, KDD 2022). If your hit rate is similar, ranking well is the main lever you have.
• Without a system, opinions win. The founder's idea, the agency's pet project, the thing a competitor just launched. A shared scoring method gives quieter, better-evidenced ideas a fair chance.

A good backlog item has: a short title, the evidence behind it ("we saw…"), the change, the expected effect, the page it runs on, a score, and a status. Our hypothesis template shows how to write the first part.

ICE in 2 minutes

ICE was popularized by Sean Ellis at GrowthHackers (Speero). You score each idea from 1 to 10 on:

• Impact: how much will this move the metric if it works?
• Confidence: how sure are we that it will work?
• Ease: how easy is it to build and test?

The score is the average (some teams multiply). An idea with I=8, C=6, E=7 scores 7.0.

Strengths: quick, intuitive, and fine for scoring 50 ideas in a meeting. Weaknesses: every number is a gut feeling. Two people score the same idea very differently, and anyone who wants their idea to win can nudge the numbers. Speero also points out a logical oddity: if you're already confident an idea will work, why test it? (Speero)

PIE in 2 minutes

PIE was created by Chris Goward of WiderFunnel (Speero). Again 1 to 10 on three criteria, but different ones:

• Potential: how much room for improvement does this page have?
• Importance: how valuable is the traffic on this page (volume, cost of that traffic)?
• Ease: how easy is it to build and test?

The score is the average.

Strengths: it forces you to think about where you test. A modest improvement on the product page template can matter more than a big one on a page few people see. Weaknesses: Potential is as subjective as ICE's Impact. And because Importance is about the page, not the idea, every idea on a busy page gets a boost, including weak ones.

PXL in 5 minutes

PXL was developed by CXL (now Speero), with Peep Laja as its best-known author (CXL, Speero). Instead of 1–10 sliders, it asks mostly yes/no questions. The idea: it's hard to argue about "is the change above the fold?", but easy to argue about "is the impact a 6 or an 8?"

The questions fall into three groups:

About the change

• Is it above the fold?
• Is it noticeable within 5 seconds? (counts double)
• Does it add or remove an element?
• Is it designed to increase motivation?
• Does it run on a high-traffic page?

About the evidence

• Was the problem found in user testing?
• Was it found in qualitative feedback (surveys, chat, reviews)?
• Is it supported by analytics data?
• Is it supported by heatmaps, scroll maps or session recordings?

About the effort

• Ease of implementation, in time brackets rather than a feeling.

A "yes" scores points; a "no" scores zero. Some questions count more than others. Different versions use slightly different weights. In this article we use these weights: "noticeable" and "adds or removes" count 2, the other yes/no questions count 1, and ease scores 3 (under 4 hours), 2 (up to 8 hours), 1 (under 2 days) or 0 (more). That gives a maximum of 14 points. (Varly's PXL preset uses the same weights and adds one question, "Confirmed by data?": 1 point when two independent sources point to the same problem, 2 points with three or more. Its maximum is 16.)

Strengths: two people scoring the same idea land much closer together. Evidence counts explicitly, so ideas backed by research rise and pure opinions sink. Weaknesses: it takes longer per idea, the yes/no questions can't express "a little", and like the others, it doesn't check whether your traffic can support the test.

One backlog, three frameworks: the results

Here's a hypothetical fashion shop with these weekly visitors and order rates:

The team has 10 ideas. Some come from research (a survey, recordings, analytics), some from opinions. We scored each with ICE, PIE and PXL as a typical team might.

ICE and PIE scores are the average of three 1–10 scores. "=" means a shared rank.

What this shows:

• They agree on the obvious winners. Delivery information and the sticky button are near the top everywhere. When a backlog has clearly strong ideas, any framework finds them.
• PIE pushes weak ideas up because of the page. The button-color test and the footer badges score as high as the free-shipping bar, because Importance rewards busy pages, whatever the idea. Meanwhile, guest checkout on the cart drops to 7th, because fewer people see the cart, even though the cart is where those visitors are closest to buying.
• ICE produces many ties. Three ideas share 5th place and three share 8th. With 1–10 gut scores, many ideas end up around 6.
• PXL rewards research. Review stars on category tiles jump to joint 2nd because a survey and a heatmap back them. The bundle offer ("a competitor does it") sinks.
• The size guide is underrated by all three. It's strongly evidenced (user test, survey, returns), but it's hard to build and not "above the fold". In a fashion shop with a returns problem, that could be the most valuable idea on the list. Frameworks are a starting point, not a verdict.

The question all three forget: does it have enough traffic?

A high score means nothing if the test can't finish. None of the three frameworks asks how many visitors the test needs. (Switas makes the same point in its comparison of the frameworks: Switas.)

So we added one column: how many weeks would this test need, given the page's traffic and a realistic expected effect? We used the standard sample size calculation (95% confidence, two-sided; 80% power; one variant; 50/50 split), rounded up to full weeks, minimum two. Above 8 weeks we call it "not feasible". You can do the same with our sample size calculator.

Only four of ten ideas can finish within eight weeks. Three of the PXL top five can't be tested as planned.

That doesn't mean you throw the others away. It tells you what to do with them:

• Change the metric. The sticky button aims to get more products into carts. Measure add-to-cart instead (say 9% of mobile visitors): at +8%, it needs 25,687 visitors per version, about 6 weeks. Keep orders as a guardrail.
• Make it bolder. A free-shipping bar alone may move orders by a few percent. A new shipping offer plus the bar might move them enough to test.
• Just ship it. Review stars on category tiles are low risk and well supported by research. If you'd keep them even if a test showed "no difference", skip the test and monitor.
• Drop it. The button color and footer badges have no evidence and need years of traffic. Delete them from the backlog.

Our article on A/B testing with low traffic goes deeper into these options.

Making scores less subjective

Whatever framework you pick, these habits make scores more reliable:

1. Write anchors. Define what a score means. "Impact 8+ = changes what visitors see in the first screen of a page with 10,000+ visitors a week." "Confidence 7+ = at least two independent sources of evidence." Put the anchors at the top of the sheet.
2. Score separately, then compare. Have two or three people score new ideas on their own. Discuss only the ones where scores differ by more than 2 points (or one PXL answer). The discussion is often more useful than the score.
3. Link evidence, not opinions. Every "yes" on an evidence question should link to the survey result, the recording or the analytics report. No link, no point.
4. Use time, not effort feelings. "Under 4 hours / up to 8 hours / under 2 days / more" is easier to agree on than "Ease 7".
5. Check your scores against results. Every quarter, compare the scores of finished tests with their outcomes. If high-scoring ideas don't win more often than low-scoring ones, your framework needs work.
6. Re-score when you learn something. A new survey, a failed test on a related idea, a site change: update the scores.

Custom frameworks for your business

The best framework is often one of the three, adjusted. Common additions:

• Traffic feasibility (as above): "Can this test finish within 8 weeks?" Yes = points, no = zero or a flag.
• Revenue impact: visitors on the page × conversion rate × average order value × expected effect. It turns "impact" into money, which makes it easier to compare a cart idea with a homepage idea.
• Strategic fit: does it support this quarter's goal (more first orders, higher order value, fewer returns)?
• Validated by earlier tests: did a similar idea win before?
• Risk: could it hurt brand, legal compliance or operations (for example, a promise of next-day delivery)?

Keep the total number of criteria small (8–12). More criteria means slower scoring and more fake precision.

Running a backlog for several clients

Agencies and freelancers face extra problems: several shops, different traffic, different stakeholders.

• One backlog per client. Don't mix ideas across clients; the evidence and traffic are different.
• One framework across clients, with client-specific weights only where the business really differs (for example, a returns criterion for fashion clients). That keeps your team's scores comparable and makes reporting easier.
• Score with the client, not for them. Let the client answer the business questions (strategic fit, risk) and your team the evidence and effort questions. It builds trust and avoids "why is my idea at the bottom?"
• Make evidence visible to the client. A backlog where every idea links to the recording or survey answer behind it is much easier to defend in a meeting.
• Review monthly. Archive old ideas, re-score with new research, and show the client what moved and why.

In Varly, every idea is scored with PXL (the default), PIE, ICE or your own framework, answers about research are filled in from the heatmaps, recordings, surveys and click data linked to the idea, and each idea shows whether the page has enough traffic and how many weeks a test would take. Agencies can set one default framework for all clients or a separate one per client. See Prioritize ideas.

Build your own scoring sheet

You don't need a special tool to start. A spreadsheet works. Set up these columns:

Sort by "Feasible?" first, then by score. Review the top five every week, and the whole list every month.

To go from a prioritized idea to a running test, read our A/B testing guide. To report the outcome, see our A/B test report template.

FAQ

Which framework is best: PXL, ICE or PIE? For most ecommerce teams, PXL (or a PXL-style yes/no framework) gives the most consistent scores, because it ties points to evidence. ICE is fine for a quick first pass. PIE is useful when deciding which pages to work on. Whatever you choose, add a traffic-feasibility check.

How often should I re-score my backlog? Score new ideas when they come in, and review the full backlog monthly or whenever you finish a round of research.

Should I test the highest-scoring idea even if it needs 20 weeks? No. Change the metric, make the change bolder, test on a larger template, or ship it without a test. A test that can't finish is not a test.

What about RICE? RICE (Reach, Impact, Confidence, Effort) comes from product management. It's similar to ICE with an explicit "reach" (how many people are affected). It works for CRO too, but has the same subjectivity problem as ICE.

How many ideas should be in a backlog? Enough to fill the next three months of tests, plus a tail of ideas waiting for evidence. If the list has more than 100 items, archive the bottom half.

---

Score ideas against your own data. Varly links every idea to the research behind it, scores it with PXL, PIE, ICE or your own framework, and tells you whether the test can finish with your traffic. Varly is in private beta. Join the waitlist (agencies: join the agency waitlist).