AB Testing Mistakes That Produce False Winners

The most common AB testing mistakes are stopping early when a result looks significant, running underpowered tests, testing too many things at once, ignoring seasonality, and acting on weak hypotheses. Each produces false winners. The fixes are mostly procedural: plan the sample, write the rule, and look once.

1. Peeking and stopping early

Watching a live dashboard and stopping when significance appears is the classic error. Because random noise crosses a threshold surprisingly often, the false positive rate climbs well above the nominal 5%. Evan Miller’s essay How Not To Run An A/B Test is the standard explanation.

Fix: decide the sample size beforehand and read the result once, or use a method designed for continuous monitoring and follow its rules. Some tools help by hiding results early; BusinessMCP’s A/B testing page says the confidence column stays hidden until the minimum sample and 14 days have passed.

Key takeaway: a result is only as trustworthy as the plan made before seeing it.

2. Underpowered tests

Running a test on a page that cannot reach a valid sample produces noise with a confident label. A test that can only detect a 50% lift will miss the 10% lift that is actually there, and the occasional “win” it does find is likely exaggerated.

Fix: calculate sample size first. If the numbers are out of reach, test a bigger change or a higher-traffic page.

3. Changing the plan mid-test

Extending a test because the result is disappointing, changing the traffic split, or adding a variant midstream all distort the statistics. Extending to chase significance is peeking in disguise.

Fix: pause only for genuine breakage, note the gap, and keep the planned stopping rule. See how long to run an AB test.

4. Too many metrics and variants

If you track twenty metrics, one will look significant by chance. If you run six variants, the number of comparisons rises. Both inflate false positives.

Fix: name one primary metric before launch. Treat other metrics as secondary and exploratory. Keep the number of arms small.

5. Overlapping tests

Two tests on the same page, or on pages in the same funnel, interact. A visitor in both gets a combination neither test planned for.

Fix: keep a test calendar, run one test per page or funnel at a time, and note any overlaps in the log.

6. Ignoring seasonality and business cycles

A test started on a holiday weekend measures holiday behaviour. A test that spans a campaign launch measures the campaign.

Fix: run whole weeks, avoid known distortions, and cover at least one business cycle.

A quick reference table

Mistake	Symptom	Fix
Peeking	Winner appears on day two, vanishes later	Fixed sample or valid sequential method
Underpowered test	Huge lift claimed from a small sample	Plan sample size first
Moving the goalposts	Test extended until significant	Written stopping rule
Metric fishing	“Winner” on a secondary metric	One primary metric
Overlap	Odd interaction effects	One test per page at a time
Bad timing	Result differs after the holiday	Whole weeks, full cycle
Sample ratio mismatch	50/50 split shows 55/45	Check assignment and tracking

7. Sample ratio mismatch

If you planned an even split and the observed groups differ noticeably, something is wrong: a redirect that drops visitors, a bot filter applied to one arm, or a script that fails in a browser. The result cannot be trusted until you find the cause.

Fix: compare observed counts with the plan before reading conversion rates. Flicker-free assignment helps; BusinessMCP’s page says an inline script sets the arm before paint and the server re-derives it, which is one design for keeping assignment consistent.

8. Testing without a hypothesis

Random changes teach little even when they win, because you cannot say why. A good hypothesis states an observation, a change and an expected effect on a metric.

Fix: base ideas on evidence. Our guide to AB testing ideas from heatmaps shows how to find it, and the same source describes ranking by impact, confidence and ease.

9. Treating a win as permanent

Wins can fade because of novelty effects, changing traffic mix or later site changes.

Fix: keep a holdout where practical, monitor the metric after launch, and re-test major assumptions occasionally.

10. Not recording losers

Teams repeat failed ideas when nobody wrote them down.

Fix: keep a log with hypothesis, dates, sample, result and decision, including negative results.

Pre-launch checklist

1. One primary metric written down.
2. Sample size and duration calculated.
3. Stopping rule agreed.
4. Calendar checked for overlap and seasonality.
5. Split verified after the first day.
6. Variant reviewed on mobile and desktop.
7. Log entry started.

If your tool makes any of these hard, that is information about the tool. Our best AB testing tools guide and the explanation of the two-proportion z-test help you choose and read results with the right level of caution.

Mistakes specific to automation

As teams adopt tools that propose and build tests, new errors appear. The first is approving proposals without reading the hypothesis: a ranked list is a starting point and the person confirming is still responsible for whether the idea makes sense. The second is launching many automated tests at once on overlapping pages. The third is accepting a plain-language summary without checking the counts underneath. Our piece on AI for AB testing goes through where automation helps and where it adds risk.

Mistakes in reporting

Even a well-run test can be undermined in the write-up. Report the sample sizes, the dates, the primary metric, the observed rates and an interval for the difference, not just “B won”. State what you did not test. If you changed the stopping rule, say so. And keep the negative results: a log of failed ideas saves the next team from repeating them.

For a code-based example of a tool built around some of these guardrails, such as hidden confidence until the minimum is met and a human-reviewed pull request, read the description on BusinessMCP’s session replay and testing pages, and treat it as one vendor’s design rather than a standard. Whatever you use, the habits above matter more than the brand.