AB Test Sample Size: How Many Visitors You Really Need

AB test sample size is the number of visitors each variant needs before a result can be trusted. It depends on four inputs: your baseline conversion rate, the smallest lift you want to detect, the significance level (usually 5%) and statistical power (usually 80%). Smaller lifts and lower baselines need far larger samples.

The four inputs

Every sample size calculator asks for the same things, and understanding them stops you from treating the output as magic.

1. Baseline conversion rate. The current rate for your goal on this page, for example 5% of visitors sign up.
2. Minimum detectable effect (MDE). The smallest change worth finding, either absolute (5% to 6%) or relative (a 20% lift).
3. Significance level (alpha). The tolerated chance of a false positive, conventionally 5%.
4. Power. The chance of detecting a real effect of the MDE size, conventionally 80%.

You choose the MDE, not the calculator. If you would only ship a change that lifts conversion by at least a fifth, there is little point planning to detect a 2% improvement. Be honest about this number, because it drives everything.

Key takeaway: sample size scales with the inverse square of the effect. Halve the lift you want to detect and you need roughly four times the visitors.

A rule of thumb with worked numbers

For a two-sided test at 5% significance and 80% power, a common approximation for visitors per variant is 16 multiplied by p(1-p) and divided by the squared absolute difference. It slightly understates the exact answer, so use it to sense-check, then confirm with a real calculator such as Evan Miller’s sample size calculator.

Baseline	Target	Absolute change	Approx. visitors per variant	Approx. total
10%	12%	2 points	3,600	7,200
5%	6%	1 point	7,600	15,200
2%	2.5%	0.5 points	12,500	25,000

These are approximations from the rule of thumb, not outputs of a specific vendor tool. The pattern is the point: low baselines and small lifts push the numbers up quickly. A page converting at 2% needs far more traffic to detect half a point than a page at 10% needs to detect two.

Turning visitors into time

Divide the required total by your weekly eligible traffic. If the 5% to 6% test above needs about 15,200 visitors in total and the page receives 3,000 eligible visitors a week, the test needs about five weeks. If it receives 500 a week, it needs about 30 weeks, which is not a test but a standing experiment that your site will have outgrown before it ends.

That is where honest options appear. Test a bigger change, where a 30% relative lift is plausible. Test on a higher-traffic page. Move the goal earlier in the funnel, to a click rather than a purchase, accepting that a click is a weaker proxy. Or skip the split test. The next article in this cluster, how long to run an AB test, covers scheduling, including why to round up to whole weeks.

Why peeking breaks the plan

The sample size you calculated is valid only if you commit to it. Checking the dashboard daily and stopping when significance first appears inflates false positives well above 5%, a point Evan Miller’s essay How Not To Run An A/B Test makes forcefully. Some tools guard against this. BusinessMCP’s A/B testing page says its results page hides the confidence column until the minimum sample and 14 days have passed, and its comparison page states that each arm needs 300 visitors and 25 conversions before results show. Those are that vendor’s own minimums, not a substitute for your own calculation based on the lift you care about.

Common sample size errors

• Using visits instead of visitors. If the same people return many times, count unique visitors per variant.
• Ignoring segments. A sample sufficient for all traffic is not sufficient for “mobile visitors from paid search”. Each subgroup needs its own sample, or treat segment cuts as hypotheses for the next test.
• Testing many variants. Every extra arm adds comparisons and needs more traffic. Keep to one change and two arms when traffic is limited; see AB testing versus multivariate testing.
• Moving the MDE after the fact. Deciding, after seeing a small lift, that it is what you wanted all along defeats the plan.
• Forgetting the baseline drifts. Seasonality changes conversion rates. Recalculate if conditions shift.

What to do when you cannot reach the sample

Say so. A before/after comparison, a qualitative study or a decision made on judgement are all acceptable if labelled honestly. BusinessMCP’s page describes before/after testing for changes that cannot be split, such as redesigns, pricing and SEO titles, with results labelled confounded. That label is the right instinct: weaker evidence should be presented as weaker evidence.

A planning checklist

1. Pull the current conversion rate for the exact goal and page.
2. Decide the smallest lift you would act on.
3. Run the numbers in a calculator; record alpha, power and the inputs.
4. Convert to weeks of traffic, then round up to whole weeks.
5. Write the stopping rule: “we read the result once, at this sample”.

If steps three and four tell you the test is impractical, read free AB testing tools and the two-proportion z-test explainer before spending money on a platform that cannot fix a traffic problem.

A worked planning conversation

Imagine a team that wants to test a new hero section on a landing page that converts 3% of 5,000 weekly visitors. They would be happy with a lift to 3.6%, a 20% relative improvement. The rule of thumb gives about 16 x 0.03 x 0.97 / 0.000036, close to 12,900 visitors per variant, so roughly 25,800 in total. At 5,000 visitors a week that is over five weeks. The team now has a real choice rather than a vague hope: wait five or six weeks, test a bolder change that might lift conversion by 40%, or decide the test is not worth running.

That conversation is the main value of sample size planning. It converts “let’s test it” into a cost in calendar time, which you can compare with the expected benefit. If the expected gain is small, the answer is often to ship the change on judgement and spend testing capacity elsewhere.

Where to run the numbers and what tools offer

Free calculators handle the arithmetic, and several testing platforms include one. BusinessMCP’s automated A/B testing page describes up-front sample-size planning as part of its statistics, and its web analytics product is where a team on that platform would find baseline conversion rates. If you use other analytics, pull the baseline from there. Whatever you use, record the inputs next to the result, so a later reader can see what the test was designed to detect. For a tool-level comparison of how vendors approach statistics, see our AB testing tools comparison.