Two-Proportion Z-Test for AB Testing: Worked Example

The two-proportion z-test compares conversion rates between variant A and variant B. It pools both groups to estimate the natural variation, computes how many standard errors apart the two rates are, and converts that into a p-value. It is valid when the sample size was fixed in advance and groups are large enough.

The formula in plain terms

Let A have n1 visitors and x1 conversions, B have n2 and x2. The rates are p1 = x1/n1 and p2 = x2/n2. The pooled rate is p = (x1 + x2) / (n1 + n2). The test statistic is:

z = (p2 - p1) / sqrt( p(1 - p) x (1/n1 + 1/n2) )

A large absolute z means the difference is large relative to what chance would produce. For a two-sided test, |z| above about 1.96 corresponds to a p-value below 0.05. The test is described in general terms in the Wikipedia entry on the Z-test.

Key takeaway: the z-test answers a narrow question: how surprising is this gap if there were no real difference? It does not tell you how big the effect is or whether it is worth shipping.

A worked example

Variant A: 8,000 visitors, 400 conversions (5.0%). Variant B: 8,000 visitors, 464 conversions (5.8%).

Quantity	Value
p1	0.050
p2	0.058
Difference	0.008 (0.8 points)
Pooled p	864 / 16,000 = 0.054
Standard error (pooled)	about 0.00357
z	about 2.24
Two-sided p-value	about 0.025

Step by step: the pooled variance term is 0.054 x 0.946 = 0.0511. Multiply by (1/8,000 + 1/8,000) = 0.00025 to get 0.0000128. The square root is about 0.00357. Dividing 0.008 by 0.00357 gives z of roughly 2.24, and a two-sided p-value near 0.025. That clears the 5% threshold.

Report the interval, not just the p-value

A p-value says nothing about size. A confidence interval for the difference does. Using the unpooled standard error, sqrt(p1(1-p1)/n1 + p2(1-p2)/n2), which here is about 0.00357 as well, the 95% interval is 0.008 plus or minus 1.96 x 0.00357, or roughly 0.1 to 1.5 percentage points.

Read that carefully. The test is “significant”, yet the plausible lift ranges from almost nothing to nearly a third of the baseline. If your decision threshold was a full point, this result does not clearly meet it. Good reporting states both. BusinessMCP’s A/B testing page says its statistics include lift confidence intervals and up-front sample-size planning alongside the z-test, which is the right pairing.

Assumptions that must hold

1. Random, independent assignment. Each visitor is assigned once and stays in that group. Reassignment on every visit breaks independence.
2. A pre-set sample. The p-value is valid when you look once, at the planned sample. Repeated peeking invalidates it; see how long to run an AB test.
3. Large enough counts. The normal approximation works well when both groups have many conversions and non-conversions. A common rule is at least several dozen of each. Vendors often set explicit gates, for example BusinessMCP says 300 visitors and 25 conversions per arm before showing results.
4. One primary metric. Testing many metrics and reporting the one that worked raises false positive risk.
5. No contamination. Overlapping tests or shared bot traffic distort results.

Common misreadings

• “There is a 95% chance B is better.” That is a Bayesian statement. The z-test p-value does not say it.
• “Not significant means no difference.” It means the test could not detect one at this sample. Absence of evidence, not evidence of absence.
• “p = 0.049 is a win and p = 0.051 is a loss.” The threshold is a convention. Treat results near it as uncertain and consider the interval and the cost of being wrong.
• “A bigger lift in a small sample is more convincing.” Small samples produce large random swings. Check the interval width.

Z-test versus other approaches

Method	Idea	Strength	Caution
Fixed-horizon z-test	Plan sample, test once	Simple, auditable	Intolerant of peeking
Bayesian	Probability one variant beats the other	Intuitive reading	Priors and stopping rules need care
Sequential	Valid under continuous monitoring	Allows early stops	More complex to explain

BusinessMCP’s comparison lists VWO and AB Tasty as Bayesian, Optimizely and Kameleoon as sequential, Convert as offering both, and itself as using the two-proportion z-test. None is wrong; what matters is knowing which you have. Our AB testing tools comparison puts the statistics next to pricing and workflow.

Doing it yourself

Spreadsheet formulas can compute everything above, and online calculators exist for a quick check. Before the test, use a calculator such as Evan Miller’s to set the sample size. After the test, compute z and the interval from the raw counts. If a tool will not show you those raw counts, that is a reason for caution about its verdicts. Record the numbers in your experiment log so that anyone can recheck the decision later.

Rare conversions and small counts

The normal approximation behind the z-test works well when each group has plenty of conversions and non-conversions. When conversions are rare, for instance a handful of purchases per arm, the approximation weakens and exact methods such as Fisher’s exact test are the safer choice. That is one reason vendors set minimum counts before showing results. BusinessMCP’s comparison page states 300 visitors and 25 conversions per arm, which keeps results from appearing while the counts are tiny.

If your page produces fewer conversions than that in a reasonable time, the better response is not a cleverer test but a different design: a bolder change, an earlier funnel event, or a qualitative study.

Checking a vendor’s number by hand

A useful habit is to recompute a tool’s verdict from raw counts. Export visitors and conversions per arm, compute the pooled rate, the standard error, z and the interval as in the example above, and compare. Small differences can come from one-sided versus two-sided tests or from continuity corrections, so check which convention the tool uses. If the results differ materially, ask the vendor why. A transparent tool will say.

This is also a good use of free analytics: if your web analytics or event log records the variant a visitor saw, you can verify the split and the conversion counts independently of the testing tool. Independent verification takes an hour and can prevent a bad launch. For the planning side of the same calculation, see how long to run an AB test, and for the wider tool landscape, the comparison on BusinessMCP’s site lists the statistical approach each vendor uses, with the usual caveat that it is a vendor document.