How Long to Run an AB Test: Duration Rules That Hold Up

Run an AB test until you reach the sample size you calculated in advance, rounded up to whole weeks, and not less than two weeks in total. Stopping the moment a dashboard shows significance produces false winners. Duration is a plan you write before launch, not a verdict you discover.

Duration is derived, not chosen

Start from the sample size you need per variant. Divide the total by your weekly eligible traffic, and you have the minimum duration. Everything else is a safeguard layered on top.

Key takeaway: the right duration is the longest of three numbers: time to reach the planned sample, time to cover at least one full business cycle, and any minimum your tool enforces.

Constraint	Typical rule	Why
Sample size reached	Calculated per test	Statistical power
Whole weeks	Round up to 7-day multiples	Weekday and weekend behaviour differ
Business cycle	At least one, ideally two	Pay cycles, billing cycles, campaign rhythms
Tool minimum	e.g. 14 days	Vendor guardrail against early calls

On the last row, BusinessMCP’s A/B testing page says a winner requires both the minimum sample and at least 14 days, and that a winner is never named on the p-value alone. Its comparison page puts the sample gate at 300 visitors and 25 conversions per arm. Treat vendor minimums as floors beneath your own calculation, not substitutes for it.

Why whole weeks matter

Suppose your B2B site gets most of its visits and nearly all of its demo requests on weekdays. A test that starts on Monday and stops on Friday has never seen a weekend, and the weekend visitors may behave differently, perhaps browsing on mobile and converting less. Whole weeks give each variant the same mix of days. If you can only run ten days, include the same number of each weekday where possible, or extend to fourteen.

Business cycles stretch the same logic. If customers are paid at month end, or your campaigns run in a four-week rhythm, a two-week test may sample an unrepresentative stretch. For consumer ecommerce, check whether promotions, holidays or paydays fall inside the window; if they do, either extend the test or acknowledge the limitation.

The early stopping trap

Imagine checking daily and stopping the first time the p-value dips below 0.05. Random fluctuation will cross that line surprisingly often even when the variants are identical, so a “significant” result can appear by chance. Evan Miller’s essay How Not To Run An A/B Test walks through why this happens and why a fixed sample is the safe habit with a standard test.

There are legitimate alternatives. Sequential and some Bayesian methods are designed for continuous monitoring and keep error rates valid when used as intended; BusinessMCP’s comparison lists Optimizely and Kameleoon as sequential, and VWO and AB Tasty as Bayesian. But those guarantees only hold when you follow the tool’s own rules. If you use a fixed-horizon test, the rule is simple: pick the sample, wait, look once. Our two-proportion z-test explainer covers the fixed-horizon calculation.

Worked planning example

A pricing page gets 4,000 eligible visitors a week with a 4% trial sign-up rate. You want to detect a lift to 5%, a one-point change.

1. Using the rule of thumb from the sample size article, about 16 x 0.04 x 0.96 / 0.0001 gives roughly 6,100 visitors per variant, so about 12,200 in total (confirm with a calculator).
2. At 4,000 a week, that is about three weeks.
3. Round up to three whole weeks; it already exceeds a 14-day vendor minimum.
4. Check the calendar for holidays and campaigns in those three weeks, and shift if needed.
5. Write down: “read the result once, at the end of week three.”

If the same page got only 600 visitors a week, the plan would need about 20 weeks, and that should make you test something bolder or reconsider testing at all.

When tests run too long

Long tests carry their own risks: people clear cookies and may be reassigned, site changes creep in, seasonality shifts, and attention drifts. As a rough practical stance, plans that need more than a couple of months deserve scrutiny. Options include a larger change, a higher-traffic page, or an earlier funnel metric; the last trades some validity for speed, so state it openly.

Pausing, extending and ending

• Do not extend a test because the result is disappointing. Extending to chase significance is peeking by another name.
• Do pause for genuine breakage, such as a tracking outage or a bug in the variant, and note the gap in your log.
• Do not launch overlapping tests on the same page, since interaction effects blur the result. AB testing mistakes lists more habits to avoid.

Duration checklist

• Sample size calculated before launch.
• Duration rounded up to whole weeks, at least two.
• Calendar checked for promotions and holidays.
• Stopping rule written down.
• One reading at the end, unless the tool supports continuous monitoring.

If the numbers say your site cannot support the test, the guidance on free AB testing tools and heatmap-driven bolder changes in AB testing ideas from heatmaps is where to look next.

Seasonality, campaigns and special cases

Some situations call for extra care beyond whole weeks. B2B sites often see end-of-quarter surges, ecommerce sites see promotional spikes, and anything tied to school terms or tax deadlines has predictable peaks. If a peak falls inside your window, you have two honest choices: extend the test so that normal days dominate, or run it on purpose across the peak and say that the result applies to peak behaviour.

Paid campaigns matter too. If you switch a campaign on halfway through, the mix of visitors changes and the two arms may not be comparable across the period. Because assignment is random, both arms receive the new visitors equally, so the comparison is not invalid, but the result then describes a blended audience. Keep notes on anything that changes during the test.

Returning visitors deserve a mention. A test that measures sign-ups over a short window will undercount visitors who need several visits to decide. If your sales cycle is long, define the goal at a point you can observe within the window, such as a demo request, and be explicit that it is a proxy.

What tools can and cannot enforce

A tool can hide results until a minimum is met, and that helps. BusinessMCP’s session replay and heatmap product feeds the same platform that runs its tests, so a team can check whether traffic changed mid-test by looking at the same data. You still own the plan. The vendor’s 14-day floor is a safeguard, not a calculation, and it does not know your weekly traffic or the lift you care about. Our overview of the best AB testing tools looks at which tools help with planning and which leave it to you.

For the vendor’s side of the duration rules, including the 300-visitor and 25-conversion gate per arm, see the notes in BusinessMCP’s AB testing tools comparison, which is a vendor document and should be read as such.