Skip to content

A/B Test Significance Calculator

Type how many visitors saw each version and how many converted to see whether B…

Type how many visitors saw each version and how many converted to see whether B really beats A, how big the lift could be, and how long to run the test.

A (control)

B (variant)

4% converted

5% converted

Hypothesis
More options95% confidence
Confidence level

Significance at α = 1 − the confidence level (0.05).

ExampleLift, B vs A

+25%

B converts at 5%, A at 4%. B beats A, and a gap this big would be unlikely by chance (p = 0.0159). The 95% confidence interval for the lift runs from +4.2% to +49.9%.

Type your own visitors and conversions.

2.9%3.7%4.6%5.4%6.2%A 4%B 5%Lift of B over A, 95% interval+4.2%+49.9%

Each curve shows how far that version’s rate could wobble from one sample of visitors to the next; the more they overlap, the less the gap means. The bar is the lift’s confidence interval.

p-value (two-sided)
0.0159
Difference, 95% CI
+1 pts [0.19, 1.81]
z
2.412

Checking the result every day and stopping the moment p dips under 0.05 makes false wins far more likely. Fix the sample size first (below), then look once.

How many visitors a test needs

Set the smallest lift worth finding before you start, run until each version has this many visitors, and judge it once at the end.

%
%

39,475 visitors in each version (78,950 in all) to detect +10% from a 4% baseline with 80% power at α = 0.05 (two-sided).

With 5,000 visitors in each version, this test could reliably detect a lift of about +29.3% or more.

For a survey’s sample size, see the sample size calculator.

How this works: 10 sources,

How this works

Method

An A/B test shows two versions of a page, email or button to random halves of your visitors and compares how many of each convert. The example: 200 of 5,000 converted on A (4.0%) and 250 of 5,000 on B (5.0%), a 25% relative lift. The question is whether that gap is real or the kind of wobble random traffic produces. Full method

How it’s tested

3 worked examples for this page are checked by automated tests before every release: given the inputs, the tool must show the expected answer.

Privacy

On your device. Nothing you enter or open leaves this browser. About privacy labels

Changes

  • First published: live curves with draggable cut-offs, and tests on pasted data (t-tests, ANOVA, chi-square, Fisher’s exact, correlation, regression, A/B), checked against SciPy.

Every change on What’s new

How we make toolsReport a mistake

How an A/B test is judged

An A/B test shows two versions of a page, email or button to random halves of your visitors and compares how many of each convert. The example: 200 of 5,000 converted on A (4.0%) and 250 of 5,000 on B (5.0%), a 25% relative lift. The question is whether that gap is real or the kind of wobble random traffic produces.

The two-proportion z-test pools both versions to estimate the shared rate if nothing had changed (4.5% here), works out the standard error of the difference from it, and divides: z = (p_B − p_A) ÷ √(p̂(1 − p̂)(1/n_A + 1/n_B)) = 2.41, a two-sided p-value of 0.016. The interval for the lift is computed on the log of the ratio p_B ÷ p_A, Katz’s method, because a ratio’s uncertainty is lopsided: here it runs from about +4% to +50%.

The biggest trap is peeking. A p-value is only valid if you decide the sample size first and look once. Checking every day and stopping as soon as p dips under 0.05 turns a 5% false-positive rate into much more: with five looks it is about 14%, with 20 looks about 25% (Pocock, 1977). Set the smallest lift worth finding, let the planner give the visitors needed, and wait for them.

z = (p_B − p_A) ÷ √(p̂(1 − p̂)(1/n_A + 1/n_B)) · n = (z₁₋α/₂√(2p̄q̄) + z₁₋β√(p₁q₁ + p₂q₂))² ÷ (p₂ − p₁)²

Visitors needed in each version

Visitors per version for a two-sided test at 5% with 80% power to detect the relative lift shown, by baseline conversion rate (Fleiss, Levin and Paik, without continuity correction).

Visitors per version for a two-sided test at 5% with 80% power to detect the relative lift shown, by baseline conversion rate (Fleiss, Levin and Paik, without continuity correction).
Baseline rateLift +5%Lift +10%Lift +20%Lift +50%
0.5%1,280,619327,92285,86215,599
1%637,010163,09542,6937,750
2%315,20680,68221,1093,826
3%207,93853,21113,9142,518
5%122,12431,2348,1581,471
10%57,76314,7513,841686
20%25,5836,5101,683294
30%14,8563,763963163
50%6,2751,56538858

Why peeking misleads (Pocock, 1977)

Times you checkChance of a false “significant” result
1 (decided in advance)5%
28.3%
310.7%
514.2%
1019.3%
2024.8%
5032.0%
10037.4%

Common questions

What lift should I plan for?

The smallest one that would change your decision. Detecting a 5% relative lift on a 2% baseline takes over 300,000 visitors per version; a 20% lift needs about 21,000. If you can’t get that traffic, test bolder changes.

Can I stop the test early if it is clearly winning?

Not with this test. Early leads are often noise, and stopping on a good day inflates false wins (see the table above). If you must monitor, use a sequential design with adjusted thresholds, planned before the test starts.

Is this the same as a chi-square test?

For a two-sided test, yes: the pooled z-test’s z² equals the chi-square statistic of the 2 × 2 table without continuity correction, and the p-values are identical.

Why is the lift interval not symmetric?

Because a ratio can’t go below −100% but can grow without limit. The interval is symmetric on the log scale, which makes it lopsided when turned back into a percentage.