A/B Test Significance Calculator
Type how many visitors saw each version and how many converted to see whether B…
Type how many visitors saw each version and how many converted to see whether B really beats A, how big the lift could be, and how long to run the test.
A (control)
B (variant)
4% converted
5% converted
More options95% confidence
Significance at α = 1 − the confidence level (0.05).
ExampleLift, B vs A
+25%
B converts at 5%, A at 4%. B beats A, and a gap this big would be unlikely by chance (p = 0.0159). The 95% confidence interval for the lift runs from +4.2% to +49.9%.
Type your own visitors and conversions.
Each curve shows how far that version’s rate could wobble from one sample of visitors to the next; the more they overlap, the less the gap means. The bar is the lift’s confidence interval.
- p-value (two-sided)
- 0.0159
- Difference, 95% CI
- +1 pts [0.19, 1.81]
- z
- 2.412
Checking the result every day and stopping the moment p dips under 0.05 makes false wins far more likely. Fix the sample size first (below), then look once.
How many visitors a test needs
Set the smallest lift worth finding before you start, run until each version has this many visitors, and judge it once at the end.
39,475 visitors in each version (78,950 in all) to detect +10% from a 4% baseline with 80% power at α = 0.05 (two-sided).
With 5,000 visitors in each version, this test could reliably detect a lift of about +29.3% or more.
For a survey’s sample size, see the sample size calculator.
How this works: 10 sources,
How this works
Method
An A/B test shows two versions of a page, email or button to random halves of your visitors and compares how many of each convert. The example: 200 of 5,000 converted on A (4.0%) and 250 of 5,000 on B (5.0%), a 25% relative lift. The question is whether that gap is real or the kind of wobble random traffic produces. Full method
Sources
- Pocock, Group sequential methods in the design and analysis of clinical trials (Biometrika, 1977)
- Armitage, McPherson and Rowe, Repeated significance tests on accumulating data (Journal of the Royal Statistical Society A, 1969)
- Katz, Baptista, Azen and Pike, Obtaining confidence intervals for the risk ratio in cohort studies (Biometrics, 1978)
- NIST/SEMATECH e-Handbook of Statistical Methods
- Abramowitz and Stegun, Handbook of Mathematical Functions, chapters 6, 7 and 26 (1964)
- SciPy 1.18 scipy.stats (reference values for the tests)
- statsmodels 0.15 (reference values for the A/B test and regression intervals)
- Welch, The generalization of Student’s problem when several different population variances are involved (Biometrika) (1947)
- Fleiss, Levin and Paik, Statistical Methods for Rates and Proportions, 3rd edition (sample size for two proportions) (2003)
- Hyndman and Fan, Sample quantiles in statistical packages (The American Statistician) (1996)
How it’s tested
3 worked examples for this page are checked by automated tests before every release: given the inputs, the tool must show the expected answer.
Privacy
On your device. Nothing you enter or open leaves this browser. About privacy labels
Changes
- First published: live curves with draggable cut-offs, and tests on pasted data (t-tests, ANOVA, chi-square, Fisher’s exact, correlation, regression, A/B), checked against SciPy.
How an A/B test is judged
An A/B test shows two versions of a page, email or button to random halves of your visitors and compares how many of each convert. The example: 200 of 5,000 converted on A (4.0%) and 250 of 5,000 on B (5.0%), a 25% relative lift. The question is whether that gap is real or the kind of wobble random traffic produces.
The two-proportion z-test pools both versions to estimate the shared rate if nothing had changed (4.5% here), works out the standard error of the difference from it, and divides: z = (p_B − p_A) ÷ √(p̂(1 − p̂)(1/n_A + 1/n_B)) = 2.41, a two-sided p-value of 0.016. The interval for the lift is computed on the log of the ratio p_B ÷ p_A, Katz’s method, because a ratio’s uncertainty is lopsided: here it runs from about +4% to +50%.
The biggest trap is peeking. A p-value is only valid if you decide the sample size first and look once. Checking every day and stopping as soon as p dips under 0.05 turns a 5% false-positive rate into much more: with five looks it is about 14%, with 20 looks about 25% (Pocock, 1977). Set the smallest lift worth finding, let the planner give the visitors needed, and wait for them.
z = (p_B − p_A) ÷ √(p̂(1 − p̂)(1/n_A + 1/n_B)) · n = (z₁₋α/₂√(2p̄q̄) + z₁₋β√(p₁q₁ + p₂q₂))² ÷ (p₂ − p₁)²
- Repeated significance tests at the nominal 5% level on accumulating data raise the overall chance of a false positive: to about 14% with 5 looks, 19% with 10 and 25% with 20. Source: Pocock, Group sequential methods in the design and analysis of clinical trials (Biometrika, 1977).
- Armitage, McPherson and Rowe showed in 1969 that testing after each new batch of data, with no correction, makes a false “significant” result almost certain if you keep going long enough. Source: Armitage, McPherson and Rowe, Repeated significance tests on accumulating data (Journal of the Royal Statistical Society A, 1969).
- The confidence interval for a ratio of two proportions is usually computed on the log scale, with standard error √(1/x₁ − 1/n₁ + 1/x₂ − 1/n₂). Source: Katz, Baptista, Azen and Pike, Obtaining confidence intervals for the risk ratio in cohort studies (Biometrics, 1978).
Visitors needed in each version
Visitors per version for a two-sided test at 5% with 80% power to detect the relative lift shown, by baseline conversion rate (Fleiss, Levin and Paik, without continuity correction).
| Baseline rate | Lift +5% | Lift +10% | Lift +20% | Lift +50% |
|---|---|---|---|---|
| 0.5% | 1,280,619 | 327,922 | 85,862 | 15,599 |
| 1% | 637,010 | 163,095 | 42,693 | 7,750 |
| 2% | 315,206 | 80,682 | 21,109 | 3,826 |
| 3% | 207,938 | 53,211 | 13,914 | 2,518 |
| 5% | 122,124 | 31,234 | 8,158 | 1,471 |
| 10% | 57,763 | 14,751 | 3,841 | 686 |
| 20% | 25,583 | 6,510 | 1,683 | 294 |
| 30% | 14,856 | 3,763 | 963 | 163 |
| 50% | 6,275 | 1,565 | 388 | 58 |
Why peeking misleads (Pocock, 1977)
| Times you check | Chance of a false “significant” result |
|---|---|
| 1 (decided in advance) | 5% |
| 2 | 8.3% |
| 3 | 10.7% |
| 5 | 14.2% |
| 10 | 19.3% |
| 20 | 24.8% |
| 50 | 32.0% |
| 100 | 37.4% |
Common questions
What lift should I plan for?
The smallest one that would change your decision. Detecting a 5% relative lift on a 2% baseline takes over 300,000 visitors per version; a 20% lift needs about 21,000. If you can’t get that traffic, test bolder changes.
Can I stop the test early if it is clearly winning?
Not with this test. Early leads are often noise, and stopping on a good day inflates false wins (see the table above). If you must monitor, use a sequential design with adjusted thresholds, planned before the test starts.
Is this the same as a chi-square test?
For a two-sided test, yes: the pooled z-test’s z² equals the chi-square statistic of the 2 × 2 table without continuity correction, and the p-values are identical.
Why is the lift interval not symmetric?
Because a ratio can’t go below −100% but can grow without limit. The interval is symmetric on the log scale, which makes it lopsided when turned back into a percentage.