A/B Test Significance Calculator
An A/B test comes down to one question: how surprising would this difference be if the two variants were really the same? That is what a p-value answers, and it answers nothing else — not how big the difference is, not whether it will last, and not whether the variants are equivalent when it comes out high. This runs the standard two-proportion test and says all of that plainly.
What this generator does
Computes both conversion rates, the pooled proportion, the z statistic and the two-sided p-value. Every figure is recomputed from the counts before display, and the significance verdict is checked against the p-value.
How to use this tool
- Enter visitors and conversions for each variant.
- Read the two rates and the difference between them.
- Read the p-value and the verdict at the usual 5% threshold.
- Note what a high p-value does not mean.
Understanding the controls
- Visitors and conversions per variant
- The raw counts. Rates entered as percentages lose the sample size, which is what actually decides significance.
- Both variants
- The test is symmetric, so which one you call A makes no difference to the p-value.
Common use cases
- Checking whether an A/B test result is statistically significant
- Comparing two conversion rates on equal terms
- Teaching what a p-value actually answers
- Working out whether a difference is worth acting on
- Checking a test tool's figure against an independent calculation
How this generator works
Under the assumption that both variants convert at the same rate, the best estimate of that rate is the pooled proportion across both. The standard error of the difference follows from it, and the z statistic is the observed difference divided by that error. The two-sided p-value is the chance of a difference at least this large in either direction. Before display the rates, the statistic and the verdict are all recomputed from the counts.
Randomness and fairness
Nothing here is random. Four counts give exactly one p-value.
For how randomness is produced across the whole site, see how Generate Random works.
Limitations and good to know
- A p-value above the threshold is not evidence that the variants are the same — it means this test could not tell them apart, which the power calculator addresses.
- Checking a test repeatedly and stopping when it crosses the threshold invalidates it; the sample size should be fixed in advance.
- The normal approximation is poor when conversions are very few, roughly under five per variant.
- Statistical significance says nothing about whether a difference is large enough to be worth having.
Privacy and your data
The calculation runs in your browser. The counts you enter are never transmitted, stored or included in analytics.
Related generators
- Statistical Power CalculatorWorks out the chance a test would detect a real effect of a given size, and how many people it would take to reach the usual eighty per cent.
- Confidence Interval CalculatorTurn a count out of a sample into an interval, with a warning when the sample is too small for the usual approximation to be trusted.
- Simpson's Paradox GeneratorBuilds a table where one side wins every subgroup and loses overall, with the reversal verified group by group.
- Survey Sample Size CalculatorHow many people you need to ask for a given margin of error, including the finite population correction most calculators leave out.