Why does my test need so many visitors?

Because the sample needed scales with the inverse square of the effect: detecting a difference half as large takes about four times as many people. Most surprise comes from asking for a small lift on a low baseline, which is the most expensive combination there is.

What is power, and why 80%?

Power is the probability of detecting an effect of the size you specified, if it is really there. At 80% you would miss one real effect in five. It is conventional rather than principled — 90% is a defensible choice and costs about 30% more traffic.

Can I stop early if the variant is clearly winning?

Not without invalidating the result. 'Clearly winning' early is what random variation looks like early, and a test checked repeatedly will cross the threshold by chance sooner or later. Sequential testing methods exist that permit early stopping, but they use different arithmetic from this and you have to choose them before you start.

Does hitting the sample size prove the variant is better?

No. It means a difference of the size you specified would have been detectable, and that any difference you do observe clears the bar you set. It says nothing about whether the test was well designed, whether the variant caused the difference, or whether it will hold next month.

A/B Test Sample Size Calculator

Every A/B test is an argument with random variation. Two versions of a page will always convert at slightly different rates, for the same reason two hundred coin flips rarely split exactly a hundred each — and the only question that matters is how big a difference has to be, over how many visitors, before it is evidence rather than noise. Work that out before you start and you have a stopping rule. Skip it and you have a test that will eventually look like it is winning, because given enough peeks any test does.

Current conversion rate (%)

What the control converts at today.

Smallest effect worth detecting (%)

Below this you would not act on the result anyway.

Read that effect as

Confidence (%)

95 is the usual default. Higher means a stricter bar and more traffic.

Power (%)

The chance of spotting a real effect of that size. 80 is conventional.

Test direction

Traffic to the variant (%)

50 is an even split, and the cheapest.

Visitors a day (optional)

Add this to turn the sample size into a run time.

The sample size will appear here.

What this generator does

Turns a baseline rate and a minimum detectable effect into the number of visitors each variant needs. The coin run calculator on this site answers the same underlying question for coin flips — how much apparent difference random variation produces on its own — and this asks it commercially. It also converts the answer into days from your traffic, which is deliberately a field here rather than a second page: run time has exactly one input the sample size does not, so splitting it would make two pages out of one calculation.

How to use this tool

Enter what your control converts at today.
Enter the smallest improvement that would actually change what you do.
Set confidence and power, or leave the conventional 95 and 80.
Add your daily visitors to turn the sample size into a run time.

Understanding the controls

Current conversion rate: What the control does now. Lower baselines need dramatically more traffic — halving the rate roughly doubles the sample — which is why tests on rare events are so much harder than they look.
Smallest effect worth detecting: The number people get wrong most often. This is not what you hope to see, it is the smallest improvement that would change your decision. Asking to detect a tiny effect is what makes a test unrunnably long, and the honest move is usually to raise this rather than to run for a year.
Relative or absolute: A 10% relative lift takes 5% to 5.5%. Ten absolute points takes 5% to 15%. These are wildly different tests and confusing them is the single most common error in test planning, so the tool asks rather than guessing.
Confidence and power: Confidence is how rarely you would call a difference real when it is not; power is how often you would spot a real one of the size you specified. 95 and 80 are conventional. Raising either raises the sample, and power is the one people forget — an underpowered test that finds nothing has told you nothing.
Test direction: Two-sided asks whether the variant differs; one-sided asks only whether it is better and needs fewer people. Choose one-sided only if you would genuinely ship the control regardless of how much worse the variant turned out, which is rarer than it sounds.
Traffic to the variant: An even split needs the fewest people in total. Skewing traffic is sometimes worth it for other reasons, and the tool shows what it costs.

Worked examples

5% baseline, 10% relative lift: About 31,000 per variant at 95% confidence and 80% power — far more than most people expect.
20% baseline, 5 absolute points: Roughly 1,100 per variant. Bigger effects on bigger baselines are cheap.
Halving the effect: Roughly quadruples the sample. The relationship is inverse-square, which is why small effects are so expensive.
What is never claimed: That reaching the number makes the result correct.

Common use cases

Deciding whether a test is worth running before building the variant
Working out how long a landing-page test will take at current traffic
Explaining to a stakeholder why a test cannot be called after two days
Setting a stopping rule in advance so results are not read early
Checking whether a small expected lift is detectable at all with the traffic available

How this generator works

The standard two-proportion comparison, in the pooled-variance form published sample-size calculators use: the required per-variant sample is the square of [z(1−α/2)·√(2p̄q̄) + z(1−β)·√(p₁q₁ + p₂q₂)] divided by the square of the difference in rates. The two z values are the normal quantiles for your confidence and power, computed with a rational approximation refined by a Halley step so they match published critical values to eight decimal places — the whole answer is a function of those two numbers, so an error there is an error everywhere. An uneven split is handled by the standard adjustment. This is a normal approximation to the binomial, which is what essentially every calculator uses and which is reliable where tests actually run; when the design would leave only a handful of conversions in a cell, the page says the approximation is shaky rather than printing a confident number.

Randomness and fairness

Nothing here is random. This is arithmetic on your inputs, and the same inputs always give the same answer — which matters for a figure you are going to write into a test plan. The subject is randomness, but the calculation is deterministic.

For how randomness is produced across the whole site, see how Generate Random works.

Assumptions this tool makes

Conversions are independent Bernoulli trials with a stable rate over the test period.
Assignment to variants is genuinely random and does not change during the test.
The result will be analysed once, after the sample size is reached.

Limitations and good to know

It sizes a test comparing two conversion rates. Revenue per visitor, time on page, and any continuous measure need a different calculation entirely.
It assumes one test, analysed once, at the end. Peeking at results as they come in invalidates the significance level, and testing several variants at once needs a correction this does not apply.
It assumes visitors are independent and randomly assigned. Returning visitors, shared devices and traffic-source shifts all break that, and none of them is visible here.
The normal approximation degrades when expected conversions per cell are very small. The page flags that case rather than hiding it.
This is not for clinical, medical or safety-related experiments. Those have their own standards and their own statisticians, and nothing here is a substitute.

Common mistakes

Stopping as soon as the result looks significant: This is the error the whole calculation exists to prevent. Checking repeatedly and stopping at the first significant reading will produce a significant reading eventually, whether or not anything is different. Fix the sample size first, then look once.
Entering the lift you hope for as the minimum detectable effect: Enter the smallest lift that would change your decision. Those are usually very different numbers, and using the optimistic one produces a test too small to detect what you will actually get.
Confusing relative and absolute effects: Going from 5% to 5.5% is a 10% relative lift and half an absolute point. Entering 10 in the absolute box asks for 5% to 15%, a completely different and far cheaper test to detect. The tool makes you choose for exactly this reason.
Running for a fixed number of days instead of a fixed sample: Days are a proxy for visitors, and traffic varies. Use the day estimate for planning, but stop on the sample size — and run whole weeks, because weekday and weekend visitors rarely behave alike.

Practical tips

If the answer is more traffic than you will ever get, the useful response is to test a bigger change rather than to run longer.
Round up to whole weeks. Partial weeks over-weight whichever days they include.
Write the sample size down before the test starts. A stopping rule decided afterwards is not a stopping rule.

Troubleshooting

The number is enormous: Almost always a low baseline, a small minimum detectable effect, or both. Try raising the effect to the smallest lift you would actually act on — it is often much larger than the one first entered, and the sample falls with its square.
It refuses my inputs: The target rate has to stay between 0 and 100%: a large absolute effect on a high baseline pushes it past 100, which is not a test design. Confidence must be under 100% and power above 50%.

Privacy and your data

Your conversion rates and traffic figures never leave your browser. Nothing is uploaded, nothing is stored, and nothing is written into the page address — which matters because a baseline rate and a daily visitor count together say quite a lot about a business. Analytics records that the tool ran and the total sample it produced, never your inputs.

Frequently asked questions

Why does my test need so many visitors?: Because the sample needed scales with the inverse square of the effect: detecting a difference half as large takes about four times as many people. Most surprise comes from asking for a small lift on a low baseline, which is the most expensive combination there is.
What is power, and why 80%?: Power is the probability of detecting an effect of the size you specified, if it is really there. At 80% you would miss one real effect in five. It is conventional rather than principled — 90% is a defensible choice and costs about 30% more traffic.
Can I stop early if the variant is clearly winning?: Not without invalidating the result. 'Clearly winning' early is what random variation looks like early, and a test checked repeatedly will cross the threshold by chance sooner or later. Sequential testing methods exist that permit early stopping, but they use different arithmetic from this and you have to choose them before you start.
Does hitting the sample size prove the variant is better?: No. It means a difference of the size you specified would have been detectable, and that any difference you do observe clears the bar you set. It says nothing about whether the test was well designed, whether the variant caused the difference, or whether it will hold next month.