Permutation Test Generator
A permutation test asks a question you can explain in one sentence: if the group labels meant nothing, how often would shuffling them produce a gap at least this big? At small sizes there is no need to sample or approximate — every rearrangement can be tried, and the p-value that comes out is exact rather than asymptotic. This page enumerates all of them whenever the groups are small enough, and says plainly when they are not.
What this generator does
Draws two groups, one optionally shifted by a true difference, and computes the observed gap in means. It then enumerates every way of splitting the pooled values into groups of the same two sizes, counts how many produce a gap at least as extreme in either direction, and reports that proportion as the p-value.
How to use this tool
- Choose how many values in each group and how big a true difference to build in.
- Look at the two groups and judge whether the gap looks convincing.
- Generate, and read the p-value along with how many rearrangements were tried.
- Set the true difference to zero and repeat — roughly one run in twenty will look significant.
Understanding the controls
- Rows per group
- Between 3 and 10 each. Up to six each is enumerated exhaustively; above that the page samples instead and says so.
- True difference
- How far the second group's mean is shifted, as a percentage of a standard deviation. Zero builds two groups from the same distribution.
- Seed
- Any word reproduces the same two groups and the same p-value.
Common use cases
- Teaching resampling inference without any distributional assumption
- Showing where a p-value actually comes from, by enumeration
- Demonstrating false positives on data built with no real effect
- Small-sample comparisons where a t-test's assumptions are doubtful
- Setting a reproducible exercise from a seed
How this generator works
Every combination of positions for the first group is generated in turn, the difference in means recomputed for that split, and the count kept of those at least as extreme as the observed one. The observed arrangement is itself one of the permutations and is deliberately counted, which is why the p-value can never be zero. Above twelve values in total the enumeration is replaced by twenty thousand random rearrangements, and the page states which of the two it used.
Randomness and fairness
The two groups are random; the p-value is computed by exhaustive enumeration where possible. A seed reproduces the data exactly and is therefore explicitly not cryptographically secure. Without a seed the browser's cryptographically secure generator is used.
For how randomness is produced across the whole site, see how Generate Random works.
Limitations and good to know
- Difference in means is the only statistic offered; medians or trimmed means would give a different test.
- Two-sided only, using the absolute gap.
- Above twelve total values the p-value is sampled rather than exact.
- This tests the labels, not the sampling — it says nothing about whether the data represents anything wider.
- Nothing is kept, so seed a sample you intend to analyse again.
Privacy and your data
The groups and the enumeration run entirely in your browser. Nothing about the data or your seed is transmitted or kept.
Related generators
- ANOVA Dataset GeneratorGrouped data with a known separation, and the full ANOVA table whose sums of squares are checked to split exactly.
- Bootstrap Sample GeneratorResamples your own numbers with replacement to show how much the average would move if you had collected a different sample.
- Statistical Power CalculatorWorks out the chance a test would detect a real effect of a given size, and how many people it would take to reach the usual eighty per cent.
- A/B Test Sample Size CalculatorHow many visitors an A/B test needs before a difference of the size you care about means anything.