Test Matrix Generator

What this generator does

Chooses which combinations to test, which is a different problem from inventing test data — the synthetic data generator makes up plausible names and addresses to put into a test, whereas this decides the shape of the test run itself. It matters because a full sweep of five parameters with four values each is 1,024 runs, and all-pairs coverage of the same input is usually around twenty.

How to use this tool

1. List each parameter on its own line as name: value, value, value.
2. Choose a strategy — pairwise is the usual answer.
3. Press Generate matrix to see the cases and the coverage.
4. Copy the CSV straight into a spreadsheet or a data-driven test.

Understanding the controls

Parameters

One per line as name: value, value. Repeated values within a parameter are removed and a repeated parameter name is skipped with a note, so a pasted list that has drifted does not silently produce a wrong matrix.

Strategy

Pairwise covers every two-way interaction and is what most teams want. Full factorial produces every combination and is usually far too many. Random sampling gives a quick smoke set with no coverage guarantee at all, which the coverage figure will show.

Cases to sample

Only used by the random strategy. Cases are distinct from one another, so asking for more than the full product exists will simply produce all of them and say so.

Seed

Pairwise ties are broken randomly, so two runs on the same input give different but equally valid matrices. A seed pins the result, which is what you want if the matrix is going to be committed to a repository and diffed.

Worked examples

4 browsers, 3 operating systems, 3 plans, 4 locales

144 combinations in full. Pairwise typically covers all 118 pairs in about 16 cases — roughly a 90% reduction.

Five yes/no flags

32 combinations in full; pairwise covers every two-way interaction in about 6 cases.

Random sample of 8 from the same input

Eight distinct cases, typically covering somewhere around two-thirds of the pairs — the figure shown tells you exactly.

Common use cases

• Deciding which browser, operating system and locale combinations to actually run
• Building a data-driven test table without hand-writing every row
• Reducing an overnight test suite that has grown to thousands of combinations
• Planning manual QA passes across plan tiers, roles and feature flags
• Documenting which configurations a release was verified against

How this generator works

Pairwise generation is greedy: it repeatedly builds a case by choosing, parameter by parameter, the value covering the most pairs not yet seen, then retires every pair that case covers and starts another. It stops when no uncovered pairs remain. Coverage is then measured by counting the distinct two-way value combinations present in the emitted cases against the total that exist, so the percentage shown is computed from the output rather than assumed from the algorithm. Greedy search is not guaranteed to find the smallest possible set — that problem is NP-hard — so the page reports a lower bound alongside the count it achieved.

Randomness and fairness

Randomness enters in exactly one place in pairwise mode: breaking ties when several values would cover the same number of new pairs. That is why the same input can produce different matrices of the same quality, and why a seed is offered for anyone committing the output to source control. A seeded run uses a deterministic, non-cryptographic generator; nothing here is security-sensitive, so reproducibility is the only property that matters.

For how randomness is produced across the whole site, see how Generate Random works.

Limitations and good to know

• Pairwise catches faults triggered by one value or by an interaction between two. A defect that needs three specific values together can be missed, and no amount of two-way coverage will find it.
• There is no way to express constraints such as Safari never running on Windows, so a matrix may contain combinations that cannot exist and will need pruning by hand.
• Parameters are treated as unordered labels, so the tool has no concept of boundary values or of one value being riskier than another. Weighting a known-fragile configuration is a manual step.
• Generation stops at 500 cases, which only affects full factorial on large inputs — the warning says so when it happens rather than silently truncating.
• The output is a list of configurations, not tests. What to assert in each one is still yours to write.

Common mistakes

Treating pairwise coverage as full coverage

It covers every two-way interaction and nothing stronger. Keep a small set of hand-written scenarios for the three-way cases you already know are risky.

Including impossible combinations and then investigating the failures

Prune combinations that cannot occur before running the suite, or split the matrix into two runs so each one only contains valid configurations.

Regenerating the matrix on every CI run

Set a seed and commit the output. A matrix that changes every run makes failures hard to reproduce and diffs meaningless.

Adding a parameter with many values without checking the count

Watch the pair count shown above the button. Widening one parameter raises the floor for every strategy, pairwise included.

Practical tips

• Put your most-valued parameters first — the greedy algorithm fixes earlier parameters first, so the ones at the top get slightly better spread.
• Keep the parameter list in a file next to the tests and paste it in when it changes, so the matrix and its inputs stay together.
• Compare the pairwise count against the lower bound shown: if they are close, there is nothing to gain from a smarter tool.

Privacy and your data

Parameter names and values are read and combined entirely in your browser — nothing is sent to a server, which matters here because internal feature-flag names and plan tiers are often the sort of thing an organisation would rather not paste into a remote service. Nothing is stored between visits and nothing appears in the address bar; analytics records only how many cases were produced.

Frequently asked questions

What is pairwise or all-pairs testing?

A technique that ensures every combination of any two parameter values appears in at least one test case, on the basis that most configuration defects involve one value or an interaction between two.

How much does it reduce the test count?

Typically by an order of magnitude. Four parameters with four values each is 256 combinations in full and usually under 25 for full pair coverage.

Is the generated matrix the smallest possible?

Not necessarily. Finding the true minimum is NP-hard, so the tool uses a greedy search and shows a lower bound so you can judge how close it got.

Can I exclude invalid combinations?

Not within the tool. Generate the matrix and remove impossible rows, or split the parameters into separate valid runs.