Pairwise Testing

What it is

Pairwise testing (also called all-pairs testing) is a combinatorial test design technique. Rather than testing every possible combination of input values, you select a minimal set of test cases that covers every pair of values at least once. The insight is that most defects are caused by interactions between one or two variables — not by complex interactions among many variables simultaneously.

This is distinct from equivalence partitioning or boundary value analysis, which test individual inputs in isolation. Pairwise is for when you have multiple inputs that could interact with each other, and you need to balance coverage against the cost of running many tests.

The problem it solves

Imagine testing a checkout form with four configuration parameters, each with four possible values:

• Payment method: Credit card, POLi, Afterpay, Gift card
• Delivery type: Standard, Express, Same-day, Click & Collect
• Customer type: Guest, Registered, VIP, Business
• Discount code: None, 10% off, Free shipping, Bundle deal

Full combinatorial coverage: 4 × 4 × 4 × 4 = 256 test cases. Running 256 test cases for a single feature is rarely realistic in a sprint.

Pairwise testing covers all two-way interactions (every pair of values tested together at least once) in approximately 20–25 test cases. That is the same defect-catching power for combinations, at less than 10% of the testing cost.

When to use it

• Testing systems with many configuration options where combinations matter (software settings, checkout options, filter combinations)
• Form fields with multiple valid values that interact (discount code + payment method + delivery type)
• Feature flags — when multiple flags can be on or off, pairwise covers the interactions without testing every permutation
• Compatibility matrices — OS × browser × screen resolution
• Regression suites where scope needs to be cut without eliminating coverage of interactions

NZ examples where pairwise applies

• NZ e-commerce checkout: payment method (credit card / POLi / Afterpay) × delivery type (NZ Post standard / CourierPost / Click & Collect) × discount code (none / percentage / free shipping)
• NZ banking: account type (everyday / savings / term deposit) × transaction type (transfer / bill payment / international) × amount range (small / medium / large)
• Software configuration screen: browser (Chrome / Firefox / Safari / Edge) × OS (Windows / macOS / iOS / Android) × locale (en-NZ / mi-NZ)

Worked NZ example

A NZ online booking system has three parameters:

• Payment method: Credit card, POLi, Afterpay
• Delivery type: Standard (3–5 days), Express (next day), Click & Collect
• Location: Auckland, Wellington, Christchurch

Full combinations: 3 × 3 × 3 = 27 test cases

Pairwise covers all pairs in 9 test cases:

Every pair is covered at least once across these 9 tests. If Afterpay + Click & Collect has a bug (a known issue with some NZ payment providers and in-store pickup), test #9 will catch it. If Express delivery to Christchurch has a routing issue, test #5 will catch it.

Why it works

Research by NIST (National Institute of Standards and Technology) analysed hundreds of software defects and found that:

• ~70% of defects are caused by a single parameter (covered by equivalence partitioning)
• ~90% of defects are caused by interactions between at most 2 parameters (covered by pairwise testing)
• ~98% of defects are caused by interactions between at most 3 parameters (covered by 3-way combinatorial testing)

This means pairwise testing — covering all 2-way interactions — catches approximately 90% of the defects that full combinatorial testing would catch, at a fraction of the test count. The remaining ~10% require 3-way or higher-order coverage and are typically reserved for safety-critical systems.

Tools

• PICT (Pairwise Independent Combinatorial Testing, Microsoft, free CLI tool) — the most widely used pairwise generator. You define your parameters and values in a text file; PICT outputs the minimum test set. Runs on Windows, macOS, and Linux.
• AllPairs (free, Python-based) — similar to PICT; useful in Python-heavy automation environments
• Hexawise (commercial, web-based) — more user-friendly interface; produces pairwise and higher-order combinatorial sets; good for larger parameter sets

Using PICT: a quick example

Create a .txt file with your parameters and values:

Run pict booking.txt and PICT outputs the 9-row test set above. For 10 parameters with 5 values each (full combinations: 5^10 = ~10 million), PICT generates approximately 35 test cases.

When not to use it

• Safety-critical systems — medical devices, aviation, nuclear control systems may require full combinatorial testing or even higher-order coverage. Pairwise is not sufficient when all combinations must be verified for safety.
• Known 3-way (or higher) interactions — if you already know that three specific parameters interact in a way that causes defects, use 3-way combinatorial testing (or targeted decision table testing) rather than pairwise.
• Very few parameters or values — if you have only 2 parameters with 3 values each, full combinatorial is only 9 tests. There’s no need for pairwise reduction.

Bugs pairwise testing typically catches

• A discount code that fails only when combined with Afterpay as the payment method (not with credit card or POLi)
• Click & Collect orders that fail for Wellington customers but not Auckland or Christchurch
• A form that only breaks when a long company name is entered and the city is Christchurch (due to a character limit in the Christchurch address database lookup)
• Express delivery unavailable for certain NZ regions, but only when a specific discount code is applied (the two interact in the pricing engine)
• A configuration screen where two specific feature flags enabled simultaneously cause a conflict, even though each flag works correctly in isolation

ISTQB mapping

The ISTQB Foundation exam expects you to apply pairwise testing (K3). You must be able to identify parameters and values from a specification, understand what a pairwise table covers, and know when the technique is appropriate.

Tips

• Identify parameters carefully — the quality of your pairwise set depends entirely on correctly identifying the parameters that can interact. Think about which inputs go through the same code path together.
• Add a few targeted tests beyond pairwise — if you have business knowledge that suggests a particular 3-way interaction is risky, add a specific test for it. Pairwise is a baseline, not a ceiling.
• Use pairwise for compatibility matrices — if you need to test across browsers, operating systems, and screen sizes, pairwise dramatically reduces the matrix size without abandoning cross-environment coverage.
• Combine with risk-based prioritisation — not all pairs are equally risky. Use business knowledge to flag which parameter combinations are highest-risk and verify those are included in your generated set.

Parameters (3, each with 3 values):
- Store brand: New World, PAK'nSAVE, Four Square
- Collection slot: next hour, same day, next day
- Payment: credit card, account, gift card

Full combinatorial: 3 x 3 x 3 = 27 test cases.

Pairwise: about 9 cases. With three 3-value parameters, every pair of values can be covered in 9 well-chosen rows (the same shape as a 3x3 Latin-square layout).

Why it is so much smaller: pairwise only guarantees that every PAIR of values appears together once, not every full triple. Each test case covers three pairs at once (brand+slot, brand+payment, slot+payment), so a handful of cases mops up all the pairs. Full coverage forces every three-way triple to appear, which multiplies out fast.

Missing pairs in the drafted set:
- mobile + SMS code (never together)
- mobile + Chrome (never together)
- SMS code + Safari (never together)

The draft only ever puts mobile with the authenticator app and Safari, so several mobile and SMS pairs go untested.

A clean completing addition:
4. mobile + SMS code + Chrome  (covers mobile+SMS, mobile+Chrome, SMS+Chrome already seen but fine)
5. desktop + SMS code + Safari (covers SMS+Safari, desktop+Safari)

After adding rows 4 and 5, every pair across the three parameters appears at least once. The lesson: a set is not "pairwise" just because it has a few rows — you must audit it against the full list of required pairs.

Three 3-value parameters cover all pairs in 9 cases. One valid set:

1. adult    — $5   — EFTPOS
2. adult    — $20  — credit card
3. adult    — $50  — cash
4. child    — $5   — credit card
5. child    — $20  — cash
6. child    — $50  — EFTPOS
7. tertiary — $5   — cash
8. tertiary — $20  — EFTPOS
9. tertiary — $50  — credit card

Every card-type+amount pair, card-type+payment pair, and amount+payment pair appears at least once. (A solver can sometimes squeeze 3x3x3 below 9, but 9 is the safe, easy-to-build answer.)

Tool for larger sets: PICT (Microsoft's free CLI) or AllPairs. Never hand-build beyond 3-4 parameters — humans miss pairs, exactly the mistake in Exercise 2.

Self-Check