Test Strategy & Planning
Automation without strategy is just expensive scripting. Learn how to decide what to automate, when to automate it, and how to measure whether it's working.
1 The Hook — Why This Matters
A large NZ government agency spent $1.2M over two years automating 2,000 test cases. At the end, their regression cycle had shrunk from five days to four and a half. The problem? They automated everything indiscriminately: one-off data migration scripts, exploratory test charters, and UI tests for features that changed every sprint. The maintenance burden consumed 70% of the automation team's capacity. The ROI was negative for eighteen months straight.
Strategy is the multiplier. A well-chosen 200-test suite can deliver more value than a poorly-chosen 2,000-test suite. The test lead's job is making that choice.
Senior engineer insight
The most valuable thing I ever did on an automation strategy was map every proposed test against a calendar of planned changes — not just current stability. At Harbour Bank NZ we had a solid ROI case for automating the payment confirmation flow, but the UI was scheduled for a complete redesign in Q2. Automating it in Q1 would have cost three sprints of rework four months later. Pyramid ratios matter less than timing your automation to land after features stabilise.
Most common mistake: teams build the strategy document once at project kickoff and never revisit it — the automation portfolio drifts out of alignment with the actual risk profile as the product evolves.
2 The Rule — The One-Sentence Version
Automate what repeats, stabilises, and matters. Manual test what explores, judges, and changes.
The automation pyramid is not a suggestion; it's a cost model. Every E2E test you write costs 10-50x more to maintain than a unit test. If your pyramid is upside down, your budget is upside down too.
From the field
A government agency running a Revenue NZ tax-filing integration assumed their biggest automation ROI was in the E2E happy path — submit return, receive confirmation. They automated 80 UI scenarios across six browser/device combinations before anyone ran the numbers on execution time. The suite took 4.5 hours to complete in CI, which meant it ran overnight, not on every PR, which meant bugs still reached staging.
When we rewrote the strategy — shifting 60% of that coverage to API-layer contract tests — the same confidence came back in 22 minutes. The lesson that generalises: measure CI execution time as a first-class metric in your strategy, not an afterthought. A test suite nobody waits for is a test suite nobody trusts.
3 The Analogy — Think Of It Like...
Hiring a full-time chef to make toast every morning.
Toast is repetitive, predictable, and low-skill. A toaster (unit test) handles it perfectly. A chef (manual tester) is wasted on toast but essential for a tasting menu (exploratory testing). Automation strategy is about matching the right tool to the right task. Hire chefs for creativity. Buy toasters for repetition.
4 Watch Me Do It — Step by Step
Here is the test lead's framework for automation strategy.
- Apply the automation pyramid
Layer Ratio Purpose Execution Time Unit Tests 70% Validate individual functions Milliseconds Integration/API 20% Verify service contracts Seconds E2E/UI 10% Validate critical user journeys Minutes The "Ice Cream Cone" anti-pattern (heavy E2E, sparse unit tests) creates slow feedback, high maintenance, and fragile pipelines. The 2025 DORA Report found teams with robust integration test coverage dominated top-tier performance metrics.
- Use the ROI decision matrix
Factor Automate if... Frequency >3 executions per release cycle Stability Feature is mature (not in active redesign) Execution Time Manual takes >30 min; automation <5 min Risk High business impact if flow breaks - Estimate effort realistically
Mature teams average 3-5 medium-complexity automation cases per sprint per engineer. Use three-point estimation: (Optimistic + 4 x Most Likely + Pessimistic) / 6. TMMi Level 3+ organisations report 60% improvement in test estimation accuracy.
5 When to Use It / When NOT to Use It
✅ Automate first...
- Regression suites (high repetition)
- Smoke/sanity checks (run per build)
- Data-validation heavy scenarios
- Cross-browser/device matrix tests
❌ Keep manual...
- Exploratory testing (cognitive, creative)
- Usability/UX assessment
- One-off edge cases
- Tests requiring human judgment
6 Common Mistakes — Don't Do This
🚫 Automating everything
I used to think: More automated tests equals better quality.
Actually: Every test has a maintenance cost. Automating volatile features or one-off scenarios creates debt that drags the team down. Apply the ROI matrix ruthlessly. Saying "no" to low-value automation is a strategic skill.
🚫 Ignoring the pyramid
I used to think: E2E tests give the most confidence because they test like a user.
Actually: E2E tests are expensive, slow, and brittle. A single API test can validate the same business logic in 1/100th the time. The pyramid exists because economics matters. Heavy E2E investment is the Ice Cream Cone anti-pattern.
🚫 No environment parity strategy
I used to think: Dev, test, and staging are "close enough" to production.
Actually: Tests that pass in staging and fail in production are worse than no tests at all because they create false confidence. Define environment parity tiers: local (synthetic), CI (mocked), integration (shared DB), staging (masked production subset), production (smoke + synthetic).
7 Now You Try — Interview Warm-Up
Scenario: Your CEO wants 100% of tests automated by end of year. The team has 500 manual tests. You estimate automation capacity at 4 tests per sprint per engineer (3 engineers). There are 8 sprints left.
How do you respond?
Your response:
"We can automate approximately 96 tests this year (3 engineers x 4 tests x 8 sprints). I recommend we prioritise using the ROI matrix: automate the 96 tests with highest repetition, stability, and business risk. The remaining 404 tests should be assessed quarterly. Some will never justify automation cost. 100% automation is not a goal; optimal automation is." Then present the risk-prioritised backlog with estimated ROI per test.
Why teams fail here
- Scope agreed verbally, never written down — the test lead and delivery manager have different mental models of what "automation coverage" means; when the sprint ends with 30% of the target unbuilt, both sides are surprised and blame each other.
- Coverage targets set as percentages without defining the denominator — "80% automated" is meaningless until you agree whether that's 80% of test cases, 80% of critical paths, or 80% of lines of code; NZ health sector projects have stalled at sign-off because auditors asked this question and nobody could answer it.
- Tooling decided before the test pyramid is shaped — choosing Playwright or Cypress before knowing your unit/integration ratio locks you into a UI-first mindset and makes the Ice Cream Cone anti-pattern almost inevitable; pick the framework after the layer proportions are agreed.
- Phased rollout without a maintenance budget line — automation is not a capital project that ends at go-live; TransitNZ and several NZ councils have decommissioned automation suites not because they failed but because no one budgeted ongoing engineering time to keep them green as the product changed.
Key takeaway
A test strategy is a resource allocation decision disguised as a technical document — if it doesn't say what you're choosing not to automate and why, it's incomplete.
8 Self-Check — Can You Actually Do This?
Click each question to reveal the answer. If you got all three, you're ready to practice.
Q1. What is the "Ice Cream Cone" anti-pattern?
Heavy E2E/UI tests at the top with few unit tests at the bottom. It creates slow feedback, high maintenance costs, and fragile pipelines. The correct shape is the pyramid: 70% unit, 20% integration, 10% E2E.
Q2. What four factors should you consider in the ROI decision matrix?
Frequency (how often it runs), Stability (is the feature mature), Execution Time (manual vs. automated), and Risk (business impact if it breaks).
Q3. What types of testing should remain manual even in a mature automation program?
Exploratory testing, usability/UX assessment, one-off edge cases, and anything requiring human judgment (aesthetics, trust, subjective quality).