Test Strategy & Planning
Automation without strategy is just expensive scripting. Learn how to decide what to automate, when to automate it, and how to measure whether it's working.
1 The Hook — Why This Matters
A large NZ government agency spent $1.2M over two years automating 2,000 test cases. At the end, their regression cycle had shrunk from five days to four and a half. The problem? They automated everything indiscriminately: one-off data migration scripts, exploratory test charters, and UI tests for features that changed every sprint. The maintenance burden consumed 70% of the automation team's capacity. The ROI was negative for eighteen months straight.
Strategy is the multiplier. A well-chosen 200-test suite can deliver more value than a poorly-chosen 2,000-test suite. The test lead's job is making that choice.
2 The Rule — The One-Sentence Version
Automate what repeats, stabilises, and matters. Manual test what explores, judges, and changes.
The automation pyramid is not a suggestion; it's a cost model. Every E2E test you write costs 10-50x more to maintain than a unit test. If your pyramid is upside down, your budget is upside down too.
3 The Analogy — Think Of It Like...
Hiring a full-time chef to make toast every morning.
Toast is repetitive, predictable, and low-skill. A toaster (unit test) handles it perfectly. A chef (manual tester) is wasted on toast but essential for a tasting menu (exploratory testing). Automation strategy is about matching the right tool to the right task. Hire chefs for creativity. Buy toasters for repetition.
4 Watch Me Do It — Step by Step
Here is the test lead's framework for automation strategy.
- Apply the automation pyramid
Layer Ratio Purpose Execution Time Unit Tests 70% Validate individual functions Milliseconds Integration/API 20% Verify service contracts Seconds E2E/UI 10% Validate critical user journeys Minutes The "Ice Cream Cone" anti-pattern (heavy E2E, sparse unit tests) creates slow feedback, high maintenance, and fragile pipelines. The 2025 DORA Report found teams with robust integration test coverage dominated top-tier performance metrics.
- Use the ROI decision matrix
Factor Automate if... Frequency >3 executions per release cycle Stability Feature is mature (not in active redesign) Execution Time Manual takes >30 min; automation <5 min Risk High business impact if flow breaks - Estimate effort realistically
Mature teams average 3-5 medium-complexity automation cases per sprint per engineer. Use three-point estimation: (Optimistic + 4 x Most Likely + Pessimistic) / 6. TMMi Level 3+ organisations report 60% improvement in test estimation accuracy.
5 When to Use It / When NOT to Use It
✅ Automate first...
- Regression suites (high repetition)
- Smoke/sanity checks (run per build)
- Data-validation heavy scenarios
- Cross-browser/device matrix tests
❌ Keep manual...
- Exploratory testing (cognitive, creative)
- Usability/UX assessment
- One-off edge cases
- Tests requiring human judgment
6 Common Mistakes — Don't Do This
🚫 Automating everything
I used to think: More automated tests equals better quality.
Actually: Every test has a maintenance cost. Automating volatile features or one-off scenarios creates debt that drags the team down. Apply the ROI matrix ruthlessly. Saying "no" to low-value automation is a strategic skill.
🚫 Ignoring the pyramid
I used to think: E2E tests give the most confidence because they test like a user.
Actually: E2E tests are expensive, slow, and brittle. A single API test can validate the same business logic in 1/100th the time. The pyramid exists because economics matters. Heavy E2E investment is the Ice Cream Cone anti-pattern.
🚫 No environment parity strategy
I used to think: Dev, test, and staging are "close enough" to production.
Actually: Tests that pass in staging and fail in production are worse than no tests at all because they create false confidence. Define environment parity tiers: local (synthetic), CI (mocked), integration (shared DB), staging (masked production subset), production (smoke + synthetic).
7 Now You Try — Interview Warm-Up
Scenario: Your CEO wants 100% of tests automated by end of year. The team has 500 manual tests. You estimate automation capacity at 4 tests per sprint per engineer (3 engineers). There are 8 sprints left.
How do you respond?
Your response:
"We can automate approximately 96 tests this year (3 engineers x 4 tests x 8 sprints). I recommend we prioritise using the ROI matrix: automate the 96 tests with highest repetition, stability, and business risk. The remaining 404 tests should be assessed quarterly. Some will never justify automation cost. 100% automation is not a goal; optimal automation is." Then present the risk-prioritised backlog with estimated ROI per test.
8 Self-Check — Can You Actually Do This?
Click each question to reveal the answer. If you got all three, you're ready to practice.
Q1. What is the "Ice Cream Cone" anti-pattern?
Heavy E2E/UI tests at the top with few unit tests at the bottom. It creates slow feedback, high maintenance costs, and fragile pipelines. The correct shape is the pyramid: 70% unit, 20% integration, 10% E2E.
Q2. What four factors should you consider in the ROI decision matrix?
Frequency (how often it runs), Stability (is the feature mature), Execution Time (manual vs. automated), and Risk (business impact if it breaks).
Q3. What types of testing should remain manual even in a mature automation program?
Exploratory testing, usability/UX assessment, one-off edge cases, and anything requiring human judgment (aesthetics, trust, subjective quality).