Defect Analysis
Logging a bug is junior work. Understanding why it happened, why it escaped, and how to stop the next one is senior work.
1 The Hook — Why This Matters
On 15 April 2023, a Christchurch retailer's website went live with a new checkout flow. Customers selecting POLi as their payment method found the button simply missing. The feature had passed QA in staging. It worked in the demo. But in production, the POLi button was gone.
The team fixed it within four hours. The developer pushed a corrected deployment script. Everyone moved on. Except the same thing happened again six weeks later, this time with Afterpay. Then again with Stripe. Three times the same class of failure. Three times the same fix-and-forget response.
Defect analysis is what turns "fix the bug" into "fix the system." Without it, you are destined to meet the same bug wearing a different mask.
Senior engineer insight
The shift that changed how I think about defect analysis was realising that the most dangerous number is not your total defect count — it is your escaped defect rate by module. Once I started tracking which components produced production escapes rather than just counting raw defects, I could see that two modules were responsible for 80% of our customer-reported issues even though they appeared fine in sprint metrics. That changed where we invested test effort entirely.
The most common mistake senior testers make is running the analysis but letting the corrective actions die in a Confluence page nobody revisits — close the loop by tracking DDP before and after each intervention.
From the field
On a Wellington government services project, the team assumed their 96% DDP across two consecutive sprints meant defect analysis was a low-value activity — they were catching almost everything before release. What the aggregate number hid was that three escaped defects were all GST rounding errors in edge-case invoice amounts, filed quietly by one Revenue NZ-compliant client who almost churned because of it. When we broke DDP out by feature area and severity, the payment and tax calculation modules had a 71% DDP for critical-severity defects — catastrophically below the overall figure. The lesson that generalised: aggregate quality metrics are organisational comfort food; cohort metrics by module and severity are the ones that actually tell you where to send the fire truck.
2 The Rule — The One-Sentence Version
Every production defect is a signal. Fix the system that produced it, not just the symptom.
A bug in production is evidence of a gap somewhere upstream: missing test coverage, unclear requirements, process failure, or knowledge silos. If you only patch the code, you leave the gap open. Defect analysis closes the gap.
3 The Analogy — Think Of It Like...
A plumber who keeps replacing the same burst pipe.
A junior plumber swaps the pipe. A senior plumber asks: why did it burst? Was the pressure too high? Is the pipe material wrong for this water chemistry? Is the thermostat cycling too aggressively? If you keep replacing the pipe without asking why, you'll be back next winter. Defect analysis is the "why" work. The code change is just the pipe swap.
4 Watch Me Do It — Step by Step
Here is the POLi button defect from 15 April, analysed properly.
- Classify the defect Document type (functional / UI / deployment), severity (critical), phase introduced (deployment), and phase detected (production). Use a standard taxonomy such as ISO/IEC/IEEE 24765 for consistency across teams.
-
Perform root cause analysis with 5 Whys
- Why was the POLi button missing? → The feature was not deployed to production.
- Why was it not deployed? → The deployment script referenced an outdated file path.
- Why was the script outdated? → The integration changed in sprint 4, but the runbook was not updated.
- Why was the runbook not updated? → There was no review process for deployment documentation.
- Why was there no review process? → There was no deployment checklist in the Definition of Done.
-
Map the Fishbone (Ishikawa) categories
- People: Who was responsible for updating the runbook? Was it clear?
- Process: What step should have caught the outdated script before deployment?
- Technology: Could the deployment pipeline validate file paths automatically?
- Environment: Did staging match production closely enough to surface this?
- Perform test gap analysis Ask three questions: Did we have a test case for this? Was it executed? Why did it not catch the defect? In this case, the staging environment test passed because the file path was correct there. The gap was environmental parity, not test case coverage.
-
Calculate Defect Detection Percentage (DDP)
DDP = (Bugs found pre-production / Total bugs found) × 100%
This project found 47 bugs in testing and 3 in production. Total = 50.
DDP = 47 / 50 = 94%. This is excellent, but the 6% that escaped were all deployment-related. That cluster is a signal.
- Propose corrective actions and track effectiveness Add a deployment checklist to the Definition of Done. Automate path validation in the CI pipeline. Measure DDP again next quarter. If deployment escapes drop to zero, the action worked.
5 When to Use It / When NOT to Use It
✅ Use defect analysis when...
- A production defect has business or reputational impact
- You see a cluster of similar defects in telemetry
- A defect recurs after being "fixed"
- You need to justify test process improvements to management
- You are preparing for an audit or compliance review
- You want to measure and improve DDP over time
❌ Don't use it when...
- The defect is trivial, one-off, and already understood
- The team is in active incident response (stabilise first)
- There is no time or mandate to implement corrective actions
- The culture punishes individuals for defects
- You lack data to perform meaningful RCA
6 Common Mistakes — Don't Do This
🚫 Stopping at the symptom
I used to think: The bug was a missing button. The fix is to add the button. Analysis complete.
Actually: The missing button was a deployment failure. The deployment failure was an outdated script. The outdated script was an unreviewed runbook. The unreviewed runbook was a missing checklist. Stopping at "add the button" guarantees recurrence. The 5 Whys exist to force you past the obvious.
🚫 Blaming individuals
I used to think: Someone forgot to update the runbook. They need to be more careful.
Actually: Individual blame is reductive and demotivating. If one person can forget a step and the system breaks, the system is fragile. Good defect analysis focuses on process, tooling, and documentation gaps. "How did the process allow this?" is more useful than "Who messed up?"
🚫 Not tracking corrective actions
I used to think: We agreed to add a checklist. Problem solved.
Actually: Agreements without tracking are just good intentions. Schedule a 30-day follow-up to verify the checklist is being used. Measure DDP next quarter. If deployment escapes are still happening, the corrective action failed and you need a stronger intervention, such as automated deployment validation.
7 Now You Try — Run the 5 Whys
Scenario: A user reports that GST is calculated incorrectly on orders over $1,000. The order total shows $1,150.00 but the GST line shows $143.75 instead of the expected $150.00.
Run a 5 Whys analysis. Write your whys before revealing the answer.
Sample 5 Whys:
- Why is GST wrong? → The system applies a discount before GST instead of after.
- Why does it apply discount first? → The calculation order was not specified in requirements.
- Why was it not specified? → The business analyst assumed the developer knew NZ tax law.
- Why was the assumption unchecked? → There was no tax-calculation review in the test plan.
- Why was there no review? → Financial rules are not in the standard test checklist.
Corrective action: Add a "financial rules validation" step to the test checklist for all pricing stories. Require business analyst sign-off on calculation order before development starts.
Why teams fail here
- They treat defect analysis as a post-mortem ritual rather than a living feedback loop — the report gets written, filed, and never referenced when the next sprint starts.
- They track raw defect counts instead of DDP by severity and module, so the dashboard looks green while critical escapes accumulate undetected in specific components.
- They let defect aging drift in NZ projects where sprint velocity pressure means "fix later" tickets stay open for months — aged defects lose context, making RCA nearly impossible and corrective actions superficial.
- They stop the 5 Whys at the first uncomfortable answer — usually a person rather than a process — and call it root cause, which guarantees the same failure mode resurfaces under a different ticket number.
Key takeaway
A defect you fix without understanding is a defect you are scheduling to fix again.
8 Self-Check — Can You Actually Do This?
Click each question to reveal the answer. If you got all three, you're ready for the interview prep.
Q1. What is DDP and how do you interpret it?
Defect Detection Percentage = (Bugs found pre-production / Total bugs found) × 100%. A higher percentage is better because it means fewer bugs reached users. Track DDP by severity to see where your testing investment is most effective. A drop in DDP for a specific severity or module signals a process gap worth investigating.
Q2. Why should you use 5 Whys with caution?
Because it can be reductive. Each "why" forces a single causal chain, but real failures often have multiple contributing factors. A Fishbone diagram is better for capturing systemic causes across people, process, technology, and environment. Use 5 Whys for quick alignment, then expand with Fishbone if the defect is high-impact or recurrent.
Q3. What three questions make up a test gap analysis?
One: Did we have a test case for this scenario? Two: Was the test case executed? Three: If both are yes, why did the test not catch the defect? (Common reasons: environmental mismatch, test data issues, incorrect expected result, or the defect manifested in a state the test didn't cover.)
9 Interview Prep — Questions You Might Get
Click each question to reveal a strong answer structure.
Q1. What is DDP and why does it matter?
Defect Detection Percentage measures how many bugs we catch before production. It matters because bugs found by users are more expensive to fix and more damaging to trust. I calculate DDP as (pre-production bugs / total bugs) × 100%. I also track it by severity: a 98% overall DDP looks great, but if the 2% missed are all critical defects, that's where I focus my next process improvement. DDP turns defect data into an actionable quality metric.
Q2. Describe a time you performed root cause analysis on a production defect.
I use the STAR method. Situation: A deployment script error caused a missing payment button in production. Task: I was asked to prevent recurrence. Action: I ran a 5 Whys, mapped a Fishbone across people, process, technology, and environment, and performed a test gap analysis. I discovered the staging environment did not mirror production file paths. Result: We added automated path validation to the CI pipeline and a deployment checklist to the Definition of Done. DDP improved from 94% to 98% the following quarter.
Q3. How do you distinguish between a defect and a root cause?
The defect is the observable failure: the button is missing, the calculation is wrong, the page is slow. The root cause is the underlying condition that allowed the defect to exist. A missing button might have a root cause of "no deployment checklist." A wrong calculation might have a root cause of "ambiguous requirements." Fixing the defect helps the user today. Fixing the root cause prevents the next user from being affected tomorrow. Senior testers do both.
Q4. What would you do if management pushed back on spending time on defect analysis?
I would frame it as risk reduction, not overhead. I'd present data: "Three deployment escapes cost us twelve hours of incident response. Defect analysis takes two hours and prevents the pattern." I'd start with a lightweight format: a 30-minute team retrospective on the last production defect, using 5 Whys. Once management sees recurrence drop, they usually support deeper analysis. If they still resist, I document the analysis anyway as part of my test summary report so the decision is visible and traceable.