Chaos Engineering
Standard tests prove your system works when everything is fine. Chaos engineering proves your system survives when everything is on fire.
1 The Hook — Why This Matters
A major NZ banking app had 100% automated test coverage. Their unit tests passed, their end-to-end suite was green, and performance testing showed they could handle 10,000 requests per second. But when a single downstream payment gateway experienced a 5-second network delay, the entire banking app crashed. Threads blocked waiting for the timeout, memory spiked, and the system went down for three hours.
Traditional testing checks correctness. Chaos engineering checks resilience. In 2026's distributed, microservices-heavy architectures, failures are not a possibility—they are a mathematical certainty. If you aren't testing how your system fails, you aren't ready for production.
2 The Rule — The One-Sentence Version
Deliberately inject controlled faults into your system (latency, pod deaths, dropped packets) to validate that fallback mechanisms, circuit breakers, and auto-scaling work as designed.
Hope is not a strategy. We don't hope the circuit breaker works; we trigger it intentionally in a controlled experiment to verify it.
Senior engineer insight
The moment that changed how I think about chaos engineering was watching a well-designed circuit breaker fail silently — it tripped correctly, shielded the user, and sent zero alerts. The team only discovered the downstream service had been dead for 11 hours when a product manager noticed a feature was missing. Resilience patterns don't just need to work; they need to be observable. A fallback that fires without telemetry is a hidden outage waiting to be discovered at the worst possible time.
The most common mistake: teams treat a passing chaos experiment as "the system is resilient" rather than "the system handled this specific fault at this specific scale today."
3 The Analogy — Think Of It Like...
Fire drills in an office building.
You don't wait for a real fire to find out if the emergency exits are locked. You sound the alarm when everyone is calm, measure how long it takes to evacuate, and fix the blocked doors. Chaos engineering is running a technical fire drill: terminating database instances, dropping network packets, and ensuring the "emergency exits" (failovers, retries, circuit breakers) operate smoothly.
From the field
A Wellington-based insurance platform migrated to a microservices architecture and invested heavily in Kubernetes auto-healing. The team assumed pod restarts were their primary resilience risk, so every chaos experiment they ran involved killing pods — and those always recovered cleanly. What they hadn't tested was the downstream claims-processing API provided by a third party, which had a 30-second timeout instead of the assumed 5-second one. When that API degraded during a peak claims period after the Kaikōura road re-opening, thread pools in the gateway service exhausted in 40 seconds and the entire quoting flow went dark. The lesson that generalises: map your blast radius outward, not just inward — third-party dependencies and shared infrastructure are where the real chaos lives, and they're the last things teams think to inject faults into.
4 Watch Me Do It — Step by Step
Implementing a Chaos Engineering experiment follows a scientific method approach. Here is the standard workflow:
- Define the Steady State
Identify the measurable metric that indicates normal behavior. Do not use CPU or memory; use business metrics. Example: "The checkout success rate is 99.9% with a p95 latency under 200ms."
- Formulate a Hypothesis
State what you expect to happen when a failure occurs. Example: "If the recommendation engine microservice fails or delays, the checkout process will ignore it and succeed anyway, though recommendations will not display."
- Inject the Fault (The Blast Radius)
Use tools like Gremlin, Chaos Mesh, or AWS Fault Injection Simulator to introduce the failure. Start small (e.g., impact 5% of traffic in staging) before expanding.
# Chaos Mesh Example: Injecting 2s latency into the Recommendation Pod apiVersion: chaos-mesh.org/v1alpha1 kind: NetworkChaos metadata: name: recommend-delay spec: action: delay mode: one selector: namespaces: - production labelSelectors: app: recommendation-service delay: latency: '2s' correlation: '100' duration: '60s' - Observe and Halt (if necessary)
Monitor the steady state metric. If the checkout success rate drops below 95%, hit the "Big Red Button" to abort the experiment immediately. If the system handles it gracefully, the hypothesis is proven.
| Fault Type | What it Tests |
|---|---|
| Pod Termination (Chaos Monkey) | Auto-healing (Kubernetes ReplicaSets) and statelessness. |
| Network Latency | Timeout configurations, circuit breakers (e.g., Resilience4j). |
| Packet Drop / Blackhole | TCP retransmissions, retry storm prevention (exponential backoff). |
| Time Travel (Clock Skew) | Token expiration, certificate validation, distributed consensus. |
5 When to Use It / When NOT to Use It
✅ Use Chaos Engineering when...
- You have a distributed architecture (microservices, cloud-native).
- Your foundational QA (unit, integration, E2E) is mature and stable.
- You have high-quality observability to measure the "steady state."
❌ Skip Chaos Engineering when...
- You lack basic monitoring and alerting.
- Your system is a monolithic application with clear failure domains.
- You are already dealing with constant unplanned production outages.
6 Common Mistakes — Don't Do This
🚫 Starting in Production
I used to think: It's only true chaos if we do it in production.
Actually: You should practice chaos engineering in staging first. Once your confidence is high and you've automated the safety nets (abort conditions), *then* graduate to Game Days in production. Hurting real users unnecessarily is negligence, not engineering.
🚫 Measuring Infrastructure Instead of Business Impact
I used to think: The metric to watch is CPU utilization on the database.
Actually: High CPU is fine if the user experience is unaffected. The steady state must be a business or UX metric: "Orders placed per minute" or "Video streams started." Measure what matters to the customer.
7 Now You Try — Interview Warm-Up
Scenario: You architect a microservice system. Service A calls Service B. Service B has an SLA to respond within 50ms. To protect Service A, you implement a 100ms timeout. You decide to run a chaos experiment where Service B is delayed by exactly 150ms.
When you inject the fault, Service A immediately starts throwing 500 Internal Server Errors to the user. Why did the experiment fail the hypothesis, and what pattern is missing?
The root cause & fix:
Service A correctly timed out, but it didn't know how to handle the timeout gracefully. A timeout is just an error if there is no Fallback or Circuit Breaker. The system should have returned cached data, degraded the UI feature gracefully, or queued the request, rather than bubbling the 500 error up to the user.
Why teams fail here
- No abort conditions defined before the experiment starts. Teams inject faults then improvise the "stop" decision under pressure. By the time consensus forms in Slack, real users are already affected. Define the kill threshold (e.g., "halt if p99 latency exceeds 1s") and wire an automated abort before the first fault fires.
- Running chaos against a system that is already broken. If you have open P1 incidents, undeployed hotfixes, or degraded monitoring, chaos experiments produce noise not signal — and they can turn a manageable incident into an all-hands outage. Chaos requires a stable baseline to test against.
- Treating game days as a one-time event rather than a continuous practice. A chaos experiment run once tells you about resilience on that day, at that load, with that codebase. Systems change weekly. Revenue NZ-scale batch runs, CoverNZ payment windows, and TransitNZ peak-traffic events all create conditions that only continuous chaos testing will expose before they expose you.
- Skipping the hypothesis and going straight to destruction. Without a written hypothesis ("we expect checkout to succeed at 99.5% when the recommendations service is delayed by 2s"), you cannot objectively evaluate the result. Teams end up watching dashboards and having arguments about whether the numbers look "okay." Science requires a falsifiable prediction before the experiment, not a post-hoc narrative after it.
Key takeaway
Chaos engineering is not about breaking things — it's about discovering, under controlled conditions, which assumptions your architecture makes that the real world will eventually violate.
8 Self-Check — Can You Actually Do This?
Click each question to reveal the answer. If you got all three, you're ready to practice.
Q1. What is the "steady state" and why is it required before running a chaos experiment?
The steady state is a measurable, business-centric metric defining normal system behavior (e.g., successful logins per minute). Without knowing normal, you cannot accurately measure the impact of the injected fault.
Q2. What is the "blast radius" in chaos engineering?
The scope of the injected fault. Best practice is to start with the smallest possible blast radius (e.g., one pod, or 1% of traffic) to minimize potential user impact, and only expand once confidence is gained.
Q3. If you suspect an experiment will break the system, should you run it to confirm?
No. Chaos engineering is for exploring the unknown, not proving known defects. If you know a vulnerability exists, fix it through standard engineering practices first.
9 Interview Prep — Architect Q&A
Interviewers for chaos engineering architects ask about safety, measurement, and systems thinking. These are the questions you'll face.
Q. "How do you design a chaos experiment that's safe to run in production?"
Safety comes from constraint and observation. First, define the "blast radius" precisely: 1% of traffic, one Availability Zone, or one specific pod. Test in staging first to build confidence. In production, start small and scale up only after each experiment succeeds. Second, establish abort conditions: if checkout success rate drops below 95%, the experiment stops immediately. Third, ensure you have good observability to detect impact quickly (sub-minute granularity). Finally, involve the team that owns the system—they know where the hidden dependencies are and can spot anomalies you might miss. Communication beats cleverness.
Q. "What's the relationship between chaos testing and traditional load testing?"
Load testing checks: "Can the system handle expected volume?" Chaos testing checks: "How does the system behave when something breaks?" Load testing is about capacity. Chaos is about resilience. A system that handles 10,000 requests per second might still fall over if 1% of those requests time out due to a network hiccup. Load tests validate the "happy path at scale." Chaos tests validate the recovery path. Both are necessary. Run load tests first to establish baselines. Then run chaos experiments to validate fault-tolerance mechanisms.
Q. "How do you measure the impact of a chaos test?"
Never measure infrastructure metrics like CPU or memory alone. Those are symptoms, not the disease. Measure business metrics: order completion rate, payment processing time, API response time at p95. Before the experiment, establish the "steady state": "Checkout success is 99.9%." During the fault injection, watch that metric. If it holds steady despite the injected fault, the hypothesis is proven (your system is resilient). If it drops, investigate: Did a fallback activate? Did a timeout fire? Did the application gracefully degrade? The richness of your observability determines how quickly you learn what happened.
Q. "What scenarios would you test with chaos engineering on a fintech system?"
Priority: payment processor timeout (test fallback, retry logic), database replication lag (test read consistency), authentication service down (test graceful feature degradation), network packet loss on cross-zone calls (test circuit breakers), clock skew on payment authorization servers (test token expiry), and downstream API rate limiting (test queuing strategies). For fintech, never test scenarios you know will break without mitigation in place. Start with the "known fragile" paths that have fallbacks, and gradually expand to edge cases. Always have a kill switch within arm's reach. Financial customers won't forgive an experiment that causes a single failed transaction.