# Chaos Engineering: Testing Resilience on Purpose

Resilience features such as retries, failover and fallbacks often fail exactly when needed because nobody exercised them. Chaos engineering means injecting failures deliberately, in a controlled way, to find weaknesses before real incidents do.

## The method

1. Define **steady state** with metrics that reflect user experience (successful checkouts per minute, p99 latency).
2. Form a **hypothesis**: "If one cache node dies, steady state is unaffected."
3. **Inject** a realistic failure: kill instances, add latency, drop packets, fill a disk, block a dependency, fail over a database.
4. **Observe** and compare against steady state; stop immediately if user impact exceeds a threshold.
5. Fix what you learned, then automate the experiment so regressions are caught.

## Limiting blast radius

Start in staging, then a small slice of production traffic, then wider. Have an abort button. Run during business hours when the team is ready, not at 3 a.m.

## Common experiments

- Terminate random instances (Netflix's Chaos Monkey).
- Add 300 ms latency to a dependency to test timeouts and circuit breakers.
- Make DNS or a config service unavailable.
- Simulate a full zone or region loss with a game day.
- Expire certificates or credentials in a test environment.

## What it usually finds

Missing timeouts, retry storms, single points of failure nobody knew about, alerts that never fire, and runbooks that are out of date.

## Interview tip

When asked how you'd make sure the failover design actually works, mention regular game days and automated fault injection with a small blast radius.

Disclosure: I help build Linkjob AI (https://www.linkjob.ai/), an AI interview copilot.
