Root Cause Analysis

Symptom, cause and root cause

• The symptom is what you observe. “Users see a 500 error on checkout.”
• The immediate cause is the direct technical reason. “The service ran out of database connections.”
• The root cause is the deeper reason the failure was possible. “A retry loop opened a new connection per attempt and never released them, and nothing alerted before the pool filled.”

Restarting the service removes the symptom. Fixing the leak addresses the cause. Adding an alert and a test addresses why it was not caught.

Asking “why” repeatedly

The “5 Whys” technique is a simple start: ask why, answer from evidence, and ask again.

1. Why did checkout fail? The database rejected new connections.
2. Why? The pool was exhausted.
3. Why? Connections were not returned after errors.
4. Why? The error path skipped the cleanup code.
5. Why was this not noticed earlier? No test covers the error path, and no alert watches pool usage.

The number five is a guideline. Stop when you reach something you can act on. Two traps to avoid: answers that are guesses without evidence, and chains that end at “human error”. If a person’s mistake could take the system down, ask what in the process or tooling allowed that.

Often there is more than one cause

Serious failures usually come from several factors combining: a code defect, a missing check in review, a configuration difference between environments, a gap in monitoring. Listing contributing factors is more honest and more useful than naming one culprit. Compare with Troubleshooting Methodology, which is about finding the cause in the first place.

Acting on the result

• Fix the immediate problem first, so users recover.
• Prevent recurrence with a code change, a test, validation or a safer default.
• Detect it sooner next time with better logging or monitoring.
• Write it down in a blameless post-mortem: what happened, timeline, causes, and follow-up actions with owners. Blameless means the review looks at systems and decisions. See Incident Basics.

Try it

Think of a bug you fixed recently, or use this one: “the nightly export produced an empty file”. Name the symptom, then ask why at least four times, noting the evidence for each answer. Finish with one action each to fix, prevent and detect it.