Troubleshooting Methodology

Investigation steps

1. Describe the problem precisely. What is expected, what happens instead, for whom, since when? “It is broken” is not enough.
2. Gather evidence. Read the logs, error messages, metrics and recent changes before touching anything.
3. Form a hypothesis that explains all the evidence, and rank candidates by how likely and how cheap they are to check.
4. Test one thing at a time. Design a check that can prove the hypothesis wrong, and change a single variable so you know what helped.
5. Record what you tried and what you saw, including dead ends.
6. Confirm the fix by showing the symptom is gone and nothing else broke.

Where to look

• Ask what changed. A deployment, configuration change, expired certificate, full disk, new data or traffic spike is behind many problems. Compare with the last time it worked.
• Read the logs of the failing component and its neighbours; see Log Analysis.
• Check configuration and environment: environment variables, connection strings and settings that differ between environments.
• Check dependencies: network, DNS, database, credentials and third-party services. Test whether they are reachable from where the code runs, because your laptop may reach things the server cannot.
• Reproduce the problem, locally if you can; see Reproducing Bugs and Debugging Fundamentals.

Narrowing down

Divide the system into parts and check which side of a boundary the problem is on: does the request reach the server, does the server reach the database, is the response correct when it leaves? Halving the search space repeatedly beats reading everything. Compare a working case with a failing one and look at every difference. See Following a Change Through a System.

Habits that slow you down

• Fixing the cause you expect and ignoring what the evidence shows.
• Changing several things at once.
• Trusting assumptions (“that service is surely up”) without checking.
• Stopping at the first plausible explanation. Confirm it explains every symptom.
• Working alone too long. Ask for help with your notes in hand; see Asking Good Questions.

On production systems, be careful: prefer read-only checks, and know who to tell before you change anything. See Incident Basics.

Try it

Take a past bug or a deliberately broken setup (a wrong port in a configuration file, for example). Keep an investigation log with symptom, evidence, hypothesis, test and result for each step until it is fixed. Read it back: was every step based on evidence?