Incident Basics

The stages of an incident

Practices differ between organisations, but most follow similar stages:

1. Detect: an alert fires or a user reports a problem. See Monitoring Concepts.
2. Triage: judge the impact and severity (how many users, which functions, how urgent). Severity levels, often numbered, decide who gets involved.
3. Mitigate: restore service first, even with a temporary measure such as rolling back a release or switching off a feature. Finding the full cause can wait.
4. Communicate: keep stakeholders and affected users informed, with regular short updates.
5. Resolve and review: confirm recovery, then hold a blameless post-mortem; see Root Cause Analysis.

Roles and tools

• On-call engineers take turns being reachable to respond to alerts. Larger incidents often have an incident commander who coordinates, while others investigate and one person handles communication.
• A runbook is a documented procedure for a known situation, such as restarting a service or failing over.
• An incident usually has a shared channel or ticket where actions and findings are recorded with times. This record becomes the timeline for the review.

What a junior developer should do

• Know how to raise the alarm and who is on call. Reporting early is expected.
• Do not change production on your own during an incident unless asked. Uncoordinated changes confuse the picture, so follow the lead.
• Share evidence: what you saw, with times, log excerpts and what you tried, as in Troubleshooting Methodology.
• Stick to facts. Guessing in a crisis costs time, and “I don’t know yet” is a valid update.
• Look after yourself. Incidents can be stressful, and tired people make mistakes.

Related ideas include environments, which is why you test before production, and CI/CD, which makes rollbacks and fixes easier to ship safely.

Try it

Find out how your team handles incidents: where problems are reported, who is on call, and whether a runbook or past post-mortem exists. Read one post-mortem if you can, and note the timeline and the follow-up actions. If none exists, read the Google SRE book chapter on postmortem culture, linked from Root Cause Analysis.