Preparation: Before Incidents Happen

Write runbooks for every critical system: what it does, how to check if it's healthy, common failure modes, and how to fix each one. Set up monitoring and alerting: Prometheus/Grafana for metrics, PagerDuty/OpsGenie for notifications. Define severity levels: P0 (revenue impact, all hands), P1 (degraded service, on-call responds), P2 (minor issue, next business day). Establish communication channels: dedicated Slack channel for incidents, status page for customers. Preparation turns chaos into process.

During an Incident

Step 1: Acknowledge the alert (within 5 minutes). Step 2: Assess severity and impact. Step 3: Communicate — update the status page, notify stakeholders. Step 4: Investigate using observability tools — metrics, logs, traces. Step 5: Mitigate — restore service first, root cause later. Step 6: Monitor the fix. Step 7: Communicate resolution. For P0 incidents, assign roles: Incident Commander (coordinates), Communications Lead (updates stakeholders), and Technical Lead (drives the fix).

On-Call Best Practices

Rotate weekly among team members — no one should be on-call more than 1 week per month. Provide clear escalation paths: if the on-call can't resolve in 30 minutes, who do they call? Compensate on-call time — either financially or with time off. On-call should not be expected to do feature work during their week — their job is responding to issues and improving reliability. After every on-call week, document what happened for the next person.

Blameless Post-Mortems

After every P0/P1 incident, conduct a post-mortem within 48 hours. Structure: (1) Timeline of events, (2) Root cause analysis (use "5 Whys"), (3) Impact assessment, (4) Action items with owners and deadlines, (5) What went well in the response. The critical rule: blameless. People make mistakes because systems allow them. Ask "what can we change in the system?" not "who messed up?" Publish post-mortems openly — transparency builds trust and learning.

SLOs and Error Budgets

SLI (Service Level Indicator): Measurable metric — latency, availability, error rate. SLO (Service Level Objective): Target for SLI — "99.9% availability" means 43 minutes downtime/month allowed. Error Budget: The allowed amount of unreliability. If you've used your error budget, freeze deployments and focus on reliability. If you have error budget remaining, deploy faster. Error budgets create a rational framework for balancing velocity and reliability instead of arguing about it.

Building Reliability Culture

Everyone owns reliability — not just SRE. Developers who build it should be on-call for it ("you build it, you run it"). Invest in: automated testing, canary deployments, feature flags for gradual rollouts, circuit breakers for graceful degradation. Celebrate reliability wins alongside feature delivery. Track MTTR (Mean Time To Recovery) — getting faster at recovery matters more than preventing every outage. Reliability is a feature, not an afterthought.