Interview Questions › Root Cause Analysis

Major 2-hour downtime during peak. Full RCA required

A catastrophic 114-minute total service outage occurred during global peak traffic hours, dropping 100% of API and web traffic across millions of customer websites. Conduct a Blameless Post-Mortem and Root Cause Analysis.

Context

Automated failover mechanisms failed to engage, and internal engineer access consoles were locked out during the incident.

Clarifying questions to ask first

Was the trigger an external attack or an internal configuration change?
Why did automated traffic rerouting fail to isolate the affected edge nodes?

How to approach it

Trace Incident Timeline: Triggering Event -> Blast Radius Expansion -> Monitoring Detection Latency -> Internal Engineering Lockout -> Remediation Deployment -> Service Recovery
Identify Root Cause: Configuration push vs BGP routing error vs Database deadlock vs DDoS cascade
Analyze why redundant multi-region automated failovers failed
Draft Action Items across Prevention, Detection, Mitigation, and Process Governance

Framework: Incident Timeline Construction -> Triggering Root Cause -> Cascading Failure & Lockout Mechanism -> Corrective Action Prevention Items

Sample answer

Malformed regex in routing rule caused 100% CPU exhaustion across all edge nodes simultaneously
BGP route leak broadcast invalid routing tables to tier-1 transit providers
Circular dependency: authentication system relied on the network it was trying to recover

Investigation reveals: A global router configuration push bypassed canary staging; invalid syntax caused a kernel panic loop on edge routers; circular dependency locked out engineers from SSH access

Comprehensive post-mortem: Global config rollout bypassed canary gates with circular SSH dependencies; implemented air-gapped out-of-band management console and mandatory automated 3-phase canary rollouts.

What interviewers look for

Mean Time to Detect (MTTD)
Mean Time to Mitigate (MTTM)
SLA Financial Penalty Liability
Blast Radius %