Incident Report: Homepage Degradation Event

Incident ID: INC-2009-000417
Severity: SEV-1, downgraded to SEV-3 once everyone calmed down
Status: Resolved, allegedly
Duration: 00:00 – 00:03:12 (approximate)

Summary

At approximately midnight, the production homepage began exhibiting cascading failures consistent with a process that should have been decommissioned seventeen years ago. Service was restored. Confidence was not.

Timeline

All times approximate, because the clock was also affected.

Incident timeline
Time	Event
00:00:00	Scheduled maintenance window opens. Nothing is scheduled.
00:00:04	A cron job with no associated documentation, owner, or Slack mention executes for the first time since 2009.
00:00:09	CPU temperature normal. Everything else is not.
00:00:17	On-call engineer (me) receives a page. Page reads only: “it's happening again.”
00:02:48	Kernel panic observed on the homepage. Technically impossible, given the homepage does not have a kernel. Noted anyway.
00:03:01	Incident declared. Coffee declared mandatory.
00:03:12	Service restored, by means nobody is prepared to explain in writing.

Root Cause

A cron job scheduled in 2009 — before version control conventions existed in this environment, before the current hosting provider existed, possibly before some readers of this report existed — finally fired. It had been waiting. It is unclear what it was waiting for. It is unclear if it is still waiting for something else.

Contributing Factors

No documentation for the cron job in question. (“It's probably fine.” — email, 2009.)
No owner identified. The original author's account was deprovisioned during a reorg of unknown vintage.
Monitoring alerted correctly, but the alert text has not been updated since the incident it was written for, which was a different incident, about a different cron job.
Runbook step 3 says “see Dave.” Dave's role here has never been confirmed.

Impact

Homepage briefly displayed artifacts inconsistent with reality.
One (1) engineer briefly considered a career change.
Zero data was lost, mostly because there was nowhere left for it to go.

Action Items

Identify and document the 2009 cron job. (Assigned: future self)
Determine whether “Dave” is a person, a process, or a philosophy.
Add monitoring for jobs that have been silent for more than one (1) presidential administration.
Stop saying “it's probably fine.”
Buy more coffee.

Lessons Learned

Systems reflect their architecture. So, apparently, do incident reports.