When It Breaks

Stop the bleeding first, understand second. Roll back, disable the feature, put up a message. Restoring service is not the same as fixing the bug, and doing them in the wrong order means people stay broken while you investigate.

Write it down afterwards, without blame. What happened, when you noticed, what you did, why it happened, and what would prevent it or make it noticed faster.

Blameless is not politeness. A write-up that lands on a person produces people who hide problems, and hidden problems are the expensive ones. Almost every incident that looks like human error is a system that made the error easy: a missing check, a confusing interface, a deploy with no rehearsed rollback.

✦ The most useful line in any write-up

How long between it breaking and you knowing. If the answer is 'a user told me', the fix is not in the code, it is in the monitoring, and it is the cheapest improvement available to you.