Guardrails & fallbacks
An automation without a fallback is a liability. Add the rails before you add the throttle.
"It works for me" is the start of every postmortem. By the time you read it, you've already shipped something that does the right thing on the wrong day.
The four guardrails
- Input validation — does the thing you got match the thing you expected? Wrong format, missing field, weird encoding — bounce it before you touch anything else.
- Output validation — does the thing you produced match what you promised? Schema check, length check, tone check if it matters.
- Rate limits — yours and theirs. Don't trust upstream APIs to throttle you politely; they won't.
- Human checkpoints — pre-defined points where "confidence below X" means a human looks. Not optional for anything customer-facing.
The fallback ladder
When something fails, walk down rungs in order:
- Retry. Backoff and try again. Transient errors are 90% of all errors.
- Degrade. Run a simpler version that still produces something useful.
- Human. Hand it to a queue with full context attached.
- Kill switch. Stop the loop. Tell someone. Don't keep going on hope.
Every automation in production should have at least the first three rungs. The kill switch is non-negotiable.
Telemetry: log what surprised you
You don't need to log everything. You need to log the things that, when something goes wrong, you'll wish you had. Input snapshots, decision points, retry counts, latency. Alerts on rates of weird things, not on weird things themselves.
The scream test
Before every release we ask: "If this thing goes wrong silently for a week, who screams first, and how loud?" If we can't name the person, we add a check. If they'd scream too loud, we add a checkpoint.
Trust comes from boring reliability. Build the rails, then drive fast.