When the aquarium went dark
2020.228The checklist ports to Slack. The hazard lights are still wired. Nobody filed the ticket.
The checklist ports to Slack. The hazard lights are still wired. Nobody filed the ticket.
The board put us ahead of every customer ticket - until it stayed green while the data underneath it went wrong.
The index was there. The query was right. Postgres stopped using it anyway.
Points are integers. The hard part is time, and users who think faster than your rules.
Autoscale never helped in the time of crisis. What helped was prescaling -- capacity ready before the traffic arrives.
Service discovery, health checks, and dynamic config across a growing fleet of microservices -- and the moment KV store got too popular.
How a growing team adopted Go -- not through boot camps, but through code review, shadowing, and a compiler that wouldn't let anyone be sloppy.
Transaction pooling, one SET that didn't stick, and a standup that moved on.
The checklist is the same as last year. That's the point.
A GraphQL gateway that started as a unified API layer for clients became the place where we could intervene without touching the backends.
The prep runs out ninety minutes in. The rest is improvised on a floor where nobody leaves.
I've built a rules engine twice, in two languages, at two scales. The tree is the same.
What a digital wallet looks like underneath -- a balance, a ledger, idempotency keys, and the gap between moving money and moving money correctly.
A rules engine that answers yes or no during checkout, with condition trees built by people who don't code.
We wired physical lights to 5xx alerts. It changed how the team reacts to production issues.
Algorithm selection, passive health checking, per-endpoint rate limiting, and upstream timing -- the decisions that matter when Nginx sits in front of production traffic.
Pair-writing a real service, asking questions instead of correcting syntax, and waiting for the style guide until after the team has scars.
Picking the first piece to cut from a Perl monolith, rewriting it in Go, and switching over before you lose your nerve.
Migrating a payment processing API from Node.js to Go -- what changed, what improved, and what we gave up.
What a codebase looks like when two people build everything -- 350 commits, and a todo list with two items.