Hitting five-nines on a payment settlement service

99.999% availability is roughly five minutes of downtime per year. At 20M transactions per month, every minute of full outage is ~460 transactions. Every minute of degraded write path can be much worse, because retries stack up and queues fill behind the failure.

The bar is unreachable by adding redundancy alone. You hit it by making the system safe to retry, safe to degrade, and safe to deploy on a Tuesday afternoon.

The biggest failure class was not raw outage. It was correctness under retry. Settlement is a write path with money attached. A naive retry on a 5xx can double-charge, double-credit, or settle the same batch twice if the upstream did not give you a clean error contract.

The second was tail latency on the database. Postgres p99 latency on the settlement tables crept up steadily as the dataset grew. A 100ms p99 at year one became 800ms at year three. Once p99 crosses your retry timeout, every minor blip cascades.

Idempotency tokens on every state-changing API, scoped per merchant per operation, with a 24-hour window. The token plus a content hash decided whether the request was a replay. Replays returned the original response, byte-for-byte, without touching state.

Outbox pattern on settlement events. Writes to the settlement table and writes to the downstream-event topic happened in the same Postgres transaction via an outbox table. A separate worker shipped outbox rows to Kafka with at-least-once semantics. The consumers were idempotent against the event id.

Postgres tuning that actually mattered: aggressive autovacuum on settlement tables, BRIN indexes on the time-partitioned audit tables, and a hard rule against unbounded queries from any service path. We added a query review step to PRs that touched settlement.

Five-nines is not a code property. It is a deployment-and-on-call property. We banned breaking schema changes during business hours, ran a two-phase deploy for any settlement-touching change (read-new-write-old, then read-new-write-new), and held a blameless retro after every Sev-1 with a written action item.

The Sev-1 count fell 40% year over year. Most of the drop came from the retro discipline, not from the code changes. Patterns repeat. If you do not write the postmortem, you will hit the same incident next quarter.

The outbox. It is one of those patterns that looks like extra work until the day a downstream consumer is down for three hours and your settlement write went through but the event did not. With the outbox, you replay. Without it, you reconcile by hand.

The idempotency surface. Make idempotency a first-class contract in every write endpoint from day one, even before you think you need it. Bolting it on later requires migrations and client changes that nobody wants to do at 2am.