Everything you depend on will be unavailable at some point this year. Your payment provider, your identity checks, your cloud region, your own database during a bad deploy. The arithmetic here is unforgiving. Chain together six services that each manage 99.9% uptime and the combined figure lands near 99.4%, which is roughly two days a year when some part of your product is not working properly.
That number changes what you are designing for. The goal stops being a system that does not fail and becomes a system where you decided in advance what failure is allowed to look like. Those are very different engineering projects, and only one of them is achievable.
Losing a request and losing money are different problems
A dropped request is an inconvenience. The user sees an error, presses the button again, and it works. Mildly irritating, forgotten within a minute.
Losing money is a different category entirely. So is losing a consent record or a medication instruction. The damage from these does not end when the outage does. Somebody gets charged twice and emails support. A duplicated dose instruction reaches a clinician. A human being now has to reconcile the mess by hand, and manual reconciliation is where small technical errors turn into expensive operational ones.
The engineering consequence is blunt. It is acceptable for a request to fail. It is not acceptable for a request to fail in a way where nobody can afterwards determine whether it happened.
Idempotency in plain terms
An operation is idempotent when doing it twice has the same effect as doing it once. Pressing the lift button five times still summons one lift. That is the whole idea.
It matters because you often cannot tell whether the first attempt worked. Your service calls the payment provider. The connection times out before a response comes back. Did the charge go through? You do not know. The provider may well have taken the money and lost the reply on the way home.
There are two poor options here and one good one. Retry and risk charging the customer twice. Do not retry and risk never charging at all. Or make the operation idempotent, retry as often as you like, and let the far side recognise that this is a request it has already handled.
Mechanically this is usually one field. The caller generates a unique key for the attempt and sends the same key with every retry. The receiver stores that key alongside the outcome. If the key turns up again, it returns the original result and does nothing else. Every serious payments API supports a key like this, which makes it worth checking two things in your own code: that you are sending one at all, and that the key stays stable across retries. Regenerating the key on each attempt is a common mistake and it defeats the entire mechanism while looking correct in review.
Naive retries make outages worse
The instinct when a call fails is to try again immediately. One client doing this is harmless. Ten thousand clients doing it in the same second is a load-generating machine aimed at a service that is already struggling. The provider recovers, every queued client retries at once, and it falls over again. Engineers call this a thundering herd, and it reliably turns a two-minute blip into a forty-minute incident.
- Back off. Wait longer after each failed attempt instead of hammering at a fixed interval.
- Add jitter. A random offset on every wait stops all your clients retrying on the same tick of the clock.
- Cap the attempts. A call that has failed five times is not about to succeed on the sixth, and something should be told about it.
- Break the circuit. After a run of failures, stop calling entirely for a period and fail fast, then let a trickle of requests through to test whether it has come back.
Every integration needs a defined answer to “they are down right now”
This is the question teams skip. You integrate identity verification, a payment rail, a records system, an SMS gateway. The happy path gets built and tested thoroughly. What the product does during the hour that provider is unreachable gets decided at two in the morning by whoever is on call, under pressure. That is the worst possible moment to be setting policy about somebody's money.
Decide it at build time instead, per integration. Often the answer is to queue the work and process it when they return. For some flows you can degrade gracefully, letting people in with reduced permissions and completing verification afterwards. Where proceeding could cause harm, the correct behaviour is a hard stop with an honest message. The answers differ, and what matters is that somebody chose deliberately rather than defaulting into whatever the code happened to do.
One line per dependency is enough. What we do while it is down, and who decides when to switch back.
Audit trails earn their keep long before the auditor arrives
An audit trail is sold to you as a compliance cost. Treat it as an engineering tool and it becomes the thing that lets your system explain itself.
When something goes wrong in a system moving money or clinical data, the first question is always what actually happened, in order. Current state tells you the balance is wrong. An event log tells you which of eleven steps ran twice, at which second, with which response from the provider. One of those is a debugging session. The other is guesswork with confidence.
The practical version is to record events as well as results. Every state change gets a durable record carrying a timestamp, the actor, the input that caused it, and the outcome. Keep it append-only. If a record can be quietly edited afterwards then it is not evidence of anything, and the first person to discover that will be a lawyer.
The test: can you replay last Tuesday at 3pm?
Here is the check worth applying to any system that touches money or health records. Pick an arbitrary moment in the recent past. Last Tuesday, three in the afternoon. Now answer these questions from what the system already stores: which operations were in flight, what each external service returned, what state each account was in, and which of those operations actually completed.
If your team can reconstruct that, the design is sound and an incident is a bounded piece of work. If the answer involves inferring things from application logs that rotate away after a week, you have a system that works and cannot account for itself. That gap is where the expensive incidents live.
None of this is exotic. Idempotency keys, backoff with jitter, a written fallback for every third party, an append-only event log. The reason it gets skipped is that all of it is work you do before you need it, and that always loses an argument with the feature shipping this week. Do it anyway in the parts of the system where the damage would be permanent, and leave the rest of the product alone.