Loading lesson…
AI Workflows & Automation / Reliability
Surviving failure without doing work twice.
Reviewed by Yuvaraj
In a single process on your laptop, a function either returns or throws, and you move on. In a distributed workflow, a chain of API calls, queue handlers, database writes, and non-deterministic AI steps, failure is not the exception, it is the steady state. Networks drop, services return 503, a worker is killed mid-step, a rate limit trips. A reliable workflow is not one that never fails; it is one that fails, retries, and still ends in exactly the right state. This lesson covers the four mechanisms that make that possible: failure classification, retries with backoff, idempotency, and durable state.
Not every error deserves a retry. Retrying a malformed request just burns latency and budget while producing the same rejection. The first decision in any resilient step is to classify the failure.
Ask about this lesson, or about anything in AI. Answers cite the lessons they draw on.
Finished this lesson?
Mark it complete to earn XP, keep your streak, and schedule a review.
Permanent failures should fail fast and, when they cannot be resolved automatically, be routed to a dead-letter queue, a holding area for records a human or separate process inspects later, so one poisoned message never blocks the whole pipeline.
When a failure is transient, retrying immediately is the worst thing you can do: if a service is overloaded, a thundering herd of instant retries keeps it down. Instead, space attempts out with exponential backoff, double the wait after each attempt:
where n is the zero-based attempt number. With a base of one second:
Attempt n | Formula | Base delay | With full jitter |
|---|---|---|---|
| 0 | 1s | random in [0s, 1s] | |
| 1 | 2s | random in [0s, 2s] | |
| 2 | 4s | random in [0s, 4s] | |
| 3 | 8s | random in [0s, 8s] |
The last column adds jitter, randomizing each delay within its window, so thousands of clients that failed at the same instant do not all retry in lockstep. AWS's analysis of this pattern shows full jitter sharply reduces contention. Always cap both the attempt count and the maximum delay.
Backoff makes retries polite; it does not make them safe. Here is the classic trap. A worker calls a charge API, the charge succeeds, but the response is lost to a timeout. The worker sees a failure and retries:
POST /v1/charges
{ "amount": 5000, "currency": "usd", "customer": "cus_42" }
# Attempt 1 -> charge succeeds, but the 200 response is lost to a timeout.
# The worker records a failure and retries...
POST /v1/charges
{ "amount": 5000, "currency": "usd", "customer": "cus_42" }
# Attempt 2 -> a SECOND, separate charge is created. Customer billed twice.
The fix is an idempotency key: a token the client attaches to the request. The server does the work once, stores the result under that key, and replays the stored result for any later request carrying the same key.
POST /v1/charges
Idempotency-Key: order-4711
{ "amount": 5000, "currency": "usd", "customer": "cus_42" }
# Attempt 1 -> creates charge ch_abc, stores it under order-4711.
POST /v1/charges
Idempotency-Key: order-4711
{ "amount": 5000, "currency": "usd", "customer": "cus_42" }
# Attempt 2 -> server sees order-4711 already processed,
# returns the SAME ch_abc. No second charge.
The key must be tied to the business operation, order-4711, not a fresh UUID per attempt, so every retry of the same logical step sends the same key.
For a curious beginner
Pressing a crosswalk button once or ten times summons the same single walk signal. An idempotent operation is like that button: repeating it changes nothing beyond the first press.
How it is actually used
The client attaches a unique key to a request. The server processes the
first request carrying that key, saves the response, and for any later
request with the same key it replays the saved response instead of redoing
the work. GET, PUT, and DELETE are naturally idempotent; POST is
not, which is exactly why payment APIs bolt an idempotency key onto it.
The underlying mechanism
An operation is idempotent when applying it twice equals applying it once: . Setting a value is idempotent; incrementing a counter is not, because .
A five-step workflow that crashes at step four should not start over from step one, that would re-run steps one through three and, without idempotency, repeat their side effects. Durable execution engines persist the outcome of each completed step to storage. On restart, the workflow replays from its last checkpoint: completed steps return their saved results, and execution continues at the step that failed. State and idempotency are complementary, state avoids re-running what already finished, and idempotency makes the unavoidable re-runs harmless.
Why AI steps make this urgent
A deterministic step returns the same output for the same input, so a re-run is at worst wasteful. An LLM call is non-deterministic: retry a summarization or an extraction and you may get different text, a different tool call, or a different cost. Cache each AI step's output under an idempotency key so a downstream retry reuses the exact result the workflow already committed to, otherwise a crash-and-resume can silently change decisions already made upstream.
Common mistakes
400, 401,
404), turning a fast failure into a slow one. - Immediate retries with no
backoff or jitter, effectively DDoSing your own dependency. - Unbounded
retries with no cap and no dead-letter queue, so a poisoned record loops
forever. - Never expiring stored idempotency results, decide how long a key is
honored before you rely on it.