Idempotency, and exactly-once as a design pattern

What it is

An operation is idempotent if performing it more than once has the same effect as performing it once. SET x = 5 is idempotent; x += 5 is not.

The reason this matters in distributed systems is a fact you cannot engineer away: a client that does not receive a response cannot know whether the operation happened. The request may have been lost before arrival, or executed and the response lost on the way back. Those two cases are indistinguishable to the client, and it has exactly two choices: retry (risking a duplicate) or not retry (risking a lost operation).

Idempotency is what makes retrying safe, which is what makes the whole at-least-once world workable. "Exactly-once" is not a delivery guarantee, it is an observable property built from at-least-once delivery plus deduplication. Anybody who claims exactly-once delivery over a network is describing something that does not exist; the Two Generals problem says so.

The problem it solves

Timeouts are not rare events. A client timeout on a successful request happens continuously at any real volume: a slow garbage collection, a network blip, a load balancer idle timeout, a mobile client losing signal after the server committed.

Without idempotency the choices are both bad. Retry, and a payment is taken twice, an email is sent twice, an order is created twice. Do not retry, and a payment silently fails and the customer is told nothing.

With an idempotency key the retry is free: the second request returns the first one's result, and the client cannot tell whether it was the first or the fifth attempt. That is the whole point, and it converts an unsolvable distributed consensus problem into a database unique constraint.

Mechanics

The three levels

Level 1: naturally idempotent operations. Design so the question does not arise.

-- Idempotent: same result whether run once or five times.
UPDATE users SET email = 'a@b.com' WHERE id = 42;
INSERT INTO seen (event_id) VALUES ('evt_9f2a') ON CONFLICT DO NOTHING;
DELETE FROM sessions WHERE id = 'sess_1';

-- Not idempotent: each execution changes the result.
UPDATE accounts SET balance = balance - 100 WHERE id = 42;
INSERT INTO orders (customer_id, total) VALUES (42, 4299);

Absolute state assignment is idempotent; relative mutation is not. Where you can express an operation as "set to this value" rather than "change by this amount", do, because it removes the problem rather than managing it.

Level 2: idempotency keys. For operations that are inherently non-idempotent (create an order, charge a card), the client supplies a unique key and the server guarantees at most one execution per key.

def charge(idempotency_key, customer_id, amount_minor):
    """The whole contract in one function. Three properties matter:
    the key is scoped, the result is stored (not just the fact of having
    seen the key), and in-flight requests are handled explicitly."""
    scoped = f"{customer_id}:charge:{idempotency_key}"

    with db.transaction():
        existing = db.query(
            "SELECT status, response FROM idempotency WHERE key = %s FOR UPDATE",
            scoped)

        if existing and existing.status == "completed":
            return existing.response            # replay: identical result

        if existing and existing.status == "in_flight":
            # A concurrent duplicate. Do NOT execute; tell the client to retry.
            raise ConflictError(retry_after=1)

        db.execute("INSERT INTO idempotency (key, status, request_hash) "
                   "VALUES (%s, 'in_flight', %s)", scoped, hash_request(...))

    try:
        result = payment_gateway.charge(customer_id, amount_minor,
                                        idempotency_key=scoped)  # pass it down
        db.execute("UPDATE idempotency SET status='completed', response=%s "
                   "WHERE key=%s", result, scoped)
        return result
    except PermanentError as e:
        db.execute("UPDATE idempotency SET status='failed', response=%s "
                   "WHERE key=%s", e.as_response(), scoped)
        raise

Five design decisions are encoded there, and each is a question an interviewer can push on:

Scope the key. {tenant}:{operation}:{key} so two customers cannot collide and a key reused across endpoints does not shadow.
Store the result, not just the key. A replay must return what the first call returned, or the client sees a different answer to the same request.
Handle in-flight explicitly. Two concurrent duplicates are common (a client retrying on timeout while the original is still running). Returning a conflict and asking the client to retry is honest; executing twice is the bug you were preventing.
Hash the request body. If the same key arrives with different parameters, that is a client bug and should be a 422, not a silent replay of a different operation.
Propagate the key downstream. Your idempotency is worthless if the payment gateway charges twice.

Two operational details: expire keys (24 hours to 7 days is typical, and the window must exceed the client's maximum retry horizon), and note that failures are recorded too, so a retry of a permanently-failed operation returns the failure rather than trying again.

Level 3: effectively-once processing. For asynchronous pipelines where the message broker delivers at least once, the consumer deduplicates.

def handle(message):
    """The dedupe table is the whole mechanism. Note that the insert and the
    side effect share one transaction: without that, a crash between them
    either loses the work or repeats it."""
    with db.transaction():
        try:
            db.execute("INSERT INTO processed (message_id) VALUES (%s)", message.id)
        except UniqueViolation:
            return                                  # already handled; ack and move on
        apply_side_effect(message)                  # same transaction

The subtlety is that this only works when the side effect is in the same transactional boundary as the dedupe record. If the side effect is an HTTP call to another service, you are back to level 2 and need an idempotency key at that boundary.

Where the key comes from

Source	Example	When
Client-generated UUID	`Idempotency-Key: 8f2a-...`	Public APIs; the client owns the retry
Natural key from the domain	`order_id`, `invoice_number`	When one already exists and is unique
Deterministic hash of the request	`sha256(customer, amount, day)`	Batch jobs where the client cannot store a key
Broker message id	Kafka `(topic, partition, offset)`	Stream consumers

The client-generated UUID is the right default for an API, and the requirement to state is that the client must generate the key once and reuse it across retries. A client that generates a fresh key per attempt has an idempotency header and no idempotency.

The dual-write problem, and the outbox

The classic failure: write to the database, then publish an event. If the process dies between them, the database and the event stream disagree permanently, and no retry fixes it because the database write already succeeded.

-- The transactional outbox. The event is written in the SAME transaction as
-- the state change, so they cannot diverge. A separate process (a poller, or
-- change data capture on this table) publishes and marks it sent.
BEGIN;
  INSERT INTO orders (id, customer_id, total) VALUES (...);
  INSERT INTO outbox (id, topic, payload)
    VALUES (gen_random_uuid(), 'orders.created', '{"order_id": ...}');
COMMIT;

Publication is then at-least-once (the publisher can crash after sending and before marking), which is fine, because consumers deduplicate. The outbox turns a distributed transaction into a local transaction plus at-least-once delivery plus consumer-side dedupe, and that composition is the general shape of every solution in this area.

A worked example

A checkout API. 3,000 orders per second at peak. Mobile clients on unreliable networks with a 10-second timeout and automatic retry.

Without idempotency, measured over a week: 0.4 percent of requests time out after the server committed. That is 12 duplicate orders per second at peak, roughly 1 million per week, each one a charged customer with two orders and a support ticket.

With idempotency keys:

POST /v1/orders
Idempotency-Key: 8f2a91c4-...
{ "items": [...], "total": 4299 }

The key is scoped {customer_id}:orders.create:{key} and stored with the response, expiring after 72 hours (well beyond the client's 60-second retry horizon).
Duplicate arrives: returns the original 201 with the same order id. The client cannot distinguish it from the first response, which is the requirement.
Concurrent duplicate: the FOR UPDATE on the idempotency row serialises them, the second sees in_flight and gets a 409 with Retry-After: 1.
The key is passed to the payment gateway, so its charge is deduplicated too.

The bug that shipped anyway, which is worth telling because it is the common one: the mobile client generated a new UUID on every retry attempt. The server was correct, the header was present, and the duplicate rate did not move. The fix was one line in the client (generate the key when the user taps the button, not when the request is constructed) and it is the failure mode to check first when idempotency "is not working".

Capacity note. The idempotency table takes one write per request, so it is on the critical path at 3,000 writes per second. Two mitigations: partition by day and drop old partitions rather than deleting rows, and consider a Redis fast path for the existence check with the database as the durable record. Both worth mentioning, because the honest cost of idempotency is an extra write and an extra read on every mutating request.

Production evidence

Stripe made idempotency keys a documented, first-class part of their public API: clients supply Idempotency-Key, the server stores the result, replays return the original response, and keys expire after 24 hours. Their engineering writing on retries and idempotency is the standard reference for how a payments API should behave, and it is what most other APIs copied.

AWS requires client tokens on many mutating operations for the same reason (EC2 ClientToken, SQS deduplication ids on FIFO queues), and SQS FIFO's five-minute deduplication window is a good concrete example of the "at-least-once plus dedupe" composition.

Kafka's exactly-once semantics are built exactly this way: an idempotent producer deduplicates retries by producer id and sequence number, transactions make the output writes and the offset commit atomic, and the guarantee is explicitly scoped to Kafka. See Kafka exactly-once for the mechanics.

Debezium deliberately provides at-least-once change data capture and documents that consumers must be idempotent, which is a mature system declining to promise something it cannot deliver.

The Two Generals problem is the formal reason exactly-once delivery is impossible over an unreliable channel: no finite protocol lets both parties agree on whether a message was received. Naming it is the crisp answer to "why can't we just do exactly-once".

The debate

The alternative to idempotency keys is at-most-once semantics: never retry, and surface failures to the user. It is simpler and it is occasionally right, for operations where a duplicate is much worse than a miss and the user can retry manually with full information.

Its weakness is that it converts every transient network failure into a user-visible error, and transient failures are constant. It also does not actually avoid the problem: the user retries manually, and now you have an uncontrolled duplicate.

The other alternative is distributed transactions (two-phase commit) across services, which gives you atomicity without dedupe. It is available and it is avoided for good reasons: the coordinator is a single point of failure, participants hold locks while blocked, and availability is the product of all participants' availability.

My position: design operations to be naturally idempotent where possible, because that removes the problem. Where not possible, idempotency keys on every mutating endpoint as a contract rather than a feature, scoped, storing the result, handling in-flight explicitly, and propagated downstream. For asynchronous work, the transactional outbox plus consumer-side dedupe, because it turns a distributed transaction into a local one. And say plainly that exactly-once is at-least-once plus deduplication, because the alternative framing leads people to look for a delivery guarantee that does not exist.

Idempotency keys are the wrong mechanism when a natural key already exists (use it), when the operation is genuinely idempotent already (do not add machinery), or when the extra write per request is unaffordable and duplicates are cheap, for example analytics events where a small duplicate rate is statistically irrelevant.

Follow-up Q&A

"Why is exactly-once delivery impossible?" Two Generals. Over an unreliable channel, no finite protocol lets both parties agree that a message was received, because the acknowledgement can be lost, and the acknowledgement of the acknowledgement can be lost. So the sender cannot know, and its only options are to retry or not. What you can build is exactly-once effects: at-least-once delivery plus deduplication at the receiver, which is what every system claiming exactly-once is actually doing.

"Design an idempotency key mechanism for a payments API." Client supplies a UUID generated once and reused across retries. Server scopes it as tenant plus operation plus key, stores it with the response and a hash of the request, and replays return the stored response. Concurrent duplicates take a row lock, and the loser sees in-flight and gets a 409 with Retry-After rather than executing. Same key with a different body is a 422, because that is a client bug. Keys expire well beyond the client's maximum retry horizon. And the key is propagated to the payment gateway, or your idempotency stops at your boundary.

"A client sends the same idempotency key twice concurrently. What happens?" They serialise on the idempotency row. The first inserts the row as in-flight and proceeds; the second finds in-flight and must not execute. Returning a 409 with Retry-After is the honest answer, because you genuinely do not know the outcome yet. Waiting for the first to complete and returning its result is nicer for the client and holds a connection open, which is a capacity tradeoff. What you must not do is treat "not completed" as "not started" and execute.

"What is the dual-write problem and how do you fix it?" Writing to a database and then publishing an event are two operations with no shared transaction, so a crash between them leaves them permanently inconsistent and no retry helps, because the first write already succeeded. The fix is the transactional outbox: write the event into an outbox table in the same transaction as the state change, and have a separate publisher read and send it. Publication is at-least-once, which is fine, because consumers deduplicate. The composition is local transaction plus at-least-once plus consumer dedupe, which is the general shape.

"Idempotency is implemented and the duplicate rate hasn't moved. What do you check?" The client, first, because the most common bug is generating a fresh key per retry attempt rather than per logical operation. Then whether the key is scoped correctly, since an unscoped key can collide or be shadowed. Then whether the key is propagated to downstream services, because your dedupe does not help if the payment gateway charges twice. Then whether the stored record includes the response, because if it only records "seen" then a replay returns something different and the client may treat it as a new operation.

Common misconceptions

The most common is that exactly-once is a delivery guarantee some systems provide. It is an effect built from at-least-once plus deduplication, and every system advertising it is doing that.

The second is that idempotency is about retries. It is about the fact that the client and server can disagree about whether a request succeeded, which is unavoidable over a network. Retries are the consequence, not the cause.

The third is that recording the key is enough. You must record the result, because a replay has to return what the first call returned; otherwise the client sees two different answers to the same request and cannot reconcile them.

Interview delivery note

Say this: "A client that doesn't get a response can't know whether the operation happened, so it either retries and risks a duplicate or doesn't and risks a loss. Idempotency makes the retry free. I'd design operations to be naturally idempotent where I can, absolute assignment rather than relative mutation, and where I can't, an idempotency key on every mutating endpoint: scoped by tenant and operation, stored with the response rather than just the fact of having seen it, in-flight handled explicitly with a 409, and propagated downstream so the payment gateway dedupes too."

Then the framing that shows you understand the theory: "Exactly-once isn't a delivery guarantee, it's at-least-once plus deduplication. Two Generals says you can't do better over an unreliable channel, so every system claiming exactly-once is doing this underneath."

The depth signal is the client-side bug: "the failure I'd check first is the client generating a new key per retry attempt instead of per logical operation, which makes a perfectly correct server useless." And the outbox, because it shows you know where the boundary between local and distributed sits.

The Staff & Team Lead Interview Reference