Idempotency for LLM retries and agent tool calls

TL;DR: Agent tool calls chain real side effects — create an order, charge a card, send an email. When a step is retried, the chain can re-execute. The usual defence, hashing the request, fails because the model is non-deterministic, and streaming makes the success boundary unreadable. Practitioners describe finding out from the duplicate charges, not from an error.

The evidence

PainHunt's developer tools and infrastructure categories hold 10,685 high-scoring signals (10+/15), average score 11.4, average pain intensity 7.4/10 across 56,801 extracted pain points. Sources are Mastodon (1,902), GitHub (1,770), Medium (1,741), BlueSky (1,122), Dev.to (847) and Discourse (803) — practitioners writing for practitioners.

The reliability cluster inside it is unusually mechanical for a complaint:

• Retries duplicate calls with no trace. No error log, no failed request, no alert — the duplicate shows up on the bill.
• Non-determinism defeats deduplication. Above temperature zero the same prompt returns different responses, so the usual "hash the request, skip if seen" pattern has nothing stable to hash.
• Tool calls chain stateful side effects. create_order → charge_card → send_email re-executed from the top on retry.
• Streaming blurs the success boundary. A connection dropped after 500 tokens leaves the caller unable to tell whether a retry is safe.

Note what is absent: nobody is complaining that the model was wrong. Every item is about not knowing what already happened.

Why now

Agents got permission to act. A year ago an LLM produced text a human then acted on. Now it calls tools directly, and the tools move money, provision accounts and send mail. The blast radius of a retry changed category.

Retry wrappers became default. Provider SDKs and agent frameworks retry transparently on 5xx and timeouts, which is correct for a read and dangerous for a write. Most teams inherit that behaviour without choosing it.

Streaming is now the default UX. Waiting for a complete response is unacceptable in a chat interface, so almost everything streams — and streaming is precisely the transport that makes "did it finish?" unanswerable from the client.

The wedge

The buildable thing is a stable identity for an agent's intent, and a ledger of what that intent already did.

• Derive the key from the decision, not the request. Hash the tool name plus its resolved arguments plus a conversation-scoped step counter. That is stable across retries of the same step and different across genuinely new steps — which is the property a request hash cannot provide once the model is non-deterministic.
• Write the intent before executing it. A durable row saying "step 7 will charge $40" written before the call, and marked done after. On retry, the row answers the question the transport cannot: this already ran.
• Make the boundary explicit in the stream. Emit a terminal marker after tool dispatch so a dropped connection is distinguishable from an incomplete generation. This costs a few bytes and removes the ambiguity entirely.
• Ship the wrapper, not the platform. The buyer wants idempotent(tool) around an existing function, not a migration. A library plus a hosted ledger for teams who do not want to own the storage is the shape with the shortest path to first use.

Start with the tools that move money. That is where a duplicate is expensive enough to justify adopting anything.

Risks and honest caveats

Frameworks will absorb this. Durable execution is an active area — Temporal, Inngest, Restate and the agent frameworks themselves are all moving toward it. Anything built here should assume the primitive gets commoditised and be worth using for the agent-specific part: deriving identity from a non-deterministic decision.

The correct key is domain knowledge, not a library. Whether two "charge the customer" calls are the same intent or two legitimate charges is a question about the business, not about the code. A tool that guesses will sometimes suppress a real second charge, which is a worse failure than the one it prevents.

Adoption requires wrapping every write. A ledger with one unwrapped path has the reliability of the unwrapped path. That is a real integration burden and the main reason teams defer this until after an incident.

The pain is invisible until it is expensive. Teams do not go looking for this. They find it on a bill or in a support ticket, which makes the go-to-market reactive — and means the honest channel is writing about the failure mode, not selling the fix.

Where this came from

This is one cluster inside PainHunt's developer tools categories. The Pain Point Browser shows the underlying signals with their intensity and commercial scoring, and the Idea Validator will score a specific version of this idea against the same dataset. Two adjacent clusters cover neighbouring failures of the same stack: checkpointing for long AI agent tasks, which is the resume side of the same reliability problem, and AI agent permissions and access audit logs, which asks what the agent was allowed to do in the first place.