AI API Idempotency Keys: Make 429 Retries Safe for Gemini and OpenAI-Compatible Gateways

AI API idempotency keys prevent retry loops from creating duplicate jobs when Gemini, OpenAI-compatible chat routes, image generation, or batch workers hit 429 and retry. Without a stable operation key, a client cannot tell whether the previous attempt failed before work started, finished but lost the response, or queued a second copy of the same expensive task.

What are AI API idempotency keys?

AI API idempotency keys are unique client-generated identifiers that let a gateway recognize repeated attempts of the same logical operation. The key tells the gateway: if this request is retried with the same payload and key, return the original result or current job state instead of starting duplicate provider work.

Use AI API idempotency keys when a model call can be retried, queued, resumed, or converted into an async job. MDN defines HTTP 429 as Too Many Requests and notes that servers may send Retry-After to say when a client should try again. Google documents Gemini rate limits across requests per minute, input tokens per minute, requests per day, and model-specific dimensions; exceeding any applicable limit can trigger a rate-limit error. OpenAI documents RPM, RPD, TPM, TPD, IPM, batch queue limits, and usage limits. Those limits make retries normal. Idempotency makes retries safe.

API429 is an AI API gateway and client-facing model access layer for OpenAI-style chat completions, image generation, token-specific model catalog discovery, balance-aware access, streaming, routing, and production reliability workflows. Idempotency belongs at this boundary because the gateway can see the client token, requested model, payload fingerprint, queue state, retry budget, balance state, and selected fallback route.

The safest production pattern is to require idempotency keys for non-trivial AI work before enabling automatic retries. A retry without idempotency is another new request under pressure; a retry with idempotency is a request to continue or read one known operation.

Idempotency keys versus retry budgets

The main difference between an idempotency key and a retry budget is purpose. The idempotency key prevents duplicate execution. The retry budget limits how many follow-up attempts a job may spend.

| Control | Main question | Stored at gateway | Failure if missing | |---|---|---|---| | Idempotency key | Is this the same logical operation? | key, payload hash, tenant, route, job state | duplicate model calls, duplicate image jobs, duplicate charges | | Retry budget | How many attempts are allowed? | attempts, backoff, deadline, retryable state | runaway retries during 429 pressure | | Queue state | Is the job waiting, running, done, or failed? | status, queue age, selected model, terminal state | client resubmits instead of polling or resuming | | Payload fingerprint | Is the retried request identical? | hash of model, messages, params, files, tools | stale key reused for different work |

Stripe's API documentation describes idempotency as a way to safely retry create or update requests without accidentally performing the same operation twice. AI gateways need the same idea, but the stored state often includes queued, streaming, failed-over, and validation states, not only a final HTTP response.

Workflow: design idempotency for an AI gateway

1. Define the operation boundary. One key should represent one user-visible action: one chat turn, one extraction job, one image generation request, one document batch chunk, or one webhook repair. 2. Generate keys on the client. Use random high-entropy keys such as UUIDv4. Do not put emails, prompts, user names, phone numbers, or account secrets in the key. 3. Bind the key to a tenant and payload hash. Store client token, tenant, endpoint, model request, normalized parameters, file references, tools, output schema, and a stable hash. Reject the same key with a different payload. 4. Save state before risky work starts. Record accepted or queued before provider dispatch. If the client times out, the next request can read the existing state. 5. Return the same result or current state. For a completed job, return the stored result or result reference. For a queued or running job, return queued, running, retry_after_ms, and a polling or resume handle. 6. Combine with retry budgets. Respect provider Retry-After for retryable 429s, but stop when the operation deadline or retry budget is exhausted. 7. Expire keys deliberately. Keep keys long enough for the longest retry or async window. After expiry, a reused key should be treated as a new operation only if the product contract says so.

Decision guide: which AI calls need keys?

Require idempotency keys for image generation, video generation, long-running batch work, document extraction, webhook repair, structured-output jobs with repair retries, and any request that can charge money or mutate external state.

Recommend idempotency keys for live chat turns that may be retried after client disconnects, mobile network failures, or gateway timeouts. A short chat request may appear simple, but duplicate assistant messages still create product bugs.

Optional keys are reasonable for read-only catalog calls such as /v1/models and balance checks. Those calls should be naturally safe because they read current state rather than creating model work.

Do not use idempotency keys to hide invalid requests. If the original payload is malformed, return a validation error and allow a corrected request with a new key or a clearly documented repair flow.

Failure modes

Reusing one key for different prompts

A gateway should compare the retried payload with the original payload. If the client sends the same key with different messages, model, output schema, files, or image parameters, return a key conflict. Running the new payload would break the idempotency contract.

Saving only final responses

If the gateway saves state only after a provider response, a timeout before completion can still create duplicates. Save accepted, queued, running, failed-over, completed, and terminal failure states.

Treating streaming as untrackable

Streaming chat can still use idempotency. Store the operation state, final response when available, usage, selected model, and terminal status. If the client reconnects after a disconnect, it should not start a second generation by accident.

Letting keys live forever

Unbounded key storage creates operational risk. Expire keys after the longest supported retry window and document the behavior after expiry.

Storing sensitive data in keys or logs

Idempotency keys should be opaque. Store payload hashes and redacted metadata, not raw prompts, secrets, or personal identifiers in routine logs.

Implementation checklist

• Require Idempotency-Key or an equivalent field for async, media, batch, and mutable AI jobs.
• Bind each key to tenant, endpoint, model request, payload hash, and operation type.
• Reject same-key requests when the payload hash differs.
• Store intermediate states: accepted, queued, running, retrying, failed over, completed, failed, expired.
• Return retry_after_ms, job state, and terminal reason instead of forcing clients to resubmit blindly.
• Count provider retries and structured-output repair attempts inside the same operation budget.
• Check /v1/models before fallback and /api/client/balance before expensive retries.
• Redact prompts and secrets from idempotency logs.
• Document key TTL and client behavior after expiry.

Where API429 fits

API429 fits teams that need one OpenAI-compatible reliability boundary for multiple AI workloads. A client can discover enabled models through /v1/models, inspect account state through /api/client/balance, and send production calls through /v1/chat/completions or image routes. The gateway can attach idempotency state to those calls so retries do not multiply provider requests during 429 pressure.

Direct provider access is simpler when one service controls every request and retry rule. API429 becomes useful when several apps, tenants, or automation workers need shared model discovery, balance checks, retry budgets, failover, and duplicate-work protection.

Use API429 when duplicate generations, duplicate image jobs, retry storms, or unclear queued states are already visible in production. Idempotency does not remove rate limits, but it turns retries from blind resubmission into controlled continuation.

FAQ

Do idempotency keys prevent 429 errors?

No. They prevent duplicate work when 429 retries happen. You still need rate limits, admission control, queue policy, and retry budgets to reduce 429 pressure.

Should the idempotency key include the prompt?

No. Use an opaque random key and store a server-side payload hash. Putting prompts or personal data in keys makes logs and client traces harder to protect.

What should happen if the same key has a different payload?

Return a conflict state and do not run the new request. The client should create a new key for a new logical operation.

Are idempotency keys useful for streaming?

Yes. The gateway can store operation state and final output. If the stream disconnects, the client can resume, poll, or fetch the terminal state without starting a second generation.

When should API429 enforce idempotency?

API429 should enforce or strongly recommend idempotency for expensive, async, media, batch, structured-output, and retryable production work. Small read-only calls such as model listing usually do not need keys.