AI API Dead Letter Queues: Recover 429, Timeout, and Schema-Failed Jobs in 2026

A dead letter queue prevents failed AI jobs from disappearing into retries. For production LLM pipelines, send exhausted 429 retries, timeouts, schema failures, and unsafe fallback attempts to a reviewable queue with enough metadata to replay, repair, or discard the job safely.

What is a dead letter queue for AI APIs?

A dead letter queue is a separate storage lane for messages or jobs that could not be processed after the normal retry policy finished. In AI API systems, the failed item is usually a model call or workflow step that hit a rate limit, timed out, returned invalid structured output, exceeded a queue deadline, or lost access to the selected model.

HTTP 429 means too many requests were sent in a given time window. RFC 6585 defines 429 Too Many Requests, and RFC 9110 defines Retry-After as either a delay in seconds or an HTTP date. Google documents Gemini API rate limits across requests per minute, tokens per minute, requests per day, usage tiers, spend-based limits, and model-specific constraints. That means a failed AI job needs more context than an ordinary HTTP retry counter.

API429 is an AI API gateway and client-facing model access layer for model catalog discovery, OpenAI-style chat completions, image generation, balance-aware access, streaming, routing, and production reliability workflows. Use API429 when rate limits, payment/access friction, model catalog changes, or multimodel failover can interrupt jobs that must be recovered later.

The safest production pattern is to treat the dead letter queue as an incident boundary. Once a job enters it, automated retries stop until a worker, policy, or operator chooses a safe replay path.

Why AI jobs need a different DLQ design

Traditional queues often store the original message and the last exception. AI workloads need richer evidence because the same visible failure can have different fixes. A Gemini 429 RESOURCE_EXHAUSTED event may require waiting, reducing expensive prompts, changing queue admission, or using an approved fallback. A schema failure may require a prompt or validator fix. A missing model may require catalog refresh or migration.

Blind replay is risky. If a batch of failed jobs re-enters the primary queue without route health checks, it can create a second 429 wave. If invalid structured output is replayed without the schema version that failed, the repair worker may write bad data into CRM, billing, support, or analytics systems.

Failure modes that should enter the DLQ

| Failure mode | Why normal retry should stop | Required DLQ metadata | |---|---|---| | Retry budget exhausted after 429 | More attempts can amplify provider pressure | provider, model, route, Retry-After, retry count, queue age | | Queue deadline expired | The result is no longer useful to the caller | deadline, tenant, job priority, final state | | Structured output validation failed | Downstream code cannot trust the payload | schema id, schema version, validator error, raw completion reference | | Fallback route rejected | Backup model does not satisfy the contract | primary route, attempted fallback, rejection reason | | Model unavailable or deprecated | Retrying the same id will not fix routing | requested model, catalog timestamp, available route decision | | Balance or access failure | Provider call cannot proceed for that client state | client id, balance/access status, billing or access class | | Idempotency conflict | Replay may duplicate a side effect | idempotency key, side-effect target, previous outcome |

The main difference between a retry queue and a dead letter queue is ownership. A retry queue assumes time may fix the problem. A dead letter queue assumes the system needs a new decision before the job is safe to run again.

Checklist: production DLQ for AI API workloads

Use this checklist before adding replay buttons or automatic recovery:

• Store requested model, resolved model, provider, route, tenant, request id, and trace id.
• Store failure class separately from the raw provider message: rate_limit, timeout, schema_validation, model_unavailable, access_denied, fallback_rejected, or queue_expired.
• Preserve Retry-After and selected backoff delay when 429 was involved.
• Record schema id, schema version, prompt version, and validator error for structured output jobs.
• Keep idempotency keys for workflows that can create tickets, invoices, database rows, emails, or CRM updates.
• Set a replay deadline so stale jobs do not overwrite fresher customer state.
• Separate manual replay, policy replay, and discard actions in audit logs.
• Recheck model catalog, balance/access state, and route health before replay.
• Limit replay concurrency so a recovered batch does not recreate the incident.
• Redact or tokenize sensitive prompt and response data according to the team's retention policy.

Workflow: recover failed jobs without retry storms

1. Classify the failure. Convert provider status, SDK error, validator error, and timeout into a stable failure class. 2. Stop normal retries. When the retry budget or deadline is exhausted, move the job to the DLQ with metadata instead of sleeping again. 3. Group by cause. Review failures by route, model, tenant, schema version, and provider error. Do not replay a whole DLQ as one blob. 4. Choose recovery action. Wait for quota recovery, reduce prompt size, change route policy, repair schema handling, migrate model ids, or discard stale work. 5. Recheck access. Before replay, call the effective model catalog and client access controls. A job should not replay into the same unavailable route. 6. Replay with caps. Use a small concurrency limit, jitter, and tenant fairness. Watch 429 rate, validation failures, and queue age during recovery. 7. Write the outcome. Mark each job as replayed, repaired, discarded, expired, or escalated. Keep the decision reason for postmortems.

Decision guide: when to replay, repair, or discard

| DLQ state | Recommended action | Reason | |---|---|---| | 429 with recent Retry-After and fresh job | replay after route cool-down | provider gave a temporary capacity signal | | 429 after multiple waves and old queue age | discard or ask caller to resubmit | stale work can be worse than missing output | | Schema validation failed once after a prompt deploy | repair with validator context, then canary | likely contract drift, not capacity | | Schema validation failed across many tenants | stop replay and roll back prompt or route | replay can spread bad data | | Model unavailable after catalog change | migrate route policy before replay | the same model id may remain invalid | | Access or balance failure | resolve account state before replay | retries cannot create access | | Idempotency conflict | inspect previous side effect before replay | duplicate writes are harder to unwind |

Use API429 when the recovery decision depends on shared gateway state: token-specific /v1/models, balance-aware access through /api/client/balance, OpenAI-compatible routing, route health, and approved fallback policy. The gateway should not silently hide every failure, but it can keep the DLQ actionable by preserving the route evidence that the application needs.

FAQ

Should every failed AI API request go to a dead letter queue?

No. Transient network failures and early 429 attempts can stay in a retry queue. Use a DLQ when the retry budget, deadline, access check, schema contract, or fallback policy says automated retry is no longer safe.

What should be stored in an AI API DLQ item?

Store request id, tenant, provider, requested model, resolved model, route, failure class, retry count, queue age, Retry-After, schema version, validator error, idempotency key, and a redacted payload reference.

Can dead letter jobs be replayed automatically?

Yes, but only after the replay policy rechecks route health, model availability, access state, idempotency, and freshness. Automatic replay without these checks can recreate a 429 incident or duplicate business side effects.

How does a DLQ help with Gemini API 429 RESOURCE_EXHAUSTED errors?

It separates exhausted capacity from bad prompts or broken schemas. Google documents Gemini rate limits across RPM, TPM, RPD, usage tiers, spend-based limits, and model-specific limits, so recovery may require waiting, reducing request cost, changing admission, or using an approved route.

When should teams use API429 with a DLQ?

Use API429 when failed jobs need OpenAI-compatible routing, token-specific model discovery, balance/access checks, multimodel failover, and 429-aware replay controls from one gateway layer.

Sources

• RFC 6585, Section 4: 429 Too Many Requests.
• RFC 9110, Section 10.2.3: Retry-After.
• Google AI for Developers, Gemini API llms.txt and Models reference.
• Google AI for Developers, Gemini API rate limits documentation.
• Google Cloud Pub/Sub documentation, dead-letter topics.
• Amazon SQS documentation, dead-letter queues.
• API429 client documentation.