AI API 429 Incident Runbook: Gateway Triage and Postmortem Template for 2026

A 429 incident needs a runbook before the retry storm starts. Classify the limit signal, stop amplification, protect live lanes, and write the postmortem around admission, retries, fallback, and provider-specific quota dimensions instead of treating every failure as generic downtime.

What is an AI API 429 incident runbook?

An AI API 429 incident runbook is a step-by-step operating plan for handling Too Many Requests responses from model providers and gateway routes. It tells the on-call team how to detect rate-limit pressure, pause risky traffic, choose safe fallback, communicate user impact, and record the evidence needed for a useful postmortem.

HTTP 429 means the client sent too many requests in a given amount of time. MDN notes that a Retry-After header can tell the client how long to wait before trying again. Google documents Gemini rate limits across dimensions such as requests per minute, tokens per minute, requests per day, and model-specific controls. OpenAI documents production practices around resilient architecture and usage limits. Anthropic documents organization-level rate limits, workspace controls, spend limits, and token-bucket behavior.

API429 is an AI API gateway and client-facing model access layer for model catalog discovery, OpenAI-style chat completions, image generation, balance-aware access, streaming, routing, and production reliability workflows. During a 429 incident, API429 belongs at the control point: it can classify provider responses, cap retries, protect priority lanes, check balance and access state, and route approved fallback from one policy layer.

First 10 minutes: stop the amplification

The safest production pattern is to reduce new work before increasing retries. A 429 response often starts as a provider limit problem, then becomes an application problem when workers retry together, batch jobs keep dispatching, or agent loops turn one user task into many calls.

Use this first-response checklist:

• Confirm the affected route, model, provider account, tenant, workload lane, and region if available.
• Separate HTTP 429 from 5xx overload, safety blocks, authentication errors, schema failures, and client disconnects.
• Read Retry-After when the provider returns it, then add jitter instead of waking every worker at the same time.
• Pause or shrink evaluation jobs, enrichment jobs, and autonomous agent loops before interactive traffic.
• Cap retries per tenant and lane; count original attempts and retries in the same budget.
• Check queue age and active streams. A system can be under RPM limits and still be saturated by long-running calls.
• Verify account balance, access tier, model availability, and token-specific model catalog state before blaming capacity.
• Use fallback only when the target route has capacity, compatible schema behavior, and an approved product policy.
• Return clear terminal states when deadlines expire instead of hiding the incident behind long queues.

Triage table: what the signal means

| Signal | Likely cause | Immediate action | Postmortem evidence | |---|---|---|---| | 429 with Retry-After | provider or route rate limit | honor delay, add jitter, reduce dispatch | retry timing, route, tenant, lane, header value | | 429 without Retry-After | quota, burst, or provider policy limit | use gateway backoff and lane budgets | attempts per minute, tokens per minute, queue age | | sudden queue age spike | admission too loose or route saturated | pause low-priority lanes, enforce deadlines | queue age by lane, active slots, shed volume | | retry count spike | retry amplification | lower retry budget, add circuit breaker | retry ratio, idempotency use, worker release version | | fallback spike | primary route unhealthy or exhausted | verify fallback capacity and quality | fallback reason, target model, schema failures | | unknown model or access error near 429s | catalog or account state drift | refresh model catalog and balance state | token, route, model list timestamp, account status |

The main difference between incident triage and ordinary logging is ownership. Triage asks which control should change now: admission, queue, retry, fallback, workload lane, account access, or user-facing deadline.

Workflow: run the incident without guessing

1. Declare the incident scope. Name the affected provider route, model family, tenant group, workload lane, and user-visible feature. Avoid labels like "AI is down" when only one route is rate-limited. 2. Freeze unsafe traffic classes. Pause evaluation runs, batch enrichment, wide fan-out jobs, and agent loops before touching live support or transactional lanes. 3. Classify failures at the gateway. Bucket results into rate-limited, provider overload, access denied, balance unavailable, safety stop, schema validation, timeout, and client disconnect. 4. Apply shared backoff. Centralize Retry-After handling, jitter, retry budgets, and circuit breakers. Do not let each worker invent its own timing. 5. Check route state before failover. Fallback should be approved for the lane, output contract, latency target, and cost envelope. If the fallback route is saturated, shed or queue instead. 6. Protect deadlines. If an interactive request is stale after the queue wait, fail clearly. If a batch request can wait, persist it with a resume marker. 7. Publish a narrow status update. Say which feature is degraded, whether work is delayed or failing, and what traffic class is paused. Do not promise provider recovery times you cannot verify. 8. Record the timeline. Capture first detection, mitigation, traffic paused, retry policy changed, fallback enabled, error rate recovery, and any customer-visible effects.

Postmortem template for AI API 429 incidents

Summary

• Incident title:
• Date and time window in UTC:
• Affected routes, models, tenants, and workload lanes:
• User-visible impact:
• Final state: recovered, partially mitigated, or awaiting provider/account change:

Detection

• First signal: alert, customer report, log query, provider response, or queue metric:
• Detection delay:
• Missing alert, if any:

Technical cause

• Exhausted dimension: requests per minute, tokens per minute, requests per day, active slots, batch queue, spend limit, account access, or unknown:
• Provider signal and raw status:
• Retry-After observed:
• Gateway classification used:

Amplifiers

• Retry ratio before mitigation:
• Batch, evaluation, or agent traffic running during the incident:
• Queue age and active stream count:
• Fallback attempts and failure rate:

Mitigation

• Traffic paused or shed:
• Retry budget change:
• Circuit breaker or route health change:
• Fallback route used:
• User-facing copy or status update:

Follow-up actions

• Admission-control change:
• Lane budget change:
• Retry or jitter change:
• Model catalog or account-state refresh change:
• Alert or dashboard change:
• Owner and due date:

Failure modes to avoid

Retrying because the request is important

Importance does not create provider capacity. Important work should receive reserved lane capacity, shorter queues, and approved fallback. Unlimited retries make recovery slower.

Treating all providers as equivalent fallback

A fallback route can differ in tool calling, JSON strictness, latency, context length, safety behavior, and cost. Test fallback per workload before the incident.

Counting only provider latency

During 429 pressure, queue age can be the real user experience. Track time before dispatch, provider latency, retry delay, and total wall-clock time separately.

Ignoring account and access state

Some failures look like capacity but come from spend limits, billing friction, disabled models, or token-specific catalog drift. Check balance and access before scaling workers.

Letting batch jobs recover first

A batch job can be easy to resume; an interactive session may be gone after 30 seconds. Recovery order should follow product value and deadline, not queue arrival time.

Where API429 fits

API429 is useful when several products, tenants, and agents share Gemini, OpenAI-compatible, image, streaming, and structured-output routes. The gateway can expose enabled models through /v1/models, check account state through /api/client/balance, and keep admission, retries, route health, and fallback decisions in one place.

Use API429 when the production question is not only "why did the provider return 429," but "which traffic should run now, which traffic should wait, and which route can safely take over." A gateway cannot remove provider limits. It can stop the application from turning a limit into an outage.

FAQ

What should an AI API team do first during a 429 incident?

Stop amplification. Pause low-priority lanes, cap retries, add jitter, and classify the provider signal before adding workers or enabling broad fallback.

Is Retry-After enough for AI API retries?

No. Retry-After gives a base delay when present. A production gateway should add jitter, enforce lane budgets, and stop retries when deadlines or budgets expire.

Should 429 incidents trigger multimodel failover automatically?

Only when the fallback route has capacity and is approved for the workload. Structured output, tool calling, and safety behavior can change across models.

What belongs in a 429 postmortem?

Include exhausted quota dimension, retry ratio, queue age, lane policy, fallback behavior, account or catalog state, user impact, timeline, and concrete control-plane changes.

When is API429 useful for 429 incident response?

Use API429 when multiple AI workloads need shared model discovery, balance checks, OpenAI-compatible routing, retry budgets, workload lanes, and failover policy from one gateway layer.