Short answer: a 429 means you hit a limit measured in requests per minute, tokens per minute, or concurrency. Read the Retry-After header, retry with exponential backoff plus jitter, and cap retries. Fixing the architecture — batching, caching, queueing — matters more than tuning the retry loop.
What is actually being limited
Providers rarely limit one number. Expect several ceilings at once: requests per minute, input plus output tokens per minute, concurrent requests, and a daily or monthly quota. A burst of small calls can pass the request limit while a single long prompt trips the token limit.
Response headers usually tell you which ceiling you hit and how long to wait. Log the limit headers on every response for a week; the pattern is more useful than any published table.
Retry rules that work
- Retry only idempotent calls, or make writes idempotent with a request key.
- Honour
Retry-Afterwhen present; otherwise use exponential backoff with full jitter. - Cap total retries at four or five and set a deadline for the whole operation.
- Separate retryable errors (429, 500, 502, 503, 504, timeouts) from permanent ones (400, 401, 403, 404, 422).
- Circuit-break per provider so one failing endpoint does not stall the whole pipeline.
import random, time
def with_backoff(call, retries=5, base=0.5, cap=20):
for attempt in range(retries):
try:
return call()
except RateLimited as e:
if attempt == retries - 1:
raise
wait = min(cap, base * 2 ** attempt)
time.sleep(wait * random.uniform(0.5, 1.5))Reduce the number of calls instead
| Pattern | Effect |
|---|---|
| Batch independent prompts into one request | Fewer requests, better throughput per token |
| Cache deterministic answers by prompt hash | Removes repeat spend entirely |
| Queue background work with a worker pool | Smooths bursts instead of hammering the limit |
| Cap output tokens | Output tokens dominate rate-limit math |
| Route cheap tasks to a smaller model | Lower cost and lower token pressure |
Design for a second provider
Keep model names, base URLs and keys in configuration, and put a thin adapter in front of each provider. When one API returns 429 for minutes, fail over to the fallback model instead of retrying into the wall. The model switching guide covers the adapter pattern; DeepSeek vs Qwen vs Kimi compares cost and context trade-offs.
Monitoring that catches limits early
Track 429 rate, p95 latency, tokens per request and retry count per provider. Alert when the 429 rate crosses a few percent for five minutes; that is usually a traffic-shape problem, not bad luck.
Frequently asked questions
How long should I wait after a 429?
Use Retry-After if the API sends it. Otherwise start around half a second and double each attempt with jitter, capping at 20–30 seconds.
Is a 429 always my fault?
Usually it is your traffic shape, but shared capacity and regional incidents also produce 429s. Compare your error rate with the provider status page before rewriting code.
Should I retry streaming requests?
Retry only if the stream failed before any content was delivered. Once tokens have been streamed, restart with a fresh request and deduplicate on your side.
No comments yet. Be the first to share your thoughts!