AI API Rate Limiting Guide 2026

The Three Types of Rate Limits

Rate Limits by Model (2026)

* Limits vary by pricing tier and may change. Always check the provider's latest documentation.

Understanding HTTP 429 Errors

When you hit a rate limit, the API returns HTTP 429 with a Retry-After header telling you how many seconds to wait.

HTTP/1.1 429 Too Many Requests
Retry-After: 12
Content-Type: application/json

{
  "error": {
    "type": "rate_limit_exceeded",
    "message": "Rate limit exceeded. Please retry after 12 seconds."
  }
}

Some providers also include headers showing your current usage:

x-ratelimit-limit-requests: 500
x-ratelimit-remaining-requests: 12
x-ratelimit-limit-tokens: 2000000
x-ratelimit-remaining-tokens: 450000

Exponential Backoff with Jitter

The industry-standard retry pattern. On 429, wait progressively longer between retries, plus random jitter to avoid thundering herd.

import random, time

def call_with_retry(func, max_retries=5):
    for attempt in range(max_retries):
        try:
            return func()
        except RateLimitError as e:
            if attempt == max_retries - 1:
                raise
            # Exponential backoff: 1s, 2s, 4s, 8s, 16s + random jitter
            delay = (2 ** attempt) + random.uniform(0, 1)
            time.sleep(delay)

Rate Limiting Best Practices

1. Monitor Your Headers

Always read the rate limit headers on every response. Track remaining quota and proactively slow down before hitting limits.

2. Use Token Buckets for Self-Throttling

Implement a client-side token bucket that mirrors the API's limits. This prevents 429s entirely by pacing requests before they leave your application.

3. Prioritize Requests

Not all requests are equal. Queue user-facing chat requests above background batch jobs. When approaching limits, defer non-critical work.

4. Batch When Possible

If you have 50 independent prompts, batching them into fewer requests with multiple messages saves RPM. But watch TPM — batched requests use more tokens per call.

5. Use Multiple Keys or a Gateway

For high-volume applications, distribute load across multiple API keys or use a gateway (like TokenEase) that automatically load-balances across provider endpoints.

6. Set Reasonable Timeouts

A hung request consumes your RPM slot. Set 30-60 second timeouts and retry with backoff rather than waiting indefinitely.

Rate Limits with TokenEase

TokenEase handles rate limiting automatically:

• Automatic retry with exponential backoff on 429 errors
• Load balancing across multiple provider endpoints
• Intelligent model fallback when one endpoint is throttled
• Real-time usage dashboard with RPM/TPM tracking
• Per-key rate limiting with configurable quotas