AI API Failover & Load Balancing

AI APIs fail. Rate limits kick in. Latency spikes during peak hours. A provider goes down for maintenance. If your application depends on a single AI provider, any of these events becomes a user-facing outage. This guide shows you how to build resilient AI systems with automatic failover, intelligent load balancing, and circuit breakers — using TokenEase's multi-provider architecture as your foundation.

Why AI API Resilience Matters

Failure Mode	Frequency	Impact	Without Resilience
Rate limiting (429)	Daily	Request blocked	User sees errors
Provider downtime	Monthly	All requests fail	Complete outage
Latency spikes	Weekly	Slow responses	Poor UX, timeouts
Model deprecation	Quarterly	API changes	Broken integrations
Region-specific issues	Occasional	Partial failure	Inconsistent experience

Resilience Patterns

Production AI systems need four resilience patterns:

1. Retry with Backoff: Automatically retry failed requests
2. Circuit Breaker: Stop sending requests to failing providers
3. Failover: Switch to backup providers when primary fails
4. Load Balancing: Distribute traffic across healthy providers

Pattern 1: Retry with Exponential Backoff

The simplest resilience pattern. When a request fails with a retryable error, wait and try again.

Which Errors Are Retryable?

Status Code	Retry?	Backoff	Reason
429 Too Many Requests	Yes	Exponential	Rate limit, will clear
500 Internal Server Error	Yes	Exponential	Transient provider error
502 Bad Gateway	Yes	Exponential	Upstream issue
503 Service Unavailable	Yes	Exponential	Provider overloaded
504 Gateway Timeout	Yes	Exponential	Request timeout
400 Bad Request	No	-	Client error, fix request
401 Unauthorized	No	-	Auth error, fix key
404 Not Found	No	-	Invalid endpoint

Implementation

import asyncio
import random

class RetryConfig:
    def __init__(self, max_retries=3, base_delay=1.0, max_delay=30.0):
        self.max_retries = max_retries
        self.base_delay = base_delay
        self.max_delay = max_delay

async def retry_with_backoff(func, config: RetryConfig, retryable_statuses=None):
    if retryable_statuses is None:
        retryable_statuses = {429, 500, 502, 503, 504}
    
    last_exception = None
    
    for attempt in range(config.max_retries + 1):
        try:
            return await func()
        except Exception as e:
            last_exception = e
            status = getattr(e, 'status_code', None)
            
            if status not in retryable_statuses:
                raise  # Non-retryable error
            
            if attempt == config.max_retries:
                break
            
            # Exponential backoff with jitter
            delay = min(
                config.base_delay * (2 ** attempt),
                config.max_delay
            )
            jitter = random.uniform(0, delay * 0.1)
            await asyncio.sleep(delay + jitter)
    
    raise last_exception

Pattern 2: Circuit Breaker

A circuit breaker stops sending requests to a provider that is consistently failing, preventing cascading failures and giving the provider time to recover.

Circuit Breaker States

• Closed: Normal operation, requests flow through
• Open: Provider failing, requests immediately fail fast
• Half-Open: Testing if provider has recovered

Implementation

import time
from enum import Enum

class CircuitState(Enum):
    CLOSED = "closed"
    OPEN = "open"
    HALF_OPEN = "half_open"

class CircuitBreaker:
    def __init__(
        self,
        failure_threshold=5,
        recovery_timeout=30,
        half_open_max_calls=3
    ):
        self.failure_threshold = failure_threshold
        self.recovery_timeout = recovery_timeout
        self.half_open_max_calls = half_open_max_calls
        
        self.state = CircuitState.CLOSED
        self.failure_count = 0
        self.last_failure_time = None
        self.half_open_calls = 0
    
    def can_execute(self) -> bool:
        if self.state == CircuitState.CLOSED:
            return True
        
        if self.state == CircuitState.OPEN:
            if time.time() - self.last_failure_time >= self.recovery_timeout:
                self.state = CircuitState.HALF_OPEN
                self.half_open_calls = 0
                return True
            return False
        
        if self.state == CircuitState.HALF_OPEN:
            return self.half_open_calls < self.half_open_max_calls
        
        return True
    
    def record_success(self):
        self.failure_count = 0
        
        if self.state == CircuitState.HALF_OPEN:
            self.half_open_calls += 1
            if self.half_open_calls >= self.half_open_max_calls:
                self.state = CircuitState.CLOSED
                self.half_open_calls = 0
    
    def record_failure(self):
        self.failure_count += 1
        self.last_failure_time = time.time()
        
        if self.state == CircuitState.HALF_OPEN:
            self.state = CircuitState.OPEN
            return
        
        if self.failure_count >= self.failure_threshold:
            self.state = CircuitState.OPEN

# Usage
breaker = CircuitBreaker(failure_threshold=3, recovery_timeout=60)

async def call_with_breaker(func):
    if not breaker.can_execute():
        raise Exception("Circuit breaker is OPEN")
    
    try:
        result = await func()
        breaker.record_success()
        return result
    except Exception as e:
        breaker.record_failure()
        raise

Pattern 3: Automatic Failover

When your primary provider fails, automatically switch to a backup. TokenEase makes this easy by providing multiple models through a single API.

Priority-Based Failover

class FailoverRouter:
    def __init__(self, tokenease_client):
        self.client = tokenease_client
        self.providers = {
            "primary": {"model": "deepseek", "priority": 1},
            "secondary": {"model": "qwen", "priority": 2},
            "tertiary": {"model": "doubao", "priority": 3}
        }
        self.breakers = {
            name: CircuitBreaker() for name in self.providers
        }
    
    async def execute_with_failover(self, messages, preferred_model=None):
        # If user specified a model, try that first
        if preferred_model:
            ordered = [preferred_model] + [
                p["model"] for p in sorted(self.providers.values(), key=lambda x: x["priority"])
                if p["model"] != preferred_model
            ]
        else:
            ordered = [p["model"] for p in sorted(
                self.providers.values(), key=lambda x: x["priority"]
            )]
        
        last_error = None
        
        for model in ordered:
            provider_name = next(
                k for k, v in self.providers.items() if v["model"] == model
            )
            breaker = self.breakers[provider_name]
            
            if not breaker.can_execute():
                continue
            
            try:
                response = await retry_with_backoff(
                    lambda: self.client.chat.completions.create(
                        model=model,
                        messages=messages,
                        timeout=30
                    )
                )
                breaker.record_success()
                return {
                    "success": True,
                    "model": model,
                    "response": response,
                    "fallback": model != (preferred_model or ordered[0])
                }
            except Exception as e:
                breaker.record_failure()
                last_error = e
        
        return {
            "success": False,
            "error": str(last_error),
            "model": None
        }

Pattern 4: Intelligent Load Balancing

Instead of failover (primary then backup), distribute traffic across all healthy providers based on real-time performance.

Latency-Aware Load Balancing

import statistics
from collections import deque

class LatencyTracker:
    def __init__(self, window_size=100):
        self.windows = {
            "deepseek": deque(maxlen=window_size),
            "qwen": deque(maxlen=window_size),
            "kimi": deque(maxlen=window_size),
            "doubao": deque(maxlen=window_size)
        }
    
    def record(self, model: str, latency_ms: float):
        self.windows[model].append(latency_ms)
    
    def get_p95(self, model: str) -> float:
        window = list(self.windows[model])
        if len(window) < 10:
            return 1000.0  # Default high latency for new models
        sorted_window = sorted(window)
        idx = int(len(sorted_window) * 0.95)
        return sorted_window[idx]
    
    def get_weights(self) -> dict:
        """Higher weight = more traffic. Inverse of latency."""
        weights = {}
        for model, window in self.windows.items():
            if len(window) < 5:
                weights[model] = 0.25  # Equal weight for new models
            else:
                p95 = self.get_p95(model)
                weights[model] = 1.0 / max(p95, 100)
        
        # Normalize to sum to 1.0
        total = sum(weights.values())
        return {k: v/total for k, v in weights.items()}

class LoadBalancer:
    def __init__(self, client, latency_tracker):
        self.client = client
        self.tracker = latency_tracker
        self.breakers = {model: CircuitBreaker() for model in latency_tracker.windows}
    
    async def route(self, messages):
        weights = self.tracker.get_weights()
        
        # Filter out circuit-open providers
        available = {
            model: weight for model, weight in weights.items()
            if self.breakers[model].can_execute()
        }
        
        if not available:
            raise Exception("All providers circuit open")
        
        # Normalize available weights
        total = sum(available.values())
        available = {k: v/total for k, v in available.items()}
        
        # Weighted random selection
        r = random.random()
        cumulative = 0
        selected_model = None
        
        for model, weight in available.items():
            cumulative += weight
            if r <= cumulative:
                selected_model = model
                break
        
        start_time = time.time()
        try:
            response = await self.client.chat.completions.create(
                model=selected_model,
                messages=messages,
                timeout=30
            )
            latency = (time.time() - start_time) * 1000
            self.tracker.record(selected_model, latency)
            self.breakers[selected_model].record_success()
            return response
        except Exception as e:
            self.breakers[selected_model].record_failure()
            raise

Putting It All Together: The Resilient Client

import openai

class ResilientAIClient:
    def __init__(self, tokenease_key: str):
        self.client = openai.AsyncOpenAI(
            base_url="https://tokenease.io/v1",
            api_key=tokenease_key
        )
        self.latency_tracker = LatencyTracker()
        self.load_balancer = LoadBalancer(self.client, self.latency_tracker)
        self.failover = FailoverRouter(self.client)
        self.retry_config = RetryConfig(max_retries=3)
    
    async def complete(
        self,
        messages,
        preferred_model=None,
        strategy="load_balance"  # or "failover"
    ):
        if strategy == "load_balance":
            return await self.load_balancer.route(messages)
        else:
            return await self.failover.execute_with_failover(
                messages, preferred_model
            )

# Usage
client = ResilientAIClient("sk-your-tokenease-key")

# Load-balanced (recommended for most applications)
response = await client.complete(messages, strategy="load_balance")

# Failover (recommended when you need a specific model)
response = await client.complete(
    messages,
    preferred_model="deepseek",
    strategy="failover"
)

Monitoring and Alerting

Resilience is only as good as your visibility. Monitor these metrics:

Metric	Alert Threshold	Action
Provider error rate	> 5% for 2 minutes	Check circuit breaker status
P95 latency	> 3 seconds	Investigate provider health
Circuit breaker open	Any provider	Page on-call engineer
Fallback rate	> 20% of requests	Primary provider degraded
Retry rate	> 10% of requests	Network or provider issues

Testing Your Resilience

Chaos engineering: deliberately inject failures to verify your resilience works.

class ChaosInjector:
    def __init__(self, failure_rate=0.1):
        self.failure_rate = failure_rate
    
    async def inject_chaos(self, func):
        if random.random() < self.failure_rate:
            raise Exception("CHAOS: Simulated provider failure")
        return await func()

# Test your resilient client under chaos
chaos = ChaosInjector(failure_rate=0.3)  # 30% failure rate

for i in range(100):
    try:
        result = await chaos.inject_chaos(
            lambda: client.complete(messages)
        )
        print(f"Request {i}: Success")
    except Exception as e:
        print(f"Request {i}: Failed - {e}")

TokenEase's Built-In Resilience

TokenEase already provides multi-provider redundancy at the infrastructure level:

• 5 model providers: DeepSeek, GLM, Qwen, Kimi, Doubao
• Automatic retries: TokenEase retries failed requests internally
• Rate limit management: Smooths out burst traffic across providers
• Health monitoring: Automatically routes away from degraded providers

Frequently Asked Questions

How quickly does failover happen?

With proper timeout settings (30 seconds), failover happens within 30-60 seconds of detecting a provider issue. TokenEase's internal failover is even faster, typically under 5 seconds.

Will failover affect response quality?

All TokenEase models are high-quality. While there may be minor differences in response style, the information accuracy remains consistent across providers.

How do I test my resilience setup?

Use chaos engineering: configure a test environment with simulated failures. Verify your circuit breakers open, fallbacks activate, and recovery happens automatically.

Should I implement all four patterns?

Start with retries and failover — these solve 90% of reliability issues. Add circuit breakers and load balancing as you scale beyond 10,000 requests per day.