A/B Testing AI Models in Production

Switching AI models without rigorous testing is risky. A model that scores higher on benchmarks might perform worse for your specific use case. A/B testing lets you compare models on real user traffic, measure actual business impact, and roll out changes confidently. This guide provides a complete framework for testing Chinese AI models in production via TokenEase.

Why A/B Test AI Models?

• Benchmarks lie: MMLU-Pro and HumanEval do not capture your users' actual experience
• Latency matters: A smarter model that takes 5 seconds longer may hurt conversion more than it helps
• Cost varies: A 5% quality improvement might cost 10x more — is it worth it?
• Behavior differs: Models have different "personalities" that affect user engagement

The A/B Testing Framework

A complete AI model A/B test has four phases:

1. Hypothesis: Define what you are testing and what success looks like
2. Setup: Configure routing, metrics, and traffic split
3. Execution: Run the test with real traffic
4. Analysis: Determine statistical significance and make a decision

Phase 1: Define Your Hypothesis

Start with a clear, testable statement:

Key Metrics to Track

Category	Metric	How to Measure
Quality	Accuracy / Correctness	Human rating, automated evaluation, user feedback
Quality	Relevance	Click-through rate, time-on-page, bounce rate
Quality	User Satisfaction	Thumbs up/down, NPS, CSAT
Performance	Latency (P50/P95/P99)	Time from request to first token, total response time
Performance	Throughput	Requests per minute, tokens per second
Performance	Error Rate	Failed requests, timeout rate, retry rate
Business	Cost per Request	Total spend / number of requests
Business	Conversion Rate	Users who complete target action
Business	Retention	Return users within 7/30 days

Phase 2: Set Up the Test Infrastructure

Traffic Splitting

Use a consistent hash of the user ID to ensure the same user always hits the same model during the test:

import hashlib
import random

class ModelRouter:
    def __init__(self, test_config):
        self.config = test_config
        self.default_model = test_config["control"]
    
    def get_model_for_user(self, user_id):
        # Deterministic assignment based on user ID
        hash_value = int(hashlib.md5(user_id.encode()).hexdigest(), 16)
        bucket = hash_value % 100
        
        cumulative = 0
        for variant, allocation in self.config["variants"].items():
            cumulative += allocation
            if bucket < cumulative:
                return variant
        
        return self.default_model
    
    def should_include_in_test(self, user_id):
        # Only include users in the test population
        hash_value = int(hashlib.md5(f"{user_id}_test".encode()).hexdigest(), 16)
        return hash_value % 100 < self.config["test_population_percent"]

# Configuration
test_config = {
    "control": "gpt-5",           # Current model
    "variants": {
        "gpt-5": 50,              # 50% control
        "deepseek": 50            # 50% treatment
    },
    "test_population_percent": 20  # 20% of total traffic
}

router = ModelRouter(test_config)

# Usage
user_id = "user_12345"
if router.should_include_in_test(user_id):
    model = router.get_model_for_user(user_id)
else:
    model = test_config["control"]  # Outside test, use control

Consistent User Experience

Logging Every Interaction

import time
import json
from datetime import datetime

class AILogger:
    def __init__(self, log_file="ab_test_log.jsonl"):
        self.log_file = log_file
    
    def log_interaction(self, user_id, model, prompt, response, metrics):
        entry = {
            "timestamp": datetime.now().isoformat(),
            "user_id": user_id,
            "model": model,
            "variant": "control" if model == "gpt-5" else "treatment",
            "prompt_tokens": metrics["prompt_tokens"],
            "completion_tokens": metrics["completion_tokens"],
            "latency_ms": metrics["latency_ms"],
            "cost": metrics["cost"],
            "success": metrics["success"],
            "error": metrics.get("error", None)
        }
        
        with open(self.log_file, "a") as f:
            f.write(json.dumps(entry) + "\n")

# Usage
logger = AILogger()

start_time = time.time()
try:
    response = client.chat.completions.create(
        model=model,
        messages=messages
    )
    latency = (time.time() - start_time) * 1000
    
    logger.log_interaction(user_id, model, prompt, response, {
        "prompt_tokens": response.usage.prompt_tokens,
        "completion_tokens": response.usage.completion_tokens,
        "latency_ms": latency,
        "cost": calculate_cost(response.usage, model),
        "success": True
    })
except Exception as e:
    logger.log_interaction(user_id, model, prompt, None, {
        "prompt_tokens": 0,
        "completion_tokens": 0,
        "latency_ms": (time.time() - start_time) * 1000,
        "cost": 0,
        "success": False,
        "error": str(e)
    })

Phase 3: Run the Test

Test Duration Guidelines

Traffic Volume	Min Duration	Target Sample Size
< 1,000 requests/day	2 weeks	10,000+ per variant
1,000 - 10,000/day	1 week	20,000+ per variant
10,000 - 100,000/day	3 days	50,000+ per variant
> 100,000/day	1 day	100,000+ per variant

Monitoring During the Test

Set up real-time dashboards for:

• Traffic split accuracy (should be 50/50 ± 2%)
• Error rate by variant (alert if treatment error rate > 2x control)
• P95 latency by variant (alert if treatment latency > 1.5x control)
• Daily cost by variant

Phase 4: Analyze Results

Statistical Significance

Use a two-proportion z-test for binary metrics (success rate, conversion) and a t-test for continuous metrics (latency, cost):

import numpy as np
from scipy import stats

def analyze_ab_test(log_file):
    control_data = []
    treatment_data = []
    
    with open(log_file) as f:
        for line in f:
            entry = json.loads(line)
            if entry["variant"] == "control":
                control_data.append(entry)
            else:
                treatment_data.append(entry)
    
    # Latency comparison
    control_latency = [d["latency_ms"] for d in control_data if d["success"]]
    treatment_latency = [d["latency_ms"] for d in treatment_data if d["success"]]
    
    t_stat, p_value = stats.ttest_ind(control_latency, treatment_latency)
    
    print(f"Control latency: {np.mean(control_latency):.1f}ms (n={len(control_latency)})")
    print(f"Treatment latency: {np.mean(treatment_latency):.1f}ms (n={len(treatment_latency)})")
    print(f"P-value: {p_value:.4f}")
    print(f"Significant: {'Yes' if p_value < 0.05 else 'No'}")
    
    # Cost comparison
    control_cost = sum(d["cost"] for d in control_data)
    treatment_cost = sum(d["cost"] for d in treatment_data)
    control_count = len(control_data)
    treatment_count = len(treatment_data)
    
    print(f"\nControl cost/request: ${control_cost/control_count:.4f}")
    print(f"Treatment cost/request: ${treatment_cost/treatment_count:.4f}")
    print(f"Cost savings: {(1 - (treatment_cost/treatment_count)/(control_cost/control_count)) * 100:.1f}%")

analyze_ab_test("ab_test_log.jsonl")

Interpreting Results

Scenario	Latency	Quality	Cost	Decision
Treatment wins	Same or better	Same or better	Lower	Roll out 100%
Mixed results	Better	Slightly worse	Much lower	Roll out with monitoring
Quality trade-off	Same	Much better	Higher	Segmented roll-out
Treatment loses	Worse	Worse	Any	Stay on control

Advanced: Multi-Model Testing

Test more than two models simultaneously to find the optimal choice:

test_config = {
    "variants": {
        "gpt-5": 25,        # Control
        "deepseek": 25,     # Treatment A
        "kimi": 25,         # Treatment B
        "qwen": 25          # Treatment C
    }
}

Use ANOVA for multi-variant comparison, followed by pairwise t-tests with Bonferroni correction.

Gradual Rollout Strategy

After a successful A/B test, do not flip to 100% immediately. Use this phased approach:

Phase	Traffic %	Duration	Goal
Validation	10%	2 days	Confirm test results hold
Expansion	50%	1 week	Validate at scale
Full rollout	100%	Ongoing	Complete migration
Holdback	5%	2 weeks	Long-term validation

Automated Model Selection

For advanced setups, automate model selection based on real-time performance:

class AutoModelSelector:
    def __init__(self, models, evaluation_window=1000):
        self.models = models
        self.scores = {m: [] for m in models}
        self.window = evaluation_window
    
    def record_score(self, model, score):
        self.scores[model].append(score)
        if len(self.scores[model]) > self.window:
            self.scores[model].pop(0)
    
    def select_model(self):
        # Epsilon-greedy: explore 10% of the time
        if random.random() < 0.1:
            return random.choice(self.models)
        
        # Exploit: pick model with highest average score
        avg_scores = {
            m: np.mean(scores) if scores else 0
            for m, scores in self.scores.items()
        }
        return max(avg_scores, key=avg_scores.get)

Common Pitfalls

Pitfall 1: Peeking at Results

Checking results daily and stopping early when you see a "significant" result invalidates your statistical tests. Pick a duration in advance and stick to it.

Pitfall 2: Multiple Comparisons

If you test 20 metrics, one will appear significant by chance alone (p < 0.05). Focus on 2-3 primary metrics and use Bonferroni correction for secondary metrics.

Pitfall 3: Seasonality Bias

Running a test during a holiday or product launch skews results. Always include at least one full business week.

Pitfall 4: Ignoring Interaction Effects

A model that works well for short queries might fail on long ones. Segment your analysis by query length, complexity, and user type.

Frequently Asked Questions

How long should I run an A/B test?

Minimum 1 week for business applications, 3 days for high-traffic consumer apps. You need at least 10,000 samples per variant for statistical power.

What if my treatment is worse?

That is valuable data. You have validated that the current model is optimal for your use case. Document the findings and revisit in 3-6 months when new models are released.

Can I A/B test more than two models?

Yes. Use ANOVA for the overall comparison, then pairwise tests for specific comparisons. TokenEase supports routing to any number of models.

Do I need custom infrastructure?

No. TokenEase's unified API means your A/B test infrastructure only needs to change the model parameter. No separate API keys, no SDK switching, no format conversion.