Measure baseline failure rates across OpenAI, Anthropic, and Google providers

Introduction

When you page on-call at 3 a.m. because OpenAI returned 8% errors for the last hour, the first question is always the same: is that abnormal, or is that just Tuesday? Teams that operate multi-provider GenAI gateways without per-provider baselines end up either swamped by false alarms — every minor rate-limit blip pages someone — or blind to slow degradations that quietly poison output quality for days. A baseline is the statistical signature of "normal" for each provider and each failure category; without it, every alert threshold is a guess. By the end of this lesson you will be able to instrument an LLM gateway to collect per-provider failure observations, compute rolling baselines per failure category across OpenAI, Anthropic, and Google, and detect anomalies that justify escalation.

Key Terminology

• Failure rate — the ratio of failure events to total request events in a time window, segmented by provider and failure category; this is the quantity you are baselining and the input to every anomaly check in the lesson.
• Baseline window — the trailing time horizon (7 days is the lesson's default) over which the rolling mean and standard deviation of failure rate are computed; choosing it trades responsiveness against stability.
• Granularity bucket — the time resolution for each rate sample (15 minutes here); too small inflates cardinality, too large hides incidents.
• Anomaly threshold — the sigma multiplier (2.5 in this lesson) above the baseline mean at which the current bucket is flagged; tightens or loosens noise without changing the underlying baseline.
• Failure category — one of the five taxonomy buckets (provider, quality, cost, security, data) under which each event is classified before being counted, so baselines are per-provider AND per-category.

Concepts

Why fixed thresholds break across providers

A single "alert at 5% errors" rule fires constantly for OpenAI's rate-limit-heavy traffic, never fires for Anthropic's lower-volume tail-latency incidents, and entirely misses Google's IAM-token-refresh failures because those don't look like ordinary HTTP errors. Each provider has its own statistical signature. The fix is per-provider, per-category baselines computed over a rolling window — five baseline tracks per provider, one for each failure category in the taxonomy.

Bucketing, baselines, and sigma-based anomalies

The collector aggregates every gateway response into 15-minute buckets keyed by provider. The analyzer computes a rolling mean and standard deviation of the bucket rates over the trailing 7 days. A current bucket is anomalous when its rate exceeds mean + 2.5 * stddev. This is the standard control-chart formulation: it adapts to each provider's noise floor without hand-tuning thresholds (see Code Walkthrough).

Drift versus spikes

Sigma anomalies catch spikes. They miss silent quality drift — the week-over-week creep where Anthropic's hallucination rate ticks from 2% to 3% to 4% without any single bucket breaking the threshold. Drift detection compares this week's baseline against last week's; a large shift in the baseline itself is the signal, not any individual bucket.

Cross-provider correlation

Once a per-provider anomaly fires, the next question is whether it is isolated or systemic. If OpenAI, Anthropic, and Google all spike provider failures within the same 15-minute window, the cause is almost certainly downstream (your gateway, your network, your auth) — not three independent vendor outages. A correlation pass over current anomalies separates "switch traffic to a healthy provider" from "page the platform team."

Pipeline shape

Code Walkthrough

The walkthrough combines collection, baseline computation, and anomaly detection into a single end-to-end snippet so you can trace one event from gateway response to anomaly verdict.

Code snippetpython
1import statistics
2from dataclasses import dataclass
3from datetime import datetime, timezone, timedelta
4from collections import defaultdict
5from threading import Lock
6from enum import Enum
7from typing import Optional
8
9class FailureCategory(Enum):
10    PROVIDER = "provider"
11    QUALITY = "quality"
12    COST = "cost"
13    SECURITY = "security"
14    DATA = "data"
15
16class Provider(Enum):
17    OPENAI = "openai"
18    ANTHROPIC = "anthropic"
19    GOOGLE = "google"
20
21@dataclass
22class Observation:
23    provider: Provider
24    category: Optional[FailureCategory]
25    timestamp: datetime
26    is_failure: bool
27
28class ProviderBaselineCollector:
29    def __init__(self, bucket_minutes: int = 15):
30        self._bucket_minutes = bucket_minutes
31        self._buckets: dict[str, list[Observation]] = defaultdict(list)
32        self._lock = Lock()
33
34    def _bucket_key(self, provider: Provider, ts: datetime) -> str:
35        truncated = ts.replace(
36            minute=(ts.minute // self._bucket_minutes) * self._bucket_minutes,
37            second=0, microsecond=0,
38        )
39        return f"{provider.value}:{truncated.isoformat()}"
40
41    def record_event(
42        self, provider: Provider,
43        failure_category: Optional[FailureCategory] = None,
44    ) -> None:
45        ts = datetime.now(timezone.utc)
46        obs = Observation(
47            provider=provider, category=failure_category,
48            timestamp=ts, is_failure=failure_category is not None,
49        )
50        with self._lock:
51            self._buckets[self._bucket_key(provider, ts)].append(obs)
52
53    def bucket_rates(
54        self, provider: Provider, window_hours: int = 168,
55    ) -> list[float]:
56        cutoff = datetime.now(timezone.utc) - timedelta(hours=window_hours)
57        prefix = provider.value + ":"
58        rates: list[float] = []
59        with self._lock:
60            for key, obs_list in sorted(self._buckets.items()):
61                if not key.startswith(prefix) or not obs_list:
62                    continue
63                if obs_list[0].timestamp < cutoff:
64                    continue
65                total = len(obs_list)
66                failures = sum(1 for o in obs_list if o.is_failure)
67                rates.append(failures / total)
68        return rates
69
70class FailureRateAnalyzer:
71    def __init__(self, collector: ProviderBaselineCollector, sigma: float = 2.5):
72        self._collector = collector
73        self._sigma = sigma
74
75    def baseline(self, provider: Provider) -> tuple[float, float]:
76        rates = self._collector.bucket_rates(provider)
77        if len(rates) < 2:
78            return 0.0, 0.0
79        return statistics.mean(rates), statistics.stdev(rates)
80
81    def is_anomalous(self, provider: Provider, current_rate: float) -> bool:
82        mean, stddev = self.baseline(provider)
83        return current_rate > mean + self._sigma * stddev
84
85    def correlate(self, current: dict[Provider, float]) -> str:
86        anomalies = [p for p, r in current.items() if self.is_anomalous(p, r)]
87        if not anomalies:
88            return "normal"
89        return "cross_provider" if len(anomalies) > 1 else f"isolated:{anomalies[0].value}"

You'll know it works when feeding a week of synthetic gateway events into record_event produces a non-zero baseline per provider, is_anomalous flags a deliberately injected 20% failure spike, and correlate returns "cross_provider" only when you spike two providers in the same window.

Do's and Don'ts

Do's

1. ✓Do baseline per provider AND per failure category — a single rolled-up rate hides which of the five categories is degrading, and the remediation for a quality drift is nothing like the remediation for a cost spike.
2. ✓Do use a rolling window — provider behavior shifts with model updates and rate-limit changes; a static baseline set at deployment is stale within weeks.
3. ✓Do run drift detection alongside spike detection — silent week-over-week creep never trips a sigma threshold but is exactly how undetected regressions reach production.

Don'ts

1. ✗Don't apply one global threshold across providers — OpenAI's 429-heavy noise floor and Anthropic's tail-latency profile are not comparable; uniform thresholds always over- or under-fire.
2. ✗Don't alert on a single anomalous bucket — 15-minute buckets are noisy; require two or three consecutive anomalies, or correlate across providers, before paging.
3. ✗Don't let record_event raise — it sits on the gateway hot path; swallow exceptions and degrade to an "unclassified" category rather than disrupting request flow.