Detect cost anomalies and spending spikes

Introduction

When you set strict budget ceilings but spend still explodes from sources hidden under the cap, you have a detection gap. Budget enforcement stops obvious overages but cannot catch unexpected patterns that stay below the limit — a misconfigured retry loop hammering a cheap model, an unauthorized upgrade quietly doubling cost-per-request, or per-user drift that no one notices until invoice day. Miss these and a single bad weekend can burn through a quarter's discretionary budget before anyone pages on-call. By the end you'll be able to compute rolling baselines, score current spend with Z-scores, classify anomalies by axis and severity, and route alerts before the damage compounds.

Key Terminology

• Baseline — the rolling mean and standard deviation of recent daily costs that defines "normal" for a user or team; matters because every Z-score in this lesson is computed against it, so a bad baseline corrupts every downstream alert.
• Z-score — the number of standard deviations a current value sits from the baseline mean; matters because it normalizes spend across users so a single sigma threshold works for both light and heavy spenders.
• Cold start — the window where a new user lacks enough history for a personal baseline; matters because naive detection produces zero alerts for new accounts (silent false-negatives) until the team-average fallback kicks in.
• Anomaly poisoning — distortion that occurs when confirmed spikes get folded back into baseline statistics; matters because the inflated standard deviation permanently raises the detection threshold, hiding future incidents of the same shape.
• Cost-per-request shift — a Z-score on spend / requests rather than on raw spend; matters because it catches model-mix changes (e.g. unauthorized upgrade from a small to a large model) even when total spend and request volume look normal.

Concepts

Rolling baselines per user and per team

Detection starts with a stable reference point. Compute a baseline from the last 14–30 days of per-user and per-team daily rollups, requiring at least 7 samples before the baseline is treated as valid. Day-of-week seasonality matters — weekday spend typically exceeds weekend spend — so either compare against same-day-of-week buckets or use a window long enough to absorb the weekly cycle. New users have no history; fall back to the team average for their first 14 days so they are not silently uncovered. (see Code Walkthrough)

Z-score scoring with severity bands

The Z-score (current - mean) / std collapses a noisy time series into one comparable number. Standard severity bands: medium at the sigma threshold (2.5), high above 3 sigma, critical above 4 sigma. Severity drives routing — medium posts to a Slack channel, critical pages on-call. Because Z-scores normalize by standard deviation, the same threshold works for a light user and a heavy team without per-cohort tuning.

Multi-axis anomaly types

A single spend check misses common failure modes. Run three independent Z-scores per evaluation: total spend (catches obvious spikes), request volume (catches retry loops dumping cheap-model calls), and cost-per-request (catches model-mix shifts where volume looks normal but each call costs more). Each anomaly type implies a different remediation, so emit a typed alert rather than a generic "spend high" event. (see Code Walkthrough)

Cold-start and anomaly-poisoning guards

Two failure modes break naive detection. Cold-start users with no history produce no alerts at all — solve with the team-baseline fallback above. Anomaly poisoning happens when a confirmed spike is folded back into the baseline; the standard deviation inflates permanently and future incidents of the same magnitude no longer cross the threshold. Exclude confirmed anomalies from baseline recomputation so the reference window keeps reflecting "normal," not "normal plus past incidents."

Code Walkthrough

Now that you have the four concepts in place — rolling baselines, Z-score severity bands, multi-axis detection, and cold-start guards — the snippet below wires them together into one evaluation pass.

Code snippetpython
1import statistics
2from dataclasses import dataclass
3
4@dataclass
5class Baseline:
6    mean: float
7    std: float
8    sample_size: int
9
10def compute_baseline(daily_values: list[float], min_samples: int = 7) -> Baseline:
11    if len(daily_values) < min_samples:
12        return Baseline(mean=0.0, std=0.0, sample_size=0)
13    return Baseline(
14        mean=statistics.mean(daily_values),
15        std=statistics.stdev(daily_values),
16        sample_size=len(daily_values),
17    )
18
19def detect_anomaly(current: float, baseline: Baseline, sigma: float = 2.5) -> dict:
20    if baseline.sample_size == 0 or baseline.std == 0:
21        return {"anomaly": False, "reason": "insufficient_baseline"}
22    z = (current - baseline.mean) / baseline.std
23    is_anomaly = z > sigma
24    severity = "critical" if z > 4.0 else "high" if z > 3.0 else "medium"
25    return {
26        "anomaly": is_anomaly,
27        "z_score": round(z, 2),
28        "severity": severity if is_anomaly else None,
29        "current": round(current, 2),
30        "expected": f"{baseline.mean:.2f} +/- {baseline.std:.2f}",
31    }
32
33def check_all_anomaly_types(current: dict, history: dict) -> list[dict]:
34    cpr = current["spend"] / max(current["requests"], 1)
35    axes = [
36        ("spend_spike",      current["spend"],   history.get("daily_spend", [])),
37        ("volume_anomaly",   current["requests"], history.get("daily_requests", [])),
38        ("cost_per_request", cpr,                history.get("daily_cpr", [])),
39    ]
40    alerts = []
41    for axis_name, current_val, hist in axes:
42        result = detect_anomaly(current_val, compute_baseline(hist))
43        if result["anomaly"]:
44            alerts.append({"type": axis_name, **result})
45    return alerts

compute_baseline enforces the 7-sample floor that protects against distorted thresholds during cold-start; when too few samples exist it returns a sentinel that detect_anomaly short-circuits on, surfacing insufficient_baseline instead of a false alert. detect_anomaly produces a typed dict that downstream routing keys on — medium goes to Slack, critical to on-call paging. check_all_anomaly_types loops three axes in one pass so a single evaluation surfaces every triggered anomaly, not just the first. You'll know it works when feeding a realistic 14-day baseline such as daily_spend=[9.1, 10.4, 8.7, 11.2, 9.8, 10.1, 8.9, 11.5, 9.4, 10.7, 8.5, 11.0, 9.6, 10.3] (mean ≈ 9.9, std ≈ 0.95) with current spend=50.0 returns a spend_spike alert with severity="critical" and z_score above 4.0.

Do's and Don'ts

Having just seen the detection logic in code, the rules below capture the operational gotchas that turn a working detector into one that actually catches incidents in production.

Do's

1. ✓Do require a minimum sample size before scoring — at least 7 daily samples; smaller windows produce distorted standard deviations that fire alerts on normal variation.
2. ✓Do run all three axes in parallel — spend, volume, and cost-per-request each catch different failure modes; a single check leaves blind spots that retry loops and unauthorized model upgrades exploit.
3. ✓Do exclude confirmed anomalies from baseline recomputation — folding spikes back into the mean and std permanently raises the threshold, creating silent false-negatives for the same incident shape.

Don'ts

1. ✗Don't alert on absolute dollar thresholds alone — a $50 spike is critical for a small user and noise for a heavy one; Z-scores normalize across cohorts so one rule works everywhere.
2. ✗Don't ignore cold-start users — fall back to the team baseline for the first 14 days; without it, new accounts are silently uncovered and runaway spend goes undetected.
3. ✗Don't wait for daily batch jobs to detect runaway loops — score against rolling 5-minute Redis counters for spend so automation incidents are caught before they burn thousands.