Detect quality anomalies with OTEL sliding-window analysis

Introduction

In production, AI systems can degrade gradually — a hallucination rate that drifts from 2% to 5% over several weeks never crosses a hard SLA threshold, yet users are already experiencing worse outcomes. Anomaly detection closes this gap by comparing incoming metrics against a statistical baseline rather than a fixed limit, surfacing problems while there is still time to act. By the end of this lesson, you will be able to implement a sliding-window z-score detector that catches gradual metric drift early, giving your on-call team time to investigate before a quiet trend becomes a visible SLA breach.

Key Terminology

• Sliding window — a fixed-size circular buffer (deque(maxlen=window_size)) that accumulates recent metric readings and automatically evicts the oldest entry when full, keeping the historical baseline current without unbounded memory growth.
• Z-score — a dimensionless measure of how many standard deviations the recent mean deviates from the window mean; check_anomaly computes it as abs(recent_mean - mean) / std_dev and raises an anomaly when it exceeds the configured sensitivity.
• Dynamic baseline — the per-metric mean and standard deviation derived from the current contents of the sliding window, which adapts to each metric's natural operating range rather than relying on a hardcoded SLA limit.
• Metric drift — a gradual, sustained shift in a production signal (such as hallucination rate rising from 2% to 4%) that may stay below hard SLA ceilings yet indicates real system degradation; the z-score approach catches drift that fixed thresholds miss entirely.
• Sensitivity threshold — the z-score cutoff stored in AnomalyConfig.sensitivity (default 2.5) that controls how far the recent mean must deviate from the historical baseline before AnomalyResult.anomalous flips to True.
• Warm-up guard — the check in check_anomaly that requires at least window_size // 2 readings before computing a baseline, preventing spurious anomalies when the detector has seen too few data points to establish a meaningful mean and standard deviation.

Concepts

Why Fixed Thresholds Miss Gradual Degradation

A hallucination rate SLA set at 10% sounds protective — until your system quietly drifts from 2% to 5% over two weeks and the alert never fires. The metric has degraded by 150% relative to its healthy level, yet it sits nowhere near the hard limit. Users experience the decline; your monitoring does not. This is the core failure mode that anomaly detection addresses: static thresholds measure absolute position, not relative movement. A system that has always operated near 2% with tight variance is behaving very differently at 4% than a system whose baseline naturally fluctuates between 3% and 7% — yet a fixed threshold treats both identically.

The Sliding Window as a Per-Metric "Normal"

The solution is to define normal dynamically, per metric, from recent history. QualityAnomalyDetector maintains a separate deque(maxlen=window_size) for each metric name. As new readings arrive via ingest_metric, old ones are automatically evicted — the window always reflects the past window_size observations without storing anything older. The mean and standard deviation computed from this buffer constitute the baseline: they capture both the central value and the natural variability of that metric's healthy state. Because each metric gets its own buffer, a high-variance latency signal and a tight-variance hallucination rate each build their own sense of "normal" independently.

Z-Score as a Relative Deviation Signal

Once a baseline exists, check_anomaly measures how surprising the most recent ten readings are relative to the full window. The z-score — abs(recent_mean - mean) / std_dev — expresses that surprise in units of historical standard deviation. A metric with tight variance will produce a large z-score from a small absolute shift; a metric that naturally fluctuates widely will require a larger absolute shift before exceeding the same threshold. This relative scaling is exactly what makes the detector sensitive to drift that fixed thresholds ignore: a hallucination rate moving from 2% to 4% on a system with a standard deviation of 0.3% yields a z-score near 6.7 — well above the default sensitivity of 2.5 — even though 4% is far below a 10% SLA ceiling (see Code Walkthrough).

Warm-Up and the Minimum Data Requirement

A baseline computed from three readings is almost meaningless — a single outlier can produce a huge standard deviation or a near-zero one, and neither reflects real system behavior. The warm-up guard (len(buffer) < window_size // 2) enforces a minimum before check_anomaly will produce a verdict, returning AnomalyResult(anomalous=False, reason="Insufficient data") until enough history has accumulated. The drift simulation in the walkthrough respects this: it seeds 40 healthy readings before injecting the rising trend, ensuring the window baseline is well-established before the anomaly check runs.

Code Walkthrough

Now that you understand how sliding windows and z-scores form a statistical baseline for SLA drift detection, let's build a QualityAnomalyDetector that puts those ideas into practice.

The detector maintains a per-metric circular buffer using Python's deque with a fixed maximum length. When a new measurement arrives via ingest_metric, it is stored as a MetricPoint alongside its timestamp. Once the buffer is at least half full, check_anomaly computes the window mean and standard deviation, then compares the mean of the most recent ten readings against that historical baseline. A z-score above the configured sensitivity threshold signals an anomaly.

Code snippetpython
1from collections import deque
2from dataclasses import dataclass, field
3from datetime import datetime
4
5@dataclass
6class MetricPoint:
7    value: float
8    timestamp: datetime
9
10@dataclass
11class AnomalyResult:
12    anomalous: bool
13    reason: str = ""
14    z_score: float = 0.0
15    window_mean: float = 0.0
16    recent_mean: float = 0.0
17    threshold: float = 0.0
18
19@dataclass
20class AnomalyConfig:
21    window_size: int = 50
22    sensitivity: float = 2.5
23
24class QualityAnomalyDetector:
25    """Detects quality anomalies via sliding windows and z-score comparison."""
26
27    def __init__(self, config: AnomalyConfig):
28        self.window_size = config.window_size
29        self.sensitivity = config.sensitivity
30        self.metrics_buffer: dict[str, deque] = {}
31
32    def ingest_metric(self, metric_name: str, value: float) -> None:
33        if metric_name not in self.metrics_buffer:
34            self.metrics_buffer[metric_name] = deque(maxlen=self.window_size)
35        self.metrics_buffer[metric_name].append(
36            MetricPoint(value=value, timestamp=datetime.utcnow())
37        )
38
39    def check_anomaly(self, metric_name: str) -> AnomalyResult:
40        buffer = self.metrics_buffer.get(metric_name, deque())
41        if len(buffer) < self.window_size // 2:
42            return AnomalyResult(anomalous=False, reason="Insufficient data")
43
44        values = [p.value for p in buffer]
45        mean = sum(values) / len(values)
46        variance = sum((v - mean) ** 2 for v in values) / len(values)
47        std_dev = variance ** 0.5
48
49        recent = values[-10:]
50        recent_mean = sum(recent) / len(recent)
51        z_score = abs(recent_mean - mean) / (std_dev if std_dev > 0 else 1)
52
53        return AnomalyResult(
54            anomalous=z_score > self.sensitivity,
55            z_score=z_score,
56            window_mean=mean,
57            recent_mean=recent_mean,
58            threshold=self.sensitivity,
59        )

The critical design decision is using the window mean and standard deviation as a dynamic baseline rather than a hardcoded threshold. A hallucination rate sitting steadily at 2% with tight variance will produce a high z-score even when drift reaches 4% — well below a typical 10% SLA ceiling — giving your monitoring system time to escalate before users notice.

The block below simulates that gradual drift scenario: it seeds the buffer with 40 healthy readings, then injects a rising trend over the next 10 samples.

Code snippetpython
1import random
2
3detector = QualityAnomalyDetector(AnomalyConfig(window_size=50, sensitivity=2.5))
4
5# Seed with a healthy baseline: ~2% hallucination rate
6for _ in range(40):
7    detector.ingest_metric("hallucination_rate", random.gauss(0.02, 0.003))
8
9# Inject a gradual upward drift over the next 10 readings
10for i in range(10):
11    detector.ingest_metric("hallucination_rate", 0.02 + i * 0.004)
12
13result = detector.check_anomaly("hallucination_rate")
14print(f"Anomalous:   {result.anomalous}")
15print(f"Z-score:     {result.z_score:.2f}  (threshold: {result.threshold})")
16print(f"Window mean: {result.window_mean:.4f}   Recent mean: {result.recent_mean:.4f}")

Once the drift pushes the z-score past 2.5, result.anomalous flips to True — even though the absolute hallucination rate has not yet crossed any hard-coded SLA limit.

Confirm that result.anomalous prints True and the reported z-score exceeds your configured sensitivity threshold when you run the drift simulation.

Do's and Don'ts

Having walked through the material above, the following Do's and Don'ts distill it into practice.

Do's

1. ✓Do size the deque buffer with maxlen=window_size — this enforces automatic eviction of stale readings so the circular buffer always reflects your intended observation window, preventing historical outliers from distorting the computed mean and standard deviation.
2. ✓Do wait until the buffer is at least half full (window_size // 2) before calling check_anomaly — computing a z-score against a sparse window yields an unstable baseline, causing false positives that erode on-call trust in the detector.
3. ✓Do compare the recent_mean of the last ten readings against the full-window baseline rather than a single incoming value — this smooths transient noise so the z-score rises reliably only under sustained drift, matching the gradual hallucination-rate scenario the detector is designed to catch.

Don'ts

1. ✗Don't replace the z-score baseline with a hardcoded SLA threshold — a fixed limit like 10% hallucination rate will never fire on the drift from 2% to 5% described in the lesson, precisely the slow degradation that the dynamic window_mean and std_dev are designed to surface early.
2. ✗Don't set std_dev to zero when variance is flat — the guard (std_dev if std_dev > 0 else 1) in check_anomaly prevents a division-by-zero crash when a metric is perfectly stable, but removing or weakening that guard turns a healthy steady-state reading into an unhandled exception.
3. ✗Don't share a single metrics_buffer entry across different metric names — ingest_metric keys each deque by metric_name, so mixing hallucination_rate and a latency metric into the same buffer contaminates both baselines and produces meaningless z-scores for both signals.