Build cost-performance analysis across providers

Introduction

When you pick an LLM by eyeballing a leaderboard, you optimize for a number that no finance team ever pays: raw accuracy. A model that scores two points higher but costs four times as much per call can quietly turn a profitable feature into a loss the moment traffic scales from a demo's 1K requests/day to production's 100K. The missing discipline is joint reasoning over accuracy and spend — knowing not just which model is best, but which model is dominated (someone else is both cheaper and better) and which is genuinely worth its price. By the end of this lesson you'll be able to compute cost_per_request and cost_per_correct_answer for every provider/model combination, latency percentiles at p50/p95/p99, build a Pareto frontier that filters out dominated models, project monthly spend across traffic tiers, and implement a CostOptimizer that names the cheapest model clearing a minimum-accuracy bar.

Key Terminology

• PriceCard — the per-model pricing record holding input_price_per_1k and output_price_per_1k (USD); the sole source of dollar figures, kept separate from eval results so a vendor price change never means re-running the harness.
• ModelProfile — the joined record for one provider/model combination carrying its measured accuracy, a list of per-request latencies_ms, and mean input_tokens / output_tokens; the unit every downstream calculation consumes.
• cost_per_request — input_tokens × input_price + output_tokens × output_price, the expected dollar cost of a single call, independent of whether the answer was correct.
• cost_per_correct_answer — cost_per_request / accuracy, the true unit economics: what you pay for one useful answer once wrong answers (which you often retry or discard) are amortized in.
• Pareto frontier — the subset of models that are non-dominated; a model is dominated when another exists that is simultaneously cheaper and at least as accurate, so the frontier is the only set worth choosing from.
• CostOptimizer — the selector that, given a minimum accuracy threshold, returns the cheapest ModelProfile clearing it — automating the "good enough, cheapest" decision.

Concepts

Now that we have the vocabulary, the analysis is a pipeline: eval output plus pricing produces per-model metrics, metrics feed a frontier filter, and the frontier feeds both projections and the optimizer.

Metrics turn tokens into economics

Accuracy comes from the eval harness; latency comes from timing each call. Neither is money until you fold in the PriceCard. cost_per_request weights each direction independently because output tokens are typically three-to-five times pricier than input tokens — a verbose model can cost more than a "more expensive" one that answers tersely. cost_per_correct_answer then divides by accuracy so a model that is cheap-but-wrong is penalized honestly: at 50% accuracy you pay for two calls to bank one correct answer.

Latency percentiles, not averages

A mean latency hides the tail that governs user experience and timeout budgets. Computing p50, p95, and p99 from the raw latencies_ms list via percentile exposes whether a model is usually fast but occasionally catastrophic — the p99 is what trips your SLA, not the mean.

The frontier and the optimizer

Building on those metrics, pareto_frontier discards every dominated model so a chart of accuracy versus cost_per_request shows only real tradeoffs. monthly_projection scales cost_per_request by traffic tiers (1K/10K/100K requests/day × 30) to make the spend concrete, and CostOptimizer.recommend walks the frontier for the cheapest model above a floor.

Code Walkthrough

Building on the metrics-then-frontier pipeline just described, the first module below defines the PriceCard and ModelProfile records and the three metric functions — cost_per_request, cost_per_correct_answer, and percentile — that convert raw eval and pricing data into economics for a single provider/model combination.

1from dataclasses import dataclass, field
2from statistics import quantiles
3
4@dataclass
5class PriceCard:
6    input_price_per_1k: float   # USD per 1,000 input tokens
7    output_price_per_1k: float  # USD per 1,000 output tokens
8
9@dataclass
10class ModelProfile:
11    provider: str
12    model_id: str
13    accuracy: float             # 0.0–1.0 from the eval harness
14    latencies_ms: list[float]
15    input_tokens: float         # mean per request
16    output_tokens: float        # mean per request
17    price: PriceCard
18
19def cost_per_request(p: ModelProfile) -> float:
20    return (
21        p.input_tokens / 1000 * p.price.input_price_per_1k
22        + p.output_tokens / 1000 * p.price.output_price_per_1k
23    )
24
25def cost_per_correct_answer(p: ModelProfile) -> float:
26    if p.accuracy <= 0:
27        return float("inf")
28    return cost_per_request(p) / p.accuracy
29
30def percentile(values: list[float], pct: float) -> float:
31    if len(values) == 1:
32        return values[0]
33    cuts = quantiles(values, n=100, method="inclusive")
34    return cuts[int(pct) - 1]

Having defined the per-model metrics, the second module computes the pareto_frontier (dropping dominated models), projects spend with monthly_projection, and implements CostOptimizer.recommend to name the cheapest model above an accuracy floor.

1def pareto_frontier(profiles: list[ModelProfile]) -> list[ModelProfile]:
2    frontier = []
3    for candidate in profiles:
4        dominated = any(
5            other is not candidate
6            and cost_per_request(other) <= cost_per_request(candidate)
7            and other.accuracy >= candidate.accuracy
8            and (cost_per_request(other) < cost_per_request(candidate)
9                 or other.accuracy > candidate.accuracy)
10            for other in profiles
11        )
12        if not dominated:
13            frontier.append(candidate)
14    return frontier
15
16def monthly_projection(p: ModelProfile, requests_per_day: int) -> float:
17    return cost_per_request(p) * requests_per_day * 30
18
19class CostOptimizer:
20    def __init__(self, profiles: list[ModelProfile]):
21        self._frontier = pareto_frontier(profiles)
22
23    def recommend(self, min_accuracy: float) -> ModelProfile | None:
24        eligible = [p for p in self._frontier if p.accuracy >= min_accuracy]
25        if not eligible:
26            return None
27        return min(eligible, key=cost_per_request)

You'll know it works when pareto_frontier drops any model you can hand-verify as dominated, and CostOptimizer(profiles).recommend(0.85) returns the cheapest frontier model at or above 85% accuracy — or None when none qualify.

Do's and Don'ts

Having walked through the metric functions, the frontier filter, and the optimizer, the following Do's and Don'ts distill them into daily practice.

Do's

1. ✓Do rank on cost_per_correct_answer, not cost_per_request — a cheap model at 60% accuracy costs more per useful answer than a pricier one at 95% once you account for retries and discards.
2. ✓Do choose exclusively from the pareto_frontier — any model off the frontier is strictly beaten on both price and accuracy, so selecting one is never justified.
3. ✓Do report p95 and p99 from percentile, not the mean latency — the tail is what breaches timeouts and SLAs, and the mean hides it.

Don'ts

1. ✗Don't hardcode dollar figures into ModelProfile — keep them in PriceCard so a vendor price change is a one-line edit that never invalidates your eval accuracy data.
2. ✗Don't compare monthly_projection outputs across models without holding token means constant — a model that emits longer answers inflates output_tokens and its projection independent of quality.
3. ✗Don't let CostOptimizer.recommend fall back to a default when it returns None — "no model clears the accuracy floor" is a real answer that should surface the gap, not silently ship an under-qualified model.