Build provisioning latency histograms — lab audio overview

Host: Welcome back. You're in AI Developer Platform Engineering — a course about building the internal tools and services that let other engineers ship AI products faster. This chapter is called Internal Developer Platform Vision. The big idea is that instead of every team reinventing how to deploy, monitor, and provision their services, you build them a self-service platform — like a well-stocked toolshed where everything they need is already there. Today's skill is about measuring how fast that platform actually serves them. Expert: Picture a mid-sized AI company — maybe two hundred engineers split across fifteen product teams. Every one of those teams needs the same things to ship: a database, a model-serving endpoint, an API gateway, some storage. So the platform team builds a self-service portal where any engineer can click a button and get those things provisioned automatically. Sounds great. But six months in, the complaints start rolling in. "Provisioning is slow." "It used to be faster." "My deploy timed out." Now the platform team is in trouble, because they have no data. They don't know if provisioning takes two seconds or two minutes. They don't know if it's slow for everyone or just slow for one team. They don't know if it's getting worse over time, or only bad during peak hours. Without measurements, every complaint is a mystery. And every mystery costs hours of debugging. The skill you're about to build is the foundation for answering all of those questions with hard numbers instead of guesses. Host: That sets the stakes. In the previous exercise, you added basic measurements to the platform's web service — counters that tally up how many requests came in, how many failed, and rough timing for each one. Now you're going deeper on the timing piece. So what exactly will we build? Expert: You're going to build a tool that tracks how long provisioning operations take — and not just the average. Averages lie. If most requests take one second but one in twenty takes thirty seconds, the average looks fine while a chunk of your users are suffering. So instead of an average, you'll use something called a histogram. A histogram is just a set of buckets. You decide the boundaries — say, under one second, one to five seconds, five to thirty seconds, over thirty seconds — and every time a provisioning operation finishes, you drop it into the right bucket. Over time, you can look at the shape of those buckets and see the full distribution of how your platform is performing. From those buckets, you'll compute what are called percentiles. A percentile answers questions like: "What's the slowest experience for the fastest ninety-five percent of my users?" That number — the ninety-fifth percentile — is the gold standard in platform engineering, because it tells you what your typical bad experience looks like, not just your typical experience. And on top of that, you'll add alert evaluation: a way to say "if the ninety-fifth percentile goes above thirty seconds, something is wrong, page someone." So three pieces: configurable buckets, percentile math, and threshold-based alerting. Host: Before someone starts coding this — what's the part that trips people up? Expert: The bucket boundaries. It's tempting to just pick round numbers — one, ten, a hundred — and move on. But your buckets determine what you can actually see. If all your provisioning operations take between two and eight seconds, and your buckets jump from one to ten, every single operation lands in the same bucket and you've learned nothing. The fix is to think about your data first. What's the fastest realistic provisioning time? What's the slowest you'd tolerate? Pack more buckets in the range where things actually happen, and use exponentially growing widths — small buckets for fast operations, wider buckets for slow ones. The other gotcha is that percentile calculation from buckets is approximate. You're not getting the exact ninety-fifth percentile, you're getting an estimate based on which bucket the boundary falls in. That's fine — but know it's an estimate, and design your buckets to be precise where precision matters most. Host: Got it. So what will the listener walk away able to do? Expert: After this exercise, you'll be able to take any operation in your platform — provisioning, deployment, model loading, anything — and produce a real distribution of how long it takes, broken down into percentiles, with automatic alerts when performance degrades. This is the measurement backbone your team needs before any serious conversation about platform reliability. It's what turns "users say it feels slow" into "the ninety-fifth percentile crossed our threshold at three p.m. on Tuesday." Next up, you'll take all these measurements — the counters from the last exercise, the histograms from this one — and build them into visual dashboards using a tool called Grafana, which is the standard open-source way to turn metrics into charts that humans can actually read. That's where your platform's health story finally becomes visible to everyone. Thanks for listening, and good luck with the build.