Build scoring API endpoints with batch evaluation and ranking — lab audio overview

Host: Welcome back. You're in AI Solution Delivery — a course about turning AI ideas into working client engagements. This chapter is about use case discovery and data readiness — the very first conversations you have with a client before any model gets built. Today's exercise is the final one in a three-part sequence, and it's where everything you've built so far becomes something your whole team can actually use. Expert: Picture this. You're a solution architect at a consulting firm. Your team runs discovery workshops with big enterprise clients — a bank, a hospital network, a manufacturer. In each workshop, the client throws out fifteen, twenty AI ideas. "Can we automate claims? Can we predict machine failure? Can we summarize contracts?" Your job is to rank those ideas — which ones are feasible, which have the data to support them, which will actually deliver business value. Right now, your team does this in spreadsheets. One analyst scores one workshop at a time. It takes days. Every analyst scores a little differently. The output is inconsistent, slow, and hard to compare across clients. What you need is a shared service — something any consultant on your team, or any internal tool, can send a list of use cases to and get back a ranked, scored evaluation. That's what you're building today. Host: So this is exercise three of three, and it's the capstone. Let's connect the dots. In the first exercise, you defined the data shapes — the structured forms that describe a use case and its score. In the second exercise, you built the scoring brain — the piece that takes one use case and uses a large language model to evaluate it across multiple weighted criteria. Now, what are we doing in this last exercise? Expert: Now you're putting a front door on that scoring brain. Specifically, you're building what's called a REST API — and let me unpack that. REST stands for Representational State Transfer, but in practice, it just means a way for one program to talk to another over the web using standard web addresses. You send a request to a web address, you get a structured answer back. The tool you'll use to build this is called FastAPI — it's a popular open-source framework in Python for building these web services quickly. You'll create three entry points. The first one takes a single use case description and returns a full evaluation. The second one takes a whole list of use cases — maybe twenty at once from a workshop — evaluates all of them, and returns them ranked from best to worst. The third one lets someone view and adjust the scoring weights — for example, if a client cares more about data quality than business value, you can tune the weights to reflect that. The key idea here is separation. The scoring logic you already built doesn't change. You're wrapping it so other people, other tools, other parts of the company can use it without needing to understand how it works inside. Host: That makes sense. Now — before someone starts coding — what's the tricky part? What trips people up on this one? Expert: The batch endpoint. That's where the trouble lives. When someone sends you twenty use cases to evaluate at once, each one requires a call to a large language model, and each of those calls takes a few seconds and costs money. If you evaluate them one by one in a simple loop, your user waits thirty, forty, sixty seconds — and the connection might even time out. So you need to think about doing them in parallel — meaning, firing off multiple evaluations at the same time and waiting for all of them to come back together. But parallel work brings its own headaches. You can overwhelm the language model provider and get rate-limited — that's when they start rejecting your requests because you're sending too many too fast. So the tip is: build the single-use-case endpoint first, get it rock solid, and only then layer the batch endpoint on top of it with sensible limits — cap how many run at once, and add graceful error handling so one failed evaluation doesn't kill the whole batch. Also, sort the results by score before returning them, because ranking is the whole point of batch mode. Host: Great warning. So what will the listener walk away able to do? Expert: After this, you'll be able to take a piece of AI logic — any piece, really, not just scoring — and expose it as a reliable web service that your whole organization can call. You'll know how to handle both single requests and batch requests, how to let clients tune the behavior through configuration, and how to make the whole thing fast and resilient under load. For your team, this is a huge unlock. That internal tool the consulting practice uses to rank use cases? It can now plug into your service. The client-facing dashboard? It can call your service. The next AI agent you build that needs to score its own ideas? Same thing. You've built a reusable building block. Host: And since this is the final exercise in the objective, that means the whole thing is now yours — the data shapes, the scoring engine, and the web service that exposes it. You have a complete, working use case scoring system you can bring back to your team's architecture discussions. That's real production infrastructure, not a demo. Go build it. Thanks for listening.