GenAI Inference Engineering
4 episodes — short audio overviews on genai inference engineering.
Production Hosted LLM Architecture — chapter audio overview
Understanding API-Based Inference Systems
Build a Live Provider Benchmarker — lab audio overview
Build a benchmarker that calls all three LLM provider APIs through proxies, measures latency and token consumption, computes costs from actual usage, and implements weighted scoring and ranking. Complete the TODO implementations to pass all tests.
Build a Token Economics Engine — lab audio overview
Build a token economics engine that calculates costs across providers using actual token counts from real API calls, tracks budget consumption, and generates cost optimization suggestions. Complete the TODO implementations to pass all tests.
Build a Provider-Aware Rate Limit Controller — lab audio overview
Build a rate limit controller that tracks API usage across providers using token bucket algorithm, monitors provider health via response timing, and implements smart request routing. Complete the TODO implementations to pass all tests.