GenAI Inference Engineering
6 episodes — short audio overviews on genai inference engineering.
Production Hosted LLM Architecture — chapter audio overview
Understanding API-Based Inference Systems
GenAI CI/CD Pipelines — chapter audio overview
Build CI/CD pipelines using Argo Workflows for prompts, model configs, RAG configs, and guardrail policies.
Build a Live Provider Benchmarker — lab audio overview
Build a benchmarker that calls all three LLM provider APIs through proxies, measures latency and token consumption, computes costs from actual usage, and implements weighted scoring and ranking. Complete the TODO implementations to pass all tests.
Build a Token Economics Engine — lab audio overview
Build a token economics engine that calculates costs across providers using actual token counts from real API calls, tracks budget consumption, and generates cost optimization suggestions. Complete the TODO implementations to pass all tests.
Build a Provider-Aware Rate Limit Controller — lab audio overview
Build a rate limit controller that tracks API usage across providers using token bucket algorithm, monitors provider health via response timing, and implements smart request routing. Complete the TODO implementations to pass all tests.
GenAI Prompt CI/CD Pipeline with Argo Workflow Simulation — lab audio overview
Build a simulated Argo Workflow pipeline for prompt CI/CD operations including linting, validation, evaluation with LLM calls, and promotion stages. Implement pipeline orchestration with artifact storage and metrics tracking for GenAI operations.