10 skill groups · 11 courses · 1682 goals
GenAI Solutions Architecture
Design enterprise GenAI reference architectures, create ADRs and technical standards, bridge GenAI with enterprise workflows.
11 courses, each building on the last. The bar shows each course's share of the curriculum. 7 goals are unlocked for preview.
Python Essentials for Agent Builders3.8%
Your Dev EnvironmentNavigate filesystem with terminal · Manage files from command line · Set up VS Code · Configure terminal in VS Code
Python, Git & Package ManagementInstall and verify Python · Write hello world script · Use Python REPL · Initialize Git repository · Track changes with Git · Install packages with pip
Variables & Basic TypesCreate and name variables · Work with strings · Work with numbers · Work with booleans · Format with f-strings
Control FlowMake decisions with if/elif/else · Iterate with for loops · Repeat with while loops · Control loop execution
FunctionsDefine and call functions · Use parameters · Return values · Document with docstrings · Understand scope
Modules & ImportsImport standard library · Create custom modules · Understand Python path · Create packages
Lists & TuplesCreate and access lists · Modify lists · Slice lists · Use list comprehensions · Work with tuples
Dictionaries & SetsCreate and access dicts · Modify dictionaries · Iterate over dicts · Work with nested dicts · Use sets
Classes & DataclassesUnderstand class basics · Create dataclasses · Add methods · Use default values · Basic inheritance
Files, JSON & Error HandlingRead and write files · Work with JSON · Use pathlib · Handle exceptions · Create custom exceptions
Basic TestingUse assert statements · Create test functions · Run pytest · Test classes
Environment Variables & ConfigurationUnderstand environment variables · Use .env files · Load with python-dotenv · Handle missing variables · Organize configuration
Decorators & Context ManagersUnderstand decorators · Write simple decorators · Use context managers · Write context managers · Combine patterns
LLM Foundations for Agent Builders5.3%
Generators & IteratorsUnderstand iteration · Create generators · Use generator expressions · Build data pipelines · Use itertools
Async Programming BasicsUnderstand async concepts · Write async functions · Run concurrent operations · Use async context managers · Handle async exceptions
Type Hints & PydanticAdd basic type hints · Use typing generics · Create Pydantic models · Validate API data · Configure Pydantic
Data Pipelines & TransformationsBuild functional pipelines · Work with tabular data · Transform data shapes · Process LLM data formats · Optimize for performance
HTTP Clients & httpxMake GET requests · Make POST requests · Use async httpx · Handle errors · Use sessions
Your First LLM CallSet up credentials · Install Gemini SDK · Make first API call · Parse response · Handle API errors
Tokenizer InternalsUnderstand tokenization basics · Learn BPE algorithm · Compare tokenizer types · Analyze cross-language efficiency · Count and optimize tokens
Context Windows, KV-Cache & MemoryUnderstand context limits · Understand KV-cache basics · Use prompt caching · Track conversation tokens · Implement truncation · Summarize for compression
LLM Architectures - Dense, MoE & KV-Cache OptimizationsUnderstand transformers · Explore dense models · Understand MoE · Master KV-cache optimizations · Compare architectures · Choose for task
LLM Inference - Prefill & DecodeUnderstand prefill phase · Understand decode phase · Run local models · Measure inference metrics · Optimize for latency
Sampling Parameters & Output ControlUnderstand temperature · Use top-p sampling · Implement determinism · Control output length · Use structured output
Multi-Provider & Prompt EngineeringBuild provider abstraction · Structure conversations · Use few-shot prompting · Implement chain-of-thought · Build prompt templates
Function Calling FundamentalsUnderstand tool use concept · Define tool schemas · Make function calls · Handle tool responses · Compare provider patterns
Embeddings & Semantic SearchUnderstand embeddings · Generate embeddings · Calculate similarity · Build simple search · Compare embedding models
RAG FundamentalsUnderstand RAG pattern · Chunk documents · Build retrieval pipeline · Compose RAG prompts · Evaluate RAG quality
Cost Awareness & Token EconomicsUnderstand pricing models · Calculate request costs · Compare provider costs · Identify cost drivers · Basic cost optimization
Retry Patterns with TenacityUnderstand retry need · Use tenacity basics · Implement exponential backoff · Handle specific exceptions · Combine with async
Kubernetes Essentials for GenAI Engineers4.4%
Containerizing LLM ApplicationsWrite a Python app that calls the Gemini API and returns structured responses · Write a Dockerfile and build a container image for the LLM app · Run the containerized LLM app with environment-based configuration · Use Docker Compose to run the LLM app with supporting services · Tag images with semantic versions and push to a container registry · Debug containers with exec, logs, and inspect
Your Kubernetes Cluster & First LLM PodUnderstand K8s architecture and connect to your vCluster · Deploy the LLM app as your first Kubernetes pod · Organize workloads with namespaces · Use labels and selectors to organize and query resources · Understand pod lifecycle and restart policies · Master kubectl debugging: exec, logs, describe, port-forward
Services & the LLM Chat BackendCreate a ClusterIP service to expose the LLM chat API internally · Deploy a multi-tier LLM chat application · Compare service types: ClusterIP, NodePort, LoadBalancer · Master DNS-based service discovery in Kubernetes · Understand endpoints and traffic routing · Debug service connectivity problems
Deployments, Scaling & Rolling UpdatesCreate a Deployment for the LLM chat API · Scale LLM app replicas to handle concurrent requests · Perform a rolling update with zero downtime · Roll back a broken deployment · Compare deployment strategies: RollingUpdate vs Recreate · Manage deployment lifecycle with kubectl rollout
ConfigMaps & Secrets for LLM AppsCreate ConfigMaps for LLM app settings · Mount ConfigMaps as files for complex configuration · Store LLM proxy credentials securely in Secrets · Manage per-environment configuration for dev, staging, and prod · Handle configuration updates and rolling restarts · Debug configuration issues in LLM app pods
Persistent Storage & StatefulSetsCreate PersistentVolumeClaims for durable storage · Deploy PostgreSQL as a StatefulSet · Connect the LLM chat API to PostgreSQL for conversation persistence · Deploy Redis as a StatefulSet for LLM response caching · Understand StatefulSet scaling and ordering guarantees · Manage PVC lifecycle: expansion, snapshots, and cleanup
Multi-Container Pods: Sidecars & Init ContainersAdd an LLM proxy sidecar to the chat API pod · Use init containers for database setup and config loading · Share data between containers via emptyDir volumes · Implement the ambassador pattern for multi-model LLM routing · Add a logging and metrics sidecar to the LLM app · Debug multi-container pods
Resource Management & Cost OptimizationSet resource requests and limits for the LLM chat API · Understand QoS classes and their impact on eviction · Enforce resource defaults with LimitRanges · Cap namespace resource usage with ResourceQuotas · Right-size LLM app containers based on actual usage · Diagnose OOMKilled and CPU throttling issues
Packaging with Helm & KustomizeCreate a Helm chart for the LLM chat application · Parameterize the chart with values.yaml for each environment · Manage Helm release lifecycle: install, upgrade, rollback · Use Kustomize bases and overlays for the LLM app · Use Kustomize patches and generators · Compare Helm vs Kustomize for different deployment scenarios
Networking, Ingress & TLSExpose the LLM chat API via an Ingress resource · Add TLS to the Ingress for HTTPS access · Isolate services with NetworkPolicies · Configure Ingress annotations for production traffic · Understand K8s networking: pod IPs, CNI, and service routing · Debug networking and connectivity issues
Health Probes, Autoscaling & Self-HealingAdd liveness and readiness probes to the LLM chat API · Configure startup probes for containers with slow initialization · Scale the chat API automatically with HPA based on CPU · Create PodDisruptionBudgets for safe maintenance · Implement health check patterns for LLM-dependent services · Combine autoscaling, probes, and PDBs for a resilient LLM service
RBAC, Security & K8s TroubleshootingCreate RBAC roles for the LLM chat application · Enforce Pod Security Standards · Apply SecurityContext for defense in depth · Debug CrashLoopBackOff and OOMKilled failures · Use kubectl debug and ephemeral containers for live debugging · Troubleshoot LLM-specific issues: timeouts, proxy errors, stale connections
Web APIs & Services for GenAI Engineers3.7%
FastAPI FundamentalsCreate a FastAPI application with path operations · Define Pydantic request and response models · Implement dependency injection for shared resources · Build CRUD endpoints with proper HTTP semantics · Configure OpenAPI documentation with examples · Handle errors with custom exception handlers
Async Python for APIsConvert sync endpoints to async with proper await patterns · Implement background tasks for non-blocking operations · Execute concurrent API calls with asyncio.gather · Manage application lifecycle with lifespan handlers · Build async generators for streaming responses · Control concurrency with semaphores and throttling
Database IntegrationConfigure SQLAlchemy async engine with connection pooling · Define ORM models with relationships and constraints · Create and manage database migrations with Alembic · Implement repository pattern for data access · Build transactional endpoints with session lifecycle · Implement filtering, sorting, and full-text search
Authentication & AuthorizationImplement user registration with password hashing · Build OAuth2 password flow with JWT tokens · Implement API key authentication for services · Enforce role-based access control with permissions · Build token refresh and revocation · Compose multiple auth strategies into dependencies
Real-time StreamingBuild SSE endpoint for streaming LLM responses · Implement WebSocket endpoint with connection lifecycle · Build WebSocket connection manager for broadcasting · Handle backpressure and slow clients · Implement heartbeat and automatic reconnection · Build real-time notification system with Redis pub/sub
Resilience PatternsImplement rate limiting with Redis sliding window · Build circuit breaker for LLM provider calls · Configure retry logic with tenacity · Isolate critical paths with bulkhead semaphores · Build fallback responses for degraded mode · Combine resilience patterns into middleware stack
API Gateway & RoutingBuild reverse proxy with path-based routing · Implement load balancing across backend instances · Transform requests and responses through the gateway · Aggregate responses from multiple backends · Implement service discovery with health checking · Build gateway authentication and request enrichment
Testing & DocumentationWrite async endpoint tests with httpx.AsyncClient · Build database fixtures with transaction rollback · Mock external services for deterministic tests · Implement contract tests for API consumers · Measure test coverage and set quality gates · Generate rich OpenAPI documentation with examples
API Versioning & EvolutionImplement URL-based API versioning with routers · Build header-based version negotiation · Manage deprecation with Sunset and Warning headers · Build request and response adapters for version translation · Detect breaking changes automatically · Generate API changelogs from schema diffs
Deployment & ObservabilityBuild production Docker images with multi-stage builds · Deploy to Kubernetes with health check probes · Instrument endpoints with Prometheus metrics · Implement distributed tracing with OpenTelemetry · Build structured logging with correlation IDs · Create Grafana dashboards for API monitoring
Data Infrastructure Essentials for GenAI3.7%
PostgreSQL & pgvectorInstall pgvector and create vector-enabled tables · Build HNSW and IVFFlat indexes for fast similarity search · Perform similarity search with distance operators · Build hybrid search combining vectors and metadata · Integrate pgvector with SQLAlchemy ORM · Build a semantic search API endpoint
Advanced PostgreSQLPartition tables by range for time-series AI data · Query and index JSONB for AI metadata · Build full-text search for document retrieval · Optimize queries with EXPLAIN ANALYZE · Implement connection pooling with PgBouncer · Build a database migration strategy for AI schemas
Redis for Caching & SessionsImplement cache-aside pattern for LLM responses · Build session storage for multi-turn conversations · Use Redis pub/sub for real-time event broadcasting · Implement distributed locks for concurrent operations · Build rate limiting with Redis sorted sets · Monitor Redis performance and memory
MinIO Object StorageCreate MinIO buckets and configure access policies · Upload and download with presigned URLs · Implement versioned storage for datasets · Build multipart uploads for large model files · Build content-addressable storage for embeddings · Monitor MinIO health and storage metrics
Kafka FundamentalsCreate topics with partition and replication strategies · Produce messages with key-based partitioning · Consume messages with consumer groups · Implement message serialization with schemas · Handle delivery guarantees and idempotency · Monitor Kafka with consumer lag metrics
Event-Driven ArchitecturesBuild an event-driven inference pipeline · Implement event sourcing for prediction audit trails · Design dead letter queues for failed processing · Build stream processing for real-time enrichment · Implement the saga pattern for multi-step workflows · Implement event replay for reprocessing
Neo4j Graph DatabaseModel a knowledge graph with nodes and relationships · Write Cypher queries for graph traversal · Build GraphRAG pipeline combining graph and LLM · Integrate Neo4j with Python using async driver · Build a knowledge graph update pipeline · Monitor Neo4j performance and queries
Data Pipeline OrchestrationDefine Argo Workflow templates for data processing · Build DAG workflows with parallel execution · Pass artifacts between workflow steps · Implement retry and error handling strategies · Build reusable WorkflowTemplates · Schedule pipelines with CronWorkflows
Data Quality & ValidationBuild schema validation for AI datasets · Implement embedding quality checks · Create data profiling reports · Build Great Expectations validation suites · Implement data quality gates in pipelines · Monitor data quality metrics over time
Data Infrastructure OperationsDeploy data services on Kubernetes with StatefulSets · Configure Prometheus monitoring for data services · Implement automated PostgreSQL backup and restore · Build MinIO backup and replication · Build auto-scaling for data services · Build operational runbooks and incident response
GenAI Inference Engineering6.5%
Production Hosted LLM ArchitectureProvider Landscape Analysis · Request Patterns and Streaming · Token Economics · Rate Limit Management · Secure API Management
Prompt Caching MasteryAnthropic Prompt Caching · OpenAI Automatic Caching · Google Context Caching · Prompt Structure Optimization · Cache Hit Monitoring · Multi-Turn Conversation Caching
Batch API StrategiesOpenAI Batch API · Anthropic Message Batches · Google Gemini Batch Processing · Batch vs Real-Time Decisions · Combined Caching and Batching · Batch Job Orchestration
Model Routing and SelectionModel Tier Strategy · Query Complexity Classification · Routing Frameworks · Quality-Aware Routing · Cost-Quality Optimization · A/B Testing Models
Structured OutputsOpenAI Structured Outputs · Anthropic Tool Use with Strict Mode · Gemini Structured Outputs · Schema Design Best Practices · Error Handling and Validation · Cross-Provider Structured Abstraction
Function Calling and Tool OrchestrationTool Schema Design · Multi-Tool Orchestration · Anthropic Tool Use Patterns · Parallel Tool Execution · Tool Caching Optimization · Tool Error Recovery
Reasoning ModelsOpenAI Reasoning Strategies · Anthropic Extended Thinking · Gemini Thinking Mode · Reasoning Model Router · Reasoning ROI Analysis · Reasoning Prompt Engineering
Test-Time Compute StrategiesChain-of-Thought Prompting · Self-Consistency Voting · Tree-of-Thought Exploration · Self-Reflection and Correction · Best-of-N with Scoring · Compute Budget Optimization
Agentic PatternsReAct Pattern · Planner-Executor Pattern · Actor-Critic Pattern · Multi-Agent Orchestration · State Management · Human-in-the-Loop Patterns
Context and Memory ManagementContext Window Analysis · Selective Context Injection · Rolling Summarization · RAG vs Long Context · Dual-Layer Memory · Context Cache Optimization
Reliability and ResilienceRate Limit Handling · Circuit Breakers · Multi-Provider Fallback · Timeout Management · Error Classification · Health Monitoring
Cost Optimization and ObservabilityToken Optimization · Combined Cost Strategies · Cost Attribution · Usage Forecasting · Quality Monitoring
Multi-Provider OrchestrationBuild a Multi-Provider Router · Build a Circuit Breaker for Provider Health · Build a Latency-Based Router · Request Hedging · Provider SLA Tracking · Orchestration Pipeline
Semantic CachingRedis Exact-Match Cache · Embedding Pipeline · Semantic Similarity Search · Cache Invalidation · Cost Savings Dashboard · Cache Integration Test
Production Streaming PatternsSSE Event Formatting · Bounded Token Buffer · Stream Checkpoint and Resume · Stream Broadcasting · Stream Metrics Collection · End-to-End Stream Pipeline
Hosted Inference ObservatoryCustom Metrics Exporter · Provider SLA Dashboard · Cost Attribution Engine · Anomaly Detection Alerts · Distributed Tracing · Executive FinOps Dashboard
Rate Limit EngineeringDistributed Token Bucket · Priority Queue System · Burst Absorption · Cross-Key Token Pooling · Rate Limit Dashboard · Rate Limit Integration Test
Inference Cost SimulatorWorkload Profiler · Cost Model Engine · What-If Simulator · Batch Optimization Modeling · FinOps Report Generator · Simulator Capstone
GenAI Agent Engineering19.0%
The LLM ClientOpenAI client setup · Anthropic client setup · Google Gemini client setup · Build a unified LLM client interface · Error handling and provider fallback · Async LLM client patterns · Practical use cases — security, parameters, observability
Token EconomicsUnderstand tokenization · Count tokens across providers · Cost forecasting and budgeting · Track LLM API usage in production · Implement budget controls · Optimize tokens · Advanced context engineering
Prompt CachingImplement Anthropic cache_control · Leverage OpenAI automatic caching · Design cache-friendly prompt architectures · Build cache monitoring systems
The Function CallerOpenAI function schemas · Anthropic function schemas · Gemini function schemas · Handle tool call responses · Execute tools safely with Pydantic validation · Handle parallel tool calls · Framework integration with LangGraph
The Tool DefinerWrite clear tool descriptions for LLMs · Define parameter schemas · Use Pydantic for tool schemas · Implement tool decorators · Handle complex parameter types · Validate tool inputs at runtime · Framework tool patterns — LangGraph, CrewAI, OpenAI, Gemini, Anthropic
The Raw Agent LoopThe core agent while-loop · Manage context as a mutable list · Handle stop sequences · Track iteration limits · Tool execution in the loop · Build a conversation state tracker · Build with LangGraph StateGraph
The Prompt Engineer (Dynamic)Master Jinja2 templating for prompts · Implement dynamic few-shot example selection · Enforce Chain-of-Thought reasoning · Structure system prompts with a builder pattern · Inject dynamic context into prompts safely · Build prompt versioning and A/B testing
The ReAct Pattern (Manual)Build the Thought-Action generator · Tool execution and observation injection · Complete ReAct agent implementation · Advanced ReAct patterns — validation, retry, confidence · Optimize ReAct performance · Common ReAct pitfalls and solutions
The Planner PatternPlan generation · Step execution · Dynamic replanning · Hierarchical planning · Plan optimization · Monitoring and observability · Practical considerations — strategy selection
The Pydantic ToolPydantic fundamentals for tool definitions · Generate JSON Schema from Pydantic models · Input validation with custom validators · Build a Pydantic tool library · Advanced Pydantic patterns · Integrate Pydantic tools with agent frameworks · Common pitfalls and solutions
The Safe Executor (Sandboxing)Understand code execution risks · Static code analysis · Sandboxed execution · Apply resource limits · Build a complete safe executor
The Web NavigatorWeb navigation fundamentals · Web navigation tools — locating elements and forms · Browser automation with Playwright · Session management · Complete web navigator system
The MCP Protocol (Basics)JSON-RPC 2.0 message format and handler · Transport mechanisms — stdio and HTTP/SSE · Protocol lifecycle — initialization, runtime, shutdown · Capability negotiation
The MCP ServerCreate an MCP server with lifecycle management · Define MCP tools · Implement MCP resources · Create prompt templates · Error handling in MCP servers · Composable MCP server architecture
The MCP ClientMCP client architecture and stdio transport · Discover available tools and translate schemas · Proxy tool invocation · Fetch and use MCP resources · Manage MCP server lifecycle · Build multi-server MCP clients
The Tool RouterTool routing architecture and implementation · Namespace-based routing · Capability-based routing · Fallback chains · Routing performance optimization
Short-Term MemorySliding window memory · Token-aware memory management · Message summarization strategies · Memory persistence layers · Memory retrieval optimization · Integrate memory with agents · Memory performance considerations · Non-functional requirements (privacy + safety)
Long-Term Memory (RAG)Document chunking strategies · Embedding pipelines · Vector database integration · Hybrid search implementation · Retrieval optimization · RAG response generation
Agentic RAG PatternsSelf-reflective RAG · Multi-hop retrieval · Query routing · Adaptive retrieval · Retrieval feedback loops
Semantic MemoryKnowledge extraction pipelines · Entity and relationship extraction · Knowledge graph construction · Memory consolidation · Integrate semantic memory with agents · Build semantic memory with LangGraph
Context OptimizerContext economics · Dynamic context prioritization · Context compression techniques · Prompt optimization · Context utilization metrics · Complete context optimizer
The State GraphStateGraph fundamentals — config and lifecycle · Design state schemas with TypedDict · Add nodes to StateGraph · State initialization patterns · Tracing, debugging, validation
The Conditional EdgeUnderstand conditional edges · Design routing functions · Fan-out and fan-in patterns · Handle unknown routes and errors · Multi-stage routing
The Checkpointer (Time Travel)Resumable workflows · Inspect, replay, and time-travel · Retention, large state, and performance · Thread management — IDs and namespaces
Human-in-the-LoopLangGraph interrupt patterns · Approval workflow patterns · Interactive agent conversations · Feedback integration · State management for HITL · Practical use cases — escalation and analytics
The Streaming AgentStreaming modes in LangGraph · Token streaming from LLMs · Custom events with `astream_events` · Build streaming APIs · Error handling in streams · Backpressure and flow control
The Subgraph (Composition)Subgraph fundamentals — compile + test in isolation · State schema mapping · Subgraph checkpointers + namespace isolation · Compose subgraphs into a parent · Catch subgraph exceptions and recover · Define subgraph interfaces and build a registry · Build a multi-agent orchestrator
The Supervisor PatternDesign supervisor architectures · Worker agent specialization · Build the complete supervisor graph · Manage inter-agent communication · Handle failures and edge cases · Implement task aggregation · Build the supervisor pattern with CrewAI
The Hierarchical PatternDesign hierarchical agent architectures · Implement team-lead agents · Build cross-team coordination · Build the complete hierarchical graph
The Reflector Pattern (Critique)Design reflection architectures · Implement critic agents · Build the evaluation and convergence system · Build the complete reflection graph · Handle reflection edge cases · Practical use cases for reflection
Input GuardrailsDesign layered guardrail architectures · Format and schema validation · Build content filtering systems · Create injection / jailbreak detection · Implement policy-based guardrails · Assemble the complete guardrail system
Output GuardrailsDesign output validation architectures · Implement factual validation (hallucination detection) · Build content safety filters · Create PII redaction · Implement policy compliance · Assemble the complete output guardrail system
Prompt Injection DefenseIdentify injection vulnerabilities · Detect direct injections · Detect indirect injections · Implement defense layers · Build red-team suites · Implement canary tokens · LangGraph injection defense pipeline
Evaluations (Evals)Design evaluation frameworks · Implement automated evaluation pipelines · Create task-specific metrics · Human evaluation protocols · Regression testing · Set baselines and track progress
Agent BenchmarkingUnderstand the GAIA benchmark · Implement ToolBench evaluation · Use AgentBench · Design domain-specific benchmarks · Cross-model performance comparison · Build benchmark dashboards
Tracing & ObservabilityUnderstand distributed tracing · Add tags and metadata · Context propagation · Build feedback collection · Integrate with Langfuse · Trace visualization
Tool Use DebuggingTool selection failures and solutions · Argument validation systems · Build tool use dashboards and visualization · Schema mismatch detection · Tool call replay · Interactive tool debugger
Serving Agents (FastAPI)Async endpoints, request validation, error handling · Server-Sent Events (SSE) streaming · Background tasks · Design request and response schemas · Authentication — API keys, middleware, errors · OpenAPI metadata and documentation · FastAPI + LangGraph + uvicorn deployment
Podman & Containerization for K8sBuild optimized container images · Container health checks · Advanced image optimization · Security best practices · Build multi-container agent pods
Production Databases (Postgres/Redis)Async PostgreSQL configuration · Connection pool management · Redis caching for LLM responses · Database migrations for agent stacks · Backup and disaster recovery · Monitoring database health
Scaling & Load BalancingStateless service design · Session externalization · Load balancing algorithms · Scaling metrics for LLM workloads · Horizontal Pod Autoscaler configuration · Load testing your scaling design · Rate limiting at the load balancer
Multi-Tenant AgentsTenant context middleware · Database-level tenant isolation · Tenant-specific rate limiting and quotas · Per-tenant configuration templates · Usage metering for billing and SLA · Enforcing tenant data segregation at the API
Kubernetes (K8s) BasicsCreating Kubernetes Deployments · Resource management for LLM workloads · Kubernetes Services · ConfigMaps and Secrets · Rolling updates and CronJobs · Cluster planning and scheduling · Deployment planning synthesis · NetworkPolicies and prompt-injection defense at the edge
CI/CD for AgentsGitHub Actions for agent testing · Agent evaluation scripts in CI · Kubernetes deployment pipeline · GitOps deployment pattern · Quality gates and pipeline optimization · Rollback mechanisms · Pipeline observability and notifications
Monitoring & AlertingPrometheus metrics for agents · Grafana dashboards · Alerting configuration · SLOs and SLIs · Runbook creation for agent incidents · Tracking business KPIs for agent platforms · Agent-specific monitoring patterns (RED, USE, golden signals)
Model Routing & FallbacksCost-optimized routing · Latency-optimized routing · Building the resilient LLM client · Provider health checking · Cost tracking and optimization · Capability-based routing · Multi-model routing inside LangGraph
Long-Running AgentsCross-session persistence · Checkpoint serialization · Workflow resumption · Task queue integration with Celery · Progress tracking and SSE streaming · Timeout handling and graceful shutdown · Long-running agents with CrewAI — synthesis
Production Architecture PatternsSystem components, interfaces, and integration points · Cost modeling and projection · Production checklists and audit · Architecture Decision Records (ADRs) · Disaster recovery planning · System component diagrams · Architecture pattern evaluation — synthesis
Alternative Frameworks (CrewAI/AutoGen)CrewAI core concepts and agent personas · AutoGen conversational architecture · Framework comparison and integration patterns · Framework migration strategies and validation · Hybrid multi-framework systems · Picking a framework for a real use case
Deep Memory (GraphRAG)Knowledge graph fundamentals · Graph traversal patterns · Entity extraction with LLMs · Hybrid retrieval strategies (vector + graph) · Entity resolution and de-duplication · Incremental graph updates and provenance · Graph export, embeddings, and summarization
Agent Swarms & CollaborationSwarm architecture patterns · Agent communication and pub/sub · Consensus and weighted voting · Emergent behavior and stigmergy
Enterprise LLM Customization17.3%
Enterprise Data PipelineBuild DataPipeline with Instructor · Implement distillation data generation · Build a data quality dashboard · Build a data validation pipeline · Optimize data pipeline throughput · Build a data lineage tracker
Synthetic Data FactoryBuild SyntheticFactory with DSPy · Use OpenAI Batch API for bulk generation · Filter and validate synthetic data · Build synthetic data quality evaluator · Optimize Batch API cost and throughput · Build synthetic data versioning
Fine-Tuned Enterprise ModelFine-tune with OpenAI · Tune with Vertex AI · Build model comparison framework · Build fine-tuning hyperparameter search · Implement model distillation pipeline · Build model comparison report generator
RLVR-Trained Reasoning ModelBuild programmatic graders · Train via RFT API · Analyze training dynamics · Build grader reliability testing · Optimize RFT training cost · Build training run monitoring dashboard
Reward Engineering ToolkitBuild composite reward functions · Build LLM-as-judge graders · Validate grader reliability with Promptfoo · Build reward function A/B testing · Optimize composite reward weighting · Build reward engineering documentation generator
Model Eval DashboardConfigure Promptfoo eval suites · Use Batch API for bulk eval · Build regression detection · Build eval suite versioning and management · Optimize eval pipeline cost with sampling · Build eval regression root cause analyzer
Model Selection EngineBuild ModelSelector · Build cost-quality optimizer · Benchmark HuggingFace open models · Build model selection test harness · Optimize model selection latency · Build model catalog and recommendation engine
Reasoning Model BenchmarkBenchmark reasoning across providers · Analyze reasoning patterns · Build task-specific reasoning profiles · Build testing and validation for reasoning model benchmark · Optimize performance for reasoning model benchmark · Build operational runbook for reasoning model benchmark
TTC ControllerBuild TTC scaling curves · Implement compute-aware routing · Build TTC cost savings dashboard · Build testing and validation for ttc controller · Optimize performance for ttc controller · Build operational runbook for ttc controller
Best-of-N Inference EngineBuild BestOfN engine · Optimize N and cost tradeoff · Build BestOfN + RLVR hybrid strategy · Build testing and validation for best-of-n inference engine · Optimize performance for best-of-n inference engine · Build operational runbook for best-of-n inference engine
Self-Refining AgentBuild SelfRefiner · Implement long-horizon planning · Build refinement convergence analysis · Build testing and validation for self-refining agent · Optimize performance for self-refining agent · Build operational runbook for self-refining agent
Compute-Optimal RouterBuild ComputeRouter · Evaluate routing performance · Build routing feedback loop with learning · Build testing and validation for compute-optimal router · Optimize performance for compute-optimal router · Build operational runbook for compute-optimal router
DSPy Enterprise ModulesBuild 5 enterprise DSPy modules · Compare DSPy vs hand-written · Test DSPy modules with Promptfoo · Build testing and validation for dspy enterprise modules · Optimize performance for dspy enterprise modules · Build operational runbook for dspy enterprise modules
DSPy Optimizer as Meta-RLRun MIPROv2 optimization · Analyze optimization dynamics · Compare cross-provider optimization · Build testing and validation for dspy optimizer as meta-rl · Optimize performance for dspy optimizer as meta-rl · Build operational runbook for dspy optimizer as meta-rl
DSPy + Skills PipelinesBuild multi-stage pipeline · Test pipeline reliability · Build pipeline monitoring and alerting · Build testing and validation for dspy + skills pipelines · Optimize performance for dspy + skills pipelines · Build operational runbook for dspy + skills pipelines
Structured Extraction with CitationsBuild CitedExtractor · Validate extraction with Promptfoo · Compare multi-provider extraction · Build testing and validation for structured extraction with citations · Optimize performance for structured extraction with citations · Build operational runbook for structured extraction with citations
LiteLLM GatewayDeploy LiteLLM proxy · Implement failover and circuit breakers · Load test and capacity plan · Build testing and validation for litellm gateway · Optimize performance for litellm gateway · Build operational runbook for litellm gateway
Smart RouterBuild intelligent routing · Implement context caching · Build routing analytics dashboard · Build testing and validation for smart router · Optimize performance for smart router · Build operational runbook for smart router
Provider Caching ComparisonImplement all caching systems · Build caching strategy recommender · Build cache invalidation and consistency · Build testing and validation for provider caching comparison · Optimize performance for provider caching comparison · Build operational runbook for provider caching comparison
FinOps ControllerBuild budget and spend tracking · Auto-route to Batch API · Build cost anomaly detection · Build testing and validation for finops controller · Optimize performance for finops controller · Build operational runbook for finops controller
Langfuse ObservabilityDeploy and integrate Langfuse · Build drift detection and alerts · Build custom eval pipelines in Langfuse · Build testing and validation for langfuse observability · Optimize performance for langfuse observability · Build operational runbook for langfuse observability
A/B Testing + Prompt CI/CDBuild prompt CI/CD pipeline · Build model deployment lifecycle · Build multi-environment promotion pipeline · Build testing and validation for a/b testing + prompt ci/cd · Optimize performance for a/b testing + prompt ci/cd · Build operational runbook for a/b testing + prompt ci/cd
Agent Mesh: A2A + ADKBuild A2A agent mesh · Build AI governance framework · Build agent mesh observability dashboard · Build testing and validation for agent mesh: a2a + adk · Optimize performance for agent mesh: a2a + adk · Build operational runbook for agent mesh: a2a + adk
Guardrails PipelineBuild guardrails pipeline · Test guardrails under adversarial input · Benchmark guardrail performance · Build security testing for guardrails pipeline · Optimize throughput for guardrails pipeline · Build compliance reporting for guardrails pipeline
Prompt Injection Red TeamRun OWASP LLM Top 10 attacks · Build security assessment report · Build automated security regression pipeline · Build security testing for prompt injection red team · Optimize throughput for prompt injection red team · Build compliance reporting for prompt injection red team
Data Classification RouterBuild DataClassifier · Integrate DLP controls · Build data lineage tracking · Build security testing for data classification router · Optimize throughput for data classification router · Build compliance reporting for data classification router
Compliance Audit TrailBuild audit logging system · Build compliance reporting · Build audit data retention and archival · Build security testing for compliance audit trail · Optimize throughput for compliance audit trail · Build compliance reporting for compliance audit trail
Compliance Test SuiteBuild regulatory compliance tests · Build bias and fairness tests · Build continuous compliance monitoring · Build security testing for compliance test suite · Optimize throughput for compliance test suite · Build compliance reporting for compliance test suite
Production RAG PipelineBuild EnterpriseRAGPipeline · Build RAGEvaluator with RAGAS · Build HybridRetriever with reranking · Build testing and evaluation for production rag pipeline · Optimize performance for production rag pipeline · Build operational monitoring for production rag pipeline
Vector Database EngineeringBuild VectorDBBenchmark · Build MultiIndexManager · Build EmbeddingOptimizer · Build testing and evaluation for vector database engineering · Optimize performance for vector database engineering · Build operational monitoring for vector database engineering
LangGraph Agentic OrchestrationBuild LangGraphAgent with stateful workflows · Build OrchestrationComparator · Build LangGraphRAGAgent · Build testing and evaluation for langgraph agentic orchestration · Optimize performance for langgraph agentic orchestration · Build operational monitoring for langgraph agentic orchestration
GraphRAG & Knowledge GraphsBuild KnowledgeGraphBuilder · Build GraphRAGRetriever · Build HybridKnowledgeSearch · Build testing and evaluation for graphrag & knowledge graphs · Optimize performance for graphrag & knowledge graphs · Build operational monitoring for graphrag & knowledge graphs
Agent Memory & Stateful SystemsBuild AgentMemorySystem · Build MemoryManager · Build StatefulAgentBenchmark · Build testing and evaluation for agent memory & stateful systems · Optimize performance for agent memory & stateful systems · Build operational monitoring for agent memory & stateful systems
Advanced RAG PatternsBuild AdaptiveRAG · Build RAGCIPipeline · Build ProductionRAGDashboard · Build testing and evaluation for advanced rag patterns · Optimize performance for advanced rag patterns · Build operational monitoring for advanced rag patterns
Multi-Provider Reliability & ScalingBuild InferenceReliabilityManager · Build RateLimitPoolManager · Build ProviderSLAMonitor · Build testing and validation for multi-provider reliability & scaling · Optimize scalability for multi-provider reliability & scaling · Build operational runbook for multi-provider reliability & scaling
Streaming & Real-Time AIBuild StreamingAPI · Build StreamingGuardrails · Build StreamingObservability · Build testing and validation for streaming & real-time ai · Optimize scalability for streaming & real-time ai · Build operational runbook for streaming & real-time ai
MLOps for LLM ApplicationsBuild LLMOpsRegistry · Build ExperimentTracker · Build LLMOpsCI · Build testing and validation for mlops for llm applications · Optimize scalability for mlops for llm applications · Build operational runbook for mlops for llm applications
Advanced Evaluation & ObservabilityBuild EvalFrameworkComparator · Build CustomEvalPipeline · Build ObservabilityStack · Build testing and validation for advanced evaluation & observability · Optimize scalability for advanced evaluation & observability · Build operational runbook for advanced evaluation & observability
Multi-Tenant AI Platform on K8sBuild TenantManager · Build PlatformDeployment · Build PlatformDashboard · Build testing and validation for multi-tenant ai platform on k8s · Optimize scalability for multi-tenant ai platform on k8s · Build operational runbook for multi-tenant ai platform on k8s
Voice AI & Conversational SystemsBuild VoicePipeline · Build ConversationalAgent · Build VoiceQualityBenchmark · Build testing and validation for voice ai & conversational systems · Optimize scalability for voice ai & conversational systems · Build operational runbook for voice ai & conversational systems
Healthcare: Clinical AdvisorBuild clinical advisory pipeline · Implement HIPAA compliance · Build clinical accuracy validation · Build end-to-end testing for healthcare: clinical advisor · Optimize cost and performance for healthcare: clinical advisor · Build production operations for healthcare: clinical advisor
Finance: Deal AnalyzerBuild deal analysis pipeline · Build FinOps controls · Build cross-department usage analytics · Build end-to-end testing for finance: deal analyzer · Optimize cost and performance for finance: deal analyzer · Build production operations for finance: deal analyzer
Legal: Regulatory MonitorBuild regulatory monitoring mesh · Validate legal accuracy · Build regulatory change impact scoring · Build end-to-end testing for legal: regulatory monitor · Optimize cost and performance for legal: regulatory monitor · Build production operations for legal: regulatory monitor
Customer Ops: ResolutionBuild customer resolution system · Security and observability · Build resolution quality feedback loop · Build end-to-end testing for customer ops: resolution · Optimize cost and performance for customer ops: resolution · Build production operations for customer ops: resolution
Multimodal EnterpriseBuild multimodal processor · Validate multimodal accuracy · Build document type router with fallbacks · Build end-to-end testing for multimodal enterprise · Optimize cost and performance for multimodal enterprise · Build production operations for multimodal enterprise
Capstone: Enterprise AI PlatformBuild enterprise AI platform · Run production readiness assessment · Build disaster recovery and runbook · Build end-to-end testing for capstone: enterprise ai platform · Optimize cost and performance for capstone: enterprise ai platform · Build production operations for capstone: enterprise ai platform
Enterprise LLM Platform CapstonePlatform Integration · Multi-Tenant Operations · Production Readiness Assessment · Compliance Reporting · Customer Onboarding Demo · Platform Capstone Report
GenAI Operations23.9%
GenAI Failure CatalogClassify GenAI failures into five categories: provider, quality, cost, security, and data failures · Instrument a multi-provider LLM gateway to detect each failure category · Build typed failure event models that feed into alerting and incident management · Measure baseline failure rates across OpenAI, Anthropic, and Google providers · Implement failure prediction from leading indicators · Create failure impact assessment system
GenAI SLI FrameworkDefine latency SLIs: TTFT, tokens-per-second, end-to-end response time across providers · Define quality SLIs: faithfulness, hallucination rate, format compliance, retrieval precision · Define cost SLIs: cost-per-request, cost-per-token, cache hit rate, budget burn rate · Instrument all SLIs with Prometheus metrics and Langfuse traces · Build SLI aggregation and reporting API · Implement SLI validation and testing
GenAI SLO EngineDefine SLO targets for latency, quality, and cost SLIs with business-justified thresholds · Compute error budgets and track consumption over rolling windows · Build multi-window burn-rate alerts that detect SLO violations before budget exhaustion · Create SLO status dashboards showing budget remaining and projected exhaustion · Implement SLO negotiation framework · Build cross-SLO dependency tracking
GenAI Toil AnalyzerIdentify GenAI-specific toil patterns: manual model updates, prompt tweaking, provider failover, cache invalidation · Measure toil using time-tracking instrumentation and classify by automation potential · Automate the highest-impact toil items with operational scripts and scheduled workflows · Track toil reduction over time with team-level reporting · Build automation testing and validation · Create toil reduction roadmap generator
GenAI Launch ReadinessDefine operational readiness criteria specific to GenAI services · Build automated readiness checks that verify infrastructure, monitoring, and runbook completeness · Implement launch gate enforcement that blocks deployment without readiness sign-off · Create readiness dashboards and historical tracking for continuous improvement · Implement progressive readiness rollout · Create readiness automation toolkit
GenAI Runbook EngineWrite structured runbooks for the top 5 GenAI failure modes identified in Ch 1 · Build executable runbook steps that link to operational APIs and scripts · Implement runbook testing that validates each step works as documented · Track runbook usage and effectiveness metrics · Build runbook recommendation engine · Implement cross-runbook orchestration
GenAI Helm ChartsPackage LiteLLM, Langfuse, Prometheus, and Grafana as Helm Charts with Production-Ready Defaults · Create values files with environment-specific overrides for dev, staging, and production · Implement Helm chart testing and linting with ct and helm unittest · Build chart dependency management for the full GenAI stack · Implement Helm chart documentation generation · Build chart rollback and recovery procedures
Dev/Staging/Prod EnvironmentsCreate K8s namespaces for dev/staging/prod with resource quotas and LimitRanges · Build Kustomize overlays for environment-specific configuration · Implement environment parity verification that detects configuration drift · Deploy the full GenAI stack to all three environments · Implement environment configuration validation · Create environment lifecycle automation
GitOps Control PlaneDeploy Argo CD to vCluster with Application and ApplicationSet CRDs · Implement GitOps sync for all GenAI components across dev/staging/prod · Configure drift detection and auto-remediation for configuration consistency · Build Argo CD RBAC for team-based access control · Implement performance optimization for gitops with argo cd · Build operational documentation for gitops with argo cd
GenAI CI/CD PipelinesBuild Argo Workflow templates for GenAI artifact CI/CD · Implement pipeline stages: lint, validate, eval, promote · Create artifact-specific pipelines for prompts, models, and RAG configs · Monitor pipeline health with observability metrics · Implement performance optimization for ci/cd pipelines for genai artifacts · Build operational documentation for ci/cd pipelines for genai artifacts
GenAI Secret ManagerDeploy External Secrets Operator with GCP Secret Manager provider · Implement perenvironment provider key isolation with · Build secret sync monitoring and alerting for rotation compliance · Create emergency secret rotation procedures with zero-downtime key swap · Implement performance optimization for secret management for genai · Build operational documentation for secret management for genai
Self-Service Environment APIBuild API for on-demand feature environment provisioning with full GenAI stack · Implement environment lifecycle management with TTL and auto-cleanup · Create environment templates with pre-configured GenAI stack components · Monitor environment usage and resource consumption across all feature environments · Implement performance optimization for developer self-service environments · Build operational documentation for developer self-service environments
Pipeline Health DashboardInstrument Argo Workflow metrics for comprehensive pipeline observability · Build promotion velocity tracking across dev, staging, and production environments · Implement pipeline failure analysis with root cause categorization · Create pipeline health dashboards with bottleneck detection and DORA metrics · Implement performance optimization for pipeline observability · Build operational documentation for pipeline observability
Prompt RegistryBuild Immutable Prompt Storage with Content-Addressable Versioning · Implement Promotion Gates Between Dev, Staging, and Production · Create Prompt Diff and Review Workflow · Track Prompt Lineage Across All Deployments · Implement performance optimization for immutable prompt registry · Build operational documentation for immutable prompt registry
Model Lifecycle ManagerDeploy MLflow on vCluster for Model Registry and Experiment Tracking · Implement Model Versioning with Stage Promotion Gates · Build Model Deprecation Workflow with Consumer Notification · Track Experiments with Cost and Quality Metrics for Data-Driven Model Selection · Implement performance optimization for model registry and lifecycle · Build operational documentation for model registry and lifecycle
Progressive Delivery EngineDeploy Argo Rollouts with Canary Strategy for LiteLLM Model Config Changes · Implement Shadow Deployments for Risk-Free Model Comparison in Production · Build Automated Rollback on Quality Regression During Canary Progression · Monitor Canary vs Baseline Quality, Latency, and Cost Metrics in Real-Time · Implement performance optimization for canary and shadow deployments · Build operational documentation for canary and shadow deployments
Eval Gate PipelineBuild Promptfoo Eval Suites for Pre-Promotion Quality Verification · Implement Eval Gates in Argo Workflows That Block Promotion on Failure · Create Golden Test Sets for Regression Detection · Track Eval Pass Rates and Gate Effectiveness Metrics · Implement performance optimization for automated eval gates · Build operational documentation for automated eval gates
AI Feature Flag SystemBuild a Feature Flag Service for AI Configuration with Redis-Backed Storage · Implement Percentage-Based Rollouts for Model and Prompt Changes · Create Kill Switches for Rapid AI Behavior Reversion During Incidents · Track Feature Flag Impact on Quality and Cost Metrics · Implement performance optimization for feature flags for ai behaviors · Build operational documentation for feature flags for ai behaviors
RAG Release PipelineBuild a Release Pipeline for Embedding Model Swaps with Dual-Index Strategy · Implement Chunking Strategy Changes with A/B Comparison Using RAGAS Metrics · Create an Index Migration Workflow with Zero-Downtime Cutover · Validate RAG Releases with Retrieval Quality Metrics Before and After · Implement performance optimization for rag pipeline release management · Build operational documentation for rag pipeline release management
Distributed LLM TracerDeploy an OpenTelemetry Collector with Langfuse Exporter · Instrument Multi-Provider Request Chains with Parent-Child Trace Spans · Build Trace Correlation Across RAG Retrieval, LLM Inference, and Guardrail Processing · Create Trace-Based Latency Analysis Dashboards with Drill-Down Capability · Implement performance optimization for end-to-end llm tracing · Build operational documentation for end-to-end llm tracing
Quality Drift DetectorImplement Output Quality Drift Detection with Rolling Window Comparison · Build Embedding Drift Detection Using Distribution Divergence Metrics · Detect Retrieval Relevance Degradation with RAGAS-Based Monitoring · Configure Automated Alerts for Each Drift Type with Severity Classification · Implement performance optimization for quality drift detection · Build operational documentation for quality drift detection
GenAI Alert SystemConfigure Alertmanager with GenAI-Specific Routing Rules and Severity Classification · Deploy Grafana OnCall for On-Call Schedules, Escalation Policies, and Incident Lifecycle · Implement Alert Deduplication and Grouping for Noisy GenAI Metrics · Build Alert Effectiveness Tracking to Reduce Alert Fatigue · Implement performance optimization for alerting strategy · Build operational documentation for alerting strategy
GenAI Dashboard SuiteBuild operational dashboard with SLO status, active incidents, and system health overview · Create business dashboard with usage, cost, and adoption metrics for stakeholders · Build compliance dashboard with guardrail activity, audit coverage, and policy status · Implement dashboard-as-code with Grafana provisioning for version-controlled dashboards · Implement performance optimization for dashboard engineering · Build operational documentation for dashboard engineering
Provider SLA TrackerImplement per-provider availability tracking with synthetic probes · Build provider degradation detection using quality and latency SLIs · Create automated escalation chains for provider issues with status page integration · Track provider SLA compliance for vendor management and contract negotiation · Implement performance optimization for provider sla monitoring · Build operational documentation for provider sla monitoring
Cross-Env ComparatorImplement cross-environment metric comparison for quality regression detection · Build staging-to-prod quality correlation analysis for deployment confidence · Create environment drift detection for configuration parity monitoring · Monitor promotion impact by comparing pre/post metrics across environments · Implement performance optimization for cross-environment observability · Build operational documentation for cross-environment observability
AI Incident CommanderDefine LLM-specific incident severity classification with impact-based criteria · Build incident lifecycle management with role assignments and status tracking · Create communication templates for AI-specific incidents targeting different audiences · Track incident metrics with MTTD, MTTA, MTTR and trend analysis · Implement performance optimization for llm incident response framework · Build operational documentation for llm incident response framework
Runbook Automation EngineBuild alert-to-runbook routing that triggers automated remediation workflows · Implement human approval gates for high-impact remediation steps · Create runbook execution auditing with step-by-step logging and outcome tracking · Track automation coverage and success rates across all runbook types · Implement performance optimization for automated runbook execution · Build operational documentation for automated runbook execution
AI Post-Mortem EngineBuild structured post-mortem templates for GenAI failure modes · Implement timeline reconstruction from Langfuse traces and Prometheus metrics · Create action item tracking with follow-through verification · Analyze post-mortem trends to identify systemic issues · Implement performance optimization for post-mortems for ai failures · Build operational documentation for post-mortems for ai failures
LLM Chaos LabDeploy Chaos Mesh in vCluster · Build Provider Failover Chaos Experiments · Create Cache Invalidation Chaos Experiments · Implement Quality Degradation Injection · Implement performance optimization for chaos engineering for llm providers · Build operational documentation for chaos engineering for llm providers
Pipeline Chaos ExperimentsCreate Embedding Pipeline Chaos Experiments · Build Ingestion Interruption Tests · Implement Index Corruption Detection and Recovery Validation · Track Chaos Experiment Results and Improvement Trends · Implement performance optimization for pipeline failure chaos · Build operational documentation for pipeline failure chaos
GenAI Game DayDesign multi-failure game day scenarios for GenAI platforms · Build game day orchestration that chains chaos experiments with time delays · Implement game day scoring: response time, runbook adherence, communication quality · Create game day retrospectives with improvement tracking · Implement performance optimization for game day operations · Build operational documentation for game day operations
Cost Attribution EngineInstrument per-request cost tracking across all pipeline stages · Build cost attribution to teams, projects, and use cases · Create cost allocation models for shared infrastructure components · Implement cost anomaly detection with automated investigation · Implement performance optimization for full-stack cost attribution · Build operational documentation for full-stack cost attribution
Token Budget ControllerConfigure LiteLLM Virtual Keys with Per-Team Budget Limits · Implement Per-Request Token Limits · Build Budget Alerting at 50%, 80%, and 100% Thresholds with Escalation · Create Budget Override Workflows for Emergency Usage Beyond Limits · Implement performance optimization for token budget enforcement · Build operational documentation for token budget enforcement
Cache Economics AnalyzerDeploy Redis Semantic Cache and Measure Hit Rate vs Cost Savings · Compare Provider Caching Strategies for OpenAI, Anthropic, and Google · Build Cost-Benefit Analysis with Break-Even Calculations · Recommend Optimal Caching Mix Per Use Case · Implement performance optimization for caching roi analysis · Build operational documentation for caching roi analysis
Batch API SchedulerImplement workload classification: real-time vs batch-eligible based on latency requirements · Build Batch API job scheduling with priority queues and SLA tracking · Create batch job monitoring with completion time SLAs and failure handling · Measure and report cost savings from batch routing vs synchronous requests · Implement performance optimization for batch api optimization · Build operational documentation for batch api optimization
Capacity ForecasterBuild token demand forecasting using historical usage patterns and trend analysis · Implement embedding volume projection for storage and compute planning · Create cost projection models for budget planning cycles · Track forecast accuracy and improve models over time with feedback loops · Implement performance optimization for capacity forecasting · Build operational documentation for capacity forecasting
FinOps Governance PlatformBuild Showback and Chargeback Reports per Team and Project with Full Cost Transparency · Create Executive FinOps Dashboards with Trend Analysis for Leadership · Implement Cost Governance Policies with Automated Enforcement · Generate Monthly FinOps Reviews with Optimization Recommendations · Implement performance optimization for finops reporting and governance · Build operational documentation for finops reporting and governance
Embedding Pipeline OpsDeploy pgvector and build embedding ingestion pipeline with operational monitoring · Implement pipeline throughput tracking and failure detection with alerting · Build reprocessing workflow for failed or stale embeddings · Create pipeline health dashboards with freshness SLA tracking · Implement performance optimization for embedding pipeline operations · Build operational documentation for embedding pipeline operations
Vector Index OpsImplement pgvector index maintenance with VACUUM and reindexing schedules · Deploy Qdrant and compare operational characteristics with pgvector · Build schema migration workflow for embedding dimension changes · Monitor index performance with query latency tracking and degradation detection · Implement performance optimization for vector index maintenance · Build operational documentation for vector index maintenance
Knowledge Graph OpsDeploy Neo4j on vCluster with backup and restore procedures · Implement graph index management and query performance monitoring · Build knowledge graph freshness tracking with entity update pipelines · Create graph operations dashboards for health monitoring · Implement performance optimization for knowledge graph operations · Build operational documentation for knowledge graph operations
RAG Quality MonitorDeploy RAGAS evaluation for production RAG quality monitoring on sampled traffic · Implement retrieval relevance tracking with precision and recall metrics · Build quality degradation alerting with root cause analysis · Compare retrieval quality across embedding models with Cohere Rerank · Implement performance optimization for rag quality monitoring · Build operational documentation for rag quality monitoring
Data Recovery PlatformImplement automated backup for pgvector, Qdrant, Neo4j, and PostgreSQL with scheduling · Build point-in-time recovery procedures with defined RTO and RPO targets · Create backup verification with data integrity checks and restore testing · Track backup compliance and recovery test results · Implement performance optimization for data backup and recovery · Build operational documentation for data backup and recovery
Data Quality OpsImplement data freshness monitoring with staleness alerting across all stores · Build completeness checks for embedding coverage and knowledge graph gaps · Detect data poisoning and anomalous ingestion patterns · Create data quality scorecards with trend tracking · Implement performance optimization for data quality monitoring · Build operational documentation for data quality monitoring
Key Rotation OperatorImplement automated key rotation for OpenAI, Anthropic, and Google provider keys · Build zero-downtime key swap using dual-key overlap windows · Create key rotation audit trails for compliance reporting · Execute emergency key revocation for compromised keys · Implement performance optimization for api key rotation automation · Build operational documentation for api key rotation automation
PII Detection PipelineDeploy Microsoft Presidio on K8s for runtime PII detection in LLM traffic · Build PII scrubbing middleware that redacts sensitive data before sending to LLM providers · Implement PII detection alerting and audit logging for compliance · Create PII detection tuning workflows to reduce false positives · Implement performance optimization for runtime pii detection · Build operational documentation for runtime pii detection
Injection Monitoring SystemImplement multi-layer prompt injection detection with pattern and embedding-based methods · Build real-time injection alerting with severity classification · Create injection attack analysis dashboards for security monitoring · Implement adaptive detection that learns from new attack patterns · Implement performance optimization for prompt injection monitoring · Build operational documentation for prompt injection monitoring
Guardrail Operations PlatformDeploy Guardrails AI and LlamaFirewall on K8s for runtime content validation · Implement hot-reload guardrail configuration without service restarts · Build A/B testing framework for guardrail thresholds to optimize block rates · Build testing and validation for guardrail operations · Implement performance optimization for guardrail operations · Build operational documentation for guardrail operations
Compliance Audit EngineImplement automated compliance scans for GenAI-specific requirements · Build evidence collection pipelines that gather audit artifacts · Schedule recurring compliance checks with drift detection · Build testing and validation for compliance audit automation · Implement performance optimization for compliance audit automation · Build operational documentation for compliance audit automation
Red Team Automation PlatformBuild automated red team attack suites using Promptfoo for systematic security testing · Implement scheduled security testing with regression detection across model changes · Build security posture scoring with trend monitoring and improvement tracking · Build testing and validation for red team operations · Implement performance optimization for red team operations · Build operational documentation for red team operations
Multi-Tenant GenAI PlatformAutomate tenant onboarding with namespace provisioning and secret management · Implement namespace isolation with network policies and resource quotas · Build noisy-neighbor detection that identifies tenants causing resource contention · Create tenant operations dashboards with per-tenant health visibility · Implement performance optimization for multi-tenant platform operations · Build operational documentation for multi-tenant platform operations
AI Developer PlatformBuild self-service deployment workflows with approval gates for AI artifacts · Create golden path templates for common GenAI patterns · Implement internal tool marketplace for reusable AI components · Build developer experience metrics and platform analytics · Implement performance optimization for internal developer platform for ai · Build operational documentation for internal developer platform for ai
GenAI Ops Maturity AssessorDefine GenAI operational maturity model with five levels across eight capability areas · Build automated maturity assessment that evaluates current operational state · Generate improvement roadmaps with prioritized actions based on assessment results · Track maturity progression over time with milestone tracking · Implement performance optimization for operational maturity model · Build operational documentation for operational maturity model
Healthcare AI Ops PlatformOperate healthcare GenAI platform with HIPAA-compliant infrastructure and audit trails · Implement clinical quality SLOs beyond standard faithfulness metrics · Build medical safety guardrails preventing harmful medical advice generation · Create healthcare-specific incident response with clinical escalation procedures · Implement performance optimization for healthcare genai operations capstone · Build operational documentation for healthcare genai operations capstone
Finance AI Ops PlatformOperate finance GenAI platform with SOX-compliant infrastructure and change management · Implement financial accuracy SLOs with numerical precision validation · Build financial regulatory guardrails preventing unauthorized financial advice · Create finance-specific compliance automation with audit trail integration · Implement performance optimization for finance genai operations capstone · Build operational documentation for finance genai operations capstone
Enterprise GenAI Ops PlatformOperate a complete enterprise GenAI platform integrating all eight operational disciplines · Implement cross-domain operational policies that adapt guardrails and SLOs per tenant domain · Build platform operational runbooks covering the full lifecycle from onboarding to decommissioning · Demonstrate operational maturity level 4+ across all eight capability areas · Implement performance optimization for full platform operations capstone · Build operational documentation for full platform operations capstone
LLM-Specific ObservabilityHallucination Detection Pipeline · Semantic Drift Monitor · Response Quality Scorer · LLM Observability Dashboard · Quality Degradation Root Cause · LLM Observability Capstone
Token Cost FinOpsCost Tracking Pipeline · Budget Management System · Chargeback Reporting · Cost Optimization Engine · FinOps Drill-Down Dashboard · FinOps Capstone
OpenTelemetry for Agentic SystemsLLM Call Instrumentation · Agent Execution Tracing · Custom Agent Attributes · Langfuse Trace Integration · K8s Distributed Tracing · Tracing Capstone
Eval Gates in CI/CDEval Suite Design · CI Pipeline Integration · Eval Result Storage · Change-Type-Specific Gates · Eval Failure Workflow · Eval Gates Capstone
Prompt Versioning and DeploymentOperational Prompt Registry · Canary Prompt Deployment · Automated Prompt Rollback · Prompt Deployment Pipeline · Multi-Prompt Coordination · Prompt Ops Capstone
Guardrail OperationsGuardrail Deployment · Guardrail Tuning · Guardrail Versioning · Effectiveness Monitoring · Guardrail Incident Response · Guardrail Ops Capstone
AI Incident ResponseAI Incident Taxonomy · Automated Detection · AI Incident Runbooks · Post-Incident Review · Escalation Path Design · Incident Response Capstone
Multi-Tenant AI Platform OpsTenant Isolation · Quota Management · Noisy-Neighbor Detection · Per-Tenant SLA Monitoring · Tenant Self-Service · Multi-Tenant Ops Capstone
AI Governance Compliance OpsEU AI Act Controls · NIST AI RMF Implementation · SOC2 AI Controls · Compliance Dashboard · Audit Preparation Workflow · Compliance Ops Capstone
Platform Operations CapstonePlatform Integration · Unified Ops Dashboard · Operational Readiness · Operational Lifecycle Demo · Operations Automation · Platform Ops Capstone Report
GenAI Evaluation, Safety & Governance2.9%
Evaluation Dataset CurationBuild a stratified evaluation dataset · Implement dataset versioning and snapshots · Detect dataset contamination and leakage · Build automated dataset refresh pipeline · Create dataset cards for documentation · Build a dataset annotation pipeline
Evaluation Observability with Langfuse v3 & OpenTelemetryDeploy Langfuse v3 on GKE with OpenTelemetry-native instrumentation · Build evaluation dashboards in Langfuse · Track costs and token usage across providers · Manage prompt versions with Langfuse · Compare observability platforms: Langfuse vs Arize Phoenix vs Braintrust · Build automated evaluation alerting
Evaluation-Driven CI/CD & Continuous Production MonitoringBuild evaluation gates with Promptfoo and DeepEval in CI · Implement progressive evaluation tiers · Implement cost-aware evaluation budgets · Build release validation pipeline · Track evaluation trends across releases · Build continuous production monitoring with async scoring
OWASP LLM Top 10 2025 & MITRE ATLASMap OWASP LLM Top 10 2025 to implemented defenses · Implement defenses for Unbounded Consumption (LLM10) · Build MITRE ATLAS threat models · Build OWASP compliance testing with Promptfoo presets and Checks by Google · Implement supply chain security (LLM03) · Create OWASP + ATLAS compliance dashboard
EU AI Act ComplianceClassify AI systems under EU AI Act risk categories · Implement the Feb 2025 AI literacy requirements · Build technical documentation for GPAI compliance · Implement risk management system · Build human oversight mechanisms · Track EU AI Act enforcement timeline compliance
Compliance FrameworksImplement NIST AI RMF Govern and Map functions · Implement NIST AI RMF Measure and Manage functions · Build ISO 42001 AI management system documentation · Create unified governance dashboard with Credo AI Agent Registry · Implement comprehensive audit trail · Build compliance automation and alerting
Red Teaming MethodologyPlan and scope a red team exercise · Execute manual red team techniques · Combine manual, Meta GOAT automated, and Inspect AI red teaming · Build red team findings reports · Track remediation and verify fixes · Build AI safety scorecard and establish red team cadence
Bias, Fairness & Continuous MonitoringDetect bias in hosted LLM outputs · Implement fairness metrics for LLM applications · Build continuous safety monitoring for production · Detect safety drift over time · Build safety incident response workflow · Generate weekly and monthly safety reports
GenAI Architecture & Design Patterns9.5%
GenAI ADR EngineBuild ADR schema and decision taxonomy for GenAI technology choices · Implement multi-provider model selection ADR workflow · Validate ADR decisions against production telemetry · Build ADR dependency graph across system decisions · Implement ADR recommendation engine using historical outcomes · Create ADR governance dashboard and compliance audit
Reference Architecture RegistryBuild reference architecture schema with C4 model layers · Catalog five canonical GenAI reference architectures · Validate architecture blueprints against quality fitness functions · Build architecture pattern matching for requirements-to-blueprint selection · Implement architecture evolution planner for incremental migration · Create architecture registry dashboard with adoption tracking
Compound AI System DesignerBuild compound AI system topology modeler · Implement cascade routing with cost-aware model selection · Validate compound system against end-to-end quality requirements · Model failure propagation and blast radius in compound systems · Optimize compound system for cost-quality Pareto frontier · Create compound system architecture review checklist and report
Eval-First Architecture EngineBuild eval gate component with pluggable evaluator registry · Integrate eval gates into retrieval, generation, and agent pipelines · Measure eval gate effectiveness with precision-recall tracking · Build eval cascade with tiered evaluation depth · Optimize eval gate latency overhead with async and sampling strategies · Create eval architecture audit report with coverage analysis
Cost-Optimized AI RouterBuild cost-aware request router with tiered model cascade · Implement semantic caching architecture with cache-aside pattern · Measure cost savings and quality impact per optimization strategy · Build token budget enforcement with per-tenant and per-request limits · Optimize batch processing architecture for non-real-time workloads · Create cost architecture review and FinOps governance report
AI Traffic GatewayBuild AI-aware API gateway with provider-specific routing rules · Implement Istio service mesh for AI microservice communication · Measure gateway latency overhead and optimize hot paths · Build intelligent rate limiting with token-aware throttling · Implement circuit breaker and bulkhead patterns for provider isolation · Create gateway operations dashboard with traffic topology view
Streaming AI PipelineBuild streaming response pipeline with SSE as default transport · Implement streaming-aware guardrails that evaluate partial output · Measure streaming quality metrics including TTFT and token throughput · Build event-driven pipeline for streaming data transformation · Optimize streaming architecture for connection efficiency and recovery · Create streaming architecture health dashboard and SLA report
Cell-Based AI PlatformBuild cell definition schema with namespace isolation on GKE Autopilot · Implement traffic routing between cells for canary and shadow deployments · Validate cell isolation with blast radius testing · Build feature-flag-driven model rollout across cells · Optimize cell resource allocation with autoscaling policies · Create cell operations dashboard with fleet-wide topology view
AI Observability StackBuild OpenTelemetry instrumentation for multi-step AI pipelines · Implement AI-specific metrics collection with quality dimensions · Validate trace completeness and data quality across the pipeline · Build cross-signal correlation linking traces to quality degradation · Optimize observability cost with intelligent data reduction · Create observability architecture assessment and maturity dashboard
Layered AI Security SystemBuild layered guardrail pipeline with pluggable security stages · Integrate NeMo Guardrails and LlamaFirewall for multi-layer defense · Measure guardrail effectiveness with red-team evaluation metrics · Build security event correlation across guardrail layers · Optimize guardrail latency with tiered security evaluation · Create security architecture audit report with compliance mapping
MCP Tool MeshBuild MCP server registry with capability discovery and health checks · Implement MCP authorization layer with per-tool permission policies · Validate MCP tool composition correctness and safety · Build MCP tool routing with load balancing and failover · Optimize MCP tool selection with intelligent pre-filtering · Create MCP ecosystem governance dashboard
A2A Agent NetworkBuild A2A agent card registry with capability advertisement · Implement A2A task delegation with streaming artifact exchange · Validate A2A communication reliability with failure injection · Build A2A agent trust and authorization framework · Optimize A2A network topology for latency and reliability · Create A2A network operations dashboard with federation view
Event-Driven AI ProcessorBuild event schema registry for GenAI domain events · Implement event backbone with Redis Streams for AI workloads · Validate event pipeline reliability with exactly-once semantics · Build event-sourced audit trail for AI decision traceability · Optimize event pipeline throughput with partitioning and batching · Create event architecture topology dashboard and maturity report
Multi-Modal AI PipelineBuild multi-modal request router with modality detection · Implement vision-language pipeline with document understanding · Validate multi-modal output quality with per-modality evaluators · Build modality-specific caching and preprocessing optimization · Optimize multi-modal pipeline for latency with parallel processing · Create multi-modal architecture capability assessment
Enterprise RAG SystemBuild hybrid retrieval pipeline combining dense, sparse, and graph search · Implement multi-index federation with routing by query intent · Validate retrieval quality with RAGAS metrics and human evaluation · Build real-time document ingestion with incremental index updates · Optimize retrieval latency with query planning and result caching · Create RAG architecture health dashboard and capacity report
Agent Orchestration PlatformBuild agent pool manager with lifecycle and capability registration · Implement supervisor hierarchy with delegation and escalation policies · Validate orchestration correctness with agent trajectory evaluation · Build task queue with priority scheduling and resource-aware dispatch · Optimize orchestration for cost with agent budget management · Create agent orchestration operations dashboard
AI Developer PlatformBuild Backstage service catalog for AI components · Implement golden path templates for AI service creation · Validate platform compliance with automated standards checking · Build self-service AI environment provisioning pipeline · Optimize developer experience with platform usage analytics · Create platform engineering maturity assessment dashboard
Provider Reliability EngineBuild provider health tracker with real-time status aggregation · Implement intelligent failover with quality-preserving model mapping · Validate failover behavior with chaos engineering experiments · Build shadow deployment for cross-provider model comparison · Optimize provider selection with cost-quality-latency Pareto routing · Create provider reliability SLA report and vendor risk assessment
Context Engineering PlatformBuild context composition engine with priority-ranked information sources · Implement context compression with summarization and pruning strategies · Validate context quality impact on generation accuracy · Build context caching with freshness-aware invalidation · Optimize context utilization with dynamic window sizing · Create context architecture audit report with optimization roadmap
AI Data ArchitectureBuild unified data access layer across vector, graph, and relational stores · Implement schema evolution strategy for embedding and knowledge data · Validate data consistency across stores with reconciliation checks · Build data lifecycle manager with retention and archival policies · Optimize data access patterns with caching and pre-computation · Create data architecture health report with capacity planning
Enterprise AI Integration HubBuild integration adapter framework with standardized connector interface · Implement data transformation pipeline between enterprise and AI formats · Validate integration correctness with contract testing · Build integration error handling with dead letter and retry strategies · Optimize integration throughput with batching and parallel processing · Create enterprise integration architecture map and health report
AI Governance PlatformBuild data classification engine for AI input and output streams · Implement model usage policy engine with approval workflows · Validate compliance coverage with automated control testing · Build comprehensive audit trail with immutable event logging · Optimize compliance overhead with risk-based control depth · Create regulatory compliance report generator for SOC2 and GDPR
HA/DR AI PlatformBuild HA topology planner with single-point-of-failure analysis · Implement stateful failover for conversation and embedding stores · Validate DR procedures with automated failover testing · Build automated DR runbook with step-by-step failover execution · Optimize recovery speed with pre-staged failover resources · Create HA/DR architecture dashboard with readiness assessment
Multi-Tenant AI PlatformBuild tenant isolation framework with namespace and network boundaries · Implement tenant-aware routing with per-tenant model and guardrail config · Validate tenant isolation with cross-tenant data leakage testing · Build per-tenant cost management with budget enforcement · Optimize tenant resource sharing with noisy-neighbor prevention · Create multi-tenant platform operations dashboard
Architecture Review EngineBuild architecture review checklist engine with domain-specific criteria · Implement technology selection framework with multi-criteria scoring · Validate review coverage with architecture surface area mapping · Build review knowledge base from historical architecture decisions · Optimize review process with risk-based prioritization · Create architecture governance report for leadership
Production AI Platform CapstoneDesign complete platform architecture with C4 model documentation · Implement architecture prototype with compound AI pipeline · Validate platform against quality, security, and reliability requirements · Build platform cost model with total cost of ownership projection · Conduct architecture review applying all course frameworks · Generate architecture handoff package for solution delivery team
Core responsibilities this discipline prepares you for. Open one to see how you build it.
Define enterprise GenAI architecture with proper documentation and governance
- Write Architecture Decision Records (ADRs) for GenAI system design choices with trade-off analysis
- Design reference architectures for common enterprise GenAI use cases
- Create ADRs, design reference architectures, and present trade-off analyses to stakeholders
Design scalable RAG systems at enterprise scale
- Architect full RAG stacks: document processing → embedding pipelines → pgvector → hybrid search with reranking
- Design multi-tenant data isolation with embedding pipeline separation and row-level security
- Benchmark RAG systems against enterprise-scale document volumes for throughput and accuracy
Architect multi-agent systems with MCP mesh and A2A network topology
- Design MCP mesh architecture for distributed tool access across organizational boundaries
- Plan A2A agent network topologies with lifecycle governance and communication protocols
- Stress-test multi-agent architectures with simulated failure scenarios and cascading fault injection
Lead PoC development and production rollouts with model selection and cost estimation
- Compare models across providers with cost-per-request modeling and quality benchmarking
- Build prototype evaluation frameworks with production readiness checklists and go/no-go criteria
- Evaluate models for specific use cases, build cost projections, and create decision frameworks
Design GenAI governance architecture — RBAC, audit trails, and compliance
- Build multi-tenant GenAI governance with role-based access control for models, prompts, and data
- Design audit trail architecture with policy-as-code enforcement and compliance reporting
- Architect governance for multi-business-unit enterprises and validate regulatory compliance
Oversee operational architecture — observability, FinOps, SLA management
- Design full-stack observability architecture spanning metrics, logs, traces, and LLM-specific telemetry
- Architect FinOps dashboards and incident response workflows with SLA definition and monitoring
- Validate operational architecture designs against production SLA targets and failure scenarios
Integrate GenAI with enterprise data platforms — pipelines, knowledge graphs, streaming
- Design data architectures integrating PostgreSQL, pgvector, Kafka streaming, Neo4j, Redis, and MinIO
- Architect data flows that support multiple GenAI use cases simultaneously across shared infrastructure
- Build data architecture designs for multi-use-case enterprise scenarios with isolation and scaling
Present architecture decisions with cost/risk analysis to leadership
- Apply ADR methodology with structured trade-off analysis and risk quantification frameworks
- Conduct architecture reviews with stakeholders and defend design decisions under scrutiny
- Write ADRs, conduct architecture reviews, and present cost/risk arguments for design choices
Each group becomes a signed node on your public skill graph — a recruiter-auditable record of the labs you actually passed. Share the URL; they see the discipline, the labs, and the signature.
Reference Architectures & Architecture Decision Records (ADRs)
Authoring enterprise reference architectures and Architecture Decision Records: canonical ADRs with rejected alternatives and explicit tradeoffs, constraint-justified decisions, failure-mode reasoning. The role's headline deliverable — design judgment, not implementation.
GenAI Solution Patterns & Architecture
Designing GenAI solution architectures and patterns: compound-AI topologies, agentic/multi-agent architecture, multi-provider gateway/routing, orchestration — chosen with pattern-selection rationale and failure-mode analysis. Absorbs the former gateway and agent-mesh bars.
Technology Selection & Build-vs-Buy
Technology-selection frameworks and build-vs-buy: vendor evaluation, weighted-criteria decision matrices, lock-in/exit-cost and TCO analysis for the GenAI stack. The reusable apparatus for making architecture decisions — distinct from authoring a single ADR.
Production-at-Scale: HA, DR & Multi-Tenancy
Designing for production at scale: high-availability and disaster-recovery architecture, resilience, blast-radius isolation, cell-based and multi-tenant platform design, and the prototype-to-production transition. Absorbs the former cell-based/multi-tenancy bar.
Governance, Trust Boundaries & Security Architecture
Governance, trust-boundary and data-residency architecture: layered security architecture, policy-as-code, compliance and risk frameworks at the architecture level. The enterprise buyer's risk surface — design, not control implementation.
Enterprise Data & Integration Architecture
Bridging GenAI with enterprise workflows and data: integration architecture, enterprise data boundaries, system-of-record integration, event/API integration patterns.
Observability & Eval-First Architecture
Observability and eval-first architecture: designing traceability, SLOs, eval gates and feedback loops as first-class architecture concerns rather than afterthoughts.
Enterprise RAG Architecture
Enterprise RAG architecture: retrieval governance, data-boundary-aware retrieval, hybrid retrieval at scale, and knowledge architecture for enterprise corpora. A deep, separately-hireable competency.
Hosted LLM API Integration
Provider SDK integration: client setup, multi-provider calls, streaming, error handling. Prerequisite plumbing.
Python for Solutions Architecture
Production-grade Python for solutions architects: prototyping, reference implementations, scripting. Prerequisite.