10 skill groups · 10 courses · 1188 goals
GenAI Solutions & Delivery
Scope GenAI solutions with estimation, risk, and success criteria. Orchestrate delivery teams, manage client relationships.
10 courses, each building on the last. The bar shows each course's share of the curriculum. 7 goals are unlocked for preview.
Python Essentials for Agent Builders5.4%
Your Dev EnvironmentNavigate filesystem with terminal · Manage files from command line · Set up VS Code · Configure terminal in VS Code
Python, Git & Package ManagementInstall and verify Python · Write hello world script · Use Python REPL · Initialize Git repository · Track changes with Git · Install packages with pip
Variables & Basic TypesCreate and name variables · Work with strings · Work with numbers · Work with booleans · Format with f-strings
Control FlowMake decisions with if/elif/else · Iterate with for loops · Repeat with while loops · Control loop execution
FunctionsDefine and call functions · Use parameters · Return values · Document with docstrings · Understand scope
Modules & ImportsImport standard library · Create custom modules · Understand Python path · Create packages
Lists & TuplesCreate and access lists · Modify lists · Slice lists · Use list comprehensions · Work with tuples
Dictionaries & SetsCreate and access dicts · Modify dictionaries · Iterate over dicts · Work with nested dicts · Use sets
Classes & DataclassesUnderstand class basics · Create dataclasses · Add methods · Use default values · Basic inheritance
Files, JSON & Error HandlingRead and write files · Work with JSON · Use pathlib · Handle exceptions · Create custom exceptions
Basic TestingUse assert statements · Create test functions · Run pytest · Test classes
Environment Variables & ConfigurationUnderstand environment variables · Use .env files · Load with python-dotenv · Handle missing variables · Organize configuration
Decorators & Context ManagersUnderstand decorators · Write simple decorators · Use context managers · Write context managers · Combine patterns
LLM Foundations for Agent Builders5.7%
Generators & IteratorsUnderstand iteration · Create generators · Use generator expressions · Build data pipelines · Use itertools
Async Programming BasicsUnderstand async concepts · Write async functions · Run concurrent operations · Use async context managers · Handle async exceptions
Type Hints & PydanticAdd basic type hints · Use typing generics · Create Pydantic models · Validate API data · Configure Pydantic
Data Pipelines & TransformationsBuild functional pipelines · Work with tabular data · Transform data shapes · Process LLM data formats · Optimize for performance
HTTP Clients & httpxMake GET requests · Make POST requests · Use async httpx · Handle errors · Use sessions
Your First LLM CallSet up credentials · Install Gemini SDK · Make first API call · Parse response · Handle API errors
Sampling Parameters & Output ControlUnderstand temperature · Use top-p sampling · Implement determinism · Control output length · Use structured output
Multi-Provider & Prompt EngineeringBuild provider abstraction · Structure conversations · Use few-shot prompting · Implement chain-of-thought · Build prompt templates
Function Calling FundamentalsUnderstand tool use concept · Define tool schemas · Make function calls · Handle tool responses · Compare provider patterns
Embeddings & Semantic SearchUnderstand embeddings · Generate embeddings · Calculate similarity · Build simple search · Compare embedding models
RAG FundamentalsUnderstand RAG pattern · Chunk documents · Build retrieval pipeline · Compose RAG prompts · Evaluate RAG quality
Cost Awareness & Token EconomicsUnderstand pricing models · Calculate request costs · Compare provider costs · Identify cost drivers · Basic cost optimization
Retry Patterns with TenacityUnderstand retry need · Use tenacity basics · Implement exponential backoff · Handle specific exceptions · Combine with async
Kubernetes Essentials for GenAI Engineers5.8%
Containerizing LLM ApplicationsWrite a Python app that calls the Gemini API and returns structured responses · Write a Dockerfile and build a container image for the LLM app · Run the containerized LLM app with environment-based configuration · Use Docker Compose to run the LLM app with supporting services · Tag images with semantic versions and push to a container registry · Debug containers with exec, logs, and inspect
Your Kubernetes Cluster & First LLM PodUnderstand K8s architecture and connect to your vCluster · Deploy the LLM app as your first Kubernetes pod · Organize workloads with namespaces · Use labels and selectors to organize and query resources · Understand pod lifecycle and restart policies · Master kubectl debugging: exec, logs, describe, port-forward
Services & the LLM Chat BackendCreate a ClusterIP service to expose the LLM chat API internally · Deploy a multi-tier LLM chat application · Compare service types: ClusterIP, NodePort, LoadBalancer · Master DNS-based service discovery in Kubernetes · Understand endpoints and traffic routing · Debug service connectivity problems
Deployments, Scaling & Rolling UpdatesCreate a Deployment for the LLM chat API · Scale LLM app replicas to handle concurrent requests · Perform a rolling update with zero downtime · Roll back a broken deployment · Compare deployment strategies: RollingUpdate vs Recreate · Manage deployment lifecycle with kubectl rollout
ConfigMaps & Secrets for LLM AppsCreate ConfigMaps for LLM app settings · Mount ConfigMaps as files for complex configuration · Store LLM proxy credentials securely in Secrets · Manage per-environment configuration for dev, staging, and prod · Handle configuration updates and rolling restarts · Debug configuration issues in LLM app pods
Persistent Storage & StatefulSetsCreate PersistentVolumeClaims for durable storage · Deploy PostgreSQL as a StatefulSet · Connect the LLM chat API to PostgreSQL for conversation persistence · Deploy Redis as a StatefulSet for LLM response caching · Understand StatefulSet scaling and ordering guarantees · Manage PVC lifecycle: expansion, snapshots, and cleanup
Resource Management & Cost OptimizationSet resource requests and limits for the LLM chat API · Understand QoS classes and their impact on eviction · Enforce resource defaults with LimitRanges · Cap namespace resource usage with ResourceQuotas · Right-size LLM app containers based on actual usage · Diagnose OOMKilled and CPU throttling issues
Packaging with Helm & KustomizeCreate a Helm chart for the LLM chat application · Parameterize the chart with values.yaml for each environment · Manage Helm release lifecycle: install, upgrade, rollback · Use Kustomize bases and overlays for the LLM app · Use Kustomize patches and generators · Compare Helm vs Kustomize for different deployment scenarios
Networking, Ingress & TLSExpose the LLM chat API via an Ingress resource · Add TLS to the Ingress for HTTPS access · Isolate services with NetworkPolicies · Configure Ingress annotations for production traffic · Understand K8s networking: pod IPs, CNI, and service routing · Debug networking and connectivity issues
Health Probes, Autoscaling & Self-HealingAdd liveness and readiness probes to the LLM chat API · Configure startup probes for containers with slow initialization · Scale the chat API automatically with HPA based on CPU · Create PodDisruptionBudgets for safe maintenance · Implement health check patterns for LLM-dependent services · Combine autoscaling, probes, and PDBs for a resilient LLM service
RBAC, Security & K8s TroubleshootingCreate RBAC roles for the LLM chat application · Enforce Pod Security Standards · Apply SecurityContext for defense in depth · Debug CrashLoopBackOff and OOMKilled failures · Use kubectl debug and ephemeral containers for live debugging · Troubleshoot LLM-specific issues: timeouts, proxy errors, stale connections
Web APIs & Services for GenAI Engineers5.3%
FastAPI FundamentalsCreate a FastAPI application with path operations · Define Pydantic request and response models · Implement dependency injection for shared resources · Build CRUD endpoints with proper HTTP semantics · Configure OpenAPI documentation with examples · Handle errors with custom exception handlers
Async Python for APIsConvert sync endpoints to async with proper await patterns · Implement background tasks for non-blocking operations · Execute concurrent API calls with asyncio.gather · Manage application lifecycle with lifespan handlers · Build async generators for streaming responses · Control concurrency with semaphores and throttling
Database IntegrationConfigure SQLAlchemy async engine with connection pooling · Define ORM models with relationships and constraints · Create and manage database migrations with Alembic · Implement repository pattern for data access · Build transactional endpoints with session lifecycle · Implement filtering, sorting, and full-text search
Authentication & AuthorizationImplement user registration with password hashing · Build OAuth2 password flow with JWT tokens · Implement API key authentication for services · Enforce role-based access control with permissions · Build token refresh and revocation · Compose multiple auth strategies into dependencies
Real-time StreamingBuild SSE endpoint for streaming LLM responses · Implement WebSocket endpoint with connection lifecycle · Build WebSocket connection manager for broadcasting · Handle backpressure and slow clients · Implement heartbeat and automatic reconnection · Build real-time notification system with Redis pub/sub
Resilience PatternsImplement rate limiting with Redis sliding window · Build circuit breaker for LLM provider calls · Configure retry logic with tenacity · Isolate critical paths with bulkhead semaphores · Build fallback responses for degraded mode · Combine resilience patterns into middleware stack
API Gateway & RoutingBuild reverse proxy with path-based routing · Implement load balancing across backend instances · Transform requests and responses through the gateway · Aggregate responses from multiple backends · Implement service discovery with health checking · Build gateway authentication and request enrichment
Testing & DocumentationWrite async endpoint tests with httpx.AsyncClient · Build database fixtures with transaction rollback · Mock external services for deterministic tests · Implement contract tests for API consumers · Measure test coverage and set quality gates · Generate rich OpenAPI documentation with examples
API Versioning & EvolutionImplement URL-based API versioning with routers · Build header-based version negotiation · Manage deprecation with Sunset and Warning headers · Build request and response adapters for version translation · Detect breaking changes automatically · Generate API changelogs from schema diffs
Deployment & ObservabilityBuild production Docker images with multi-stage builds · Deploy to Kubernetes with health check probes · Instrument endpoints with Prometheus metrics · Implement distributed tracing with OpenTelemetry · Build structured logging with correlation IDs · Create Grafana dashboards for API monitoring
GenAI Agent Engineering27.3%
The LLM ClientOpenAI client setup · Anthropic client setup · Google Gemini client setup · Build a unified LLM client interface · Error handling and provider fallback · Async LLM client patterns · Practical use cases — security, parameters, observability
Token EconomicsUnderstand tokenization · Count tokens across providers · Cost forecasting and budgeting · Track LLM API usage in production · Implement budget controls · Optimize tokens · Advanced context engineering
Prompt CachingImplement Anthropic cache_control · Leverage OpenAI automatic caching · Design cache-friendly prompt architectures · Build cache monitoring systems
The Function CallerOpenAI function schemas · Anthropic function schemas · Gemini function schemas · Handle tool call responses · Execute tools safely with Pydantic validation · Handle parallel tool calls · Framework integration with LangGraph
The Tool DefinerWrite clear tool descriptions for LLMs · Define parameter schemas · Use Pydantic for tool schemas · Implement tool decorators · Handle complex parameter types · Validate tool inputs at runtime · Framework tool patterns — LangGraph, CrewAI, OpenAI, Gemini, Anthropic
The Raw Agent LoopThe core agent while-loop · Manage context as a mutable list · Handle stop sequences · Track iteration limits · Tool execution in the loop · Build a conversation state tracker · Build with LangGraph StateGraph
The Prompt Engineer (Dynamic)Master Jinja2 templating for prompts · Implement dynamic few-shot example selection · Enforce Chain-of-Thought reasoning · Structure system prompts with a builder pattern · Inject dynamic context into prompts safely · Build prompt versioning and A/B testing
The ReAct Pattern (Manual)Build the Thought-Action generator · Tool execution and observation injection · Complete ReAct agent implementation · Advanced ReAct patterns — validation, retry, confidence · Optimize ReAct performance · Common ReAct pitfalls and solutions
The Planner PatternPlan generation · Step execution · Dynamic replanning · Hierarchical planning · Plan optimization · Monitoring and observability · Practical considerations — strategy selection
The Pydantic ToolPydantic fundamentals for tool definitions · Generate JSON Schema from Pydantic models · Input validation with custom validators · Build a Pydantic tool library · Advanced Pydantic patterns · Integrate Pydantic tools with agent frameworks · Common pitfalls and solutions
The Safe Executor (Sandboxing)Understand code execution risks · Static code analysis · Sandboxed execution · Apply resource limits · Build a complete safe executor
The Web NavigatorWeb navigation fundamentals · Web navigation tools — locating elements and forms · Browser automation with Playwright · Session management · Complete web navigator system
The MCP Protocol (Basics)JSON-RPC 2.0 message format and handler · Transport mechanisms — stdio and HTTP/SSE · Protocol lifecycle — initialization, runtime, shutdown · Capability negotiation
The MCP ServerCreate an MCP server with lifecycle management · Define MCP tools · Implement MCP resources · Create prompt templates · Error handling in MCP servers · Composable MCP server architecture
The MCP ClientMCP client architecture and stdio transport · Discover available tools and translate schemas · Proxy tool invocation · Fetch and use MCP resources · Manage MCP server lifecycle · Build multi-server MCP clients
The Tool RouterTool routing architecture and implementation · Namespace-based routing · Capability-based routing · Fallback chains · Routing performance optimization
Short-Term MemorySliding window memory · Token-aware memory management · Message summarization strategies · Memory persistence layers · Memory retrieval optimization · Integrate memory with agents · Memory performance considerations · Non-functional requirements (privacy + safety)
Long-Term Memory (RAG)Document chunking strategies · Embedding pipelines · Vector database integration · Hybrid search implementation · Retrieval optimization · RAG response generation
Agentic RAG PatternsSelf-reflective RAG · Multi-hop retrieval · Query routing · Adaptive retrieval · Retrieval feedback loops
Semantic MemoryKnowledge extraction pipelines · Entity and relationship extraction · Knowledge graph construction · Memory consolidation · Integrate semantic memory with agents · Build semantic memory with LangGraph
Context OptimizerContext economics · Dynamic context prioritization · Context compression techniques · Prompt optimization · Context utilization metrics · Complete context optimizer
The State GraphStateGraph fundamentals — config and lifecycle · Design state schemas with TypedDict · Add nodes to StateGraph · State initialization patterns · Tracing, debugging, validation
The Conditional EdgeUnderstand conditional edges · Design routing functions · Fan-out and fan-in patterns · Handle unknown routes and errors · Multi-stage routing
The Checkpointer (Time Travel)Resumable workflows · Inspect, replay, and time-travel · Retention, large state, and performance · Thread management — IDs and namespaces
Human-in-the-LoopLangGraph interrupt patterns · Approval workflow patterns · Interactive agent conversations · Feedback integration · State management for HITL · Practical use cases — escalation and analytics
The Streaming AgentStreaming modes in LangGraph · Token streaming from LLMs · Custom events with `astream_events` · Build streaming APIs · Error handling in streams · Backpressure and flow control
The Subgraph (Composition)Subgraph fundamentals — compile + test in isolation · State schema mapping · Subgraph checkpointers + namespace isolation · Compose subgraphs into a parent · Catch subgraph exceptions and recover · Define subgraph interfaces and build a registry · Build a multi-agent orchestrator
The Supervisor PatternDesign supervisor architectures · Worker agent specialization · Build the complete supervisor graph · Manage inter-agent communication · Handle failures and edge cases · Implement task aggregation · Build the supervisor pattern with CrewAI
The Hierarchical PatternDesign hierarchical agent architectures · Implement team-lead agents · Build cross-team coordination · Build the complete hierarchical graph
The Reflector Pattern (Critique)Design reflection architectures · Implement critic agents · Build the evaluation and convergence system · Build the complete reflection graph · Handle reflection edge cases · Practical use cases for reflection
Input GuardrailsDesign layered guardrail architectures · Format and schema validation · Build content filtering systems · Create injection / jailbreak detection · Implement policy-based guardrails · Assemble the complete guardrail system
Output GuardrailsDesign output validation architectures · Implement factual validation (hallucination detection) · Build content safety filters · Create PII redaction · Implement policy compliance · Assemble the complete output guardrail system
Prompt Injection DefenseIdentify injection vulnerabilities · Detect direct injections · Detect indirect injections · Implement defense layers · Build red-team suites · Implement canary tokens · LangGraph injection defense pipeline
Evaluations (Evals)Design evaluation frameworks · Implement automated evaluation pipelines · Create task-specific metrics · Human evaluation protocols · Regression testing · Set baselines and track progress
Agent BenchmarkingUnderstand the GAIA benchmark · Implement ToolBench evaluation · Use AgentBench · Design domain-specific benchmarks · Cross-model performance comparison · Build benchmark dashboards
Tracing & ObservabilityUnderstand distributed tracing · Add tags and metadata · Context propagation · Build feedback collection · Integrate with Langfuse · Trace visualization
Tool Use DebuggingTool selection failures and solutions · Argument validation systems · Build tool use dashboards and visualization · Schema mismatch detection · Tool call replay · Interactive tool debugger
Serving Agents (FastAPI)Async endpoints, request validation, error handling · Server-Sent Events (SSE) streaming · Background tasks · Design request and response schemas · Authentication — API keys, middleware, errors · OpenAPI metadata and documentation · FastAPI + LangGraph + uvicorn deployment
Podman & Containerization for K8sBuild optimized container images · Container health checks · Advanced image optimization · Security best practices · Build multi-container agent pods
Production Databases (Postgres/Redis)Async PostgreSQL configuration · Connection pool management · Redis caching for LLM responses · Database migrations for agent stacks · Backup and disaster recovery · Monitoring database health
Scaling & Load BalancingStateless service design · Session externalization · Load balancing algorithms · Scaling metrics for LLM workloads · Horizontal Pod Autoscaler configuration · Load testing your scaling design · Rate limiting at the load balancer
Multi-Tenant AgentsTenant context middleware · Database-level tenant isolation · Tenant-specific rate limiting and quotas · Per-tenant configuration templates · Usage metering for billing and SLA · Enforcing tenant data segregation at the API
Kubernetes (K8s) BasicsCreating Kubernetes Deployments · Resource management for LLM workloads · Kubernetes Services · ConfigMaps and Secrets · Rolling updates and CronJobs · Cluster planning and scheduling · Deployment planning synthesis · NetworkPolicies and prompt-injection defense at the edge
CI/CD for AgentsGitHub Actions for agent testing · Agent evaluation scripts in CI · Kubernetes deployment pipeline · GitOps deployment pattern · Quality gates and pipeline optimization · Rollback mechanisms · Pipeline observability and notifications
Monitoring & AlertingPrometheus metrics for agents · Grafana dashboards · Alerting configuration · SLOs and SLIs · Runbook creation for agent incidents · Tracking business KPIs for agent platforms · Agent-specific monitoring patterns (RED, USE, golden signals)
Model Routing & FallbacksCost-optimized routing · Latency-optimized routing · Building the resilient LLM client · Provider health checking · Cost tracking and optimization · Capability-based routing · Multi-model routing inside LangGraph
Long-Running AgentsCross-session persistence · Checkpoint serialization · Workflow resumption · Task queue integration with Celery · Progress tracking and SSE streaming · Timeout handling and graceful shutdown · Long-running agents with CrewAI — synthesis
Production Architecture PatternsSystem components, interfaces, and integration points · Cost modeling and projection · Production checklists and audit · Architecture Decision Records (ADRs) · Disaster recovery planning · System component diagrams · Architecture pattern evaluation — synthesis
Alternative Frameworks (CrewAI/AutoGen)CrewAI core concepts and agent personas · AutoGen conversational architecture · Framework comparison and integration patterns · Framework migration strategies and validation · Hybrid multi-framework systems · Picking a framework for a real use case
Deep Memory (GraphRAG)Knowledge graph fundamentals · Graph traversal patterns · Entity extraction with LLMs · Hybrid retrieval strategies (vector + graph) · Entity resolution and de-duplication · Incremental graph updates and provenance · Graph export, embeddings, and summarization
Agent Swarms & CollaborationSwarm architecture patterns · Agent communication and pub/sub · Consensus and weighted voting · Emergent behavior and stigmergy
Enterprise LLM Customization21.6%
Enterprise Data PipelineBuild DataPipeline with Instructor · Implement distillation data generation · Build a data quality dashboard · Build a data validation pipeline · Optimize data pipeline throughput · Build a data lineage tracker
Synthetic Data FactoryBuild SyntheticFactory with DSPy · Use OpenAI Batch API for bulk generation · Filter and validate synthetic data · Build synthetic data quality evaluator · Optimize Batch API cost and throughput · Build synthetic data versioning
Fine-Tuned Enterprise ModelFine-tune with OpenAI · Tune with Vertex AI · Build model comparison framework · Build fine-tuning hyperparameter search · Implement model distillation pipeline · Build model comparison report generator
RLVR-Trained Reasoning ModelBuild programmatic graders · Train via RFT API · Analyze training dynamics · Build grader reliability testing · Optimize RFT training cost · Build training run monitoring dashboard
Reward Engineering ToolkitBuild composite reward functions · Build LLM-as-judge graders · Validate grader reliability with Promptfoo · Build reward function A/B testing · Optimize composite reward weighting · Build reward engineering documentation generator
Model Eval DashboardConfigure Promptfoo eval suites · Use Batch API for bulk eval · Build regression detection · Build eval suite versioning and management · Optimize eval pipeline cost with sampling · Build eval regression root cause analyzer
Model Selection EngineBuild ModelSelector · Build cost-quality optimizer · Benchmark HuggingFace open models · Build model selection test harness · Optimize model selection latency · Build model catalog and recommendation engine
Reasoning Model BenchmarkBenchmark reasoning across providers · Analyze reasoning patterns · Build task-specific reasoning profiles · Build testing and validation for reasoning model benchmark · Optimize performance for reasoning model benchmark · Build operational runbook for reasoning model benchmark
TTC ControllerBuild TTC scaling curves · Implement compute-aware routing · Build TTC cost savings dashboard · Build testing and validation for ttc controller · Optimize performance for ttc controller · Build operational runbook for ttc controller
Best-of-N Inference EngineBuild BestOfN engine · Optimize N and cost tradeoff · Build BestOfN + RLVR hybrid strategy · Build testing and validation for best-of-n inference engine · Optimize performance for best-of-n inference engine · Build operational runbook for best-of-n inference engine
Self-Refining AgentBuild SelfRefiner · Implement long-horizon planning · Build refinement convergence analysis · Build testing and validation for self-refining agent · Optimize performance for self-refining agent · Build operational runbook for self-refining agent
Compute-Optimal RouterBuild ComputeRouter · Evaluate routing performance · Build routing feedback loop with learning · Build testing and validation for compute-optimal router · Optimize performance for compute-optimal router · Build operational runbook for compute-optimal router
DSPy Enterprise ModulesBuild 5 enterprise DSPy modules · Compare DSPy vs hand-written · Test DSPy modules with Promptfoo · Build testing and validation for dspy enterprise modules · Optimize performance for dspy enterprise modules · Build operational runbook for dspy enterprise modules
DSPy Optimizer as Meta-RLRun MIPROv2 optimization · Analyze optimization dynamics · Compare cross-provider optimization · Build testing and validation for dspy optimizer as meta-rl · Optimize performance for dspy optimizer as meta-rl · Build operational runbook for dspy optimizer as meta-rl
DSPy + Skills PipelinesBuild multi-stage pipeline · Test pipeline reliability · Build pipeline monitoring and alerting · Build testing and validation for dspy + skills pipelines · Optimize performance for dspy + skills pipelines · Build operational runbook for dspy + skills pipelines
Structured Extraction with CitationsBuild CitedExtractor · Validate extraction with Promptfoo · Compare multi-provider extraction · Build testing and validation for structured extraction with citations · Optimize performance for structured extraction with citations · Build operational runbook for structured extraction with citations
LiteLLM GatewayDeploy LiteLLM proxy · Implement failover and circuit breakers · Load test and capacity plan · Build testing and validation for litellm gateway · Optimize performance for litellm gateway · Build operational runbook for litellm gateway
Smart RouterBuild intelligent routing · Implement context caching · Build routing analytics dashboard · Build testing and validation for smart router · Optimize performance for smart router · Build operational runbook for smart router
Provider Caching ComparisonImplement all caching systems · Build caching strategy recommender · Build cache invalidation and consistency · Build testing and validation for provider caching comparison · Optimize performance for provider caching comparison · Build operational runbook for provider caching comparison
FinOps ControllerBuild budget and spend tracking · Auto-route to Batch API · Build cost anomaly detection · Build testing and validation for finops controller · Optimize performance for finops controller · Build operational runbook for finops controller
Langfuse ObservabilityDeploy and integrate Langfuse · Build drift detection and alerts · Build custom eval pipelines in Langfuse · Build testing and validation for langfuse observability · Optimize performance for langfuse observability · Build operational runbook for langfuse observability
A/B Testing + Prompt CI/CDBuild prompt CI/CD pipeline · Build model deployment lifecycle · Build multi-environment promotion pipeline · Build testing and validation for a/b testing + prompt ci/cd · Optimize performance for a/b testing + prompt ci/cd · Build operational runbook for a/b testing + prompt ci/cd
Agent Mesh: A2A + ADKBuild A2A agent mesh · Build AI governance framework · Build agent mesh observability dashboard · Build testing and validation for agent mesh: a2a + adk · Optimize performance for agent mesh: a2a + adk · Build operational runbook for agent mesh: a2a + adk
Guardrails PipelineBuild guardrails pipeline · Test guardrails under adversarial input · Benchmark guardrail performance · Build security testing for guardrails pipeline · Optimize throughput for guardrails pipeline · Build compliance reporting for guardrails pipeline
Prompt Injection Red TeamRun OWASP LLM Top 10 attacks · Build security assessment report · Build automated security regression pipeline · Build security testing for prompt injection red team · Optimize throughput for prompt injection red team · Build compliance reporting for prompt injection red team
Data Classification RouterBuild DataClassifier · Integrate DLP controls · Build data lineage tracking · Build security testing for data classification router · Optimize throughput for data classification router · Build compliance reporting for data classification router
Compliance Audit TrailBuild audit logging system · Build compliance reporting · Build audit data retention and archival · Build security testing for compliance audit trail · Optimize throughput for compliance audit trail · Build compliance reporting for compliance audit trail
Compliance Test SuiteBuild regulatory compliance tests · Build bias and fairness tests · Build continuous compliance monitoring · Build security testing for compliance test suite · Optimize throughput for compliance test suite · Build compliance reporting for compliance test suite
Production RAG PipelineBuild EnterpriseRAGPipeline · Build RAGEvaluator with RAGAS · Build HybridRetriever with reranking · Build testing and evaluation for production rag pipeline · Optimize performance for production rag pipeline · Build operational monitoring for production rag pipeline
Vector Database EngineeringBuild VectorDBBenchmark · Build MultiIndexManager · Build EmbeddingOptimizer · Build testing and evaluation for vector database engineering · Optimize performance for vector database engineering · Build operational monitoring for vector database engineering
LangGraph Agentic OrchestrationBuild LangGraphAgent with stateful workflows · Build OrchestrationComparator · Build LangGraphRAGAgent · Build testing and evaluation for langgraph agentic orchestration · Optimize performance for langgraph agentic orchestration · Build operational monitoring for langgraph agentic orchestration
GraphRAG & Knowledge GraphsBuild KnowledgeGraphBuilder · Build GraphRAGRetriever · Build HybridKnowledgeSearch · Build testing and evaluation for graphrag & knowledge graphs · Optimize performance for graphrag & knowledge graphs · Build operational monitoring for graphrag & knowledge graphs
Agent Memory & Stateful SystemsBuild AgentMemorySystem · Build MemoryManager · Build StatefulAgentBenchmark · Build testing and evaluation for agent memory & stateful systems · Optimize performance for agent memory & stateful systems · Build operational monitoring for agent memory & stateful systems
Advanced RAG PatternsBuild AdaptiveRAG · Build RAGCIPipeline · Build ProductionRAGDashboard · Build testing and evaluation for advanced rag patterns · Optimize performance for advanced rag patterns · Build operational monitoring for advanced rag patterns
Healthcare: Clinical AdvisorBuild clinical advisory pipeline · Implement HIPAA compliance · Build clinical accuracy validation · Build end-to-end testing for healthcare: clinical advisor · Optimize cost and performance for healthcare: clinical advisor · Build production operations for healthcare: clinical advisor
Finance: Deal AnalyzerBuild deal analysis pipeline · Build FinOps controls · Build cross-department usage analytics · Build end-to-end testing for finance: deal analyzer · Optimize cost and performance for finance: deal analyzer · Build production operations for finance: deal analyzer
Legal: Regulatory MonitorBuild regulatory monitoring mesh · Validate legal accuracy · Build regulatory change impact scoring · Build end-to-end testing for legal: regulatory monitor · Optimize cost and performance for legal: regulatory monitor · Build production operations for legal: regulatory monitor
Customer Ops: ResolutionBuild customer resolution system · Security and observability · Build resolution quality feedback loop · Build end-to-end testing for customer ops: resolution · Optimize cost and performance for customer ops: resolution · Build production operations for customer ops: resolution
Multimodal EnterpriseBuild multimodal processor · Validate multimodal accuracy · Build document type router with fallbacks · Build end-to-end testing for multimodal enterprise · Optimize cost and performance for multimodal enterprise · Build production operations for multimodal enterprise
Capstone: Enterprise AI PlatformBuild enterprise AI platform · Run production readiness assessment · Build disaster recovery and runbook · Build end-to-end testing for capstone: enterprise ai platform · Optimize cost and performance for capstone: enterprise ai platform · Build production operations for capstone: enterprise ai platform
Enterprise LLM Platform CapstonePlatform Integration · Multi-Tenant Operations · Production Readiness Assessment · Compliance Reporting · Customer Onboarding Demo · Platform Capstone Report
GenAI Operations8.9%
GenAI SLI FrameworkDefine latency SLIs: TTFT, tokens-per-second, end-to-end response time across providers · Define quality SLIs: faithfulness, hallucination rate, format compliance, retrieval precision · Define cost SLIs: cost-per-request, cost-per-token, cache hit rate, budget burn rate · Instrument all SLIs with Prometheus metrics and Langfuse traces · Build SLI aggregation and reporting API · Implement SLI validation and testing
GenAI SLO EngineDefine SLO targets for latency, quality, and cost SLIs with business-justified thresholds · Compute error budgets and track consumption over rolling windows · Build multi-window burn-rate alerts that detect SLO violations before budget exhaustion · Create SLO status dashboards showing budget remaining and projected exhaustion · Implement SLO negotiation framework · Build cross-SLO dependency tracking
GenAI Launch ReadinessDefine operational readiness criteria specific to GenAI services · Build automated readiness checks that verify infrastructure, monitoring, and runbook completeness · Implement launch gate enforcement that blocks deployment without readiness sign-off · Create readiness dashboards and historical tracking for continuous improvement · Implement progressive readiness rollout · Create readiness automation toolkit
AI Incident CommanderDefine LLM-specific incident severity classification with impact-based criteria · Build incident lifecycle management with role assignments and status tracking · Create communication templates for AI-specific incidents targeting different audiences · Track incident metrics with MTTD, MTTA, MTTR and trend analysis · Implement performance optimization for llm incident response framework · Build operational documentation for llm incident response framework
Runbook Automation EngineBuild alert-to-runbook routing that triggers automated remediation workflows · Implement human approval gates for high-impact remediation steps · Create runbook execution auditing with step-by-step logging and outcome tracking · Track automation coverage and success rates across all runbook types · Implement performance optimization for automated runbook execution · Build operational documentation for automated runbook execution
AI Post-Mortem EngineBuild structured post-mortem templates for GenAI failure modes · Implement timeline reconstruction from Langfuse traces and Prometheus metrics · Create action item tracking with follow-through verification · Analyze post-mortem trends to identify systemic issues · Implement performance optimization for post-mortems for ai failures · Build operational documentation for post-mortems for ai failures
Cost Attribution EngineInstrument per-request cost tracking across all pipeline stages · Build cost attribution to teams, projects, and use cases · Create cost allocation models for shared infrastructure components · Implement cost anomaly detection with automated investigation · Implement performance optimization for full-stack cost attribution · Build operational documentation for full-stack cost attribution
Token Budget ControllerConfigure LiteLLM Virtual Keys with Per-Team Budget Limits · Implement Per-Request Token Limits · Build Budget Alerting at 50%, 80%, and 100% Thresholds with Escalation · Create Budget Override Workflows for Emergency Usage Beyond Limits · Implement performance optimization for token budget enforcement · Build operational documentation for token budget enforcement
Cache Economics AnalyzerDeploy Redis Semantic Cache and Measure Hit Rate vs Cost Savings · Compare Provider Caching Strategies for OpenAI, Anthropic, and Google · Build Cost-Benefit Analysis with Break-Even Calculations · Recommend Optimal Caching Mix Per Use Case · Implement performance optimization for caching roi analysis · Build operational documentation for caching roi analysis
Batch API SchedulerImplement workload classification: real-time vs batch-eligible based on latency requirements · Build Batch API job scheduling with priority queues and SLA tracking · Create batch job monitoring with completion time SLAs and failure handling · Measure and report cost savings from batch routing vs synchronous requests · Implement performance optimization for batch api optimization · Build operational documentation for batch api optimization
Capacity ForecasterBuild token demand forecasting using historical usage patterns and trend analysis · Implement embedding volume projection for storage and compute planning · Create cost projection models for budget planning cycles · Track forecast accuracy and improve models over time with feedback loops · Implement performance optimization for capacity forecasting · Build operational documentation for capacity forecasting
FinOps Governance PlatformBuild Showback and Chargeback Reports per Team and Project with Full Cost Transparency · Create Executive FinOps Dashboards with Trend Analysis for Leadership · Implement Cost Governance Policies with Automated Enforcement · Generate Monthly FinOps Reviews with Optimization Recommendations · Implement performance optimization for finops reporting and governance · Build operational documentation for finops reporting and governance
Multi-Tenant GenAI PlatformAutomate tenant onboarding with namespace provisioning and secret management · Implement namespace isolation with network policies and resource quotas · Build noisy-neighbor detection that identifies tenants causing resource contention · Create tenant operations dashboards with per-tenant health visibility · Implement performance optimization for multi-tenant platform operations · Build operational documentation for multi-tenant platform operations
AI Developer PlatformBuild self-service deployment workflows with approval gates for AI artifacts · Create golden path templates for common GenAI patterns · Implement internal tool marketplace for reusable AI components · Build developer experience metrics and platform analytics · Implement performance optimization for internal developer platform for ai · Build operational documentation for internal developer platform for ai
GenAI Ops Maturity AssessorDefine GenAI operational maturity model with five levels across eight capability areas · Build automated maturity assessment that evaluates current operational state · Generate improvement roadmaps with prioritized actions based on assessment results · Track maturity progression over time with milestone tracking · Implement performance optimization for operational maturity model · Build operational documentation for operational maturity model
AI Incident ResponseAI Incident Taxonomy · Automated Detection · AI Incident Runbooks · Post-Incident Review · Escalation Path Design · Incident Response Capstone
AI Governance Compliance OpsEU AI Act Controls · NIST AI RMF Implementation · SOC2 AI Controls · Compliance Dashboard · Audit Preparation Workflow · Compliance Ops Capstone
GenAI Architecture & Design Patterns13.7%
GenAI ADR EngineBuild ADR schema and decision taxonomy for GenAI technology choices · Implement multi-provider model selection ADR workflow · Validate ADR decisions against production telemetry · Build ADR dependency graph across system decisions · Implement ADR recommendation engine using historical outcomes · Create ADR governance dashboard and compliance audit
Reference Architecture RegistryBuild reference architecture schema with C4 model layers · Catalog five canonical GenAI reference architectures · Validate architecture blueprints against quality fitness functions · Build architecture pattern matching for requirements-to-blueprint selection · Implement architecture evolution planner for incremental migration · Create architecture registry dashboard with adoption tracking
Compound AI System DesignerBuild compound AI system topology modeler · Implement cascade routing with cost-aware model selection · Validate compound system against end-to-end quality requirements · Model failure propagation and blast radius in compound systems · Optimize compound system for cost-quality Pareto frontier · Create compound system architecture review checklist and report
Eval-First Architecture EngineBuild eval gate component with pluggable evaluator registry · Integrate eval gates into retrieval, generation, and agent pipelines · Measure eval gate effectiveness with precision-recall tracking · Build eval cascade with tiered evaluation depth · Optimize eval gate latency overhead with async and sampling strategies · Create eval architecture audit report with coverage analysis
Cost-Optimized AI RouterBuild cost-aware request router with tiered model cascade · Implement semantic caching architecture with cache-aside pattern · Measure cost savings and quality impact per optimization strategy · Build token budget enforcement with per-tenant and per-request limits · Optimize batch processing architecture for non-real-time workloads · Create cost architecture review and FinOps governance report
AI Traffic GatewayBuild AI-aware API gateway with provider-specific routing rules · Implement Istio service mesh for AI microservice communication · Measure gateway latency overhead and optimize hot paths · Build intelligent rate limiting with token-aware throttling · Implement circuit breaker and bulkhead patterns for provider isolation · Create gateway operations dashboard with traffic topology view
Streaming AI PipelineBuild streaming response pipeline with SSE as default transport · Implement streaming-aware guardrails that evaluate partial output · Measure streaming quality metrics including TTFT and token throughput · Build event-driven pipeline for streaming data transformation · Optimize streaming architecture for connection efficiency and recovery · Create streaming architecture health dashboard and SLA report
Cell-Based AI PlatformBuild cell definition schema with namespace isolation on GKE Autopilot · Implement traffic routing between cells for canary and shadow deployments · Validate cell isolation with blast radius testing · Build feature-flag-driven model rollout across cells · Optimize cell resource allocation with autoscaling policies · Create cell operations dashboard with fleet-wide topology view
AI Observability StackBuild OpenTelemetry instrumentation for multi-step AI pipelines · Implement AI-specific metrics collection with quality dimensions · Validate trace completeness and data quality across the pipeline · Build cross-signal correlation linking traces to quality degradation · Optimize observability cost with intelligent data reduction · Create observability architecture assessment and maturity dashboard
Layered AI Security SystemBuild layered guardrail pipeline with pluggable security stages · Integrate NeMo Guardrails and LlamaFirewall for multi-layer defense · Measure guardrail effectiveness with red-team evaluation metrics · Build security event correlation across guardrail layers · Optimize guardrail latency with tiered security evaluation · Create security architecture audit report with compliance mapping
MCP Tool MeshBuild MCP server registry with capability discovery and health checks · Implement MCP authorization layer with per-tool permission policies · Validate MCP tool composition correctness and safety · Build MCP tool routing with load balancing and failover · Optimize MCP tool selection with intelligent pre-filtering · Create MCP ecosystem governance dashboard
A2A Agent NetworkBuild A2A agent card registry with capability advertisement · Implement A2A task delegation with streaming artifact exchange · Validate A2A communication reliability with failure injection · Build A2A agent trust and authorization framework · Optimize A2A network topology for latency and reliability · Create A2A network operations dashboard with federation view
Event-Driven AI ProcessorBuild event schema registry for GenAI domain events · Implement event backbone with Redis Streams for AI workloads · Validate event pipeline reliability with exactly-once semantics · Build event-sourced audit trail for AI decision traceability · Optimize event pipeline throughput with partitioning and batching · Create event architecture topology dashboard and maturity report
Multi-Modal AI PipelineBuild multi-modal request router with modality detection · Implement vision-language pipeline with document understanding · Validate multi-modal output quality with per-modality evaluators · Build modality-specific caching and preprocessing optimization · Optimize multi-modal pipeline for latency with parallel processing · Create multi-modal architecture capability assessment
Enterprise RAG SystemBuild hybrid retrieval pipeline combining dense, sparse, and graph search · Implement multi-index federation with routing by query intent · Validate retrieval quality with RAGAS metrics and human evaluation · Build real-time document ingestion with incremental index updates · Optimize retrieval latency with query planning and result caching · Create RAG architecture health dashboard and capacity report
Agent Orchestration PlatformBuild agent pool manager with lifecycle and capability registration · Implement supervisor hierarchy with delegation and escalation policies · Validate orchestration correctness with agent trajectory evaluation · Build task queue with priority scheduling and resource-aware dispatch · Optimize orchestration for cost with agent budget management · Create agent orchestration operations dashboard
AI Developer PlatformBuild Backstage service catalog for AI components · Implement golden path templates for AI service creation · Validate platform compliance with automated standards checking · Build self-service AI environment provisioning pipeline · Optimize developer experience with platform usage analytics · Create platform engineering maturity assessment dashboard
Provider Reliability EngineBuild provider health tracker with real-time status aggregation · Implement intelligent failover with quality-preserving model mapping · Validate failover behavior with chaos engineering experiments · Build shadow deployment for cross-provider model comparison · Optimize provider selection with cost-quality-latency Pareto routing · Create provider reliability SLA report and vendor risk assessment
Context Engineering PlatformBuild context composition engine with priority-ranked information sources · Implement context compression with summarization and pruning strategies · Validate context quality impact on generation accuracy · Build context caching with freshness-aware invalidation · Optimize context utilization with dynamic window sizing · Create context architecture audit report with optimization roadmap
AI Data ArchitectureBuild unified data access layer across vector, graph, and relational stores · Implement schema evolution strategy for embedding and knowledge data · Validate data consistency across stores with reconciliation checks · Build data lifecycle manager with retention and archival policies · Optimize data access patterns with caching and pre-computation · Create data architecture health report with capacity planning
Enterprise AI Integration HubBuild integration adapter framework with standardized connector interface · Implement data transformation pipeline between enterprise and AI formats · Validate integration correctness with contract testing · Build integration error handling with dead letter and retry strategies · Optimize integration throughput with batching and parallel processing · Create enterprise integration architecture map and health report
AI Governance PlatformBuild data classification engine for AI input and output streams · Implement model usage policy engine with approval workflows · Validate compliance coverage with automated control testing · Build comprehensive audit trail with immutable event logging · Optimize compliance overhead with risk-based control depth · Create regulatory compliance report generator for SOC2 and GDPR
HA/DR AI PlatformBuild HA topology planner with single-point-of-failure analysis · Implement stateful failover for conversation and embedding stores · Validate DR procedures with automated failover testing · Build automated DR runbook with step-by-step failover execution · Optimize recovery speed with pre-staged failover resources · Create HA/DR architecture dashboard with readiness assessment
Multi-Tenant AI PlatformBuild tenant isolation framework with namespace and network boundaries · Implement tenant-aware routing with per-tenant model and guardrail config · Validate tenant isolation with cross-tenant data leakage testing · Build per-tenant cost management with budget enforcement · Optimize tenant resource sharing with noisy-neighbor prevention · Create multi-tenant platform operations dashboard
Architecture Review EngineBuild architecture review checklist engine with domain-specific criteria · Implement technology selection framework with multi-criteria scoring · Validate review coverage with architecture surface area mapping · Build review knowledge base from historical architecture decisions · Optimize review process with risk-based prioritization · Create architecture governance report for leadership
Production AI Platform CapstoneDesign complete platform architecture with C4 model documentation · Implement architecture prototype with compound AI pipeline · Validate platform against quality, security, and reliability requirements · Build platform cost model with total cost of ownership projection · Conduct architecture review applying all course frameworks · Generate architecture handoff package for solution delivery team
GenAI Evaluation, Safety & Governance1.1%
EU AI Act ComplianceClassify AI systems under EU AI Act risk categories · Implement the Feb 2025 AI literacy requirements · Build technical documentation for GPAI compliance · Implement risk management system · Build human oversight mechanisms · Track EU AI Act enforcement timeline compliance
Compliance FrameworksImplement NIST AI RMF Govern and Map functions · Implement NIST AI RMF Measure and Manage functions · Build ISO 42001 AI management system documentation · Create unified governance dashboard with Credo AI Agent Registry · Implement comprehensive audit trail · Build compliance automation and alerting
AI Solution Delivery5.3%
AI Use Case Discovery & Data Readiness AssessmentScore AI use cases with weighted multi-criteria evaluation · Profile customer datasets for quality and PII exposure · Run LLM-driven discovery interviews with LangGraph state · Benchmark provider feasibility across OpenAI, Gemini, Anthropic · Generate executive discovery reports from structured assessment data
Solution Scoping & Effort EstimationDecompose AI projects into a hierarchical work breakdown structure · Classify project risks with DSPy-optimized prompts · Detect scope drift via embedding similarity · Plan resource allocation with constraint-based scheduling · Assemble versioned scope documents with diff tracking
SOW & Proposal GenerationGenerate milestone schedules with critical-path analysis · Extract SMART acceptance criteria from raw requirements · Estimate AI project pricing across tokens, infra, and labor · Generate full SOW proposals with LangGraph workflows · Detect risky contract language with NeMo Guardrails
Rapid AI PrototypingBuild a RAG prototype with pgvector retrieval · Prototype LangGraph agents with MCP tools and human approval · Compare providers side-by-side with LiteLLM showcase · Score prototype demo readiness with RAGAS and LLM-as-judge · Package prototypes with Dockerfiles, Helm charts, and K8s manifests
Customer Data Integration PipelinesBuild pluggable data connectors for SQL, REST, files, and S3 · Map customer schemas to target schemas with LLM-assisted suggestions · Detect and redact PII with Presidio and LlamaGuard 4 · Embed documents incrementally with content-hash dedup · Monitor data quality with OTEL throughput and precision metrics
Deploying in Customer EnvironmentsGenerate K8s manifests from customer-parameterized Jinja2 templates · Manage K8s secrets with rotation and init-container injection · Enforce service isolation with K8s NetworkPolicy · Validate deployments with probe checks and inference smoke tests · Log compliance events as OTEL traces with structured attributes
Stakeholder Communication & Demo EngineeringOrchestrate scripted demos with fallback scenarios · Generate technical review reports from live system data · Translate technical reports into executive briefings · Collect stakeholder feedback with sentiment analysis · Provision isolated K8s demo environments with TTL teardown
Delivery Risk Management & GovernanceTrack risks in a CRUD register with LLM-enriched descriptions · Detect scope drift with embedding similarity classification · Monitor delivery health from standup notes and velocity data · Enforce governance gates with LLM-as-judge evaluation · Automate escalations with threshold-based webhooks
POC to Production HardeningLayer guardrails with NeMo and LlamaFirewall PromptGuard 2 · Load-test LLM endpoints with Locust and SSE stream validation · Deploy with blue-green Helm charts and atomic service switching · Validate production readiness against OWASP LLM Top 10 · Auto-rollback on quality regression with RAGAS monitoring
Knowledge Transfer & Training AutomationGenerate API docs and Mermaid diagrams from source code · Generate runbooks from K8s configs with LangGraph workflows · Generate training quizzes with DSPy-optimized prompts · Search a knowledge base semantically with source attribution · Score handoff readiness from quiz, runbook, and coverage signals
Post-Delivery Support & SLA MonitoringTrack SLAs with multi-category threshold breach detection · Detect quality anomalies with OTEL sliding-window analysis · Run incident response pipelines with severity classification · Schedule maintenance from quality degradation signals · Aggregate support analytics with MTTR and cost-per-resolution
Delivery Capstone — End-to-End AI EngagementOrchestrate end-to-end delivery with LangGraph state · Coordinate a multi-agent delivery team with Google ADK and A2A · Validate phase transitions with quality gate pipelines · Extract retrospective lessons with Instructor and Anthropic · Track multi-engagement portfolios with health scoring
Core responsibilities this discipline prepares you for. Open one to see how you build it.
Lead end-to-end GenAI project delivery from discovery through production handoff
- Run the complete delivery lifecycle: discovery workshops → problem scoping → rapid prototyping → handoff
- Drive evaluation-driven iteration with measurable quality gates and knowledge transfer
- Walk through each delivery phase with realistic client scenarios including scoping and risk assessment
Design GenAI architecture for client engagements
- Apply cell-based AI, MCP mesh, and multi-tenant architecture patterns to client requirements
- Write ADR documentation with reference designs and technology evaluation rationale
- Create architecture proposals for varied client scenarios and defend design decisions under review
Build agent-based solutions for client business processes
- Design LangGraph agents with MCP tool integration and human-in-the-loop approval gates
- Customize agent behavior, tool access, and workflow logic for different business process requirements
- Build and deploy domain-specific agents adapted to varied client business scenarios
Customize enterprise LLM deployments — gateways, RAG, domain adaptation
- Operate LiteLLM gateways with multi-provider management and enterprise RAG stack customization
- Adapt LLM deployments for healthcare, finance, and legal verticals with domain-specific constraints
- Deliver end-to-end LLM customization for regulated industries with compliance validation
Manage FinOps for client GenAI projects
- Build token cost attribution models with budget forecasting and TCO analysis for proposals
- Design cost optimization strategies across providers, caching tiers, and model selection
- Build cost models, forecast annual spend, and present optimization recommendations to stakeholders
Scope project timelines and team requirements
- Apply effort estimation frameworks designed for non-deterministic GenAI project delivery
- Map team skills, assess technical risks, and develop detailed project proposals
- Estimate effort for sample GenAI projects and identify optimal team composition and skill coverage
Package solutions as deployable artifacts for client operations teams
- Build Helm charts with operational runbooks, SLA definitions, and integrated monitoring
- Create client handoff documentation with deployment guides and escalation procedures
- Package a complete GenAI solution and conduct a simulated client handoff with operational validation
Advise clients on technology roadmaps with emerging GenAI patterns
- Evaluate emerging patterns: A2A protocol, MCP mesh, cell-based AI, and multi-tenant architectures
- Assess industry trends, adoption timelines, and migration strategies for client technology stacks
- Build technology roadmap recommendations that balance innovation with operational stability
Each group becomes a signed node on your public skill graph — a recruiter-auditable record of the labs you actually passed. Share the URL; they see the discipline, the labs, and the signature.
POC-to-Production Transition
Taking a delivered GenAI POC to production: transition playbooks, observability/SLA instrumentation, robustness hardening, capacity planning. The lab-provable delivery deliverable.
Quality Gates & Eval Pipelines
Eval-driven delivery quality: CI/CD quality gates, golden-trajectory regression suites, RAGAS/DeepEval pipelines, LLM-judge calibration. The strongest lab-assessable bar for this role.
Architecture Conformance & Reviews
Ensuring a delivery conforms to the target architecture: ADR/reference-architecture conformance, technical-standards adherence, architectural-quality gates. The lab-checkable conformance face (the review-judgment half is reserved for the portfolio track).
Solution Tech Depth: Agents + RAG
Hands-on technical depth: agent loops, ReAct, MCP, multi-agent orchestration, production RAG, vector + graph DB engineering. Demonstrable in labs.
Production Ops & Cost Management
Running the delivered solution: incident-response runbooks, automation, cost attribution/chargeback, token-budget controllers, capacity forecasting, FinOps.
Delivery Risk & Compliance (Technical)
The technical/automatable face of delivery governance: compliance pre-flight (EU AI Act/SOC2/HIPAA), regulatory-artifact generation, change-control gates, risk-register tooling. Governance-committee facilitation is reserved for the portfolio track.
Solution Scoping (Technical Feasibility)
The lab-assessable feasibility face of scoping: data-readiness assessment, technical feasibility analysis, effort estimation from technical signals, success-criteria definition. Live client discovery/qualification is reserved for the portfolio track.
Prototyping & Demo Engineering
The buildable half of delivery framing: rapid prototyping for client validation, scaffold libraries, demo engineering, synthetic-data scaffolds. Delivery-framework design is reserved for the portfolio track.
Hosted LLM API Integration
Provider SDK literacy: OpenAI/Anthropic/Gemini integration, multi-provider abstraction. Prerequisite.
Python for Delivery Engineering
Production-grade Python for delivery tooling: async, Pydantic, typing, pytest. Prerequisite.