12 skill groups · 8 courses · 869 goals

LLMOps Engineering

Monitor hallucination rates and token costs, operate guardrails and eval gates, manage prompt versioning and canary deployments.

Preview 7 goals free Subscribe

cancel anytime · save 20% on 6 months

8 courses, each building on the last. The bar shows each course's share of the curriculum. 7 goals are unlocked for preview.

Python Essentials for Agent Builders7.1%62 goals

Your Dev Environment4 goalsNavigate filesystem with terminal · Manage files from command line · Set up VS Code · Configure terminal in VS Code

Python, Git & Package Management6 goalsInstall and verify Python · Write hello world script · Use Python REPL · Initialize Git repository · Track changes with Git · Install packages with pip

Variables & Basic Types5 goalsCreate and name variables · Work with strings · Work with numbers · Work with booleans · Format with f-strings

Control Flow4 goalsMake decisions with if/elif/else · Iterate with for loops · Repeat with while loops · Control loop execution

Functions5 goalsDefine and call functions · Use parameters · Return values · Document with docstrings · Understand scope

Modules & Imports4 goalsImport standard library · Create custom modules · Understand Python path · Create packages

Lists & Tuples5 goalsCreate and access lists · Modify lists · Slice lists · Use list comprehensions · Work with tuples

Dictionaries & Sets5 goalsCreate and access dicts · Modify dictionaries · Iterate over dicts · Work with nested dicts · Use sets

Classes & Dataclasses5 goalsUnderstand class basics · Create dataclasses · Add methods · Use default values · Basic inheritance

Files, JSON & Error Handling5 goalsRead and write files · Work with JSON · Use pathlib · Handle exceptions · Create custom exceptions

Basic Testing4 goalsUse assert statements · Create test functions · Run pytest · Test classes

Environment Variables & Configuration5 goalsUnderstand environment variables · Use .env files · Load with python-dotenv · Handle missing variables · Organize configuration

Decorators & Context Managers5 goalsUnderstand decorators · Write simple decorators · Use context managers · Write context managers · Combine patterns

LLM Foundations for Agent Builders10.0%87 goals

Generators & Iterators5 goalsUnderstand iteration · Create generators · Use generator expressions · Build data pipelines · Use itertools

Async Programming Basics5 goalsUnderstand async concepts · Write async functions · Run concurrent operations · Use async context managers · Handle async exceptions

Type Hints & Pydantic5 goalsAdd basic type hints · Use typing generics · Create Pydantic models · Validate API data · Configure Pydantic

Data Pipelines & Transformations5 goalsBuild functional pipelines · Work with tabular data · Transform data shapes · Process LLM data formats · Optimize for performance

HTTP Clients & httpx5 goalsMake GET requests · Make POST requests · Use async httpx · Handle errors · Use sessions

Your First LLM Call5 goalsSet up credentials · Install Gemini SDK · Make first API call · Parse response · Handle API errors

Tokenizer Internals5 goalsUnderstand tokenization basics · Learn BPE algorithm · Compare tokenizer types · Analyze cross-language efficiency · Count and optimize tokens

Context Windows, KV-Cache & Memory6 goalsUnderstand context limits · Understand KV-cache basics · Use prompt caching · Track conversation tokens · Implement truncation · Summarize for compression

LLM Architectures - Dense, MoE & KV-Cache Optimizations6 goalsUnderstand transformers · Explore dense models · Understand MoE · Master KV-cache optimizations · Compare architectures · Choose for task

LLM Inference - Prefill & Decode5 goalsUnderstand prefill phase · Understand decode phase · Run local models · Measure inference metrics · Optimize for latency

Sampling Parameters & Output Control5 goalsUnderstand temperature · Use top-p sampling · Implement determinism · Control output length · Use structured output

Multi-Provider & Prompt Engineering5 goalsBuild provider abstraction · Structure conversations · Use few-shot prompting · Implement chain-of-thought · Build prompt templates

Function Calling Fundamentals5 goalsUnderstand tool use concept · Define tool schemas · Make function calls · Handle tool responses · Compare provider patterns

Embeddings & Semantic Search5 goalsUnderstand embeddings · Generate embeddings · Calculate similarity · Build simple search · Compare embedding models

RAG Fundamentals5 goalsUnderstand RAG pattern · Chunk documents · Build retrieval pipeline · Compose RAG prompts · Evaluate RAG quality

Cost Awareness & Token Economics5 goalsUnderstand pricing models · Calculate request costs · Compare provider costs · Identify cost drivers · Basic cost optimization

Retry Patterns with Tenacity5 goalsUnderstand retry need · Use tenacity basics · Implement exponential backoff · Handle specific exceptions · Combine with async

Kubernetes Essentials for GenAI Engineers8.3%72 goals

Containerizing LLM Applications6 goalsWrite a Python app that calls the Gemini API and returns structured responses · Write a Dockerfile and build a container image for the LLM app · Run the containerized LLM app with environment-based configuration · Use Docker Compose to run the LLM app with supporting services · Tag images with semantic versions and push to a container registry · Debug containers with exec, logs, and inspect

Your Kubernetes Cluster & First LLM Pod6 goalsUnderstand K8s architecture and connect to your vCluster · Deploy the LLM app as your first Kubernetes pod · Organize workloads with namespaces · Use labels and selectors to organize and query resources · Understand pod lifecycle and restart policies · Master kubectl debugging: exec, logs, describe, port-forward

Services & the LLM Chat Backend6 goalsCreate a ClusterIP service to expose the LLM chat API internally · Deploy a multi-tier LLM chat application · Compare service types: ClusterIP, NodePort, LoadBalancer · Master DNS-based service discovery in Kubernetes · Understand endpoints and traffic routing · Debug service connectivity problems

Deployments, Scaling & Rolling Updates6 goalsCreate a Deployment for the LLM chat API · Scale LLM app replicas to handle concurrent requests · Perform a rolling update with zero downtime · Roll back a broken deployment · Compare deployment strategies: RollingUpdate vs Recreate · Manage deployment lifecycle with kubectl rollout

ConfigMaps & Secrets for LLM Apps6 goalsCreate ConfigMaps for LLM app settings · Mount ConfigMaps as files for complex configuration · Store LLM proxy credentials securely in Secrets · Manage per-environment configuration for dev, staging, and prod · Handle configuration updates and rolling restarts · Debug configuration issues in LLM app pods

Persistent Storage & StatefulSets6 goalsCreate PersistentVolumeClaims for durable storage · Deploy PostgreSQL as a StatefulSet · Connect the LLM chat API to PostgreSQL for conversation persistence · Deploy Redis as a StatefulSet for LLM response caching · Understand StatefulSet scaling and ordering guarantees · Manage PVC lifecycle: expansion, snapshots, and cleanup

Multi-Container Pods: Sidecars & Init Containers6 goalsAdd an LLM proxy sidecar to the chat API pod · Use init containers for database setup and config loading · Share data between containers via emptyDir volumes · Implement the ambassador pattern for multi-model LLM routing · Add a logging and metrics sidecar to the LLM app · Debug multi-container pods

Resource Management & Cost Optimization6 goalsSet resource requests and limits for the LLM chat API · Understand QoS classes and their impact on eviction · Enforce resource defaults with LimitRanges · Cap namespace resource usage with ResourceQuotas · Right-size LLM app containers based on actual usage · Diagnose OOMKilled and CPU throttling issues

Packaging with Helm & Kustomize6 goalsCreate a Helm chart for the LLM chat application · Parameterize the chart with values.yaml for each environment · Manage Helm release lifecycle: install, upgrade, rollback · Use Kustomize bases and overlays for the LLM app · Use Kustomize patches and generators · Compare Helm vs Kustomize for different deployment scenarios

Networking, Ingress & TLS6 goalsExpose the LLM chat API via an Ingress resource · Add TLS to the Ingress for HTTPS access · Isolate services with NetworkPolicies · Configure Ingress annotations for production traffic · Understand K8s networking: pod IPs, CNI, and service routing · Debug networking and connectivity issues

Health Probes, Autoscaling & Self-Healing6 goalsAdd liveness and readiness probes to the LLM chat API · Configure startup probes for containers with slow initialization · Scale the chat API automatically with HPA based on CPU · Create PodDisruptionBudgets for safe maintenance · Implement health check patterns for LLM-dependent services · Combine autoscaling, probes, and PDBs for a resilient LLM service

RBAC, Security & K8s Troubleshooting6 goalsCreate RBAC roles for the LLM chat application · Enforce Pod Security Standards · Apply SecurityContext for defense in depth · Debug CrashLoopBackOff and OOMKilled failures · Use kubectl debug and ephemeral containers for live debugging · Troubleshoot LLM-specific issues: timeouts, proxy errors, stale connections

Web APIs & Services for GenAI Engineers6.9%60 goals

FastAPI Fundamentals6 goalsCreate a FastAPI application with path operations · Define Pydantic request and response models · Implement dependency injection for shared resources · Build CRUD endpoints with proper HTTP semantics · Configure OpenAPI documentation with examples · Handle errors with custom exception handlers

Async Python for APIs6 goalsConvert sync endpoints to async with proper await patterns · Implement background tasks for non-blocking operations · Execute concurrent API calls with asyncio.gather · Manage application lifecycle with lifespan handlers · Build async generators for streaming responses · Control concurrency with semaphores and throttling

Database Integration6 goalsConfigure SQLAlchemy async engine with connection pooling · Define ORM models with relationships and constraints · Create and manage database migrations with Alembic · Implement repository pattern for data access · Build transactional endpoints with session lifecycle · Implement filtering, sorting, and full-text search

Authentication & Authorization6 goalsImplement user registration with password hashing · Build OAuth2 password flow with JWT tokens · Implement API key authentication for services · Enforce role-based access control with permissions · Build token refresh and revocation · Compose multiple auth strategies into dependencies

Real-time Streaming6 goalsBuild SSE endpoint for streaming LLM responses · Implement WebSocket endpoint with connection lifecycle · Build WebSocket connection manager for broadcasting · Handle backpressure and slow clients · Implement heartbeat and automatic reconnection · Build real-time notification system with Redis pub/sub

Resilience Patterns6 goalsImplement rate limiting with Redis sliding window · Build circuit breaker for LLM provider calls · Configure retry logic with tenacity · Isolate critical paths with bulkhead semaphores · Build fallback responses for degraded mode · Combine resilience patterns into middleware stack

API Gateway & Routing6 goalsBuild reverse proxy with path-based routing · Implement load balancing across backend instances · Transform requests and responses through the gateway · Aggregate responses from multiple backends · Implement service discovery with health checking · Build gateway authentication and request enrichment

Testing & Documentation6 goalsWrite async endpoint tests with httpx.AsyncClient · Build database fixtures with transaction rollback · Mock external services for deterministic tests · Implement contract tests for API consumers · Measure test coverage and set quality gates · Generate rich OpenAPI documentation with examples

API Versioning & Evolution6 goalsImplement URL-based API versioning with routers · Build header-based version negotiation · Manage deprecation with Sunset and Warning headers · Build request and response adapters for version translation · Detect breaking changes automatically · Generate API changelogs from schema diffs

Deployment & Observability6 goalsBuild production Docker images with multi-stage builds · Deploy to Kubernetes with health check probes · Instrument endpoints with Prometheus metrics · Implement distributed tracing with OpenTelemetry · Build structured logging with correlation IDs · Create Grafana dashboards for API monitoring

DevOps Foundations for GenAI Engineers5.5%48 goals

Git Workflows for AI Teams6 goalsImplement trunk-based development for AI projects · Configure branch protection and status checks · Build PR templates for prompt and model changes · Manage merge conflicts in AI data files · Implement pre-commit hooks and automated dependency updates · Version AI artifacts with Git tags and releases

CI Pipelines with GitHub Actions6 goalsCompare CI platforms: GitHub Actions vs Tekton vs Dagger · Build AI artifact validation CI checks · Implement matrix builds for multi-provider testing · Build reusable workflow templates · Configure CI to run on GKE self-hosted runners · Optimize CI pipeline performance

Container Image CI/CD6 goalsBuild optimized Docker images for AI applications · Automate image builds with GitHub Actions · Scan images with Trivy, generate SBOMs with Syft, and assess SLSA level · Manage image lifecycle and garbage collection · Sign images with Cosign and enforce Binary Authorization on GKE · Build multi-architecture images for GKE

ArgoCD & GitOps6 goalsInstall ArgoCD and deploy first application · Configure sync policies for automated deployment · Detect and resolve configuration drift · Compare GitOps controllers: ArgoCD ApplicationSet vs Flux CD · Implement sync waves and hooks for ordered deployment · Implement ArgoCD RBAC and multi-tenancy

Infrastructure as Code6 goalsBuild Helm charts with Skaffold local development workflow · Use Kustomize overlays for environment management · Enforce policies with OPA Gatekeeper and test with Conftest · Test Helm charts before deployment · Implement Helmfile for multi-chart orchestration · Build chart versioning and release pipeline

Secrets Management6 goalsStore hosted LLM API keys in Google Secret Manager · Deploy External Secrets Operator for auto-sync · Implement automatic secret rotation · Prevent secrets from leaking in logs and manifests · Secure environment variable injection patterns · Audit and monitor secret access

Deployment Strategies6 goalsConfigure rolling updates with health checks · Implement blue-green and progressive delivery with Argo Rollouts · Automate canary deployments with Flagger · Implement automated rollback on SLO breach · Configure PodDisruptionBudgets for availability · Build deployment dashboards and change tracking

DevOps for AI Artifacts6 goalsVersion prompt templates as code · Build evaluation CI for prompt changes · Implement evaluation-gated deployment · Deploy guardrails as versioned configuration · Build A/B testing infrastructure for prompts · Implement model configuration drift detection

Enterprise LLM Customization8.3%72 goals

LiteLLM Gateway6 goalsDeploy LiteLLM proxy · Implement failover and circuit breakers · Load test and capacity plan · Build testing and validation for litellm gateway · Optimize performance for litellm gateway · Build operational runbook for litellm gateway

Smart Router6 goalsBuild intelligent routing · Implement context caching · Build routing analytics dashboard · Build testing and validation for smart router · Optimize performance for smart router · Build operational runbook for smart router

Provider Caching Comparison6 goalsImplement all caching systems · Build caching strategy recommender · Build cache invalidation and consistency · Build testing and validation for provider caching comparison · Optimize performance for provider caching comparison · Build operational runbook for provider caching comparison

FinOps Controller6 goalsBuild budget and spend tracking · Auto-route to Batch API · Build cost anomaly detection · Build testing and validation for finops controller · Optimize performance for finops controller · Build operational runbook for finops controller

Langfuse Observability6 goalsDeploy and integrate Langfuse · Build drift detection and alerts · Build custom eval pipelines in Langfuse · Build testing and validation for langfuse observability · Optimize performance for langfuse observability · Build operational runbook for langfuse observability

A/B Testing + Prompt CI/CD6 goalsBuild prompt CI/CD pipeline · Build model deployment lifecycle · Build multi-environment promotion pipeline · Build testing and validation for a/b testing + prompt ci/cd · Optimize performance for a/b testing + prompt ci/cd · Build operational runbook for a/b testing + prompt ci/cd

Guardrails Pipeline6 goalsBuild guardrails pipeline · Test guardrails under adversarial input · Benchmark guardrail performance · Build security testing for guardrails pipeline · Optimize throughput for guardrails pipeline · Build compliance reporting for guardrails pipeline

Prompt Injection Red Team6 goalsRun OWASP LLM Top 10 attacks · Build security assessment report · Build automated security regression pipeline · Build security testing for prompt injection red team · Optimize throughput for prompt injection red team · Build compliance reporting for prompt injection red team

Compliance Audit Trail6 goalsBuild audit logging system · Build compliance reporting · Build audit data retention and archival · Build security testing for compliance audit trail · Optimize throughput for compliance audit trail · Build compliance reporting for compliance audit trail

Compliance Test Suite6 goalsBuild regulatory compliance tests · Build bias and fairness tests · Build continuous compliance monitoring · Build security testing for compliance test suite · Optimize throughput for compliance test suite · Build compliance reporting for compliance test suite

MLOps for LLM Applications6 goalsBuild LLMOpsRegistry · Build ExperimentTracker · Build LLMOpsCI · Build testing and validation for mlops for llm applications · Optimize scalability for mlops for llm applications · Build operational runbook for mlops for llm applications

Advanced Evaluation & Observability6 goalsBuild EvalFrameworkComparator · Build CustomEvalPipeline · Build ObservabilityStack · Build testing and validation for advanced evaluation & observability · Optimize scalability for advanced evaluation & observability · Build operational runbook for advanced evaluation & observability

GenAI Evaluation, Safety & Governance9.0%78 goals

Evaluation Observability with Langfuse v3 & OpenTelemetry6 goalsDeploy Langfuse v3 on GKE with OpenTelemetry-native instrumentation · Build evaluation dashboards in Langfuse · Track costs and token usage across providers · Manage prompt versions with Langfuse · Compare observability platforms: Langfuse vs Arize Phoenix vs Braintrust · Build automated evaluation alerting

Evaluation-Driven CI/CD & Continuous Production Monitoring6 goalsBuild evaluation gates with Promptfoo and DeepEval in CI · Implement progressive evaluation tiers · Implement cost-aware evaluation budgets · Build release validation pipeline · Track evaluation trends across releases · Build continuous production monitoring with async scoring

Cost Governance & Token Budgets6 goalsBuild per-user token usage tracking · Implement budget enforcement and rate limiting · Detect cost anomalies and spending spikes · Build intelligent model routing with LiteLLM gateway and RouteLLM semantic routing · Implement prompt compression and caching · Build cost governance dashboard and chargeback

Prompt Injection Defense6 goalsDetect prompt injection with PromptGuard 2 and custom classifiers · Defend against indirect prompt injection · Prevent system prompt leakage (OWASP LLM07) · Build canary token detection · Deploy LlamaFirewall and Google Model Armor as unified defense orchestrators · Implement output filtering and response safety

Content Safety Filters6 goalsCompare guardrail frameworks: Guardrails AI vs NeMo Guardrails 0.20 vs NemoGuard NIMs vs Google Model Armor · Build custom domain-specific safety validators · Integrate LlamaGuard 4 and hosted LLM safety APIs · Build multi-layer content safety pipeline · Tune filter sensitivity and manage false positives · Monitor content safety metrics in production

PII Detection & Redaction6 goalsDetect PII with Presidio and Google Sensitive Data Protection · Implement reversible PII redaction · Build custom PII recognizers for domain data · Implement PII audit logging for compliance · Scan LLM outputs for PII leakage · Build end-to-end PII protection pipeline

Hallucination Detection6 goalsDetect hallucinations with source grounding and Patronus AI Lynx 2.0 · Implement citation verification · Build NLI-based faithfulness scoring · Classify hallucination types · Build production hallucination monitoring · Reduce hallucinations with prompt engineering

Adversarial Robustness Testing6 goalsExecute manual adversarial attack categories · Automated red teaming with PyRIT, Promptfoo Hydra, and Meta GOAT · Vulnerability scanning with NeMo Auditor and Garak 0.14 · Build adversarial CI/CD test suite · Measure defense effectiveness against attacks · Build adversarial robustness dashboard

OWASP LLM Top 10 2025 & MITRE ATLAS6 goalsMap OWASP LLM Top 10 2025 to implemented defenses · Implement defenses for Unbounded Consumption (LLM10) · Build MITRE ATLAS threat models · Build OWASP compliance testing with Promptfoo presets and Checks by Google · Implement supply chain security (LLM03) · Create OWASP + ATLAS compliance dashboard

EU AI Act Compliance6 goalsClassify AI systems under EU AI Act risk categories · Implement the Feb 2025 AI literacy requirements · Build technical documentation for GPAI compliance · Implement risk management system · Build human oversight mechanisms · Track EU AI Act enforcement timeline compliance

Compliance Frameworks6 goalsImplement NIST AI RMF Govern and Map functions · Implement NIST AI RMF Measure and Manage functions · Build ISO 42001 AI management system documentation · Create unified governance dashboard with Credo AI Agent Registry · Implement comprehensive audit trail · Build compliance automation and alerting

Red Teaming Methodology6 goalsPlan and scope a red team exercise · Execute manual red team techniques · Combine manual, Meta GOAT automated, and Inspect AI red teaming · Build red team findings reports · Track remediation and verify fixes · Build AI safety scorecard and establish red team cadence

Bias, Fairness & Continuous Monitoring6 goalsDetect bias in hosted LLM outputs · Implement fairness metrics for LLM applications · Build continuous safety monitoring for production · Detect safety drift over time · Build safety incident response workflow · Generate weekly and monthly safety reports

GenAI Operations44.9%390 goals

GenAI Failure Catalog6 goalsClassify GenAI failures into five categories: provider, quality, cost, security, and data failures · Instrument a multi-provider LLM gateway to detect each failure category · Build typed failure event models that feed into alerting and incident management · Measure baseline failure rates across OpenAI, Anthropic, and Google providers · Implement failure prediction from leading indicators · Create failure impact assessment system

GenAI SLI Framework6 goalsDefine latency SLIs: TTFT, tokens-per-second, end-to-end response time across providers · Define quality SLIs: faithfulness, hallucination rate, format compliance, retrieval precision · Define cost SLIs: cost-per-request, cost-per-token, cache hit rate, budget burn rate · Instrument all SLIs with Prometheus metrics and Langfuse traces · Build SLI aggregation and reporting API · Implement SLI validation and testing

GenAI SLO Engine6 goalsDefine SLO targets for latency, quality, and cost SLIs with business-justified thresholds · Compute error budgets and track consumption over rolling windows · Build multi-window burn-rate alerts that detect SLO violations before budget exhaustion · Create SLO status dashboards showing budget remaining and projected exhaustion · Implement SLO negotiation framework · Build cross-SLO dependency tracking

GenAI Toil Analyzer6 goalsIdentify GenAI-specific toil patterns: manual model updates, prompt tweaking, provider failover, cache invalidation · Measure toil using time-tracking instrumentation and classify by automation potential · Automate the highest-impact toil items with operational scripts and scheduled workflows · Track toil reduction over time with team-level reporting · Build automation testing and validation · Create toil reduction roadmap generator

GenAI Launch Readiness6 goalsDefine operational readiness criteria specific to GenAI services · Build automated readiness checks that verify infrastructure, monitoring, and runbook completeness · Implement launch gate enforcement that blocks deployment without readiness sign-off · Create readiness dashboards and historical tracking for continuous improvement · Implement progressive readiness rollout · Create readiness automation toolkit

GenAI Runbook Engine6 goalsWrite structured runbooks for the top 5 GenAI failure modes identified in Ch 1 · Build executable runbook steps that link to operational APIs and scripts · Implement runbook testing that validates each step works as documented · Track runbook usage and effectiveness metrics · Build runbook recommendation engine · Implement cross-runbook orchestration

GenAI Helm Charts6 goalsPackage LiteLLM, Langfuse, Prometheus, and Grafana as Helm Charts with Production-Ready Defaults · Create values files with environment-specific overrides for dev, staging, and production · Implement Helm chart testing and linting with ct and helm unittest · Build chart dependency management for the full GenAI stack · Implement Helm chart documentation generation · Build chart rollback and recovery procedures

Dev/Staging/Prod Environments6 goalsCreate K8s namespaces for dev/staging/prod with resource quotas and LimitRanges · Build Kustomize overlays for environment-specific configuration · Implement environment parity verification that detects configuration drift · Deploy the full GenAI stack to all three environments · Implement environment configuration validation · Create environment lifecycle automation

GitOps Control Plane6 goalsDeploy Argo CD to vCluster with Application and ApplicationSet CRDs · Implement GitOps sync for all GenAI components across dev/staging/prod · Configure drift detection and auto-remediation for configuration consistency · Build Argo CD RBAC for team-based access control · Implement performance optimization for gitops with argo cd · Build operational documentation for gitops with argo cd

GenAI CI/CD Pipelines6 goalsBuild Argo Workflow templates for GenAI artifact CI/CD · Implement pipeline stages: lint, validate, eval, promote · Create artifact-specific pipelines for prompts, models, and RAG configs · Monitor pipeline health with observability metrics · Implement performance optimization for ci/cd pipelines for genai artifacts · Build operational documentation for ci/cd pipelines for genai artifacts

GenAI Secret Manager6 goalsDeploy External Secrets Operator with GCP Secret Manager provider · Implement perenvironment provider key isolation with · Build secret sync monitoring and alerting for rotation compliance · Create emergency secret rotation procedures with zero-downtime key swap · Implement performance optimization for secret management for genai · Build operational documentation for secret management for genai

Self-Service Environment API6 goalsBuild API for on-demand feature environment provisioning with full GenAI stack · Implement environment lifecycle management with TTL and auto-cleanup · Create environment templates with pre-configured GenAI stack components · Monitor environment usage and resource consumption across all feature environments · Implement performance optimization for developer self-service environments · Build operational documentation for developer self-service environments

Pipeline Health Dashboard6 goalsInstrument Argo Workflow metrics for comprehensive pipeline observability · Build promotion velocity tracking across dev, staging, and production environments · Implement pipeline failure analysis with root cause categorization · Create pipeline health dashboards with bottleneck detection and DORA metrics · Implement performance optimization for pipeline observability · Build operational documentation for pipeline observability

Prompt Registry6 goalsBuild Immutable Prompt Storage with Content-Addressable Versioning · Implement Promotion Gates Between Dev, Staging, and Production · Create Prompt Diff and Review Workflow · Track Prompt Lineage Across All Deployments · Implement performance optimization for immutable prompt registry · Build operational documentation for immutable prompt registry

Model Lifecycle Manager6 goalsDeploy MLflow on vCluster for Model Registry and Experiment Tracking · Implement Model Versioning with Stage Promotion Gates · Build Model Deprecation Workflow with Consumer Notification · Track Experiments with Cost and Quality Metrics for Data-Driven Model Selection · Implement performance optimization for model registry and lifecycle · Build operational documentation for model registry and lifecycle

Progressive Delivery Engine6 goalsDeploy Argo Rollouts with Canary Strategy for LiteLLM Model Config Changes · Implement Shadow Deployments for Risk-Free Model Comparison in Production · Build Automated Rollback on Quality Regression During Canary Progression · Monitor Canary vs Baseline Quality, Latency, and Cost Metrics in Real-Time · Implement performance optimization for canary and shadow deployments · Build operational documentation for canary and shadow deployments

Eval Gate Pipeline6 goalsBuild Promptfoo Eval Suites for Pre-Promotion Quality Verification · Implement Eval Gates in Argo Workflows That Block Promotion on Failure · Create Golden Test Sets for Regression Detection · Track Eval Pass Rates and Gate Effectiveness Metrics · Implement performance optimization for automated eval gates · Build operational documentation for automated eval gates

AI Feature Flag System6 goalsBuild a Feature Flag Service for AI Configuration with Redis-Backed Storage · Implement Percentage-Based Rollouts for Model and Prompt Changes · Create Kill Switches for Rapid AI Behavior Reversion During Incidents · Track Feature Flag Impact on Quality and Cost Metrics · Implement performance optimization for feature flags for ai behaviors · Build operational documentation for feature flags for ai behaviors

RAG Release Pipeline6 goalsBuild a Release Pipeline for Embedding Model Swaps with Dual-Index Strategy · Implement Chunking Strategy Changes with A/B Comparison Using RAGAS Metrics · Create an Index Migration Workflow with Zero-Downtime Cutover · Validate RAG Releases with Retrieval Quality Metrics Before and After · Implement performance optimization for rag pipeline release management · Build operational documentation for rag pipeline release management

Distributed LLM Tracer6 goalsDeploy an OpenTelemetry Collector with Langfuse Exporter · Instrument Multi-Provider Request Chains with Parent-Child Trace Spans · Build Trace Correlation Across RAG Retrieval, LLM Inference, and Guardrail Processing · Create Trace-Based Latency Analysis Dashboards with Drill-Down Capability · Implement performance optimization for end-to-end llm tracing · Build operational documentation for end-to-end llm tracing

Quality Drift Detector6 goalsImplement Output Quality Drift Detection with Rolling Window Comparison · Build Embedding Drift Detection Using Distribution Divergence Metrics · Detect Retrieval Relevance Degradation with RAGAS-Based Monitoring · Configure Automated Alerts for Each Drift Type with Severity Classification · Implement performance optimization for quality drift detection · Build operational documentation for quality drift detection

GenAI Alert System6 goalsConfigure Alertmanager with GenAI-Specific Routing Rules and Severity Classification · Deploy Grafana OnCall for On-Call Schedules, Escalation Policies, and Incident Lifecycle · Implement Alert Deduplication and Grouping for Noisy GenAI Metrics · Build Alert Effectiveness Tracking to Reduce Alert Fatigue · Implement performance optimization for alerting strategy · Build operational documentation for alerting strategy

GenAI Dashboard Suite6 goalsBuild operational dashboard with SLO status, active incidents, and system health overview · Create business dashboard with usage, cost, and adoption metrics for stakeholders · Build compliance dashboard with guardrail activity, audit coverage, and policy status · Implement dashboard-as-code with Grafana provisioning for version-controlled dashboards · Implement performance optimization for dashboard engineering · Build operational documentation for dashboard engineering

Provider SLA Tracker6 goalsImplement per-provider availability tracking with synthetic probes · Build provider degradation detection using quality and latency SLIs · Create automated escalation chains for provider issues with status page integration · Track provider SLA compliance for vendor management and contract negotiation · Implement performance optimization for provider sla monitoring · Build operational documentation for provider sla monitoring

Cross-Env Comparator6 goalsImplement cross-environment metric comparison for quality regression detection · Build staging-to-prod quality correlation analysis for deployment confidence · Create environment drift detection for configuration parity monitoring · Monitor promotion impact by comparing pre/post metrics across environments · Implement performance optimization for cross-environment observability · Build operational documentation for cross-environment observability

AI Incident Commander6 goalsDefine LLM-specific incident severity classification with impact-based criteria · Build incident lifecycle management with role assignments and status tracking · Create communication templates for AI-specific incidents targeting different audiences · Track incident metrics with MTTD, MTTA, MTTR and trend analysis · Implement performance optimization for llm incident response framework · Build operational documentation for llm incident response framework

Runbook Automation Engine6 goalsBuild alert-to-runbook routing that triggers automated remediation workflows · Implement human approval gates for high-impact remediation steps · Create runbook execution auditing with step-by-step logging and outcome tracking · Track automation coverage and success rates across all runbook types · Implement performance optimization for automated runbook execution · Build operational documentation for automated runbook execution

AI Post-Mortem Engine6 goalsBuild structured post-mortem templates for GenAI failure modes · Implement timeline reconstruction from Langfuse traces and Prometheus metrics · Create action item tracking with follow-through verification · Analyze post-mortem trends to identify systemic issues · Implement performance optimization for post-mortems for ai failures · Build operational documentation for post-mortems for ai failures

LLM Chaos Lab6 goalsDeploy Chaos Mesh in vCluster · Build Provider Failover Chaos Experiments · Create Cache Invalidation Chaos Experiments · Implement Quality Degradation Injection · Implement performance optimization for chaos engineering for llm providers · Build operational documentation for chaos engineering for llm providers

Pipeline Chaos Experiments6 goalsCreate Embedding Pipeline Chaos Experiments · Build Ingestion Interruption Tests · Implement Index Corruption Detection and Recovery Validation · Track Chaos Experiment Results and Improvement Trends · Implement performance optimization for pipeline failure chaos · Build operational documentation for pipeline failure chaos

GenAI Game Day6 goalsDesign multi-failure game day scenarios for GenAI platforms · Build game day orchestration that chains chaos experiments with time delays · Implement game day scoring: response time, runbook adherence, communication quality · Create game day retrospectives with improvement tracking · Implement performance optimization for game day operations · Build operational documentation for game day operations

Cost Attribution Engine6 goalsInstrument per-request cost tracking across all pipeline stages · Build cost attribution to teams, projects, and use cases · Create cost allocation models for shared infrastructure components · Implement cost anomaly detection with automated investigation · Implement performance optimization for full-stack cost attribution · Build operational documentation for full-stack cost attribution

Token Budget Controller6 goalsConfigure LiteLLM Virtual Keys with Per-Team Budget Limits · Implement Per-Request Token Limits · Build Budget Alerting at 50%, 80%, and 100% Thresholds with Escalation · Create Budget Override Workflows for Emergency Usage Beyond Limits · Implement performance optimization for token budget enforcement · Build operational documentation for token budget enforcement

Cache Economics Analyzer6 goalsDeploy Redis Semantic Cache and Measure Hit Rate vs Cost Savings · Compare Provider Caching Strategies for OpenAI, Anthropic, and Google · Build Cost-Benefit Analysis with Break-Even Calculations · Recommend Optimal Caching Mix Per Use Case · Implement performance optimization for caching roi analysis · Build operational documentation for caching roi analysis

Batch API Scheduler6 goalsImplement workload classification: real-time vs batch-eligible based on latency requirements · Build Batch API job scheduling with priority queues and SLA tracking · Create batch job monitoring with completion time SLAs and failure handling · Measure and report cost savings from batch routing vs synchronous requests · Implement performance optimization for batch api optimization · Build operational documentation for batch api optimization

Capacity Forecaster6 goalsBuild token demand forecasting using historical usage patterns and trend analysis · Implement embedding volume projection for storage and compute planning · Create cost projection models for budget planning cycles · Track forecast accuracy and improve models over time with feedback loops · Implement performance optimization for capacity forecasting · Build operational documentation for capacity forecasting

FinOps Governance Platform6 goalsBuild Showback and Chargeback Reports per Team and Project with Full Cost Transparency · Create Executive FinOps Dashboards with Trend Analysis for Leadership · Implement Cost Governance Policies with Automated Enforcement · Generate Monthly FinOps Reviews with Optimization Recommendations · Implement performance optimization for finops reporting and governance · Build operational documentation for finops reporting and governance

Embedding Pipeline Ops6 goalsDeploy pgvector and build embedding ingestion pipeline with operational monitoring · Implement pipeline throughput tracking and failure detection with alerting · Build reprocessing workflow for failed or stale embeddings · Create pipeline health dashboards with freshness SLA tracking · Implement performance optimization for embedding pipeline operations · Build operational documentation for embedding pipeline operations

Vector Index Ops6 goalsImplement pgvector index maintenance with VACUUM and reindexing schedules · Deploy Qdrant and compare operational characteristics with pgvector · Build schema migration workflow for embedding dimension changes · Monitor index performance with query latency tracking and degradation detection · Implement performance optimization for vector index maintenance · Build operational documentation for vector index maintenance

Knowledge Graph Ops6 goalsDeploy Neo4j on vCluster with backup and restore procedures · Implement graph index management and query performance monitoring · Build knowledge graph freshness tracking with entity update pipelines · Create graph operations dashboards for health monitoring · Implement performance optimization for knowledge graph operations · Build operational documentation for knowledge graph operations

RAG Quality Monitor6 goalsDeploy RAGAS evaluation for production RAG quality monitoring on sampled traffic · Implement retrieval relevance tracking with precision and recall metrics · Build quality degradation alerting with root cause analysis · Compare retrieval quality across embedding models with Cohere Rerank · Implement performance optimization for rag quality monitoring · Build operational documentation for rag quality monitoring

Data Recovery Platform6 goalsImplement automated backup for pgvector, Qdrant, Neo4j, and PostgreSQL with scheduling · Build point-in-time recovery procedures with defined RTO and RPO targets · Create backup verification with data integrity checks and restore testing · Track backup compliance and recovery test results · Implement performance optimization for data backup and recovery · Build operational documentation for data backup and recovery

Data Quality Ops6 goalsImplement data freshness monitoring with staleness alerting across all stores · Build completeness checks for embedding coverage and knowledge graph gaps · Detect data poisoning and anomalous ingestion patterns · Create data quality scorecards with trend tracking · Implement performance optimization for data quality monitoring · Build operational documentation for data quality monitoring

Key Rotation Operator6 goalsImplement automated key rotation for OpenAI, Anthropic, and Google provider keys · Build zero-downtime key swap using dual-key overlap windows · Create key rotation audit trails for compliance reporting · Execute emergency key revocation for compromised keys · Implement performance optimization for api key rotation automation · Build operational documentation for api key rotation automation

PII Detection Pipeline6 goalsDeploy Microsoft Presidio on K8s for runtime PII detection in LLM traffic · Build PII scrubbing middleware that redacts sensitive data before sending to LLM providers · Implement PII detection alerting and audit logging for compliance · Create PII detection tuning workflows to reduce false positives · Implement performance optimization for runtime pii detection · Build operational documentation for runtime pii detection

Injection Monitoring System6 goalsImplement multi-layer prompt injection detection with pattern and embedding-based methods · Build real-time injection alerting with severity classification · Create injection attack analysis dashboards for security monitoring · Implement adaptive detection that learns from new attack patterns · Implement performance optimization for prompt injection monitoring · Build operational documentation for prompt injection monitoring

Guardrail Operations Platform6 goalsDeploy Guardrails AI and LlamaFirewall on K8s for runtime content validation · Implement hot-reload guardrail configuration without service restarts · Build A/B testing framework for guardrail thresholds to optimize block rates · Build testing and validation for guardrail operations · Implement performance optimization for guardrail operations · Build operational documentation for guardrail operations

Compliance Audit Engine6 goalsImplement automated compliance scans for GenAI-specific requirements · Build evidence collection pipelines that gather audit artifacts · Schedule recurring compliance checks with drift detection · Build testing and validation for compliance audit automation · Implement performance optimization for compliance audit automation · Build operational documentation for compliance audit automation

Red Team Automation Platform6 goalsBuild automated red team attack suites using Promptfoo for systematic security testing · Implement scheduled security testing with regression detection across model changes · Build security posture scoring with trend monitoring and improvement tracking · Build testing and validation for red team operations · Implement performance optimization for red team operations · Build operational documentation for red team operations

Multi-Tenant GenAI Platform6 goalsAutomate tenant onboarding with namespace provisioning and secret management · Implement namespace isolation with network policies and resource quotas · Build noisy-neighbor detection that identifies tenants causing resource contention · Create tenant operations dashboards with per-tenant health visibility · Implement performance optimization for multi-tenant platform operations · Build operational documentation for multi-tenant platform operations

AI Developer Platform6 goalsBuild self-service deployment workflows with approval gates for AI artifacts · Create golden path templates for common GenAI patterns · Implement internal tool marketplace for reusable AI components · Build developer experience metrics and platform analytics · Implement performance optimization for internal developer platform for ai · Build operational documentation for internal developer platform for ai

GenAI Ops Maturity Assessor6 goalsDefine GenAI operational maturity model with five levels across eight capability areas · Build automated maturity assessment that evaluates current operational state · Generate improvement roadmaps with prioritized actions based on assessment results · Track maturity progression over time with milestone tracking · Implement performance optimization for operational maturity model · Build operational documentation for operational maturity model

Healthcare AI Ops Platform6 goalsOperate healthcare GenAI platform with HIPAA-compliant infrastructure and audit trails · Implement clinical quality SLOs beyond standard faithfulness metrics · Build medical safety guardrails preventing harmful medical advice generation · Create healthcare-specific incident response with clinical escalation procedures · Implement performance optimization for healthcare genai operations capstone · Build operational documentation for healthcare genai operations capstone

Finance AI Ops Platform6 goalsOperate finance GenAI platform with SOX-compliant infrastructure and change management · Implement financial accuracy SLOs with numerical precision validation · Build financial regulatory guardrails preventing unauthorized financial advice · Create finance-specific compliance automation with audit trail integration · Implement performance optimization for finance genai operations capstone · Build operational documentation for finance genai operations capstone

Enterprise GenAI Ops Platform6 goalsOperate a complete enterprise GenAI platform integrating all eight operational disciplines · Implement cross-domain operational policies that adapt guardrails and SLOs per tenant domain · Build platform operational runbooks covering the full lifecycle from onboarding to decommissioning · Demonstrate operational maturity level 4+ across all eight capability areas · Implement performance optimization for full platform operations capstone · Build operational documentation for full platform operations capstone

LLM-Specific Observability6 goalsHallucination Detection Pipeline · Semantic Drift Monitor · Response Quality Scorer · LLM Observability Dashboard · Quality Degradation Root Cause · LLM Observability Capstone

Token Cost FinOps6 goalsCost Tracking Pipeline · Budget Management System · Chargeback Reporting · Cost Optimization Engine · FinOps Drill-Down Dashboard · FinOps Capstone

OpenTelemetry for Agentic Systems6 goalsLLM Call Instrumentation · Agent Execution Tracing · Custom Agent Attributes · Langfuse Trace Integration · K8s Distributed Tracing · Tracing Capstone

Eval Gates in CI/CD6 goalsEval Suite Design · CI Pipeline Integration · Eval Result Storage · Change-Type-Specific Gates · Eval Failure Workflow · Eval Gates Capstone

Prompt Versioning and Deployment6 goalsOperational Prompt Registry · Canary Prompt Deployment · Automated Prompt Rollback · Prompt Deployment Pipeline · Multi-Prompt Coordination · Prompt Ops Capstone

Guardrail Operations6 goalsGuardrail Deployment · Guardrail Tuning · Guardrail Versioning · Effectiveness Monitoring · Guardrail Incident Response · Guardrail Ops Capstone

AI Incident Response6 goalsAI Incident Taxonomy · Automated Detection · AI Incident Runbooks · Post-Incident Review · Escalation Path Design · Incident Response Capstone

Multi-Tenant AI Platform Ops6 goalsTenant Isolation · Quota Management · Noisy-Neighbor Detection · Per-Tenant SLA Monitoring · Tenant Self-Service · Multi-Tenant Ops Capstone

AI Governance Compliance Ops6 goalsEU AI Act Controls · NIST AI RMF Implementation · SOC2 AI Controls · Compliance Dashboard · Audit Preparation Workflow · Compliance Ops Capstone

Platform Operations Capstone6 goalsPlatform Integration · Unified Ops Dashboard · Operational Readiness · Operational Lifecycle Demo · Operations Automation · Platform Ops Capstone Report

Core responsibilities this discipline prepares you for. Open one to see how you build it.

1Design CI/CD pipelines for LLM application deployment

Build ArgoCD GitOps workflows with Helm-based deployments and environment promotion
Implement canary and blue-green rollout strategies with automated quality-based rollback
Wire complete CI/CD pipelines that trigger rollbacks when evaluation metrics degrade

2Monitor LLM systems in production — latency, errors, costs, quality

Instrument with OpenTelemetry and Langfuse v3 for OTEL-native distributed tracing
Build Grafana dashboards with Logfire for Python application monitoring and alerting
Set up monitoring stacks that detect anomalies, fire alerts, and enable trace-based root cause analysis

3Manage LLM gateway operations — key rotation, failover, quota management

Operate LiteLLM gateway: API key lifecycle management, provider health monitoring, per-team quotas
Handle zero-downtime model version switching with traffic draining and validation
Simulate provider outages and quota exhaustion to validate failover and degradation behavior

4Implement FinOps practices — cost attribution, budgets, and optimization

Track token costs by team, feature, and model with Prometheus-based budget alerting
Implement cost optimization through semantic caching, model tiering, and prompt compression
Build FinOps dashboards that demonstrate measurable cost reduction across optimization strategies

5Build continuous evaluation pipelines for production LLM quality

Run RAGAS and DeepEval evaluation pipelines alongside production traffic as shadow evaluators
Set up Langfuse-based quality tracking with automated quality gates and threshold alerting
Detect quality degradation in real time and trigger automated alerts when scores drop below baselines

6Detect and respond to prompt attacks and safety incidents in production

Monitor NeMo Guardrails operationally for prompt injection and jailbreak detection patterns
Classify incident severity and execute structured response workflows with containment procedures
Simulate attack scenarios end-to-end: detection, triage, remediation, and post-incident analysis

7Manage data quality for RAG systems — freshness, drift, accuracy

Monitor embedding drift and retrieval accuracy with continuous RAGAS evaluation
Set up automated reindexing triggers and stale content detection pipelines
Build monitoring for live RAG systems that detects quality degradation and triggers reindexing workflows

8Implement capacity planning — predict demand and right-size deployments

Forecast token demand using historical usage patterns and run load tests for LLM services
Model SLA capacity requirements and configure KEDA-based autoscaling policies
Run load tests that predict capacity requirements and validate SLA compliance under variable traffic

Each group becomes a signed node on your public skill graph — a recruiter-auditable record of the labs you actually passed. Share the URL; they see the discipline, the labs, and the signature.

Output Quality Eval & Gating

Is the answer good? Hallucination/groundedness detection, eval-gate pipelines that block bad deploys, quality-drift detection, RAGAS/DeepEval, LLM-as-judge (and validating the judge), and RAG retrieval-quality monitoring.

Cost & Token FinOps

Keep it cheap: per-request cost attribution, token-budget controllers and gates, cache economics, model-tier arbitrage, batch scheduling, and capacity forecasting — driving spend down (per-tenant chargeback belongs to the platform engineer).

Incident Response, On-Call & Resilience

Keep it not-on-fire: incident command, triage and post-mortems, runbook automation, on-call and alerting hygiene, error-budget policy, and provider-outage failover/fallback drills.

Compliance & Governance Ops

Keep it compliant: audit engines and trails, EU AI Act / SOC2 / HIPAA / GDPR controls, policy gates, bias and fairness monitoring, and regulatory reporting.

Prompt/Model Lifecycle & Release Safety

Safely change what's in prod: prompt/model versioning and registry, provenance/lineage, canary analysis, automated rollback triggers, progressive delivery, feature flags, and shadow/replay testing.

Service Observability & SLOs

Is the service up, fast, and within SLO? OpenTelemetry tracing, Langfuse, Prometheus/Grafana, latency/throughput/error telemetry, and SLI/SLO engines.

Security Ops & Guardrails

Operate the safety layer in production: runtime guardrails with canary/kill-switch, key rotation, PII detection, prompt-injection monitoring, content-safety filters, and red-team automation.

CI/CD & GitOps for AI

Ship safely and repeatably: GitHub Actions/ArgoCD GitOps, Infrastructure-as-Code, secrets management, environment promotion, and deployment automation.

Container & Kubernetes for LLMOps

Operate the serving substrate: Helm, multi-tenant isolation and quota, HPA, gateway/guardrail sidecars, and namespace policy.

Retrieval Substrate Health (Ops)

Keep the retrieval substrate up in production: health checks, failure detection, recovery, and capacity/staleness alarms — operating it, not building it.

Hosted LLM API Integration

Baseline LLM access in ops code: provider SDKs, multi-provider gateways, smart routers, and provider failover.

Python for LLMOps

Production Python for ops tooling: async/await, Pydantic, typing, dataclasses, pytest, and error handling.