GenAI Data Pipelines
Master data pipeline engineering for GenAI applications using hosted LLM APIs (OpenAI, Anthropic, Gemini, DeepSeek, LLaMA via Groq/Together), modern retrieval patterns (Contextual Retrieval, Agentic RAG, LazyGraphRAG, ColBERT reranking), and production deployment on GKE Autopilot. Covers VLM-based document ingestion, NeMo Curator data cleaning, context engineering, multimodal embeddings, AlloyDB vector operations, hybrid search with multi-layer caching, knowledge graph construction with LightRAG, evaluation-driven quality engineering (RAGAS, DeepEval), PII detection with Presidio and NeMo Guardrails, agentic pipeline orchestration with Dagster and MCP, data flywheels, and a full production capstone with Crossplane IaC and OTel observability. All labs run on Kubernetes (GKE).
Learning Path
8 phases • 18 chaptersFoundations
Python essentials and development environment for agent development
Tools & Topics
Virtual environments, async programming, type hints, Pydantic, error handling, testing, debugging, logging, project structure
Goals
- •Set up professional development environments
- •Write async Python code fluently
- •Use type hints and Pydantic for robust data handling
- •Implement error handling, testing, logging, and debugging
Chapters
LLM Fundamentals
Core LLM concepts: API clients, token economics, caching, and function calling basics
Tools & Topics
LLM APIs, OpenAI/Anthropic/Gemini clients, prompt caching, token economics, function calling basics
Goals
- •Call multiple LLM providers (OpenAI, Anthropic, Gemini)
- •Implement prompt caching and token cost management
- •Build function calling and tool definitions
- •Understand token economics and cost optimization
Chapters
Agent Fundamentals
Agent patterns: ReAct, planning, tool execution, sandboxing, web navigation, and MCP protocol
Tools & Topics
ReAct loop, planning patterns, tool execution, sandboxing, web navigation, MCP servers, MCP clients, tool routing
Goals
- •Create agent loops with ReAct and planning patterns
- •Build and consume MCP servers for tool integration
- •Implement sandboxing and web navigation
- •Design structured outputs and prompts
Chapters
Agent State & Memory
Memory systems, RAG patterns, context optimization, and LangGraph state machines
Tools & Topics
Short-term memory, long-term memory (RAG), agentic RAG patterns, semantic memory, context optimization, state graphs, conditional edges, checkpointing, human-in-the-loop, streaming, subgraphs
Goals
- •Implement short-term and long-term memory
- •Build RAG and agentic RAG systems
- •Create state machines with LangGraph
- •Implement checkpointing, streaming, and human-in-the-loop
Chapters
Multi-Agent Systems
Multi-agent patterns, guardrails, evaluations, and observability
Tools & Topics
Supervisor pattern, hierarchical pattern, reflector pattern, input guardrails, output guardrails, prompt injection defense, evaluations, benchmarking, tracing, observability
Goals
- •Implement supervisor, hierarchical, and reflector patterns
- •Build input and output guardrails
- •Defend against prompt injection attacks
- •Evaluate agents with benchmarks
Chapters
Production & Operations
Production deployment: APIs, containers, databases, scaling, CI/CD, and monitoring
Tools & Topics
FastAPI, Docker, production databases, scaling, CI/CD, monitoring, alerting, model routing, fallbacks, system design
Goals
- •Serve agents via FastAPI with Docker
- •Deploy to Kubernetes with CI/CD
- •Monitor with Prometheus/Grafana
- •Build multi-tenant agent platforms
Chapters
Advanced Topics
Alternative frameworks, protocols, specialized agents, autonomous workflows, and cutting-edge capabilities
Tools & Topics
CrewAI/AutoGen, A2A protocols, GraphRAG, local models, vision agents, voice agents, code agents, autonomous workflows, streaming data, agent swarms
Goals
- •Use alternative frameworks (CrewAI, AutoGen)
- •Implement A2A protocol for agent communication
- •Build GraphRAG for complex knowledge
- •Build vision, computer use, and voice agents
Chapters
Agent Production Excellence
Production excellence: trajectory evaluation, safety, cost control, enterprise patterns, and governance
Tools & Topics
Agent trajectory evaluation, safety boundaries, cost control, enterprise agent patterns, load testing, versioning, fleet dashboards, autonomous agent governance
Goals
- •Score multi-step agent reasoning with LLM-as-judge pipelines
- •Build safety boundaries with permissions and kill switches
- •Implement per-agent cost budgets and cost-aware routing
- •Deploy enterprise agent patterns for document processing and code review