Generate runbooks from K8s configs with LangGraph workflows

Introduction

When customer engineering teams onboard a new service, assembling deployment runbooks by hand means hunting through scattered Kubernetes manifests, Helm values, and environment files — then writing procedures that are already stale by the next release. LangGraph's directed state graphs and Instructor's structured output solve this by automating the extract-generate-validate cycle in a single repeatable pipeline. By the end of this lesson, you'll be able to build a runbook generator that parses configuration artifacts, produces validated step-by-step procedures for each discovered service, and loops back for refinement when quality checks fail.

Key Terminology

• Directed State Graph — A LangGraph StateGraph in which nodes represent discrete processing steps (analyze, generate, validate) and edges define the allowed transitions between them, with a shared TypedDict like RunbookState flowing through each node as the pipeline's memory.
• RunbookState — The TypedDict that carries pipeline data across graph nodes, holding config_artifacts (raw input), config_analysis (the extracted service map), and procedures (validated runbook entries); each node returns a partial dict that LangGraph merges into this shared state.
• Structured Output — The technique of passing Instructor's response_model parameter to an LLM call so that the raw completion is automatically parsed and validated against a Pydantic schema — here RunbookProcedure — before the result is accepted into the pipeline.
• RunbookProcedure — The Pydantic model that enforces the schema for a single service's runbook entry, requiring service_name, a list of steps, an expected_output string, and a rollback string on every LLM response before it is appended to procedures.
• Conditional Edge — A LangGraph add_conditional_edges call that routes graph execution to different successor nodes based on a routing function; in this lesson should_regenerate sends failing procedures back to generate or exits at END when all procedures pass.
• Refinement Loop — The pipeline pattern in which the validate node checks quality criteria and, on failure, routes execution back to generate rather than aborting the entire workflow, enabling targeted retries without discarding already-passing procedures.

Concepts

Mapping the Extract-Generate-Validate Cycle to a Graph

Runbook generation involves three distinct, sequenced responsibilities: reading raw configuration artifacts to discover what services exist, prompting an LLM to produce step-by-step procedures for each service, and verifying those procedures against quality criteria. A LangGraph StateGraph makes this structure explicit — each responsibility becomes a named node, and edges encode the allowed order. The benefit over a flat script is that the graph itself communicates the pipeline's logic: reading workflow.add_edge("analyze", "generate") tells you the intended flow without tracing call stacks (see Code Walkthrough).

State flows through the graph as a shared TypedDict. Each node receives the full current state and returns only the keys it updated — LangGraph merges these partial dicts so the next node always sees a complete, consistent snapshot. This means analyze can write config_analysis without knowing anything about how generate will consume it, and generate can append to procedures without knowing how validate will inspect them.

Enforcing Schema at the LLM Boundary

LLMs produce free-form text. Downstream code that expects structured data — numbered steps, rollback instructions, expected outputs — cannot safely parse raw completions. Instructor's response_model parameter solves this by intercepting the LLM response before it reaches your code, parsing it into the specified Pydantic model, and raising a validation error if any required field is missing or malformed. In this lesson, every call inside generate_procedures carries response_model=RunbookProcedure, so the returned object is always a fully-validated RunbookProcedure instance rather than a raw string (see Code Walkthrough).

Pydantic validation at the LLM boundary also makes the pipeline's quality contract machine-readable: the RunbookProcedure schema documents exactly what constitutes a valid procedure, and any future change to that contract is reflected automatically in what the LLM is required to produce.

Targeted Retry via Conditional Edges and Per-Service Decomposition

When validation fails in a flat pipeline, the only options are abort or full restart. LangGraph's add_conditional_edges unlocks a third option: route selectively. The should_regenerate function inspects the validated procedures and returns either "regenerate" (routing back to generate) or "done" (routing to END). This means a failing procedure triggers another generation pass without discarding procedures that already passed — state persists across the loop (see Code Walkthrough).

The one-service-per-call decomposition inside generate_procedures reinforces this. Because each LLM call handles exactly one service, the refinement loop retries only the services that failed, rather than re-issuing a monolithic prompt for the entire deployment. This decomposition also prevents token-limit problems on complex multi-service configurations and makes the retry boundary obvious: one service, one call, one schema validation, one outcome.

Code Walkthrough

Now that you understand how LangGraph state graphs and Instructor's response_model parameter enforce structured output, you can see how they combine in the RunbookGenerator class.

The generator wires three nodes into a directed graph: analyze reads configuration artifacts — Kubernetes manifests, Helm values, and environment files — and extracts a structured service map; generate calls the LLM once per discovered service and validates each response against a Pydantic schema; and validate checks every procedure against quality criteria. A conditional edge from validate back to generate creates a refinement loop that retries failing procedures rather than the entire runbook.

Code snippetpython
1from langgraph.graph import StateGraph, END
2from typing import TypedDict
3
4class RunbookState(TypedDict):
5    config_artifacts: dict
6    config_analysis: dict
7    procedures: list
8
9class RunbookGenerator:
10    def build_workflow(self) -> StateGraph:
11        workflow = StateGraph(RunbookState)
12        workflow.add_node("analyze", self.analyze_config)
13        workflow.add_node("generate", self.generate_procedures)
14        workflow.add_node("validate", self.validate_runbook)
15        workflow.add_edge("analyze", "generate")
16        workflow.add_edge("generate", "validate")
17        workflow.add_conditional_edges(
18            "validate",
19            self.should_regenerate,
20            {"regenerate": "generate", "done": END},
21        )
22        workflow.set_entry_point("analyze")
23        return workflow.compile()

The generate_procedures node issues a separate LLM call for each service rather than requesting all procedures in a single prompt. This one-service-per-call pattern avoids token-limit problems on complex deployments and makes it straightforward to retry a single failed procedure without regenerating the rest. The response_model parameter instructs Instructor to parse and validate each LLM response against RunbookProcedure — a Pydantic model that enforces numbered steps, expected outputs, and rollback instructions — before the result is appended to the procedures list.

Code snippetpython
1from pydantic import BaseModel
2import json
3
4class RunbookProcedure(BaseModel):
5    service_name: str
6    steps: list[str]
7    expected_output: str
8    rollback: str
9
10async def generate_procedures(self, state: RunbookState) -> dict:
11    analysis = state["config_analysis"]
12    procedures = []
13    for service in analysis["services"]:
14        response = await self.client.chat.completions.create(
15            model="gpt-4o",
16            messages=[
17                {
18                    "role": "system",
19                    "content": "Generate a numbered runbook procedure for this service.",
20                },
21                {
22                    "role": "user",
23                    "content": f"Service config: {json.dumps(service)}",
24                },
25            ],
26            response_model=RunbookProcedure,
27        )
28        procedures.append(response)
29    return {"procedures": procedures}

The returned procedures list flows into the validate node as a state update. If all procedures pass, the graph exits at END; if any fail, the conditional edge routes execution back to generate for another refinement pass.

Confirm that compiling the workflow and invoking it with a sample config_artifacts dictionary produces a non-empty procedures list where every entry is a valid RunbookProcedure instance, and that deliberately introducing a validation failure causes the graph to loop back through generate at least once before exiting.

Do's and Don'ts

Having walked through the material above, the following Do's and Don'ts distill it into practice.

Do's

1. ✓Do issue one LLM call per discovered service in generate_procedures — batching all services into a single prompt risks hitting token limits on complex deployments and forces you to regenerate every procedure when one fails; the per-service loop lets you retry only the failing RunbookProcedure instance.
2. ✓Do declare RunbookState as a TypedDict and pass it to StateGraph — LangGraph uses the typed schema to route state updates between analyze, generate, and validate nodes; untyped or ad-hoc dicts bypass this contract and cause silent key-miss errors when config_analysis or procedures is absent at node entry.
3. ✓Do use Instructor's response_model=RunbookProcedure parameter on every LLM call in generate_procedures — this enforces that each response contains steps, expected_output, and rollback before the result is appended to the procedures list, so the validate node receives structurally complete data rather than free-form text that can pass silently malformed.

Don'ts

1. ✗Don't wire a static edge from validate to generate — use add_conditional_edges with should_regenerate so only failing procedures trigger a refinement loop; a static edge creates an infinite cycle that re-generates every service on every pass regardless of quality check results.
2. ✗Don't skip the analyze node and pass raw Kubernetes manifests or Helm values directly into generate_procedures — the analyze node extracts a structured service map into config_analysis["services"]; bypassing it means the per-service loop has no iterable and silently produces an empty procedures list with no error.
3. ✗Don't remove rollback from the RunbookProcedure Pydantic schema to simplify output — the validate node checks quality criteria against all required fields, and a procedure missing rollback will fail validation on every pass, causing the conditional edge to loop indefinitely rather than reaching END.