Deploy Guardrails AI and LlamaFirewall on K8s for runtime content validation

Introduction

In production, LLM services process thousands of requests per hour — without a runtime validation layer, a single malformed output or injected prompt can reach end users or downstream systems before any human reviewer can intervene. Deploying Guardrails AI and LlamaFirewall as dedicated Kubernetes services creates a two-sided defense: output validation catches policy violations after the model responds, while input scanning blocks injection attempts before the model ever sees them. By the end of this lesson, you'll be able to deploy both services via Helm on Kubernetes and configure their validators and scanners using Pydantic-validated config models that support hot-reload without pod restarts.

Key Terminology

• ValidatorConfig — A Pydantic model that encodes a single output-side validation rule for Guardrails AI, binding a ValidatorType enum value (e.g., TOXICITY_DETECTION, FORMAT_COMPLIANCE), a float threshold constrained to [0.0, 1.0], and an action_on_fail policy; a list of these is collected in GuardrailsAIConfig.validators.
• ScannerConfig — A Pydantic model that encodes a single input-side scanning rule for LlamaFirewall, specifying a ScannerType (e.g., PROMPT_INJECTION, JAILBREAK_DETECTION), a confidence_threshold defaulting to 0.85, and flags (scan_user_input, scan_system_prompt) that control which parts of an inbound message are inspected.
• fallback_on_timeout — A string field on both GuardrailsAIConfig and LlamaFirewallConfig that declares each service's behavior when validation or scanning exceeds its deadline; set to "pass" on the output side (prioritizing availability) and "block" on the input side (prioritizing security against unscanned prompts reaching the model).
• threshold constraint — A Pydantic Field(ge=0.0, le=1.0) annotation on ValidatorConfig.threshold and ScannerConfig.confidence_threshold that enforces valid confidence bounds at model-construction time, raising a ValidationError before any misconfigured value can reach a Helm deployment.
• hot-reload — The ability to update validator or scanner thresholds at runtime without restarting Kubernetes pods, achieved by serializing config via .model_dump() into a Helm-mounted ConfigMap that each service polls on a configurable interval, enabling A/B threshold experiments without downtime.
• two-sided defense — The architectural pattern in which LlamaFirewall scans inbound prompts before they reach the model (input side) while Guardrails AI validates model responses before they reach users or downstream systems (output side), with each service deployed as a dedicated Kubernetes service in the same namespace.

Concepts

Two Surfaces, Two Services

LLM deployments face content risks on two distinct surfaces: user inputs that attempt to manipulate model behavior through injection or jailbreak patterns, and model outputs that may violate policy regardless of what the input looked like. Securing only one side leaves a gap — a syntactically clean input can still produce a toxic or PII-leaking response, and a well-tuned output filter does nothing to stop a prompt injection from reaching the model in the first place.

This lesson closes both gaps by deploying two cooperating Kubernetes services. LlamaFirewall owns the input side: it intercepts each user prompt and runs configurable scanners (PROMPT_INJECTION, JAILBREAK_DETECTION, ENCODING_ATTACK, and others) before the model ever processes the message. Guardrails AI owns the output side: it evaluates model responses against validators (TOXICITY_DETECTION, FORMAT_COMPLIANCE, PII_OUTPUT_FILTER, and others) before the response reaches the user or any downstream system. Both services share a Kubernetes namespace and a common Pydantic-based configuration pattern, but their distinct threat positions produce meaningfully different defaults (see Code Walkthrough).

Pydantic as a Pre-Deployment Contract

Rather than storing validator and scanner settings in raw YAML or untyped dictionaries, this lesson models them as structured Pydantic classes — ValidatorConfig, ScannerConfig, GuardrailsAIConfig, and LlamaFirewallConfig. The practical benefit is that field-level constraints, such as Field(ge=0.0, le=1.0) on threshold and confidence_threshold, are enforced the moment a config object is constructed in Python, not when it arrives at the cluster. A threshold of 1.5 or -0.1 raises a ValidationError before any Helm release runs.

This matters most during A/B threshold experiments, where teams iteratively adjust confidence levels to tune a scanner's precision-recall balance. Without model-layer enforcement, an out-of-range threshold is syntactically valid JSON that a Helm chart will deploy without complaint — silently misconfiguring the guardrail. The Pydantic layer catches that class of error at authoring time. Both models also serialize cleanly via .model_dump(), producing a dict that can be written to a Kubernetes ConfigMap and polled by the running service for hot-reload without pod restarts.

Timeout Policies as Deliberate Risk Trade-offs

When a guardrail service exceeds its latency deadline, the system must choose: let the request through ("pass") or reject it ("block"). The right answer differs by side, and this lesson encodes that asymmetry explicitly in the fallback_on_timeout field.

For Guardrails AI on the output side, "pass" prioritizes user experience — a validation timeout means the response reaches the user rather than hanging indefinitely. The accepted risk is that a policy-violating output may slip through during transient latency spikes. For LlamaFirewall on the input side, "block" prioritizes security — an unscanned prompt forwarded to the model under scanner timeout represents unmitigated risk, so the request is rejected outright. The cost is a hard failure for the user when the scanner is slow.

These are not arbitrary defaults. They encode the asymmetric consequences of a false pass on each side: a missed output violation is recoverable with logging and review; a missed input injection may cause the model to act on adversarial instructions with no downstream checkpoint to catch it. Operators tuning timeout_seconds and replica counts need to hold this asymmetry in mind — tightening LlamaFirewall's timeout without adequate replicas increases false blocks under load, while loosening Guardrails AI's timeout extends the window during which a policy violation can reach users.

Code Walkthrough

Now that you understand how ValidatorConfig and ScannerConfig model threshold-bound, hot-reloadable settings, here is how to instantiate a full deployment configuration for both services.

The GuardrailsAIConfig model below assembles a named server configuration with a set of validators covering output format compliance and toxicity detection. Setting fallback_on_timeout to "pass" keeps the service available when validation latency spikes — a deliberate trade-off that prioritizes user experience over strict blocking in degraded conditions. The threshold field's ge=0.0, le=1.0 constraints enforce valid confidence bounds at the model layer, preventing misconfigured A/B threshold experiments from reaching the cluster.

Code snippetpython
1from pydantic import BaseModel, Field
2from enum import Enum
3
4class ValidatorType(str, Enum):
5    FORMAT_COMPLIANCE = "format_compliance"
6    TOPIC_RESTRICTION = "topic_restriction"
7    TOXICITY_DETECTION = "toxicity_detection"
8    PII_OUTPUT_FILTER = "pii_output_filter"
9    HALLUCINATION_CHECK = "hallucination_check"
10
11class ValidatorConfig(BaseModel):
12    validator_type: ValidatorType
13    enabled: bool = True
14    threshold: float = Field(ge=0.0, le=1.0, default=0.5)
15    action_on_fail: str = "block"
16    custom_params: dict = Field(default_factory=dict)
17
18class GuardrailsAIConfig(BaseModel):
19    server_name: str = "guardrails-ai"
20    namespace: str = "guardrails"
21    replicas: int = 2
22    validators: list[ValidatorConfig] = Field(default_factory=list)
23    timeout_seconds: float = 5.0
24    fallback_on_timeout: str = "pass"
25
26output_config = GuardrailsAIConfig(
27    validators=[
28        ValidatorConfig(
29            validator_type=ValidatorType.TOXICITY_DETECTION,
30            threshold=0.7,
31        ),
32        ValidatorConfig(
33            validator_type=ValidatorType.FORMAT_COMPLIANCE,
34            threshold=0.9,
35        ),
36    ]
37)

LlamaFirewall's config mirrors the same Pydantic pattern but targets the input side. Its confidence_threshold defaults to 0.85 — higher than the output validators — because a false positive here blocks the user's request outright, so precision matters more. The fallback_on_timeout policy is "block" rather than "pass" because an unscanned prompt arriving at the LLM during a scanner timeout represents unmitigated risk. The scan_system_prompt flag is disabled by default since system prompts are trusted operator inputs; only scan_user_input is active by default.

Code snippetpython
1from pydantic import BaseModel, Field
2from enum import Enum
3
4class ScannerType(str, Enum):
5    PROMPT_INJECTION = "prompt_injection"
6    JAILBREAK_DETECTION = "jailbreak_detection"
7    PII_DETECTION = "pii_detection"
8    ENCODING_ATTACK = "encoding_attack"
9    ROLE_SWITCHING = "role_switching"
10
11class ScannerConfig(BaseModel):
12    scanner_type: ScannerType
13    enabled: bool = True
14    confidence_threshold: float = Field(ge=0.0, le=1.0, default=0.85)
15    action_on_detect: str = "block"
16    scan_system_prompt: bool = False
17    scan_user_input: bool = True
18
19class LlamaFirewallConfig(BaseModel):
20    server_name: str = "llamafirewall"
21    namespace: str = "guardrails"
22    replicas: int = 2
23    scanners: list[ScannerConfig] = Field(default_factory=list)
24    timeout_seconds: float = 3.0
25    fallback_on_timeout: str = "block"
26
27input_config = LlamaFirewallConfig(
28    scanners=[
29        ScannerConfig(scanner_type=ScannerType.PROMPT_INJECTION),
30        ScannerConfig(scanner_type=ScannerType.JAILBREAK_DETECTION),
31    ]
32)

Both configs serialize cleanly via Pydantic's .model_dump() and can be reloaded at runtime without restarting the Kubernetes deployment — the Helm chart mounts the serialized config as a ConfigMap that each service polls on a configurable interval, enabling hot-reload of thresholds for A/B testing without downtime.

Verify by instantiating both config objects in a Python REPL and confirming that passing a threshold or confidence_threshold value outside [0.0, 1.0] raises a ValidationError — this confirms Pydantic's field constraints are active and will catch misconfigured thresholds before any Helm deployment reaches the cluster.

Do's and Don'ts

Having walked through the material above, the following Do's and Don'ts distill it into practice.

Do's

1. ✓Do set fallback_on_timeout asymmetrically across the two services — use "pass" in GuardrailsAIConfig and "block" in LlamaFirewallConfig. An output validator timing out is an availability trade-off; an input scanner timing out means a prompt reaches the LLM with zero injection or jailbreak coverage, which is unmitigated risk.
2. ✓Do rely on Pydantic's Field(ge=0.0, le=1.0) constraints on threshold and confidence_threshold as the pre-deployment gate — instantiate both config objects in a Python REPL and confirm that an out-of-range value (e.g., threshold=1.5) raises ValidationError before any Helm chart reaches the cluster, especially when iterating A/B threshold experiments.
3. ✓Do default ScannerConfig.confidence_threshold higher (0.85) than ValidatorConfig.threshold (0.5) — a false positive on the input side blocks the user's request outright, so precision is critical; output validators act after the model has responded and can tolerate a lower bar without blocking the user interaction entirely.

Don'ts

1. ✗Don't enable scan_system_prompt in ScannerConfig unless the system prompt contains user-supplied content — the field defaults to False because system prompts are trusted operator inputs; enabling it unconditionally adds latency and produces false positives on controlled operator text while providing no meaningful injection coverage.
2. ✗Don't pass threshold values to the Helm ConfigMap as raw dicts, bypassing ValidatorConfig or ScannerConfig instantiation — the ge=0.0, le=1.0 constraints only fire at Pydantic model construction time; serializing an unchecked dict directly via .model_dump() skips this gate and allows values like threshold=-0.1 to reach the cluster silently during A/B threshold rollouts.
3. ✗Don't share a single fallback_on_timeout policy between GuardrailsAIConfig and LlamaFirewallConfig — the two services sit on opposite sides of the model call (output vs. input), so their degraded-mode behavior must reflect different risk surfaces; treating them identically either leaves input prompts unscanned during LlamaFirewall timeouts or unnecessarily blocks users when Guardrails AI is slow.