Integrate NeMo Guardrails and LlamaFirewall for multi-layer defense

Introduction

When you ship a GenAI application behind a single guardrail engine, you inherit that engine's blind spots verbatim: NeMo Guardrails misses novel prompt injections its Colang rules never anticipated, and LlamaFirewall misses multi-turn social-engineering attacks that look innocent token-by-token. Defense in depth means orchestrating heterogeneous engines so each one's strengths cover the other's gaps—programmable, dialogue-aware policy enforcement from NeMo, and stateless, token-level injection detection from LlamaFirewall. Neither tool alone covers the full threat surface of a production system. By the end of this lesson you'll be able to deploy both engines behind a unified DefenseCoordinator that merges their independent verdicts into a single, auditable security decision with confidence scoring and conflict resolution.

Key Terminology

• Defense Coordinator: An orchestration component that dispatches security evaluations to multiple independent engines concurrently and merges their results into a unified decision.
• Security Verdict: A normalized assessment from a single guardrail engine containing an action recommendation, confidence score, engine attribution, and human-readable reasons.
• Merged Decision: The final security outcome produced by combining multiple independent verdicts according to a configurable merge policy.
• Conflict Resolution: The algorithmic strategy used when two or more engines disagree on whether an input is safe, typically involving threshold comparisons and escalation routing.
• Confidence Calibration: The process of normalizing confidence scores from heterogeneous engines onto a common probability scale so they can be meaningfully compared during decision merging.
• Escalation Path: A routing mechanism that sends ambiguous or conflicting security verdicts to human reviewers rather than making an automated allow/block decision.

Concepts

Why Two Engines Instead of One

A common misconception among teams first adopting guardrails is that a single framework—NeMo Guardrails or LlamaFirewall—can handle every threat category. In practice, each framework's architecture creates inherent coverage gaps.

• NeMo Guardrails uses Colang scripts to define allowed conversational flows. It catches policy violations like off-topic requests, unauthorized data access patterns, and multi-turn social engineering attempts. However, its detection relies on pattern matching against predefined canonical forms, meaning novel prompt injections that don't match existing patterns can slip through.

• LlamaFirewall deploys fine-tuned classifier models (PromptGuard and AuditLog analyzers) that score inputs against learned distributions of malicious prompts. It catches zero-day injection variants and adversarial token sequences that rule-based systems miss. However, it lacks dialogue-state awareness—it evaluates each message in isolation, missing slow-burn attacks that unfold across multiple turns.

• Content safety requires both perspectives: structural policy enforcement (NeMo) and statistical anomaly detection (LlamaFirewall). Red-team evaluation consistently shows that attacks bypassing one engine get caught by the other when both run in parallel.

The key architectural insight is that these engines are not redundant—they are complementary. NeMo Guardrails acts as a policy-layer firewall enforcing business rules, while LlamaFirewall acts as an anomaly-detection layer catching novel threats. Together, they form a defense-in-depth pipeline where each layer addresses a distinct class of risk.

Decision Conflict Resolution Strategies

The strict merge policy shown above—block if either engine says block—prioritizes safety over availability. Production deployments often require more nuanced strategies depending on the application's risk profile. Consider these alternatives:

• Weighted voting: Assign different weights to each engine based on historical precision. If LlamaFirewall has a 92% precision on prompt injection but NeMo has 98% precision on policy violations, weight their confidence scores accordingly when both fire on the same input.

• Category-specific routing: Route prompt injection detection primarily to LlamaFirewall (its strength) and topic boundary enforcement primarily to NeMo Guardrails (its strength). When a category falls outside an engine's specialty, discount its verdict.

• Confidence calibration: Raw confidence scores from different engines are not directly comparable. A 0.8 from LlamaFirewall (a neural classifier) and a 0.8 from NeMo (a rule match) represent fundamentally different levels of certainty. Apply isotonic regression or Platt scaling to each engine's historical scores to calibrate them onto a common probability scale before merging.

• Escalation budgets: Unlimited escalation to human reviewers is impractical at scale. Set a per-hour escalation budget (e.g., 50 cases/hour) and when the budget is exhausted, fall back to the strict policy (block on any disagreement) rather than silently allowing.

Production Deployment Considerations

When deploying the DefenseCoordinator in production, several operational concerns arise beyond the core merge logic:

1. Failover behavior: If LlamaFirewall's model service becomes unavailable, the coordinator must decide whether to fail-open (allow with only NeMo's verdict) or fail-closed (block all traffic). For most GenAI applications handling sensitive data, fail-closed is the correct default. Implement a circuit breaker around each adapter that trips after three consecutive timeouts and routes all traffic through the remaining engine with a heightened block threshold.

2. Latency budgets: Running two engines concurrently via asyncio.gather means total latency equals the slower engine, typically LlamaFirewall at 50–150ms for model inference versus NeMo at 10–30ms for rule evaluation. Set a per-request latency budget (e.g., 200ms) and if the slower engine hasn't responded within the budget, proceed with the available verdict while logging the timeout for security correlation analysis.

3. Audit logging integration: Every MergedDecision must be written to the compliance audit log with the full verdict chain—both individual SecurityVerdict objects, the merge policy applied, whether a conflict occurred, and the final action taken. This audit trail is mandatory for compliance audit and for retrospective red-team evaluation of guardrail effectiveness.

4. Configuration hot-reloading: NeMo's Colang rules and LlamaFirewall's model weights should be updatable without service restarts. Implement a file watcher on the NeMo config directory and a model version endpoint for LlamaFirewall that the coordinator polls at a configurable interval (e.g., every 60 seconds).

The DefenseCoordinator pattern gives you a composable, testable integration point between heterogeneous security engines. By normalizing each engine's output into a shared SecurityVerdict schema and centralizing merge logic in a single method, you can add new engines (Model Armor, custom classifiers, regex-based filters) without modifying existing adapter code—each new engine simply implements the scan interface and plugs into the coordinator's evaluation pipeline.

Code Walkthrough

Now that you understand why the two engines are complementary rather than redundant, you can place both behind a single DefenseCoordinator that fans each request out concurrently and merges their independent verdicts into one decision. The flowchart below traces a request from user input through both engines to one of three outcomes—ALLOW, BLOCK, or ESCALATE.

Each adapter normalizes its engine's native response into a shared SecurityVerdict, so the merge logic operates on one consistent schema. The coordinator runs both scans concurrently with asyncio.gather, then applies a merge policy: BLOCK if either engine flags a high-confidence threat, ESCALATE when verdicts conflict or confidence falls below threshold, and ALLOW only when both engines agree the input is safe.

Code snippetpython
1import asyncio
2from dataclasses import dataclass, field
3from enum import Enum
4
5class Action(Enum):
6    ALLOW = "allow"
7    BLOCK = "block"
8    ESCALATE = "escalate"
9
10@dataclass
11class SecurityVerdict:
12    action: Action
13    confidence: float            # 0.0 - 1.0
14    engine: str
15    reasons: list[str] = field(default_factory=list)
16
17@dataclass
18class MergedDecision:
19    action: Action
20    verdicts: list[SecurityVerdict]
21    conflict: bool = False
22
23class DefenseCoordinator:
24    def __init__(self, nemo_adapter, firewall_adapter,
25                 block_threshold: float = 0.7,
26                 escalation_threshold: float = 0.4):
27        self._nemo = nemo_adapter
28        self._firewall = firewall_adapter
29        self._block = block_threshold
30        self._escalate = escalation_threshold
31
32    async def evaluate(self, user_input: str,
33                       context: dict | None = None) -> MergedDecision:
34        nemo, firewall = await asyncio.gather(
35            self._nemo.scan(user_input, context or {}),
36            self._firewall.scan(user_input, context or {}),
37        )
38        return self._merge([nemo, firewall])
39
40    def _merge(self, verdicts: list[SecurityVerdict]) -> MergedDecision:
41        blocking = [v for v in verdicts
42                    if v.action is Action.BLOCK and v.confidence >= self._block]
43        if blocking:
44            return MergedDecision(Action.BLOCK, verdicts)
45        actions = {v.action for v in verdicts}
46        low_conf = any(v.confidence < self._escalate for v in verdicts)
47        if len(actions) > 1 or low_conf:
48            return MergedDecision(Action.ESCALATE, verdicts, conflict=True)
49        return MergedDecision(Action.ALLOW, verdicts)

You'll know it works when a high-confidence BLOCK from either engine short-circuits the merge to BLOCK, agreeing low-risk inputs return ALLOW, and any disagreement or sub-threshold confidence routes to ESCALATE for human review.

Do's and Don'ts

Do's

1. ✓Do run both NeMo Guardrails and LlamaFirewall concurrently via asyncio.gather — fanning out both scans in parallel keeps latency bounded to the slower of the two engines rather than their sum, which is critical when both engines must complete before the merge step can produce a decision.
2. ✓Do normalize each engine's native response into a shared SecurityVerdict before merging — the merge logic in _merge operates on a single consistent schema (action, confidence, engine, reasons), so adding a third engine later requires only a new adapter, not changes to the decision logic.
3. ✓Do treat verdict disagreement between NeMo and LlamaFirewall as an ESCALATE signal rather than defaulting to ALLOW — when len(actions) > 1 or any confidence falls below escalation_threshold, routing to the human review queue with conflict=True preserves the audit trail and prevents a low-confidence safe verdict from silently overriding a block from the other engine.

Don'ts

1. ✗Don't set block_threshold so high that only unanimous high-confidence verdicts trigger a BLOCK — the _merge logic intentionally short-circuits to BLOCK when either engine flags a threat above threshold, because NeMo and LlamaFirewall cover different attack surfaces; requiring both to agree eliminates the defense-in-depth guarantee.
2. ✗Don't bypass the SecurityVerdict abstraction by branching directly on engine-native response formats inside _merge — coupling the decision logic to NeMo's allow/block/warn strings or LlamaFirewall's safe/unsafe fields means any API change in either engine forces a rewrite of the merge policy rather than only the affected adapter.
3. ✗Don't omit an audit log entry on BLOCK or ESCALATE outcomes — the MergedDecision carries the full verdicts list and conflict flag specifically so every security decision is auditable; discarding that context before logging makes post-incident forensics impossible because you lose which engine fired and at what confidence.