Build Llama Guard 4 content classifier

Introduction

When you ship an LLM-powered application, every user prompt and every model response is a potential vector for harmful content — violent instructions, PII leaks, defamation, jailbreak payloads — and a single missed classification can put unsafe output in front of real users and trigger a compliance incident. This lesson teaches you how to classify both LLM inputs and outputs across the 14 MLCommons hazard categories using Llama Guard 4. You will build a ContentClassifier that wraps your primary LLM with bidirectional safety gates—evaluating user prompts before inference and model responses before delivery—and emits structured SafetyVerdict objects you can log for compliance auditing. By the end you will be able to configure category subsets per user role, parse Llama Guard 4's raw output into typed verdicts, and persist safety telemetry without leaking the underlying harmful content.

Key Terminology

• Llama Guard 4: A fine-tuned classifier built on the Llama 4 architecture that performs multi-label safety classification over conversation turns, returning either safe or unsafe along with the violated category codes (S1–S14).
• MLCommons hazard taxonomy: The 14-category schema (S1 Violent Crimes through S14 Code Interpreter Abuse) that Llama Guard 4 uses to label content; categories can be selectively enabled per application except S4, which is always-on.
• SafetyVerdict: The structured result your ContentClassifier emits for every classified turn, containing the role (input vs. output), the boolean safe flag, the list of violated category codes, and metadata needed for compliance logs.

Concepts

Understanding the MLCommons Hazard Taxonomy

Llama Guard 4 classifies content against 14 hazard categories derived from the MLCommons AI Safety v0.5 taxonomy. Each category has a short code (S1–S14) and covers a distinct class of harmful content. Understanding these categories is essential because your ContentClassifier must map model outputs back to human-readable violation descriptions and because different applications may need to enable or disable specific categories based on their domain.

• S1 — Violent Crimes: Content that enables, encourages, or depicts acts of violence against persons, including assault, murder, and human trafficking
• S2 — Non-Violent Crimes: Content facilitating fraud, theft, cybercrime, drug trafficking, or weapons-related offenses without direct physical violence
• S3 — Sex-Related Crimes: Content depicting or enabling sexual assault, exploitation, or non-consensual sexual acts
• S4 — Child Sexual Exploitation: Any content that sexualizes minors, including generated or fictional depictions—this category is always-on and cannot be disabled
• S5 — Defamation: Content containing false statements presented as fact that damage the reputation of real individuals or organizations
• S6 — Specialized Advice: Unqualified generation of medical, legal, financial, or other professional advice without appropriate disclaimers
• S7 — Privacy: Content that exposes personal identifiable information (PII), doxxing, or surveillance instructions—this category intersects with your NeMo Guardrails PII detection flow
• S8 — Intellectual Property: Content that reproduces copyrighted material, generates trademark-infringing content, or facilitates IP theft
• S9 — Indiscriminate Weapons: Content providing instructions for chemical, biological, radiological, nuclear, or high-yield explosive weapons (CBRNE)
• S10 — Hate: Content attacking individuals or groups based on protected characteristics including race, religion, gender identity, or disability
• S11 — Suicide & Self-Harm: Content that encourages, instructs, or glorifies suicide or self-injury
• S12 — Sexual Content: Explicit sexual material that falls outside the criminal categories (S3/S4) but exceeds application content policies
• S13 — Elections: Content containing false information about electoral processes, voter suppression tactics, or election manipulation guidance
• S14 — Code Interpreter Abuse: Content that attempts to exploit code execution environments to access unauthorized system resources or data

The category numbering matters because Llama Guard 4 returns violation codes in its raw output (e.g., "unsafe\nS6,S7"), and your classifier must parse these codes to produce structured safety verdicts. Note that category S4 cannot be disabled through prompt configuration—the model always classifies against it regardless of your category selection.

Selective Category Configuration for Domain-Specific Applications

Not every application needs all 14 hazard categories active. A medical education platform might intentionally disable S6 (Specialized Advice) because providing medical information is the application's purpose. A creative writing tool might relax S12 (Sexual Content) while keeping S3 and S4 strictly enforced. The enabled_categories parameter in your ContentClassifier constructor enables this customization. When you pass a subset of categories, only those categories appear in the classification prompt, and Llama Guard 4 evaluates content only against the included categories. Category S4 (Child Sexual Exploitation) remains active regardless of your configuration—this is enforced at the model level, not in your code.

This selective configuration ties directly into your JWT authentication layer. Different user roles—authenticated via the PyJWT tokens you implemented earlier—may warrant different safety profiles. An admin user running safety research might need relaxed S10 (Hate) classification to evaluate content moderation effectiveness, while a standard user gets the full 14-category sweep. Your Redis-backed session can store the user's safety profile alongside their session data, and the ContentClassifier instance can be configured per-request based on the authenticated user's role. This pattern avoids creating multiple classifier instances; instead, you instantiate one per request with the appropriate category list derived from the session's role-based access control metadata.

Logging Safety Verdicts for Compliance

Every SafetyVerdict your classifier produces should be persisted for compliance auditing. At minimum, log the timestamp, the user's JWT subject claim (never the raw token), the classification role (input vs. output), the boolean verdict, any violated category codes, and the conversation turn index. Do not log the raw message content in the safety audit log—store a reference to the conversation ID in your Redis session store instead. This separation ensures that your safety audit trail does not itself become a repository of harmful content, while still enabling investigators to reconstruct the full context when needed by cross-referencing the conversation ID with the session data. Structure these logs as JSON lines emitted to a dedicated safety audit stream, separate from your application logs, so that compliance teams can query them independently without access to your operational infrastructure.

Code Walkthrough

How Llama Guard 4 Differs from Embedding-Based Filters

Traditional content moderation uses embedding similarity or keyword matching to flag unsafe content. These approaches suffer from two critical limitations: they cannot understand context (the word "kill" means different things in "kill the process" versus "kill the person"), and they cannot generalize to novel phrasings. Llama Guard 4 is a fine-tuned LLM—specifically built on the Llama 4 architecture—that performs multi-label classification by understanding the semantic meaning of the input text within the context of a conversation. It accepts the full conversation history, not just the latest message, which means it can detect unsafe content that only becomes apparent in context (e.g., a user gradually steering a conversation toward harmful instructions across multiple turns).

The model supports two classification modes. In prompt classification mode, it evaluates user input before it reaches your primary LLM, blocking unsafe queries before they consume inference resources. In response classification mode, it evaluates the LLM's generated output before it reaches the user, catching cases where your primary model produces unsafe content despite its own alignment training. A production pipeline should use both modes, creating a bidirectional safety gate.

Loading diagram...

This bidirectional classification flow shows how Llama Guard 4 wraps your primary LLM. The input classifier catches unsafe user prompts before they consume inference compute. The output classifier catches hallucinated or misaligned responses before they reach the user. Both classifiers return structured verdicts containing the specific hazard categories violated, enabling your application to log granular safety telemetry for compliance reporting.

Building the ContentClassifier Class

The core implementation centers on a ContentClassifier class that communicates with Llama Guard 4 through Together.ai's OpenAI-compatible API. This design choice is deliberate: by using the standard OpenAI SDK with a custom base_url, you avoid vendor-specific client libraries and can switch inference providers (Together.ai, Fireworks, self-hosted vLLM) by changing a single URL. The classify_message method formats the conversation into Llama Guard 4's expected prompt structure, sends it for classification, and parses the model's response into a structured SafetyVerdict object. The _build_prompt helper method constructs the safety prompt with configurable category selection, while _parse_response extracts the safe/unsafe verdict and any violated category codes from the model's raw text output.

1from openai import OpenAI
2from dataclasses import dataclass, field
3from enum import Enum
4
5class HazardCategory(Enum):
6    S1 = "Violent Crimes"
7    S2 = "Non-Violent Crimes"
8    S3 = "Sex-Related Crimes"
9    S4 = "Child Sexual Exploitation"
10    S5 = "Defamation"
11    S6 = "Specialized Advice"
12    S7 = "Privacy"
13    S8 = "Intellectual Property"
14    S9 = "Indiscriminate Weapons"
15    S10 = "Hate"
16    S11 = "Suicide & Self-Harm"
17    S12 = "Sexual Content"
18    S13 = "Elections"
19    S14 = "Code Interpreter Abuse"
20
21@dataclass
22class SafetyVerdict:
23    is_safe: bool
24    violated_categories: list[HazardCategory] = field(default_factory=list)
25    raw_response: str = ""
26    classification_role: str = "user"  # "user" or "assistant"
27
28class ContentClassifier:
29    MODEL = "meta-llama/Llama-Guard-4-12B"
30
31    def __init__(self, api_key: str,
32                 enabled_categories: list[HazardCategory] | None = None):
33        self.client = OpenAI(
34            api_key=api_key,
35            base_url="https://api.together.xyz/v1",
36        )
37        self.enabled_categories = enabled_categories or list(HazardCategory)
38
39    def classify_message(self, conversation: list[dict],
40                         role: str = "user") -> SafetyVerdict:
41        guard_messages = [
42            {"role": "user", "content": self._build_prompt(conversation, role)}
43        ]
44        response = self.client.chat.completions.create(
45            model=self.MODEL,
46            messages=guard_messages,
47            max_tokens=100,
48            temperature=0.0,
49        )
50        raw = response.choices[0].message.content.strip()
51        return self._parse_response(raw, role)
52
53    def _build_prompt(self, conversation: list[dict], role: str) -> str:
54        category_block = "\n".join(
55            f"{cat.name}: {cat.value}" for cat in self.enabled_categories
56        )
57        formatted_turns = "\n".join(
58            f"{msg['role']}: {msg['content']}" for msg in conversation
59        )
60        return (
61            f"Task: Check if there is unsafe content in "
62            f"'{role}' messages.\n\n"
63            f"<BEGIN UNSAFE CATEGORIES>\n{category_block}\n"
64            f"<END UNSAFE CATEGORIES>\n\n"
65            f"<BEGIN CONVERSATION>\n{formatted_turns}\n"
66            f"<END CONVERSATION>"
67        )
68
69    def _parse_response(self, raw: str, role: str) -> SafetyVerdict:
70        lines = raw.strip().split("\n")
71        is_safe = lines[0].lower() == "safe"
72        violated = []
73        if not is_safe and len(lines) > 1:
74            codes = [c.strip() for c in lines[1].split(",")]
75            for code in codes:
76                try:
77                    violated.append(HazardCategory[code])
78                except KeyError:
79                    continue
80        return SafetyVerdict(
81            is_safe=is_safe, violated_categories=violated,
82            raw_response=raw, classification_role=role,
83        )

Bidirectional Classification in the Request Pipeline

With the ContentClassifier built, you need to wire it into your FastAPI request pipeline so that every conversation turn passes through both input and output classification. The following classify_bidirectional function demonstrates this integration pattern. It accepts the user's new message along with the existing conversation history, runs input classification first (to block unsafe prompts before they hit your primary LLM), then runs output classification on the generated response. The function returns a ClassificationResult that contains the safety verdicts for both directions along with the response text, giving your endpoint handler everything it needs to decide whether to return the response or a safety fallback message. This function also integrates with your Redis-backed session management by storing safety metadata alongside the conversation history.

1@dataclass
2class ClassificationResult:
3    input_verdict: SafetyVerdict
4    output_verdict: SafetyVerdict | None
5    response_text: str
6    blocked: bool
7
8async def classify_bidirectional(
9    classifier: ContentClassifier,
10    conversation: list[dict],
11    user_message: str,
12    generate_fn,  # async callable that produces LLM response
13) -> ClassificationResult:
14    updated_convo = conversation + [{"role": "user", "content": user_message}]
15
16    input_verdict = classifier.classify_message(updated_convo, role="user")
17    if not input_verdict.is_safe:
18        categories = ", ".join(c.value for c in input_verdict.violated_categories)
19        return ClassificationResult(
20            input_verdict=input_verdict,
21            output_verdict=None,
22            response_text=f"Your message was flagged for: {categories}. "
23                          f"Please rephrase your request.",
24            blocked=True,
25        )
26
27    llm_response = await generate_fn(updated_convo)
28    full_convo = updated_convo + [{"role": "assistant", "content": llm_response}]
29
30    output_verdict = classifier.classify_message(full_convo, role="assistant")
31    if not output_verdict.is_safe:
32        categories = ", ".join(c.value for c in output_verdict.violated_categories)
33        return ClassificationResult(
34            input_verdict=input_verdict,
35            output_verdict=output_verdict,
36            response_text="I'm unable to provide that information as it "
37                          "may involve sensitive content.",
38            blocked=True,
39        )
40
41    return ClassificationResult(
42        input_verdict=input_verdict,
43        output_verdict=output_verdict,
44        response_text=llm_response,
45        blocked=False,
46    )

Do's and Don'ts

Do's

1. ✓Do run classify_message twice per turn — once with role="user" before inference and once with role="assistant" before delivery — a unidirectional gate only on inputs lets your primary LLM's own misaligned outputs reach users unchecked, which is exactly the failure mode the bidirectional pipeline is designed to close.
2. ✓Do pass the full conversation history to classify_message, not just the latest message — Llama Guard 4's LLM architecture detects hazards that only emerge across multiple turns (e.g., a user gradually steering toward harmful instructions), and truncating history to the current prompt makes those multi-turn jailbreaks invisible to the classifier.
3. ✓Do configure enabled_categories as a list[HazardCategory] subset per user role rather than enforcing all 14 S1–S14 categories globally — blanket enforcement can over-block legitimate professional queries (e.g., S6 Specialized Advice for credentialed users), and the ContentClassifier.__init__ accepts this list precisely so you can dial the hazard surface to the audience.

Don'ts

1. ✗Don't replace Llama Guard 4 with keyword matching or embedding similarity — neither technique understands semantic context, so "kill the process" and "kill the person" score identically, and novel phrasings trivially bypass filters that the fine-tuned Llama Guard 4 model generalizes across.
2. ✗Don't set temperature above 0.0 when calling the meta-llama/Llama-Guard-4-12B model — any nonzero temperature introduces sampling randomness into safety verdicts, meaning the same user prompt can flip between is_safe=True and is_safe=False across requests, making your compliance telemetry unreproducible and the gate unpredictably permeable.
3. ✗Don't persist raw_response or message content when a SafetyVerdict records a violation — the violated_categories list of HazardCategory codes (e.g., [S4, S11]) is sufficient for compliance logging; storing the harmful text itself creates a secondary exposure vector and complicates data-retention obligations the lesson explicitly calls out.