Build content safety middleware with Llama Guard 4

Introduction

When you wire a multi-modal API into production, every uploaded image, transcribed audio clip, and free-text prompt becomes a liability vector — a single CSAM image or doxxing request that slips through to your vision pipeline can trigger billable GPT-4o and Gemini calls, get persisted in logs, and put you on the wrong side of a takedown order before you've even returned a response. Llama Guard 4 is Meta's multimodal safety classifier that scores text and images against 14 MLCommons hazard categories (S1–S14), and Together.ai exposes it through an OpenAI-compatible endpoint, so you can bolt it onto a FastAPI stack without standing up GPU inference yourself. By the end of this lesson you will be able to deploy Llama Guard 4 as request-blocking middleware that classifies content before it fans out to downstream models, parse the category codes it returns, and short-circuit unsafe requests with structured 422 responses that name the violated categories.

Key Terminology

• Llama Guard 4: Meta's multimodal safety classifier (meta-llama/Llama-Guard-4-12B) that scores text and image inputs against the 14 MLCommons hazard categories (S1–S14) and returns either safe or unsafe\n<category codes>.
• Together.ai inference endpoint: An OpenAI-compatible chat-completions API (https://api.together.xyz/v1) that hosts Llama Guard 4, letting middleware call the classifier via AsyncOpenAI without standing up dedicated GPU inference.
• MLCommons hazard taxonomy (S1–S14): The 14-category AI safety taxonomy (v0.5) that Llama Guard 4 emits as short codes (e.g. S1, S10) which middleware maps to human-readable labels for logging and 422 error responses.
• Fail-closed middleware: A safety-classifier failure policy where any Together.ai error or timeout rejects the request rather than letting it proceed to downstream models — the production-default behavior for ContentSafetyMiddleware.
• Short-circuit response: The 422 status returned by the middleware when Llama Guard 4 flags content as unsafe, listing the violated category codes and descriptions before any GPT-4o, Gemini, or Whisper call is billed.

Concepts

Understanding Llama Guard 4's Hazard Taxonomy

Llama Guard 4 classifies content against the MLCommons AI Safety taxonomy v0.5, which defines 14 distinct hazard categories (S1 through S14). Your middleware must parse the model's output and map category codes back to human-readable labels for logging and error responses.

• S1 (Violent Crimes): Physical violence including assault, murder, and terrorism planning
• S2 (Non-Violent Crimes): Fraud, theft, cybercrime, and drug trafficking
• S3 (Sex-Related Crimes): Sexual assault, exploitation, and trafficking
• S4 (Child Sexual Exploitation): Any CSAM content—zero-tolerance category
• S5 (Defamation): False statements of fact intended to damage reputation
• S6 (Specialized Advice): Unqualified medical, legal, or financial advice
• S7 (Privacy): PII exposure, doxxing, and surveillance enablement
• S8 (Intellectual Property): Copyright infringement and trade secret disclosure
• S9 (Indiscriminate Weapons): Instructions for CBRN weapons
• S10 (Hate): Content targeting protected characteristics
• S11 (Suicide & Self-Harm): Content promoting or instructing self-harm
• S12 (Sexual Content): Explicit material not involving crimes
• S13 (Elections): Voting misinformation and election interference
• S14 (Code Interpreter Abuse): Exploiting code execution environments

When Llama Guard 4 determines content is unsafe, it returns unsafe followed by a newline and category codes (e.g., unsafe\nS1,S10). Safe content returns safe.

Operational Considerations

When deploying Llama Guard 4 as middleware in a production multi-modal API, several operational factors determine whether it becomes a bottleneck or a seamless safety layer.

Latency budget: Together.ai's inference for Llama Guard 4 typically completes in 200-400ms for text inputs and 400-800ms for image inputs. Since this classification runs before the concurrent GPT-4o and Gemini 2.5 Flash vision calls (which themselves take 1-3 seconds each), the safety check adds proportionally less overhead to image-heavy requests. For audio transcription via Whisper, classify the transcript text after transcription rather than the raw audio—Llama Guard 4 does not accept audio inputs directly.

Batching strategy: If a request contains multiple images, classify each individually rather than packing them into a single call. The model processes one content item at a time, and individual classification produces per-image category mappings.

Failure mode: If Together.ai returns an error or times out, your middleware must decide whether to fail-open or fail-closed. For production systems, fail-closed is the safer default. Set the AsyncOpenAI client's timeout parameter lower than your overall API timeout.

Audit logging: Always persist the raw_response field from SafetyResult alongside the request ID, timestamp, and content hash. This audit trail is essential for compliance and false-positive analysis.

Code Walkthrough

Architecture: Where Safety Classification Sits

In a multi-modal API pipeline, the content safety middleware must execute before any content reaches the concurrent vision analysis, grounding queries, or audio transcription endpoints. This placement ensures that a single unsafe image does not trigger billable API calls to GPT-4o, Gemini, and Whisper simultaneously before being rejected.

This architecture routes every request through ContentSafetyMiddleware, which invokes Llama Guard 4 via Together.ai before any model call executes. Safe requests fan out to vision analysis, grounding, and transcription handlers, while unsafe requests short-circuit with a 422 status and violated category details.

Loading diagram...

This architecture ensures that the safety classification acts as a gate. The ContentSafetyMiddleware class wraps the Together.ai API call and exposes a classify_content() method that accepts a Pydantic ContentInput model. If classification returns unsafe with any triggered categories, the middleware short-circuits the request pipeline and returns a structured error response containing the specific violated category codes and their human-readable descriptions.

Building the Pydantic Models and Safety Classifier

The foundation of the content safety middleware is a set of Pydantic models that define the input contract and classification result, plus the ContentSafetyMiddleware class that wraps the Together.ai call. The ContentInput model handles both text-only and multimodal inputs because Llama Guard 4 accepts both. The SafetyResult model captures whether content is safe, which categories were violated, and the raw model response for audit logging. The HAZARD_CATEGORIES dictionary maps short codes like S1 to full descriptions. The middleware exposes a single classify_content() method that builds the appropriate message format for text or image, sends it to Llama Guard 4 at temperature=0.0, and parses the response — safe returns a clean result, while unsafe\n<codes> is parsed into a list of violated categories with human-readable descriptions.

1from pydantic import BaseModel, Field
2from openai import AsyncOpenAI
3from enum import Enum
4
5class ContentType(str, Enum):
6    TEXT = "text"
7    IMAGE_URL = "image_url"
8
9class ContentInput(BaseModel):
10    content_type: ContentType
11    text: str | None = Field(default=None, description="Text content to classify")
12    image_url: str | None = Field(default=None, description="Image URL to classify")
13
14class SafetyResult(BaseModel):
15    is_safe: bool
16    violated_categories: list[str] = Field(default_factory=list)
17    category_descriptions: dict[str, str] = Field(default_factory=dict)
18    raw_response: str = Field(default="", description="Raw model output for audit")
19
20HAZARD_CATEGORIES: dict[str, str] = {
21    "S1": "Violent Crimes", "S2": "Non-Violent Crimes",
22    "S3": "Sex-Related Crimes", "S4": "Child Sexual Exploitation",
23    "S5": "Defamation", "S6": "Specialized Advice",
24    "S7": "Privacy", "S8": "Intellectual Property",
25    "S9": "Indiscriminate Weapons", "S10": "Hate",
26    "S11": "Suicide & Self-Harm", "S12": "Sexual Content",
27    "S13": "Elections", "S14": "Code Interpreter Abuse",
28}
29
30class ContentSafetyMiddleware:
31    MODEL = "meta-llama/Llama-Guard-4-12B"
32
33    def __init__(self, api_key: str):
34        self._client = AsyncOpenAI(
35            api_key=api_key,
36            base_url="https://api.together.xyz/v1",
37        )
38
39    async def classify_content(self, content: ContentInput) -> SafetyResult:
40        messages = self._build_messages(content)
41        response = await self._client.chat.completions.create(
42            model=self.MODEL,
43            messages=messages,
44            max_tokens=100,
45            temperature=0.0,
46        )
47        raw = response.choices[0].message.content.strip()
48        return self._parse_response(raw)
49
50    def _build_messages(self, content: ContentInput) -> list[dict]:
51        if content.content_type == ContentType.IMAGE_URL:
52            user_content = [
53                {"type": "image_url", "image_url": {"url": content.image_url}},
54                {"type": "text", "text": content.text or "Classify this image."},
55            ]
56        else:
57            user_content = content.text
58        return [{"role": "user", "content": user_content}]
59
60    def _parse_response(self, raw: str) -> SafetyResult:
61        lines = raw.strip().split("\n")
62        verdict = lines[0].strip().lower()
63        if verdict == "safe":
64            return SafetyResult(is_safe=True, raw_response=raw)
65
66        violated = []
67        descriptions = {}
68        if len(lines) > 1:
69            codes = [c.strip() for c in lines[1].split(",")]
70            for code in codes:
71                if code in HAZARD_CATEGORIES:
72                    violated.append(code)
73                    descriptions[code] = HAZARD_CATEGORIES[code]
74
75        return SafetyResult(
76            is_safe=False,
77            violated_categories=violated,
78            category_descriptions=descriptions,
79            raw_response=raw,
80        )

Integrating as FastAPI Middleware

The middleware must intercept requests before they reach your vision, grounding, or transcription handlers. Rather than using ASGI middleware (which operates on raw bytes and complicates Pydantic model access), the idiomatic FastAPI pattern is a dependency function that runs the safety check and raises an HTTPException if content is unsafe. This dependency can be injected into any route that accepts user-generated content, ensuring consistent safety enforcement across the concurrent GPT-4o/Gemini vision pipeline, Gemini Google Search grounding endpoint, and Whisper audio transcription handler. The following code defines the verify_content_safety dependency function that extracts a ContentInput from the request body, runs classification, and either passes through or rejects the request with a 422 status containing the specific violated hazard categories.

1from fastapi import Depends, HTTPException, status
2import os
3
4_middleware = ContentSafetyMiddleware(
5    api_key=os.environ["TOGETHER_API_KEY"]
6)
7
8async def verify_content_safety(content: ContentInput) -> ContentInput:
9    result = await _middleware.classify_content(content)
10    if not result.is_safe:
11        raise HTTPException(
12            status_code=status.HTTP_422_UNPROCESSABLE_ENTITY,
13            detail={
14                "error": "content_safety_violation",
15                "violated_categories": result.violated_categories,
16                "descriptions": result.category_descriptions,
17            },
18        )
19    return content
20
21# Usage in route definitions:
22# @app.post("/api/v1/vision/analyze")
23# async def analyze_image(
24#     content: ContentInput = Depends(verify_content_safety),
25# ):
26#     # Content is guaranteed safe here — proceed with
27#     # concurrent GPT-4o + Gemini 2.5 Flash analysis
28#     ...

Do's and Don'ts

Do's

1. ✓Do invoke classify_content() as the first step in the request pipeline, before any downstream model call — placing ContentSafetyMiddleware at the gateway ensures a single unsafe image cannot trigger simultaneous billable calls to GPT-4o, Gemini, and Whisper before being rejected; running it after fan-out defeats the entire middleware pattern.
2. ✓Do set temperature=0.0 in the chat.completions.create call to meta-llama/Llama-Guard-4-12B — Llama Guard 4's output is a deterministic label (safe or unsafe\nS4\nS12); any non-zero temperature introduces randomness into a binary safety decision that must be reproducible across identical inputs for auditing and compliance.
3. ✓Do store raw_response in SafetyResult alongside the parsed violated_categories list — the unmodified Llama Guard 4 output is the only audit evidence that a specific request was blocked; without it you cannot reconstruct which S-codes triggered the rejection or investigate false-positive blocks after the fact.

Don'ts

1. ✗Don't parse Llama Guard 4's response by reading only the first line — when content is unsafe the model returns unsafe\nS1\nS3 with category codes on subsequent lines; stopping after unsafe leaves violated_categories empty, makes the 422 response meaningless to callers, and loses the HAZARD_CATEGORIES lookup that converts codes like S4 to "Child Sexual Exploitation".
2. ✗Don't send image inputs to _build_messages() as a plain text string — Llama Guard 4's multimodal classification requires the structured {"type": "image_url", "image_url": {"url": ...}} content block; passing an image URL as the text field bypasses vision entirely and produces unreliable text-only classification that silently misses image-borne hazards.
3. ✗Don't instantiate AsyncOpenAI with the default api.openai.com base URL when targeting Together.ai — Llama Guard 4 is served at https://api.together.xyz/v1; omitting the base_url override routes the request to OpenAI's endpoint, which does not host the model and returns a 404 or a model-not-found error with no safety classification performed.