Build a Llama 4 Maverick streaming adapter via Together.ai

Introduction

When you ship a multi-provider chat UI, the moment you add an open-weight model the integration story usually fragments — different SDKs, different stream formats, different auth flows. Teams that try to bolt Llama onto a working OpenAI streaming pipeline often end up maintaining two parallel SSE adapters, and a single drift between them shows up in production as half-rendered tokens or stuck cursors. Llama 4 Maverick on Together.ai sidesteps that by exposing the OpenAI wire protocol, so the same AsyncOpenAI client you already use for GPT-4o can talk to a 400-billion-parameter MoE model with a one-line base_url swap. By the end of this lesson you'll be able to build a Llama4StreamAdapter that streams Maverick tokens as SSE frames through FastAPI, with client-disconnect detection that cancels upstream requests so a closed browser tab does not keep billing your account.

Key Terminology

• Server-Sent Events (SSE): A one-way HTTP streaming protocol where the server emits data: <payload>\n\n frames over a long-lived response, consumed in the browser via the EventSource API.
• base_url override: The AsyncOpenAI constructor parameter that redirects all SDK HTTP traffic from api.openai.com to any OpenAI-compatible endpoint such as https://api.together.xyz/v1.
• StreamingResponse: FastAPI's response class that consumes an async generator and writes each yielded chunk to the wire immediately, bypassing the framework's default full-body buffering.
• Client-disconnect detection: Polling request.is_disconnected() between yielded frames so the server can stop reading the upstream provider stream as soon as the browser closes the connection.

Concepts

Why Together.ai for Llama 4 Maverick

Llama 4 Maverick is a 17B-active-parameter, 128-expert MoE model that routes each token through a subset of experts, achieving throughput comparable to dense models one-tenth its total size. Together.ai provisions this model across clustered NVIDIA H100 nodes with speculative decoding enabled, delivering first-token latencies under 300 milliseconds for typical chat prompts. The critical architectural advantage is API compatibility: Together.ai implements the /v1/chat/completions endpoint with identical request and response schemas to OpenAI, including the stream: true parameter, ChatCompletionChunk objects, and finish_reason signaling. This means your adapter needs zero custom HTTP logic — the official openai Python SDK handles connection pooling, retry backoff, and SSE parsing natively.

• Together.ai: A managed inference platform that hosts open-weight models behind OpenAI-compatible API endpoints, eliminating the need for self-hosted GPU infrastructure.
• Llama 4 Maverick: Meta's mixture-of-experts chat model using 17 billion active parameters per forward pass out of 400 billion total, optimized for instruction-following and multi-turn dialogue.
• OpenAI-compatible API: A REST interface that mirrors the OpenAI /v1/chat/completions request and response schema, enabling SDK reuse across providers.
• Mixture-of-Experts (MoE): A model architecture where each input token activates only a small subset of "expert" sub-networks, reducing compute cost while maintaining capacity equivalent to the full parameter count.

Error handling and retry strategy

Together.ai's API enforces rate limits that differ from OpenAI's — free-tier accounts are capped at 60 requests per minute and 1,000 tokens per second of throughput. When limits are exceeded, the server returns HTTP 429 with a Retry-After header. The openai SDK handles this transparently through its built-in retry mechanism: it reads the Retry-After value, waits the specified duration, and replays the request up to max_retries times (defaulting to 2). You can override this during client initialization by passing max_retries=5 to AsyncOpenAI for production deployments where brief throttling spikes are expected.

Beyond rate limits, two Together.ai-specific failure modes require attention. First, the FP8-quantized Maverick checkpoint occasionally produces null content deltas mid-stream — not at the start where None deltas are normal, but interspersed between valid tokens. Your delta extraction guard (if delta_content is not None) already handles this. Second, Together.ai returns finish_reason="length" when the response hits max_tokens without a natural stop. Your application should surface this to the user so they know the response was truncated, not complete.

Provider normalization considerations

When integrating Llama4StreamAdapter alongside the OpenAI, Gemini, and Anthropic adapters from earlier goals, the normalization surface is minimal because Together.ai already speaks the OpenAI wire protocol. However, three differences require attention in a production multi-provider system:

1. Model capability metadata — Llama 4 Maverick supports a 1,048,576-token context window compared to GPT-4o's 128,000 tokens. Your routing layer must track per-model limits to avoid silent truncation.

2. Token counting — Together.ai does not return usage objects in streaming responses. If you need token accounting, you must either count tokens client-side using the Llama tokenizer (tiktoken does not support Llama vocabulary) or make a non-streaming call with stream_options={"include_usage": True}, which Together.ai supports as of their March 2026 API update.

3. System prompt handling — Llama 4 Maverick treats the system message as a prefix to the first user turn rather than a persistent instruction. For multi-turn conversations, repeat critical system instructions in a developer-role message every N turns to prevent instruction drift.

Code Walkthrough

Having just covered the architectural compatibility, retry behavior, and normalization differences in the Concepts section, you can now translate those ideas into a working adapter. The walkthrough below demonstrates the base_url override, the chunk-delta extraction guard, and the disconnect-detection pattern in concrete code.

Adapter class — client initialization and streaming generator

The core insight behind the Together.ai adapter is that openai.AsyncOpenAI accepts a base_url constructor parameter that redirects all HTTP traffic away from api.openai.com to any compatible endpoint. Combined with a separate API key sourced from the TOGETHER_API_KEY environment variable, you get a fully isolated client instance that shares no credentials or routing with your OpenAI adapter. The implementation below defines the Llama4StreamAdapter class in one piece: an init method that reads the API key, instantiates the async client with Together.ai's base URL, and stores the FP8 model identifier as a class constant, followed by a stream_chat async generator that opens a streaming connection, iterates over ChatCompletionChunk objects, extracts token deltas, and yields W3C-compliant SSE frames. Together.ai returns chunks with identical structure to OpenAI — each chunk's choices[0].delta.content holds the token fragment and choices[0].finish_reason signals completion.

Code snippetpython
1import os
2from openai import AsyncOpenAI
3from typing import AsyncGenerator
4
5class Llama4StreamAdapter:
6    """Streaming adapter for Llama 4 Maverick via Together.ai."""
7
8    MODEL_ID = "meta-llama/Llama-4-Maverick-17B-128E-Instruct-FP8"
9
10    def __init__(self) -> None:
11        api_key = os.environ.get("TOGETHER_API_KEY")
12        if api_key is None:
13            raise EnvironmentError(
14                "TOGETHER_API_KEY environment variable is not set. "
15                "Obtain a key at https://api.together.xyz/settings/api-keys"
16            )
17        self.client = AsyncOpenAI(
18            base_url="https://api.together.xyz/v1",
19            api_key=api_key,
20        )
21        self.default_max_tokens = 2048
22
23    async def stream_chat(
24        self,
25        messages: list[dict[str, str]],
26        max_tokens: int | None = None,
27    ) -> AsyncGenerator[str, None]:
28        """Yield SSE-formatted frames from Llama 4 Maverick streaming."""
29        token_budget = max_tokens or self.default_max_tokens
30
31        stream = await self.client.chat.completions.create(
32            model=self.MODEL_ID,
33            messages=messages,
34            max_tokens=token_budget,
35            temperature=0.7,
36            stream=True,
37        )
38
39        async for chunk in stream:
40            if not chunk.choices:
41                continue
42
43            choice = chunk.choices[0]
44            delta_content = choice.delta.content
45
46            if delta_content is not None:
47                escaped = delta_content.replace("\n", "\\n")
48                yield f"data: {escaped}\n\n"
49
50            if choice.finish_reason is not None:
51                yield "data: [DONE]\n\n"
52                break

• Imports + class header: AsyncOpenAI from the official openai package gives non-blocking HTTP, and AsyncGenerator types the streaming return value. The Llama4StreamAdapter class encapsulates all Together.ai-specific configuration so calling code remains provider-agnostic.
• MODEL_ID: The Together.ai model registry identifier as a class-level constant. The FP8 suffix indicates 8-bit floating-point quantization, which Together.ai uses to maximize throughput on H100 GPUs without meaningful quality loss.
• __init__: Read TOGETHER_API_KEY from the environment; the explicit None check raises an EnvironmentError with an actionable message pointing to the Together.ai dashboard, preventing cryptic 401 errors at request time. Instantiate AsyncOpenAI with base_url pointing to Together.ai's API gateway — the SDK appends /chat/completions automatically when you call client.chat.completions.create. The 2048-token default caps generation length to prevent runaway completions that consume billing credits.
• stream_chat signature: Returns AsyncGenerator[str, None]; each yielded value is an SSE frame string and the generator does not accept sent values via .asend().
• Token budget + create call: The or operator falls back to self.default_max_tokens when the caller omits max_tokens. The await is necessary because the SDK performs the initial HTTP handshake asynchronously before returning the AsyncStream[ChatCompletionChunk] iterator. temperature=0.7 balances creativity and coherence for chat applications.
• Chunk loop: if not chunk.choices skips keep-alive frames with an empty choices list, preventing IndexError. The delta_content is not None guard skips the first chunk (which typically carries role metadata with delta.content = None) and any null deltas the FP8 checkpoint emits mid-stream. Newlines are escaped to \\n because the SSE specification uses literal newlines as frame delimiters — an unescaped \n would split one logical token into two SSE events.
• Termination: When finish_reason transitions to a terminal value ("stop" or "length"), yield the [DONE] sentinel and break. The break is defensive — Together.ai may send additional chunks after the finish signal during connection teardown.

Request flow through the adapter

The following diagram traces a single streaming request from the FastAPI endpoint through the Llama4StreamAdapter to Together.ai's inference cluster and back to the browser client. Each arrow represents an async boundary where control yields back to the event loop, enabling concurrent request handling.

Llama4StreamAdapter bridges the /chat/llama4 FastAPI endpoint to Together.ai's completion API by calling stream_chat(messages) with stream=true. Each ChatCompletionChunk carrying a token delta (δ₁, δ₂, …) is reformatted into SSE "data: δ\n\n" frames and flushed to the browser in real time. The sequence terminates when Together.ai returns finish_reason=stop, triggering the final "data: [DONE]\n\n" sentinel that signals the browser to close the event stream.

Loading diagram...

Notice that the FastAPI layer never buffers the full response — each SSE frame flows through StreamingResponse the instant the adapter yields it. This keeps memory consumption constant regardless of response length, which matters when Llama 4 Maverick generates multi-thousand-token completions.

FastAPI endpoint with disconnect detection

The final piece connects the adapter to a FastAPI route that returns a StreamingResponse with the correct text/event-stream content type. Client disconnect detection is critical for cost control: if a user navigates away mid-stream, you must cancel the upstream Together.ai request to stop billing. The implementation below wraps the adapter's generator in an outer async generator that catches asyncio.CancelledError, which FastAPI raises when the client TCP connection closes. The route handler instantiates Llama4StreamAdapter, validates the incoming request body against a Pydantic model, and returns the StreamingResponse with appropriate cache-control headers to prevent CDN buffering.

1import asyncio
2from fastapi import FastAPI, Request
3from fastapi.responses import StreamingResponse
4from pydantic import BaseModel
5
6app = FastAPI()
7
8class ChatRequest(BaseModel):
9    messages: list[dict[str, str]]
10    max_tokens: int | None = None
11
12async def guarded_stream(
13    adapter: Llama4StreamAdapter,
14    request: Request,
15    messages: list[dict[str, str]],
16    max_tokens: int | None,
17) -> AsyncGenerator[str, None]:
18    """Wrap adapter stream with disconnect detection."""
19    try:
20        async for frame in adapter.stream_chat(messages, max_tokens):
21            if await request.is_disconnected():
22                break
23            yield frame
24    except asyncio.CancelledError:
25        yield "data: [CANCELLED]\n\n"
26
27@app.post("/chat/llama4")
28async def chat_llama4(body: ChatRequest, request: Request):
29    adapter = Llama4StreamAdapter()
30    generator = guarded_stream(adapter, request, body.messages, body.max_tokens)
31    return StreamingResponse(
32        generator,
33        media_type="text/event-stream",
34        headers={
35            "Cache-Control": "no-cache",
36            "X-Accel-Buffering": "no",
37        },
38    )

Do's and Don'ts

Do's

1. ✓Do instantiate AsyncOpenAI with an explicit base_url="https://api.together.xyz/v1" and a TOGETHER_API_KEY sourced from the environment — the SDK appends /chat/completions automatically, so a single constructor override redirects all traffic to Together.ai without touching any other call site in your multi-provider pipeline.
2. ✓Do guard every chunk with if not chunk.choices: continue and if delta_content is not None before yielding an SSE frame — Together.ai's FP8 checkpoint emits keep-alive frames with empty choices lists and null-content deltas mid-stream; skipping both prevents IndexError and spurious empty frames that confuse browser EventSource parsers.
3. ✓Do escape newline characters to \\n before yielding each token delta as data: {escaped}\n\n — the SSE wire format uses literal newline pairs as frame delimiters, so an unescaped \n inside a token splits one logical delta into two events and corrupts the reconstructed text on the client.

Don'ts

1. ✗Don't omit the TOGETHER_API_KEY is None check in __init__ — without it, a missing environment variable propagates as a cryptic 401 from Together.ai at the first stream_chat call rather than an EnvironmentError with an actionable message at startup, making misconfigured deployments nearly impossible to diagnose.
2. ✗Don't reuse the same AsyncOpenAI client instance across your OpenAI and Together.ai adapters — base_url and api_key are baked into the client at construction time; sharing one instance means either provider's credentials silently route to the wrong endpoint, producing 401s or billing the wrong account.
3. ✗Don't continue iterating the async for chunk in stream loop after finish_reason becomes non-None without a break — Together.ai can emit additional chunks during connection teardown after the terminal finish signal; without the defensive break, your generator may yield content or a second [DONE] sentinel after the stream is logically complete, causing double-close errors in FastAPI's StreamingResponse.