Deploy an OpenTelemetry Collector with Langfuse Exporter

Introduction

When you run multi-provider GenAI pipelines across multiple agents, tracing data scatters across application logs and SDK callbacks with no central place to observe latency, token costs, or failure modes end-to-end. The OpenTelemetry Collector solves this by acting as a single ingestion and routing layer — your applications emit spans once, and the collector forwards them to Langfuse where AI-specific analysis, token aggregation, and cost tracking become available. By the end of this lesson, you will have a working collector configuration with an OTLP receiver, a batching and enrichment processor pipeline, and a Langfuse HTTP exporter, along with Kubernetes manifests that make those endpoints reachable to every service in your cluster.

Key Terminology

• OTLP (OpenTelemetry Protocol) — The wire protocol used to transmit spans from instrumented services to the collector; the otlp receiver opens a gRPC endpoint on port 4317 and an HTTP endpoint on port 4318 so applications using either transport can deliver spans without code changes.
• Collector pipeline — The ordered data path declared under service.pipelines.traces that routes spans through receivers, then processors, then exporters; this ordering is enforced at startup and determines how each span is transformed before reaching Langfuse.
• memory_limiter processor — A guard processor that caps the collector's heap at limit_mib: 512 with a spike_limit_mib: 128 ceiling; it must be placed first in the processor chain so it acts before batch allocates additional buffer memory during traffic spikes.
• batch processor — Groups spans into payloads of up to send_batch_size: 512 or timeout: 5s, whichever comes first, reducing the number of individual HTTP round-trips the collector makes to Langfuse's ingestion endpoint.
• Span attribute enrichment — The practice of using the attributes processor's upsert action to stamp every span with metadata fields such as environment and service.namespace at the collector layer, without modifying any application instrumentation code.
• otlphttp exporter — The pipeline stage that forwards processed spans to Langfuse's /api/public/otel ingestion endpoint over HTTP, attaching a Base64-encoded Authorization header sourced from the LANGFUSE_AUTH environment variable and retrying on transient network failures with configurable back-off.

Concepts

The Collector as a Central Ingestion Point

Multi-provider GenAI pipelines naturally scatter telemetry across SDK callbacks, application logs, and per-agent sidecars with no shared schema and no single place to ask "why did this trace take 8 seconds?" The OpenTelemetry Collector solves this by becoming the one stable address every service writes to — otel-collector.observability.svc:4317 or :4318 — regardless of which model provider, agent framework, or backend ultimately stores the data. Applications emit spans once, and the collector takes full responsibility for buffering, enriching, and forwarding them to Langfuse.

This decoupling has a practical payoff: you can swap Langfuse for a different backend, add a second exporter, or change enrichment rules entirely inside the collector config without touching a single line of application instrumentation. The collector is also where AI-specific metadata gets attached — Langfuse understands token usage, generation cost, and prompt/completion visibility in ways that generic backends like Jaeger do not, which is why routing GenAI traces to Langfuse specifically unlocks dashboards that generic OTLP backends cannot produce.

Processor Ordering and the Memory Safety Contract

The three processors in this pipeline are not interchangeable — their order encodes a deliberate safety contract. memory_limiter runs first because it must assess heap pressure before batch begins accumulating spans into buffers. If batch ran first, a traffic spike could fill buffers past the memory ceiling before the limiter had a chance to act. batch runs second to coalesce spans into efficient payloads; attributes runs last because enrichment only needs to happen once on the final batch shape, not on every raw span independently. The processors array in the service.pipelines.traces block preserves this order exactly (see Code Walkthrough), and changing the sequence is a correctness issue, not just a style preference.

Kubernetes Deployment for Continuous Collection

Running two collector replicas and externalizing configuration via a ConfigMap are deliberate availability choices, not boilerplate. A single-replica collector creates a gap in span ingestion during any pod restart; two replicas let Kubernetes drain and reschedule one pod while the other continues receiving traffic uninterrupted. The ConfigMap pattern separates the collector's YAML from the container image, so configuration updates — changing a batch timeout, adding an attribute, pointing to a new Langfuse endpoint — apply with a rolling restart rather than a full image rebuild. The Langfuse auth token lives in a Kubernetes Secret and is injected as the LANGFUSE_AUTH environment variable, ensuring the credential never appears in the collector config file or in version control (see Code Walkthrough).

Code Walkthrough

Now that you understand the receiver-processor-exporter pipeline and Langfuse's role as the AI-aware trace backend, the configurations below put that architecture into practice with two deployable artifacts: the collector config file and the Kubernetes manifests that run it.

The collector configuration follows the three-stage pipeline the Concepts section described. The receivers block opens two OTLP ports — gRPC on 4317 and HTTP on 4318 — so both gRPC-based and HTTP-based client SDKs can deliver spans without any application-side changes. The processors block chains three processors in a deliberate order: memory_limiter runs first to cap the collector's heap before any buffering begins, batch groups spans into efficient payloads, and attributes stamps every span with environment and namespace labels so Langfuse dashboards can filter across services. The exporters block targets Langfuse's OTLP-over-HTTP ingestion endpoint, injecting a Base64-encoded Authorization header and enabling retry logic for transient network failures.

Code snippetyaml
1# otel-collector-config.yaml
2receivers:
3  otlp:
4    protocols:
5      grpc:
6        endpoint: 0.0.0.0:4317
7      http:
8        endpoint: 0.0.0.0:4318
9
10processors:
11  memory_limiter:
12    check_interval: 1s
13    limit_mib: 512
14    spike_limit_mib: 128
15  batch:
16    timeout: 5s
17    send_batch_size: 512
18  attributes:
19    actions:
20      - key: environment
21        value: production
22        action: upsert
23      - key: service.namespace
24        value: genai-platform
25        action: upsert
26
27exporters:
28  otlphttp:
29    endpoint: http://langfuse:3000/api/public/otel
30    headers:
31      Authorization: "Basic ${LANGFUSE_AUTH}"
32    retry_on_failure:
33      enabled: true
34      initial_interval: 5s
35      max_interval: 30s
36
37service:
38  pipelines:
39    traces:
40      receivers: [otlp]
41      processors: [memory_limiter, batch, attributes]
42      exporters: [otlphttp]

The Kubernetes manifests deploy this configuration as a two-replica Deployment that mounts the YAML above as a ConfigMap, exposes ports 4317 and 4318 through a ClusterIP Service, and pulls the Langfuse auth token from a Secret named langfuse-credentials. Running two replicas means a pod restart does not interrupt span collection — the remaining replica continues receiving traffic while Kubernetes reschedules the failed one.

Code snippetyaml
1# otel-collector-k8s.yaml
2apiVersion: v1
3kind: Service
4metadata:
5  name: otel-collector
6  namespace: observability
7spec:
8  selector:
9    app: otel-collector
10  ports:
11    - name: otlp-grpc
12      port: 4317
13      targetPort: 4317
14    - name: otlp-http
15      port: 4318
16      targetPort: 4318
17---
18apiVersion: apps/v1
19kind: Deployment
20metadata:
21  name: otel-collector
22  namespace: observability
23spec:
24  replicas: 2
25  selector:
26    matchLabels:
27      app: otel-collector
28  template:
29    metadata:
30      labels:
31        app: otel-collector
32    spec:
33      containers:
34        - name: collector
35          image: otel/opentelemetry-collector-contrib:0.96.0
36          ports:
37            - containerPort: 4317
38            - containerPort: 4318
39          args: ["--config=/etc/otel/config.yaml"]
40          volumeMounts:
41            - name: config
42              mountPath: /etc/otel
43          resources:
44            requests:
45              memory: "256Mi"
46              cpu: "100m"
47            limits:
48              memory: "512Mi"
49              cpu: "500m"
50          env:
51            - name: LANGFUSE_AUTH
52              valueFrom:
53                secretKeyRef:
54                  name: langfuse-credentials
55                  key: auth-token
56      volumes:
57        - name: config
58          configMap:
59            name: otel-collector-config

Confirm that both collector pods reach Running status and that port 4317 is reachable from within the cluster before routing any application telemetry through the pipeline.

Do's and Don'ts

Having walked through the material above, the following Do's and Don'ts distill it into practice.

Do's

1. ✓Do order processors as memory_limiter → batch → attributes — placing memory_limiter first caps the collector's heap before any span buffering begins; if it runs after batch, the buffer itself can exhaust memory during traffic spikes before the limiter can intervene.
2. ✓Do mount the Langfuse auth token from the langfuse-credentials Secret into LANGFUSE_AUTH — the otlphttp exporter injects this as a Base64-encoded Authorization: Basic header; hardcoding credentials in otel-collector-config.yaml would expose them in the ConfigMap and in any kubectl get configmap -o yaml output.
3. ✓Do run the collector Deployment with replicas: 2 and confirm both pods reach Running before routing application telemetry — a single-replica deployment drops spans during any pod restart or node eviction, creating gaps in Langfuse traces that make latency and cost aggregations unreliable.

Don'ts

1. ✗Don't open only one OTLP protocol (gRPC 4317 or HTTP 4318) and assume all SDK clients will adapt — gRPC-based clients (e.g., the OpenTelemetry Go and Java SDKs by default) cannot fall back to HTTP, and HTTP-based clients cannot negotiate gRPC; omitting either port forces application-side SDK reconfiguration across every service in the cluster.
2. ✗Don't omit retry_on_failure from the otlphttp exporter block — without initial_interval: 5s and max_interval: 30s, transient network failures between the collector and Langfuse's /api/public/otel endpoint cause spans to be silently dropped rather than retried, producing permanent holes in distributed traces.
3. ✗Don't apply the attributes processor's environment and service.namespace upserts without verifying the values match your actual deployment environment — if both staging and production collectors stamp spans with environment: production, Langfuse dashboards cannot filter by environment, making cross-service cost and latency comparisons meaningless.