Instrument pipelines with OpenTelemetry GenAI conventions

Introduction

When you instrument a GenAI pipeline with vendor-specific tracing APIs, you lock every span emitter into that backend — switching from Phoenix to Datadog or Grafana Tempo later means rewriting every instrumentation point, and historical traces become unportable. By the end of this lesson you'll be able to instrument LLM calls using OpenTelemetry GenAI semantic conventions, configure an OTel Collector to fan out to Phoenix and Prometheus, and verify that standard attributes like gen_ai.usage.input_tokens appear in your traces.

Key Terminology

• OpenTelemetry (OTel) — a vendor-neutral observability framework that defines APIs, SDKs, and a wire protocol for traces, metrics, and logs; using it keeps your instrumentation portable across backends.
• OTLP (OpenTelemetry Protocol) — the gRPC/HTTP wire format SDKs use to ship spans to the Collector; standardizing on OTLP is what lets one instrumentation point feed multiple backends.
• GenAI semantic conventions — the OTel-defined standard attribute names for LLM operations (gen_ai.system, gen_ai.request.model, gen_ai.usage.input_tokens, gen_ai.response.finish_reason); using them ensures backends parse your LLM telemetry without custom mapping.
• OTel Collector — a stand-alone process (typically a Kubernetes DaemonSet) that receives OTLP spans, batches them, and routes to multiple exporters; it lets you change backends without redeploying instrumented services.
• Span — the unit of trace data representing a single operation (one LLM call, one retrieval step); the attributes attached to a span are what populate dashboards and trace views.

Concepts

Standard attributes beat custom ones

The OTel GenAI semantic conventions (specification v1.37+) define standard attribute names for LLM operations. Key attributes include gen_ai.system (the provider), gen_ai.request.model (the model identifier), gen_ai.usage.input_tokens and gen_ai.usage.output_tokens (token consumption), and gen_ai.response.finish_reason (why the model stopped). Using these names — rather than model_name, tokens_in, or stop_reason — means any OTel-compatible backend can parse and visualize your traces without custom configuration. Phoenix surfaces gen_ai.usage.* automatically; Grafana Tempo can compute token-cost panels from the same attributes; vendor lock-in disappears at the attribute layer (see Code Walkthrough).

Collector as a routing hub

The OTel Collector decouples instrumentation from backend choice. Pipeline pods emit OTLP spans over gRPC to a local Collector; the Collector batches, processes, and fans the same spans out to Phoenix (for LLM-specific trace UX), Prometheus (for time-series aggregates), and any other configured exporter. Swapping Phoenix for another trace backend is a Collector YAML edit, not an application redeploy.

Cost as a first-class span attribute

Token counts come back from every provider; multiplied by per-token rates they become a dollar-value attribute (gen_ai.cost_usd) you attach to each span. Combined with gen_ai.latency_ms, this gives per-call cost/latency observability that aggregates cleanly into dashboards — without a custom exporter.

Code Walkthrough

The two snippets below demonstrate the concepts above: a Collector configuration that routes OTLP spans to Phoenix and Prometheus, and a Python instrumentor that emits LLM spans with GenAI semantic-convention attributes.

Code snippetpython
1import yaml
2
3class CollectorConfigBuilder:
4    def __init__(
5        self,
6        phoenix_endpoint: str = "phoenix:4317",
7        prometheus_port: int = 8889,
8        batch_size: int = 512,
9        batch_timeout_ms: int = 5000,
10    ):
11        self.phoenix_endpoint = phoenix_endpoint
12        self.prometheus_port = prometheus_port
13        self.batch_size = batch_size
14        self.batch_timeout_ms = batch_timeout_ms
15
16    def build_config(self) -> dict:
17        return {
18            "receivers": {
19                "otlp": {"protocols": {"grpc": {"endpoint": "0.0.0.0:4317"}}},
20            },
21            "processors": {
22                "batch": {
23                    "send_batch_size": self.batch_size,
24                    "timeout": f"{self.batch_timeout_ms}ms",
25                },
26            },
27            "exporters": {
28                "otlp/phoenix": {
29                    "endpoint": self.phoenix_endpoint,
30                    "tls": {"insecure": True},
31                },
32                "prometheus": {"endpoint": f"0.0.0.0:{self.prometheus_port}"},
33            },
34            "service": {
35                "pipelines": {
36                    "traces": {
37                        "receivers": ["otlp"],
38                        "processors": ["batch"],
39                        "exporters": ["otlp/phoenix"],
40                    },
41                    "metrics": {
42                        "receivers": ["otlp"],
43                        "processors": ["batch"],
44                        "exporters": ["prometheus"],
45                    },
46                },
47            },
48        }
49
50    def render_yaml(self) -> str:
51        return yaml.dump(self.build_config(), default_flow_style=False)

The OTLP gRPC receiver listens on port 4317; the batch processor (size 512, 5s timeout) trades a small amount of latency for export throughput. Traces export to Phoenix; metrics scrape via Prometheus — one instrumentation point, two backends, no code change to swap either.

Code snippetpython
1from opentelemetry import trace
2import time
3
4tracer = trace.get_tracer(
5    "genai-pipeline",
6    schema_url="https://opentelemetry.io/schemas/1.37.0",
7)
8
9class GenAIInstrumentor:
10    def __init__(
11        self,
12        cost_per_input_token: float = 0.00015,
13        cost_per_output_token: float = 0.0006,
14    ):
15        self.cost_input = cost_per_input_token
16        self.cost_output = cost_per_output_token
17
18    def traced_llm_call(self, model: str, messages: list[dict], llm_client) -> dict:
19        with tracer.start_as_current_span("gen_ai.chat") as span:
20            span.set_attribute("gen_ai.system", "openai")
21            span.set_attribute("gen_ai.request.model", model)
22            start = time.perf_counter()
23            response = llm_client.chat.completions.create(
24                model=model, messages=messages,
25            )
26            elapsed_ms = (time.perf_counter() - start) * 1000
27            usage = response.usage
28            span.set_attribute("gen_ai.usage.input_tokens", usage.prompt_tokens)
29            span.set_attribute("gen_ai.usage.output_tokens", usage.completion_tokens)
30            span.set_attribute(
31                "gen_ai.response.finish_reason",
32                response.choices[0].finish_reason,
33            )
34            cost = (
35                usage.prompt_tokens * self.cost_input
36                + usage.completion_tokens * self.cost_output
37            )
38            span.set_attribute("gen_ai.cost_usd", cost)
39            span.set_attribute("gen_ai.latency_ms", elapsed_ms)
40            return {
41                "content": response.choices[0].message.content,
42                "tokens": usage.prompt_tokens + usage.completion_tokens,
43                "cost_usd": cost,
44                "latency_ms": elapsed_ms,
45            }

The tracer is initialized with the 1.37.0 schema URL — that signal alone lets compliant backends auto-map the gen_ai.* attributes. Every span carries gen_ai.system, gen_ai.request.model, both token counts, gen_ai.response.finish_reason, and the derived gen_ai.cost_usd / gen_ai.latency_ms.

You'll know it works when Phoenix shows a gen_ai.chat span for each LLM call with non-empty gen_ai.usage.input_tokens and gen_ai.usage.output_tokens attributes, and Prometheus exposes the same data via the Collector's scrape endpoint on port 8889.

Do's and Don'ts

Now that you have worked through the implementation, the practices below separate a durable approach from a fragile one.

Do's

1. ✓Do set the schema URL on every tracer — without schema_url="https://opentelemetry.io/schemas/1.37.0", backends cannot reliably auto-map gen_ai.* attributes.
2. ✓Do compute gen_ai.cost_usd at span-emit time — historical traces are immutable; you can't backfill cost if you only stored tokens.
3. ✓Do route through a Collector, not direct from app to backend — the Collector is the swap point that keeps backend choice out of application code.

Don'ts

1. ✗Don't invent custom attribute names — tokens_in or model_name break every dashboard built on gen_ai.usage.input_tokens and gen_ai.request.model.
2. ✗Don't disable batching to "see spans faster" — unbatched OTLP exports flood the Collector and degrade pipeline throughput; tail Collector logs in dev instead.
3. ✗Don't bypass the semantic conventions for "just this one" custom attribute — every exception accrues into vendor lock-in.