Invoke Provider-Native Extended Thinking and Structured Output — lab audio overview

Host: Let me start with something I ran into last week. A team ships an internal support assistant. It works. Then two requests land the same day: legal wants to see the model's reasoning on escalation decisions, and the data team wants every response as JSON they can drop straight into a table. Suddenly the "just call the model and print the string" approach falls apart. Expert: That's exactly the pivot point. And it's worth being precise about why. Both of those requests are asking the provider to do something structurally different, not just prompt it differently. Extended thinking changes the shape of the response — you get back multiple content blocks with different types. Structured output changes your obligations on the parsing side. Neither one is a client-side concern like retries or connection pooling. Host: Which is the gap this lab is closing, right? Because there's a companion lab already. Expert: Right, obj_11_7_lab_1, the Async LLM Client Patterns lab. That one builds real plumbing — async calls, retry logic, the client-side scaffolding. But it never actually reaches for a provider-native capability. You end up with an excellent pipe that only ever carries plain text. This lab makes you exercise the capabilities end to end. Host: So what are we building? Expert: A class called ProviderFeatureClient. It wraps the Anthropic SDK, and it exposes four things you'll implement: the constructor, invoke_with_thinking, parse_thinking_response, and call_with_structured_output. Two of those are the thinking path, one is the structured-output path, and the constructor is where the transport gets decided. Host: Start with the constructor, because there's a design choice baked in there that I want you to defend. Expert: Happy to. The signature is `__init__(self, create_message: Callable[..., Any] = None)`. You're injecting the message-creation callable. Inside, you set `self.api_key` to the literal string "student-token" and read `self.base_url` from the ANTHROPIC_PROXY_URL environment variable, defaulting to `http://anthropic-proxy:8080`. Then the branch: if `create_message` is None, you build a real `Anthropic(api_key=self.api_key, base_url=self.base_url)` and set `self.create_message = self._sdk_client.messages.create`. Otherwise you take the injected callable as-is. Host: So it's dependency injection, but the default is the real thing. Expert: And that ordering matters. Notice what the answer code does not do — there's no try/except around the SDK import that quietly falls back to a stub. The comment in the reference is explicit about it: no silent fallback. If the client can't be built, the lab fails loudly rather than pretending to exercise the provider. That's the whole point of this exercise. A lab that silently degrades to a fake is a lab that teaches you nothing about the provider. Host: Let's do extended thinking. What is it actually doing on the wire? Expert: You pass a `thinking` parameter into `messages.create`. In `invoke_with_thinking` you build it as a dict: `{"type": "enabled", "budget_tokens": int(budget_tokens)}`. The method signature is `invoke_with_thinking(self, prompt: str, budget_tokens: int = 4000)`. That budget is a token allowance for the model's internal reasoning — it's telling the provider how much room it has to think before answering. Host: And there's a subtlety in the max_tokens line that I'd bet people get wrong. Expert: They do. It's `max_tokens=int(budget_tokens) + 1024`. The thinking tokens come out of your output budget. If you set max_tokens to something at or below your thinking budget, you've allocated the model zero room to actually answer — it thinks itself right up to the ceiling and stops. The `+ 1024` is headroom for the final response. That's a real production failure mode, and it produces a confusing symptom: an empty answer with a perfectly good thinking trace. Host: There's a data model for this config too. Expert: There's a `ThinkingConfig` dataclass in models.py with `type: str = "enabled"` and `budget_tokens: int = 4000`. It documents the shape. The reference implementation constructs the dict inline in `invoke_with_thinking`, which is fine — but the dataclass is there so the contract is legible. Host: You also pin a specific model for the thinking path. Expert: Two constants. `THINKING_MODEL = "claude-sonnet-4-20250514"` and `DEFAULT_MODEL = "claude-haiku-4-5-20251001"`. `invoke_with_thinking` uses THINKING_MODEL; the structured-output path uses DEFAULT_MODEL. And a standing rule for this lab: current-generation identifiers only. No deprecated model strings. Host: Okay, the call comes back. Now parse_thinking_response. Expert: This is where the response-shape change becomes concrete. The signature is `parse_thinking_response(self, response: Any) -> ExtendedThinkingResult`. You iterate `getattr(response, "content", []) or []` and check each block's `type`. If the type is `"thinking"`, you pull `block.thinking` and append it to `thinking_parts`. If the type is `"text"`, you pull `block.text` and append to `text_parts`. Then you join each list with newlines, strip, and return an `ExtendedThinkingResult` with `thinking`, `answer`, and `model` from `getattr(response, "model", "")`. Host: Why the defensive getattr everywhere instead of just `response.content`? Expert: Because you're parsing a structure you didn't build. Different block types carry different attributes — a thinking block has `.thinking`, a text block has `.text`. Reaching for the wrong one raises. The `getattr` with a default keeps the parser from exploding on a block type you didn't anticipate, and the `or []` handles a content field that's present but null. Parsers over provider responses should degrade, not crash. Host: And ExtendedThinkingResult is the dataclass that separates the two. Expert: `thinking: str = ""`, `answer: str = ""`, `model: str = ""`. Three fields, all defaulted. The reason it exists at all is the legal request from your opening story — once thinking and answer are separate typed fields, you can log one, display the other, and audit both independently. If you return a concatenated string, that separation is gone and you can't get it back. Host: Let's switch to structured output. The failure mode here is different. Expert: Completely different. Extended thinking is a first-class provider parameter — you flip it on and the response shape changes. Structured JSON output in this lab is instruction-plus-validation. You steer with a system prompt, and then you defend on the parsing side, because a model asked for JSON can still hand you something that isn't. Host: What's the system prompt? Expert: `STRUCTURED_JSON_SYSTEM`, a module constant. It says: you are a weather reporter, reply with a single valid JSON object matching the schema `{"city": string, "temperature_celsius": number, "conditions": string}`, and then — this part earns its keep — "Return ONLY the JSON object, no prose, no code fences." Host: And you still handle code fences in the parser. Expert: You do. Because the instruction reduces the fence rate; it doesn't eliminate it. `call_with_structured_output(self, prompt: str) -> WeatherReport` calls `create_message` with DEFAULT_MODEL, `max_tokens=512`, the system prompt, and the user message. Then it walks the content blocks looking for the first `"text"` block and grabs its text as `raw`. If `raw` is empty, it raises `ValueError("No text block in response")`. Host: Then the cleanup. Expert: Strip whitespace. If the result starts with three backticks, strip the backticks and strip again — and then if what's left starts with "json" case-insensitively, slice off four characters and strip. That handles the very common ` ```json ` opener. Then `json.loads(cleaned)` inside a try, and on `json.JSONDecodeError` you raise `ValueError(f"Malformed JSON: {e}")` chained with `from e`. Host: Two ValueErrors so far. Is that deliberate? Expert: It is. The caller gets one exception type for "the provider gave me something I can't use," with a message that says which way it failed. And `from e` preserves the original traceback so you can still see the underlying decode error when you're debugging. Host: There's a third raise. Expert: The schema check. Parsing valid JSON only tells you it's syntactically well-formed — it says nothing about whether the fields you need are there or the right type. So you construct `WeatherReport(city=str(data["city"]), temperature_celsius=float(data["temperature_celsius"]), conditions=str(data["conditions"]))` inside a try that catches `KeyError`, `TypeError`, and `ValueError`, and re-raises as `ValueError(f"JSON did not match WeatherReport schema: {e}")`. Host: Walk me through why each of those three exception types can fire. Expert: `KeyError` if the model omitted a field. `TypeError` if it gave you something un-coercible — a list where a string belongs. `ValueError` from `float()` if `temperature_celsius` came back as the string "warm". Catching all three and re-raising as one schema error means the caller has a single thing to handle. And the coercion is load-bearing: `float(data["temperature_celsius"])` means a JSON `22` and a JSON `"22"` both land as a Python float. Your dataclass says `temperature_celsius: float` and that's what it gets. Host: Now, grading. You said it makes real calls. Expert: Both `run_lab.sh` and `run_tests.sh` hit the real Anthropic provider through ANTHROPIC_PROXY_URL. No fakes on either path. The `main()` function is what run_lab.sh executes: it builds a bare `ProviderFeatureClient()` — no injected callable, so the real SDK — calls `invoke_with_thinking("What is 12 + 30? Think through it step by step.", budget_tokens=2048)`, parses it, prints the first two hundred characters of thinking and of answer, then calls `call_with_structured_output` asking for Paris weather as JSON, and prints the city, temperature, and conditions off the returned `WeatherReport`. Host: If the tests make real calls, how does that not get expensive? Expert: Session-scoped fixtures. The suite makes exactly one extended-thinking call and one structured-output call, and every test reuses those two results. That bounds cost and latency regardless of how many assertions you write. Host: And what do the assertions actually check, given the responses aren't deterministic? Expert: Structural properties, never canned strings. The extended-thinking response must yield a non-empty thinking trace. The answer to "What is 12 + 30" must contain "42" — that's an assertion about correctness that survives any phrasing the model picks. The structured reply must parse into a `WeatherReport` with string `city`, string `conditions`, and float `temperature_celsius`. There's also a check that the thinking budget is actually passed into the create call, and a malformed-JSON case confirming `call_with_structured_output` raises rather than returning junk. Host: So the grader is testing the contract, not the wording. Expert: Which is how you should test any LLM-backed code. Assert on shape, type, and the invariant you actually care about. Assert on exact prose and your suite is a flake generator. Host: Let's land it. What's on the learner's plate? Expert: Four TODOs, bodies only — every signature is already written and fully typed, which is the convention for these labs. TODO one: the constructor, setting api_key and base_url and branching between the injected callable and a real Anthropic client with no silent fallback. TODO two: `invoke_with_thinking`, building the thinking dict with budget_tokens and calling create with THINKING_MODEL and max_tokens set to budget plus 1024. TODO three: `parse_thinking_response`, splitting content blocks into thinking versus text and returning an `ExtendedThinkingResult`. TODO four: `call_with_structured_output`, extracting the text block, stripping fences, decoding JSON, and validating into a `WeatherReport` — raising `ValueError` on every failure path. Host: And when it runs, it's really talking to the provider. Expert: Every time. You'll see an actual thinking trace scroll past, and a real JSON object become a typed Python object. That's the difference between having a client and having exercised one.