Claude API in Production: A Runtime, Not a String Function, and What It Leaves to You

What is the Claude API, really?

The first version of the AI Readiness Assessment I run parsed Claude's responses with a regular expression. It worked in testing and broke in production, and the fix was always one more line in the system prompt begging the model to hold its format. That is the failure this whole page is organized against, because the mistake underneath it was a category error: I was treating the Claude API as a string function, text in and text out, when it is a structured runtime.

A runtime is not an endpoint. An endpoint returns a value. A runtime gives you primitives and a contract and leaves the orchestration to you. The Claude API ships tool use, structured outputs, prompt caching, extended thinking, and streaming as first-class capability systems, each one solving a class of production problem that a string function solves with a prayer in the prompt. The three-part Claude API series on this site, starting with the diagnosis of the wrapper pattern, covers the wiring of each capability in detail. This page is the layer above the wiring: what the runtime gives you, what it quietly hands back, and where I have watched teams get the boundary wrong.

I run the Assessment as a production Claude API system: a roughly 20-turn conversational agent that classifies a visitor, adapts to them, scores them, and synthesizes a readout, against real users who break every assumption I made on day one. The lived experience that grounds this cornerstone is that system, not a benchmark.

What the runtime gives you

Five capability systems, named compactly. Each gets a paragraph here, because the depth belongs to the implementation post and to the narrower glossary entries, not to this page.

Tool use is a typed contract. You declare what operations exist and the shape of their inputs and outputs; the model decides when to call them and emits a structured request; your code runs the operation and feeds the result back. The boundary is strict, and Anthropic's own documentation draws it in one sentence:

The model never executes anything on its own. Anthropic, "How tool use works"

Structured outputs constrain the response to a schema through constrained decoding, which forces each generated token to fit the schema, so a downstream parser receives valid, typed data instead of prose it has to guess at. The structured outputs documentation is precise about the scope of that guarantee, and the scope matters later. Prompt caching lets the API resume from a fixed prefix instead of reprocessing it, which is the difference between paying for your system prompt once and paying for it every turn; the prompt caching documentation is the reference. Extended thinking gives the model room to reason before it answers, and streaming returns the response incrementally so a user is not staring at a spinner while two thousand tokens generate in silence.

These are not five independent switches. Caching dictates the order of your prompt, which dictates where a tool result is allowed to sit without breaking the cache; extended thinking changes what you pay per turn, which changes when caching is worth setting up at all. The choices compound, so a production integration gets designed around them together rather than reached for one at a time. That is the point of the runtime framing: these are not optional extras bolted onto a text endpoint, they are the surface, and the cost of ignoring them doesn't stay flat. It grows with every conversation turn.

What the API leaves to you

The runtime gives you primitives. It does not give you a working system. Three things stay on your side of the boundary no matter how good the model gets, and owning them is the actual production work.

State

The Messages API, the direct-model-access layer this page is built on, is stateless. Every call carries the entire conversation, because it holds nothing between turns: you send the full history each time, or the model has no memory of it. Anthropic's managed-agents tier can persist session state for you, but that hands off the very orchestration this page is about. Statelessness sounds like a billing detail until you notice what it makes yours. The context window is a budget you spend, and spending more of it is not free of consequence. Anthropic's own context documentation is blunt about the cost:

As token count grows, accuracy and recall degrade, a phenomenon known as context rot. Anthropic, "Context windows"

This isn't a small effect at the edges. Recent work measuring frontier models as they fill their windows finds the degradation persists at high context saturation, not just past some far horizon. There is an honest counter-case: for a single bounded session where pure factual recall is the only goal, paying to push the whole history into a large window can beat a curated context, because nothing gets dropped. That trade stops paying the moment cost, latency, or many concurrent sessions enter the picture, which describes most production. So "send the whole history every turn" is the naive default, and past a point it rots. The work the API leaves you is context assembly: deciding what goes into each call, what gets summarized, what gets dropped, and when a long conversation needs compaction rather than accumulation. In the Assessment, this is why a visitor does not simply get the running transcript replayed at the model. The conversation carries a curated state plus a role-specific module, not an ever-growing log.

Cost

The capabilities you skip are the bill you pay, and the bill is an architecture decision, not a model-choice decision. Prompt caching is the clearest case. A cache read costs roughly one tenth of a base input token while a cache write costs a premium over base input that grows with the cache lifetime you choose, so on an input-heavy agent that resends a large stable prefix every turn, caching is the largest lever on the bill. The scope is worth stating, because caching only discounts input tokens. On an output-heavy workload, where generation dominates the bill, the bigger lever is the model and effort choice in the next section, not the cache. The catch is that the cache is keyed on an exact prefix match: change one token near the front of the prompt and everything after it misses. Naive caching can therefore cost more latency than it saves when dynamic content sits inside the cached block and silently invalidates it every call. The discipline is to put the stable material first and the volatile material last. Even the minimum cacheable prompt length is versioned: it is 1,024 tokens on the current Opus 4.8 but 4,096 on Opus 4.7, so the number you pin depends on the model you pin. The Assessment caches its system prompt and its per-archetype module on a one hour window, behind separate cache breakpoints so a change to one does not invalidate the other.

Verification

Structured outputs guarantee the shape of the answer. They do not guarantee that the answer is right. Constrained decoding will hand you a perfectly typed object containing a confidently wrong value, and the structured outputs documentation is careful to scope the guarantee to schema conformance. The output is not even deterministic: identical inputs at temperature zero can still produce different outputs, because of floating point non-associativity and dynamic batching beneath the API. So verification of non-deterministic output is the third thing the runtime leaves you, and it is the one most often assumed to be the model's job. It isn't. What you own is the loop that checks the model's output against something outside the model: a schema for shape, your own business logic for correctness, and a confidence signal for when to escalate. The agent reliability cornerstone treats that loop in depth; here it is enough to name it as a boundary the API draws and hands to you.

The two knobs you own: model and effort

Routing a production call is two decisions, not one. The first is which model. The second is how hard it thinks, and it is the one that's easy to forget. On the current models, reasoning depth is set through the effort parameter, which affects every token the model spends, the tool calls and the prose as well as the reasoning, not a thinking budget in isolation. Anthropic's own model-selection guidance puts the priority plainly: tuning effort is often a better lever than switching models. The mechanics shifted recently in a way worth pinning, too: on the newest Opus models the manual thinking budget is gone and thinking is adaptive, with the model deciding how much to spend and the thinking tokens billed as output whether or not you display them.

The Assessment runs both knobs across one agent. The turn-zero classification runs on Haiku 4.5 at a low implied effort, because a fast cheap placement is worth more than a slow careful one and the cost of a wrong guess is bounded. Turns 1 through 19 run on Sonnet 4.6, the balance point for real-time dialogue. The final synthesis pass, the one place where reasoning depth earns its latency, runs on Opus 4.7 at the highest effort tier. That is the model and effort matrix applied to a real workload: not one model for everything, but the cheapest model that clears the bar for each turn, at the effort that turn deserves. Getting this wrong compounds, because every turn re-sends the growing context, and a high-effort model on a turn that did not need it pays that tax on every token.

Where the capabilities stop paying

The runtime framing has a failure mode of its own, which is treating every capability as mandatory. They are not. Each one earns its place against a specific problem, and outside that problem it is overhead.

Extended thinking is the clearest example. On a task the model already answers well, more reasoning is not free improvement. It is latency and output-token cost spent to second-guess a correct answer, and a verification loop layered on top of a high-confidence call can make the result worse rather than better by inventing flaws that were not there. The Assessment reflects this directly: it does not run a verification pass on every turn. It verifies at the synthesis stage, where the stakes and the ambiguity are highest, and it lets the cheap classification turns stand on a schema check alone. The advanced-patterns post states the rule I work to: if your use case is simpler, your architecture should be too.

The same logic runs across the board. Tool use is overhead when a single deterministic transform would do the same work in your own code. Prompt caching is overhead when the prefix changes every call, because then you pay the write premium without ever collecting the read discount. Streaming is overhead when nothing consumes the partial output. The boundary is not "use everything." It is "use the capability whose specific failure mode you have, not the one you assume."

What goes wrong in production?

Four failure modes I have either hit or watched others hit. Each has a structural fix, not an editorial one, which is the recurring lesson: you don't prompt your way out of an architecture problem.

The string-function wrapper. You parse the model's prose with a regex, it works in testing, and it breaks the first time the model phrases the answer differently. The fix is structured outputs or tool use, moving the contract from a hopeful prompt to a typed boundary. This is the failure the Assessment started with, and replacing the regex extractor with a tool-use call is what ended it.

Context rot on long conversations. The transcript grows, recall degrades, and the model starts losing instructions from twenty messages back. Curating the context, summarizing and compacting instead of accumulating, is what holds recall together over a long session.

Silent cache invalidation. A token of dynamic content sits near the front of a cached prefix, every call misses the cache and pays the write premium instead of the read discount, and the only symptom is a bill that climbs while latency does not improve. Prefix discipline is the fix: stable first, volatile last, with separate breakpoints for blocks that change independently. The prompt caching reference documents the exact-match rule that makes this bite.

Non-deterministic output breaking a downstream parser. The model returns a well-formed object the parser cannot use, or returns it differently across two identical calls. You close this one by validating semantics in your own code and treating the model's output as something to confirm, not a return value you can trust on sight.

How do the capabilities compose?

A production system uses several at once, and the composition is where the runtime framing pays off. The Assessment is the worked example I know best. A turn arrives. The stateless call carries a curated context plus a cached system prompt and a cached role module. The model reasons under an effort tier chosen for that turn. It calls a tool to emit structured scores, which a schema validates before anything downstream touches them. The response streams to the browser. State, cost, and verification are all being managed in a single call, by me, on top of primitives the API provided but did not orchestrate.

When one stateless agent is no longer enough, the next layer up is subagent orchestration, where the unit of isolation becomes the worker rather than the call. That is a different cornerstone. The boundary worth naming here is that the single-agent runtime is where most production Claude API work lives, and I watch teams reach for multi-agent complexity before they have exhausted what one well-orchestrated call can do. That is the same mistake in a larger frame: reaching for more machinery instead of owning the boundary you already have.

A compact form of the trade-off:

How is this page kept current?

This cornerstone carries the role: cornerstone posture, so the build does not hard-fail it on a fixed window. The cadence is editorial. I revisit it when Anthropic ships a capability that changes the boundary (the move to adaptive thinking is a recent example), when a model release shifts the routing math, or when a new production failure mode earns a place in the list above. The numbers most likely to drift are the ones pinned to a model version, like the minimum cacheable prompt length, so those are stated against a named model rather than as a standing fact.

The cornerstone is the deep companion to the tool use, prompt caching, extended thinking, and context window glossary entries, and it sits alongside the MCP servers and agent reliability cornerstones in the same production stack. The lived evidence is the Assessment, documented in this guide's primary artifact; the external anchors are listed under Sources below with their publication dates. When I scope a Claude API engagement, the work usually starts where this page does, with the boundary between what the runtime gives you and what it leaves you to own.