Structured Output and Tool-Call Reliability for Agents

Updated 2026-09-06 · guide · agents, technical, how-to

Structured output is the contract between your agent and your code: instead of the model returning prose that you have to parse, it returns data in a shape you defined — a JSON object with known fields, typed values, and no surprises. Tool-call reliability is the other half of the same problem: making sure that when the agent asks to call a tool, the call is well-formed, executed correctly, and fails gracefully when something goes wrong. Together they're the difference between an agent that feels like a demo and one that feels like software. This guide covers how to pin down structured output, how to handle tool-call failures without cascading chaos, and how to test that the whole thing holds up.

Anyone who's built an agent for more than a week has felt this: the model returns valid-sounding but slightly wrong JSON — a missing field, an extra key, a value of the wrong type — and your parser throws, the run dies, and the user sees an error for a problem that was entirely preventable. Or the model decides to call a tool with an argument that's almost right, the tool returns an error, and the agent retries the same mistake three times before giving up. These aren't exotic edge cases; they're the normal failure modes of non-deterministic models, and they're exactly what structured output and principled retry handling exist to contain.

The stakes get higher when the pattern scales. A single agent calling one tool is easy to babysit; an agent fleet calling dozens of tools across multi-step tasks — the shape of production AI in 2026 — will hit malformed output and tool errors constantly, on every run. The teams that ship reliable agents aren't luckier than the ones that don't; they've built the load-bearing structures that turn model chaos into bounded, expected, handleable failure. This guide is that structure.

Why "just parse what it returns" fails

The naive approach — ask the model for JSON, then json.loads the response and hope — fails in predictable, repeatable ways:

1. Prose-wrapped JSON. The model answers "Sure! Here's the JSON:" followed by a code block, and your parser chokes on the preamble. Models are chatty by default; they add commentary that isn't part of the data.
2. Schema drift. You asked for {name, email, plan}, and the model returns {Name, email_address, plan_type}. Keys shift, fields go missing, extra keys appear — sometimes because of a prompt edit you forgot triggered it, sometimes just because.
3. Type ambiguity. "5" comes back as a string instead of an integer; null appears where you needed a real value; an array comes back as a comma-separated string. Parsing libraries are lenient about some of this and strict about others, and the failure is inconsistent.
4. Silent truncation. Long outputs get cut off mid-JSON, and the model doesn't always notice that what it returned isn't valid. The result is a parse error that only shows up in your error logs hours later.

The fix isn't "parse more carefully" — it's structured output: the model API-level constraint that forces the response into a schema before it leaves the model. When you enable it, the model literally cannot return prose-wrapped or malformed JSON; it returns data conforming to your schema or fails cleanly. This one feature removes the entire class of parse errors.

Structured output in practice

Modern model APIs expose structured output natively, usually through a "response format" or "structured output" parameter paired with a JSON schema. The practical moves:

Use the native structured-output mode, not prompt-only JSON. Prompting "please return JSON" is a hint, not a guarantee. The API-level schema constraint is the guarantee — the model is forced into producing conformant output, and it sidesteps the prose-wrapping problem automatically. If your model provider supports it, this is the default for any data your code has to consume.

Design schemas the model can fill. Keep schemas small and specific. {company, role, years} is fillable; {everything_you_know_about_this_user} is not. Use clear field names, proper types, and sensible defaults for optional fields. The same discipline applies whether the output is consumed by your app or by another agent — the output-spec advice in the prompt-engineering guide is the human-readable version of the same contract.

Allow for refusal and partial data. Real-world inputs are messy: a resume with no email, a log line with no timestamp. If your schema requires email as a non-nullable string and the model truly can't find one, structured output forces either a fabricated value or a whole-response failure. The better design: make naturally-optional fields nullable, and give the model an explicit "not found" convention (null, or a sentinel) so it can represent reality instead of inventing it. The guardrails guide makes the same point structurally: when the model can't comply, you want it to say so — not to paper over it with fake conformance.

Validate on the way in, not just on the way out. Structured output guarantees the format is right; it doesn't guarantee the content is right. A model can return a perfectly valid JSON object with an empty name field or a phone number that's clearly a hallucination. Validate semantics at the boundary — required fields present, values within expected ranges, references resolvable — before your code acts on the data. This is the validation layer the guardrails guide names for tool and retrieved content, applied to your own model output.

Tool-call failures: the anatomy of a retry

Tools are where agents touch the real world, and the real world fails. The reliability pattern has three phases:

1. Validate the call before executing. The model chose the tool and arguments; before your code runs anything, check the arguments against the tool's schema — types, required fields, values within range. Most malformed calls are caught here, cheaply, without the tool ever being hit. The MCP server guide covers the schema side from the server's perspective: a well-declared tool makes the agent's first call correct more often, which is the cheapest failure you can prevent.

2. Retry with backoff, but bounded. Tools fail for reasons that are sometimes transient (a network blip, a rate limit) and sometimes permanent (a bad argument, an unauthorized endpoint). The pattern: a bounded number of retries — 2–3 is usually right — with exponential backoff and a small amount of jitter. Never retry forever: an unbounded retry loop is how agents burn tokens and money on a failure that was never going to succeed. The token-cost guide and the observability guide both flag retry storms as a top cost and debugging hazard; bounding retries is the fix on both fronts.

3. Escalate, don't just fail. After the retries are exhausted, the agent should change its approach, not just report "tool failed." Options: try a different tool that does the same job, ask the user a clarifying question, or fail with a clear, structured error that says what was attempted and why. This is where a good agent feels intelligent — not because it never fails, but because it fails usefully. The agent-loop guide covers the reflection step that makes "change approach" possible; a loop that reflects can turn one dead-end into a pivot.

The retry policy, spelled out

A concrete retry policy you can implement today:

Testing reliability (it's not magic)

• Classify failures. Transient (timeout, 429/5xx, connection reset) → retryable. Permanent (400 invalid argument, 401/403 auth, validation error) → not retryable, escalate immediately. Never retry a permanent error; it wastes the same resources on the same guaranteed failure.
• Cap the retries. Max 3 attempts total (initial + 2 retries). Each retry multiplies wait: e.g., 1s, 3s, with jitter. Track the attempt count in the observability layer so you can see retry storms in your agent monitoring dashboards, not just in the logs.
• Give the model feedback. When a tool call fails, feed the structured error back to the model and let it self-correct once or twice — models are surprisingly good at fixing their own tool calls when told "that argument was invalid because X". This is legitimate because the failure data is part of the agent loop, not a hidden mechanism.
• Set a global budget. A per-run token or time budget that hard-stops runaway loops. This is the "watchdog" pattern from the safety guide operating in the reliability layer instead of the security layer.
• Model context as a retry constraint. Long context can silently alter behavior — the same tool call tends to behave differently when the model is processing 80k tokens versus 5k. The context engineering guide is where that relationship gets managed; keep it in mind when a flaky tool call only flakes on long runs.

Schema validation belongs at the deployed tool boundary, not only in tests. See agent runtime deployment and hosting for tool workers, retries, and sandboxing.

All of the above is only as good as your verification. Some of the structure overlaps with testing agent skills, but reliability has its own checklist:

Common mistakes

Bottom line

• Test the failure paths, not just the happy path. Feed the agent a tool that returns a 500, a tool that times out, a malformed argument — and assert that it retries the right number of times, escalates correctly, and never loops forever.
• Test the schema edge cases. Empty strings, missing optional fields, oversized values, unexpected nulls. Structured output handles most of these automatically; your validation layer handles the rest. Prove both.
• Test across models. Structured output behaves differently across providers and model versions — one model conforms perfectly, another drifts on field naming. If you support multiple models, test the contract on each. This is the same "non-determinism requires deliberate testing" argument as the testing guide, applied to the output boundary.
• Soak test for drift. Run the same pipeline repeatedly (10–50 times) and watch for rare malformed outputs or occasional tool-call weirdness. A 2% flakiness rate is invisible in a single demo and categorically fatal in production.
• Relying on prompt-only JSON. Without API-level structured output, you're betting on a hint. Use the native constraint; reserve prompt-only JSON for cases where the provider doesn't support it and accept the flakiness.
• Retrying permanent errors. A 400 from a bad argument will 400 again. Classify failures and skip retry for permanent ones — escalation is the move.
• Retrying forever. Unbounded retry loops burn tokens and money on guaranteed-failure calls. Bound them, back off, and hard-stop with a budget.
• Not validating semantics. Structured output validates format, not truth. An empty name or a hallucinated phone number sails through schema validation. Validate content at the boundary.
• Ignoring the failure path in tests. If your test suite only exercises success, you've tested nothing about reliability. The failure paths are the product.
• Silent degradation. An agent that quietly returns a partial result as if it were complete is worse than one that fails loudly with a clear error. Fail structured, or fall back informatively.

Reliability in agents is a structure problem, not a luck problem: enforce structured output at the API level, validate format and content at the boundary, classify tool failures into retryable and permanent, bound your retries with backoff and a budget, and test the failure paths deliberately. Done right, the non-determinism of the model stops being the thing you fear and becomes the thing you've contained. Your single next action: take the one agent flow you trust least, enable structured output on its main call, and write a test for its failure path today.

FAQ