The first time I put an LLM into a product, the model did its job well and I did mine badly: I asked for JSON in the prompt, prayed, and then wrote regex to rescue the object from between apologies and markdown blocks. It worked 90% of the time, which in production is another way of saying it failed. The fix wasn't a smarter prompt — it was treating the model's output as a typed interface: structured outputs with function calling and Zod validation. This article is how I made that shift and what I got for it.
The real problem isn't the model, it's the edge
An LLM inside a product doesn't live alone: its output feeds a function, a query, or a call to another API. That contact point between natural language and typed code is the fragile edge. If the model returns free text, the edge fills up with defensive parsing: find the first {, count braces, strip the ```json, catch the JSON.parse that blows up. Every one of those lines is debt that breaks the moment the model decides to get conversational.
For a while I tried to solve it with prompt engineering. "Respond with valid JSON only, no explanations." It helps, but it guarantees nothing: the prompt is a plea, not a contract. The lesson that took me a while to accept is that reliability isn't requested, it's enforced at the API layer.
Function calling as a contract, not a trick
Modern models accept a tool or response schema and commit to returning something that fits it. That changes the game: instead of describing the format in prose, I declare it as structure and the model fills in the blanks. I stop parsing intentions and start receiving data.
I model it like this: I define the shape I need, pass it as a response schema, and the rest of my code assumes it will receive exactly that. The prompt goes back to talking about the task — what to extract, how to prioritize — and not about commas and quotes.
import { z } from 'zod';
const CandidateExtraction = z.object({
seniority: z.enum(['junior', 'mid', 'senior', 'lead']),
primary_stack: z.array(z.string()).max(8),
years_experience: z.number().int().min(0).max(50),
location: z.string(),
open_to_remote: z.boolean(),
});
type CandidateExtraction = z.infer<typeof CandidateExtraction>;
That schema is the source of truth. From it comes the TypeScript type, from it comes the JSON schema I hand the model, and against it I validate the response. One definition, three uses.
Zod: the net that catches what the model doesn't guarantee
Here's the nuance a lot of people skip: the API accepting a schema doesn't mean the result is correct in the sense I care about. The model can return structurally valid JSON that's semantically absurd: a years_experience of 200, a stack with fifteen duplicate entries, an enum that never existed. The shape is right; the content isn't.
That's why I always validate at the edge with Zod, even when using structured outputs. It isn't belt-and-suspenders paranoia: it's that the API's guarantee and my domain's guarantee are different things. Zod lets me express the second one — ranges, max lengths, closed enums — and turns a dubious response into an error I can handle before it pollutes the database.
const raw = await callModelWithSchema(prompt, CandidateExtraction);
const parsed = CandidateExtraction.safeParse(raw);
if (!parsed.success) {
// I don't propagate garbage: I retry with the error as context,
// or fall back to a manual flow. I never write without validating.
return handleInvalid(parsed.error);
}
await saveCandidate(parsed.data); // parsed.data is already the correct type
💡 An LLM in a product isn't a source of truth, it's a source of proposals. Validation is what turns a proposal into data you trust.
Retries with the error as context
When validation fails, the instinct is to retry with the same prompt. You waste tokens repeating the same mistake. What actually works is re-injecting the failure: I hand the model back what I expected and what it gave me, and ask it to fix it. In practice, 90% of validation failures resolve in a single informed retry, because they're almost always dumb slips — a missing field, a misspelled enum — that the model corrects immediately once you point at the exact spot.
I cap the retries, of course. If after two attempts it still doesn't fit, the problem isn't the model having a bad day: it's that the task is poorly framed or the schema asks for something the input doesn't contain. There, the loud failure is a gift, because it forces me to fix the cause instead of hiding it.
The trade-off I accepted
Structured outputs isn't free. Boxing the model into a schema reduces its flexibility: for genuinely open-ended tasks, where I don't know the shape of the answer ahead of time, the schema gets in the way more than it helps. And adding Zod plus retries is code and latency I didn't have before.
I chose structured outputs anyway because in a product I almost never want creativity at the edge: I want determinism. I'd rather pay in rigidity and a few lines of validation in exchange for the rest of my system being able to treat the LLM's output the way it treats any other typed API. That's the real goal: the model stops being a special exception everyone tiptoes around and becomes just another component, with its contract and its error handling, like any other piece I already trust.
