A demo needs one good answer. A product needs a good answer every time, including for the inputs nobody thought to try. That distance is where most language model projects stall, and the cause is rarely the model. It is everything nobody built around it.
The work after the demo is unglamorous and mostly familiar. Measurement, cost control, error handling, version control. What changes is that the thing at the centre gives you a slightly different answer each time you ask, so some of the usual instincts need adjusting.
Why the demo lies to you
When you test by hand, you supply the input. You know what the product does, you phrase things the way the system expects, and when an answer comes back slightly off you read past it without noticing. Real users do none of that. They paste a wall of text out of an email. They ask something adjacent to what you built. They type two words and expect the system to work out the rest.
The model will produce fluent, confident output for all of it. That is the failure mode specific to this technology. A broken API returns an error and you find out. A language model that has misunderstood the question returns a well-written paragraph, and you find out when a customer acts on it.
Get a test set before you get users
Before you ship, collect real inputs. Ones you did not write. Pull them from support tickets, sales calls, the notes your first users already send you, the search box on your marketing site. A hundred is genuinely useful. Fifty is enough to start.
For each one, write down what a good answer looks like. You do not need the exact wording. You need the facts it must contain and the things it must never say.
Then run the whole set every time you change the prompt, and read the output yourself. By hand, at first. An afternoon spent reading two hundred outputs will teach you more about your product than a month of dashboards. Once you know what failure actually looks like, you can automate the checks that matter: did it cite a real document, did it stay inside the length limit, did it decline when it should have declined.
The reason to do this before launch is simple. Afterwards, every prompt change is a gamble. You fix one complaint and quietly break four things that were working, and with no test set you will not know until someone tells you.
Tokens are a usage bill most founders have not modelled
Every request costs money in proportion to the text going in and the text coming back. Founders model the text coming back. Almost nobody models the text going in, which is where the cost actually lives.
Take a chat feature that sends the whole conversation to the model on every turn, which is how these things get built by default. Turn one sends one message. Turn ten sends ten. Across a conversation the total grows with the square of its length, so one user with a forty message thread costs far more than four users with ten message threads. Attach retrieved documents to each request and the input grows again.
Decide two numbers before launch. The maximum a single request may consume, and the point at which a long conversation gets summarised or reset. Then log cost per user, not just cost per month. A monthly total tells you nothing about whether the heaviest ten percent of your users are profitable.
Latency is a design problem before it is an infrastructure one
A model that takes eight seconds to answer feels broken even when the answer is excellent. The same model, streaming its first words in under a second and still finishing at eight, feels fast. Nothing about the underlying work changed. Streaming changes what the user experiences while waiting, and that turns out to matter more than the total.
This has a real architectural consequence. If you chain three model calls together and show nothing until the last one returns, you have built something that feels slow no matter how good each step is. Show work as it arrives. Where a step genuinely cannot stream, say what is happening while it runs.
When it is wrong in front of a customer
Plan for this on the assumption it happens weekly, because it will. The mechanisms that help are cheap to build early and painful to retrofit during an incident.
- Let the system say it does not know. A product that declines ten percent of questions and is dependable on the rest is worth more than one that always answers.
- Route uncertain cases to a person. Low confidence, unusual input, anything touching money or health.
- Keep the human path permanently available rather than improvising one when something goes wrong.
- Log every input and output with a version stamp, so when a customer complains you can reproduce exactly what they saw.
A system that admits it does not know gets trusted again after its first mistake. A system that is confidently wrong does not.
Your prompts belong in version control
A prompt is the specification for how your product behaves. It gets edited by whoever is closest to the complaint, usually in a hurry, often in a text box in someone's browser. Behaviour changes for every customer and there is no record of who changed what.
Put prompts in the repository. Review changes the way you review any other change. Tag each deployed version and record which version produced which output. When quality drops next quarter and someone asks what changed, that record is the difference between an answer and a shrug.
None of this needs a research team. It needs a test set, a cost model, a fallback, and a git history. Most of it fits inside a week, and doing it before launch costs a fraction of what it costs after.