You can wire up a voice agent that answers a phone call in about a day. Speech-to-text in, model in the middle, text-to-speech out. It will work, and the first call will feel like magic.
The gap between that and something you would let talk to your customers unsupervised is four to six weeks, and almost none of it is model work.
Latency is the whole experience
People forgive a voice agent for being a bit robotic. They do not forgive it for pausing.
Human conversation runs on turn gaps of roughly two hundred milliseconds. Push past about eight hundred and the caller starts talking over the agent, or assumes the line dropped. Your budget has to cover speech recognition, the model call, any tool calls it makes, and speech synthesis — end to end, on a bad mobile connection.
That budget shapes every other decision. It is why you stream partial transcripts instead of waiting for the caller to finish. It is why you start synthesising the first sentence before the model has produced the last one. It is why a database lookup that takes 1.2 seconds needs a cache or a filler phrase, and why the smaller, faster model often beats the smarter one for the conversational turns.
Optimising latency after the fact means rebuilding the pipeline. Design for it on day one.
Interruption handling is not optional
Real callers interrupt. They start answering before the question ends, they correct themselves mid-sentence, they say “no wait, sorry” and change the subject.
An agent that cannot be interrupted feels broken in a way that is hard to articulate but immediately obvious. Handling it means detecting speech while you are still talking, stopping synthesis mid-word, discarding the rest of the planned response, and working out what the caller actually meant. Getting this smooth is a meaningful chunk of the build.
Write down what it must never do
Before a line of code: what is the agent categorically not allowed to say or do?
Not permitted to quote a price outside the published range. Not permitted to promise a delivery date. Not permitted to confirm a cancellation without a reference number. Not permitted to give anything resembling medical, legal, or financial advice.
These become hard constraints in the system — checks around tool calls, validation on what leaves the agent — not polite requests in a prompt. A prompt is a suggestion; under enough conversational pressure, models take suggestions loosely. Anything with a legal or financial consequence needs a real guard.
Design the hand-off before the happy path
The most important flow in a voice agent is the one where it gives up.
Decide in advance what triggers a transfer: an explicit request for a human, repeated recognition failures, a detected complaint, a topic outside scope, or the caller simply sounding frustrated. Then decide what transfers with them — the caller should never have to repeat what they just spent two minutes explaining. Pass a transcript and a summary to whoever picks up.
An agent that hands off cleanly at the right moment is more valuable than one that handles a slightly wider range of calls badly.
Integrations are the unglamorous majority
The agent is only useful if it can actually do something: check an order, book a slot, create a ticket, look up an account.
Every one of those is an integration with real-world problems. The calendar API is rate-limited. The CRM has three records for the same person. The booking system needs a field the caller will never know. Deduplicating a customer record over the phone, when you have a name that speech recognition heard three different ways, is its own small project.
Budget for this properly. It is usually more work than the conversational layer.
Start with one call type
The temptation is to launch an agent that handles everything. Don’t.
Pick the highest-volume, lowest-risk call type — appointment scheduling, order status, opening hours and routing. Run it alongside your team, not instead of them. Listen to the recordings, all of them, for the first couple of weeks. You will find failure modes no test script would have produced, because real callers do things nobody writing tests imagines.
Widen scope only once the narrow case genuinely holds.
Measure it like a system, not a demo
Track containment rate, average handle time, transfer rate and why, recognition error rate, cost per call, and caller sentiment where you can get it. Set a target for each before launch so you have something to judge against.
Then keep watching. Voice models get deprecated. Accents your training set under-represented show up in a new market. A prompt change made for one call type quietly degrades another. The agent that worked in month one needs someone looking at the numbers in month six.
The honest summary
Four to six weeks for a first production call type, and the model is maybe a fifth of it. The rest is latency engineering, integration work, hand-off design, and the discipline to launch narrow.
Anyone quoting you a week has built the demo, not the system.