How to stop a voice agent from bluffing about what it did
Ask a voice agent to book a time and it will say "done". Sometimes nothing happened. The calendar was never called, or it was called and said no, and the agent said "booked" anyway, because that is the sentence a receptionist says next. We call this bluffing. It is the single most damaging thing a phone agent can do, because the caller hangs up happy and finds out on the day. This post is about why it happens and the three things that stop it.
What a bluff sounds like, and why it happens
"I've booked you in for Tuesday at ten." "I've sent you a text with the address." "I've passed that on to the team." "I've cancelled your order." Each one is a claim about the world. Each one is easy to say and hard to check from inside a conversation.
A language model produces the most likely next sentence. It has read a great many transcripts in which a receptionist says "booked" after the caller picks a time, so that is what it produces. Doing the booking is a separate step: the model has to call a tool, wait for the answer and only then speak. Under time pressure on a live call, with a caller breathing into the silence, the model is pulled towards speaking first. Speech-to-speech models, which are built for fast natural rhythm, feel this pull the most. We compared the engines in Pipeline, speech-to-speech or hybrid?
So the bluff is not a bug you fix once. It is a tendency you have to build against.
Rule one: the tool speaks first
The agent never says "booked" until the calendar has said so. That is a rule about the order of events, and it is the one that matters most.
In practice it means three things. The confirmation sentence is written from the tool's answer, not from the model's memory of the request: if the calendar returns Tuesday at 10:15 because ten was taken, the agent says 10:15. If the tool fails, the agent says so, in plain words, and offers a callback or a message instead of a guess. And if there is no tool connected at all, the agent has no business saying "booked"; it takes the details and says a person will confirm.
The same rule covers texts, notes, transfers and cancellations. Anything the caller could later rely on is an action, an action is a tool, and the tool's reply is the only thing that earns the word "done". Our guide to connecting a voice agent to your calendar and CRM shows what those tools look like.
Rule two: a second model reads every reply
Rule one depends on the model doing the right thing. Rule two assumes it sometimes will not.
In NordTell's Hybrid engine, a realtime model handles the conversation and a second model checks each reply that did not call a tool. It asks three questions. Did the reply announce a transfer? Then the transfer is actually made. Did it say goodbye? Then the call is ended properly rather than left hanging. Did it claim to have done something that needs a tool, like saving a note or sending a text? Then the tool is run, and if it fails, the agent says so instead of letting the claim stand.
The same check is what keeps the agent on script: a reply that wanders off the instructions is read by a model whose only job is to read and compare, not to keep the caller happy. Two models, two jobs. It uses more than one model, so it is not the cheapest engine, which is why the engine is a choice per agent. It is also the reason Hybrid is the default we recommend.
Rule three: instructions that make bluffing hard
The model's instructions are the third line of defence, and the one you control entirely.
- "Never confirm a booking, cancellation or message unless the tool has returned a success."
- "If a tool fails, tell the caller it did not go through and offer a callback."
- "Do not promise that someone will call back unless you have saved a message with their number."
- "If you do not know, say so. Do not guess opening hours, prices or availability."
- "Read every time, date and number back to the caller before using it."
Short, blunt sentences work better than paragraphs. The model reads them on every turn. We collected more of them in writing instructions for a phone agent.
Test it with a broken calendar
The only way to know your agent does not bluff is to make bluffing easy and see what it does. Disconnect the calendar tool, or point it at an address that returns an error. Call the agent from the browser and ask to book Tuesday at ten. A good agent says the booking did not go through and offers to take your number. A bluffing agent says "booked". Do the same with the text tool and the transfer target. Then reconnect everything and ask for a time you know is taken, to check that the agent reads back the time the calendar offered and not the one you asked for.
Do it again after every change to the instructions or the engine. Ten minutes, once a week, is cheap compared with one caller who turns up to an appointment that does not exist. The full routine is in testing a voice agent before launch.
Where NordTell fits
NordTell's agents never say "booked" before the calendar tool has confirmed, and the Hybrid engine adds a second model that reads every reply, catches claims of actions that did not happen, and turns "I'll put you through" into an actual transfer. The engine is a setting per agent, so you can compare it with Pipeline and speech-to-speech on the same instructions. The broken-calendar test above works from the browser on a free account.