Why voice agents are hard to test
A voice agent is a pipeline: speech recognition, a language model, tools, and speech synthesis, over a phone line or WebRTC connection. A failure can come from any stage, and many only appear in the conversation itself. The agent hears the wrong word, calls a tool with the wrong argument, answers a question the knowledge base never covered, or talks over the caller.
Two things make this harder than testing an ordinary API. The input is open-ended speech, so you cannot enumerate it. And the output can be wrong while sounding completely fluent, so passing a latency check or a happy-path script tells you very little about what real callers experienced.
The four layers of testing
Teams that ship reliable agents usually run all four layers, because each catches a different class of failure.
- 1. Text-mode tests
- Drive the agent with text instead of audio and assert on messages, tool calls, and handoffs turn by turn. They are fast, cheap, and deterministic enough to run on every change. They cannot catch anything audio-related: recognition errors, interruptions, or pauses.
- 2. Simulated callers
- A second model plays the caller, following a persona and a goal, and the conversation is scored against success criteria. This finds flow problems and edge cases at scale before launch. The simulated caller is still a model, so it behaves more politely and predictably than people do.
- 3. Real calls before launch
- Place real phone calls, with real accents, background noise and carrier audio, to a staging agent. This catches telephony and speech recognition problems that text and simulation skip. It is slow and manual, so keep it for the scenarios that matter most.
- 4. Production QA
- Grade every real call after launch, and group calls that failed for the same reason. This is the only layer that shows what your actual callers do, which failures are most common, and whether the last prompt change made things worse.
What to check on every call
Whichever layer you use, the same questions apply. Define them once and reuse them across tests and production.
- Did the caller get what they came for?
- Task completion is the headline metric: the appointment was booked, the question answered, the right person reached.
- Did the tools work?
- Every tool call should have run with correct arguments. The dangerous case is a tool that failed while the agent carried on as if it had succeeded.
- Was the answer true?
- Check prices, policies, hours and any other fact against the source of truth. Fluent, confident wrong answers are the hardest failure to notice.
- Did it follow the rules?
- Required disclosures, identity checks, and topics the agent must refuse or escalate.
- Did the conversation feel right?
- Interruptions, long pauses, repeated questions, and loops. These show up in the audio and the transcript, not in an error log.
- Did it hand off correctly?
- When the caller asks for a human, or the agent cannot help, the transfer should happen with context intact.
How to build a test set that stays useful
Start from real calls. Pull the transcripts of calls that went wrong, turn each into a scenario with an expected outcome, and add it to the regression suite. A test set built only from imagined scenarios drifts away from what callers do.
Keep scenarios small and specific: one goal, one expected outcome. Cover the happy path, then the variations that break it: the caller who changes their mind, gives the answer out of order, has a strong accent, or talks over the agent. Rerun the whole set on every prompt, model, voice, or tool change, because a fix in one place regularly breaks another.
Common mistakes
- Testing only the happy path
- Scripts that follow the ideal conversation pass while real callers fall off it.
- Trusting latency and completion dashboards
- A call can be fast, completed, and wrong. Metrics that don't read the conversation miss it.
- Sampling a few calls by hand
- Listening to a handful of calls finds the loudest problems, not the most frequent ones. Grade every call so frequency drives priority.
- Treating launch as the finish line
- Callers change, upstream models change, and every prompt edit is a deploy. Production is where regressions appear.
What each platform gives you
Most platforms ship some pre-launch testing. Vapi has Evals and test suites; Retell has simulation and batch testing; LiveKit Agents supports unit tests with pytest or Vitest, a CLI debugger, and agent simulations; Pipecat has built-in Evals with text-mode runs and an LLM judge. Features change often, so check each vendor's documentation.
These cover layers one and two. Layer four, finding which failures your real callers hit and how often, is what Zelto does: it grades every production call, groups calls that failed for the same reason into a Finding ranked by volume, and drafts the fix.