Measure outcomes first, then the pipeline
Voice agent metrics fall into two groups. Outcome metrics describe whether the call achieved what the caller and the business wanted. Pipeline metrics describe how the speech, model, and tools behaved. Outcomes tell you if there is a problem; pipeline metrics help you find where.
Start with outcomes. A call can be fast, low-latency and completed, and still have given a wrong answer or booked the wrong slot. Metrics that never read the conversation cannot see that.
Outcome metrics
- Task success
- The share of calls where the caller's goal was met: appointment booked, question answered, order changed. Define success per call type, because a support call and an outbound sales call are not the same task.
- Tool-call accuracy
- Whether each tool ran, with correct arguments, and whether the agent handled failures honestly. The key failure is a tool that errored while the agent told the caller it worked.
- Answer accuracy
- Whether stated facts (prices, policies, hours, eligibility) match your source of truth. Track hallucinated answers and answers outside the knowledge base separately.
- Policy and compliance adherence
- Required disclosures, identity verification, and refusals or escalations on restricted topics. Score these as pass or fail per call.
- Escalation correctness
- Whether the agent handed off when it should have, and did not hand off when it could have resolved the call. Both directions cost money.
- Caller sentiment and drop-off
- Where callers hang up, get frustrated, or ask for a human. Useful as a signal to investigate, less so as a score.
Conversation-quality metrics
- Loops and repetition
- The agent asking the same question again, or the caller repeating themselves. A reliable sign of a flow or recognition problem.
- Interruptions and overlap
- The agent cutting off the caller, or answering while the caller is still speaking.
- Dead air
- Pauses longer than a natural turn gap. Long silences make callers hang up.
Pipeline metrics
- Response latency
- Time from the caller finishing a sentence to the agent starting to speak. Track the median and a high percentile, because the slow tail is what callers remember.
- Transcription accuracy
- How often speech recognition got the caller's words wrong, especially names, numbers, and addresses.
- Tool latency and error rate
- How long tools take and how often they fail. Slow tools create dead air; failed tools create wrong outcomes.
- Cost per call
- Useful for budgeting, and worth reading beside task success. A cheaper stack that fails more often is not cheaper.
Turning metrics into a rubric
A rubric is the list of checks every call is graded against. Keep it short enough to apply consistently: task success, tool correctness, answer accuracy, policy adherence, and conversation quality cover most agents. Add checks specific to your use case, and write each as a yes-or-no question with a clear definition, so two graders (human or model) reach the same answer.
Calibrate against human judgment. Have a person grade a sample, compare with the automated grades, and fix the definitions where they disagree. Then keep spot-checking, because both the agent and the callers change over time.
Grade every call, then group the failures
Averages hide what matters. A task success rate of 90% means little until you know that most of the missing 10% comes from two causes. Grade every call rather than a sample, then group the failed calls by cause so the biggest cluster is obvious and fixing it moves the number.
This is how Zelto works: it learns the rubric from your own traffic, grades every production call, and groups calls that failed for the same reason into a Finding ranked by how many callers it affects, with example calls and a drafted fix.