Voice agents that act
Measure completed tasks, state changes, recovery, confirmation, reversibility, and truthful status—not just valid tool-call JSON.
“Voice AI” is not one evaluation problem. A benchmark for speech recognition may say nothing about task completion; a tool-call suite may never execute the action; and a transcript benchmark may start after the microphone work is already done.
This guide maps 13 public starting points reviewed from first-party sources on September 1, 2026. It is a landscape, not a leaderboard, and inclusion does not imply endorsement.
Article updated:
Select the track that matches the user result you need to verify. Then use component metrics to explain why that result succeeded or failed.
Measure completed tasks, state changes, recovery, confirmation, reversibility, and truthful status—not just valid tool-call JSON.
Combine recognition accuracy with entity handling, formatting, latency, correction burden, and reliable insertion into the target app.
Keep capture, transcription, diarization, summary faithfulness, decisions, and action-item quality as separate layers.
The matrix below answers a narrow question: which public project is a reasonable first stop for a given evaluation need? “Partial” means the capability appears in only part of a suite or is measured indirectly.
| Benchmark | Best starting point for | Spoken I/O | Tools | Task completion | Computer action |
|---|---|---|---|---|---|
| EVA | End-to-end conversational agent accuracy and experience | Input + output | Yes | Yes | No |
| tau2-bench / tau-Voice | Dynamic user-agent-tool interaction in realistic domains | Input + output | Yes | Yes | No |
| VoiceComputerBench / TalkAct | Real-time voice conversation plus browser action | Input + output | Yes | Yes | Yes |
| NVIDIA voice-agent evaluation | Reproducible live harness over EVA and tau2 scenarios | Input + output | Yes | Yes | No |
| VoiceAgentBench | Speech-based tool selection, orchestration, and safety | Input only | Yes | Partial | No |
| Audio2Tool | Spoken tool-call selection across intent complexity | Input only | Yes | No | No |
| Voice Agent Latency | Caller-perceived time to first agent audio | Input + output | No | No | No |
| OpenBench | Reproducible ASR, diarization, and streaming tests | Input only | No | No | No |
| ELITR-Bench | Long-context reasoning over meeting transcripts | Transcript text | No | Partial | No |
Capability coverage, evidence notes, licenses, and links for all 13 reviewed projects are available in the machine-readable JSONL dataset.
Start with the outcome, then inspect the evidence behind a capability label. In the frozen landscape, filtering audio_input=yes and tool_use=yes returns seven records. Adding goal_completion=yes leaves four. The other three have partial or no goal-completion coverage; they are not necessarily worse systems. This is a metadata shortlist, not a benchmark run or a ranking.
NeMo provides a useful next check. Its airline scenario source adds natural-language assertions for cases where the expected database remains unchanged. Scenario 0 requires the agent to explain that the reservation cannot be cancelled and not claim cancellation. Its success signals combine database-state matching, those assertions and clean exit. An unchanged database alone would not establish that the user received the required answer. These are simulated scenario checks, not evidence of a production booking change.
Choose the remaining measurement by task. For dictation, OpenBench supplies speech-recognition and streaming components; separately check the final text and its insertion into the intended application. For conversational delay, OpenBenchmarks measures caller-audio stop to agent-audio start from recordings. Report excluded turns alongside latency, and assess answer correctness separately.
Reproduce the filters below, then inspect the selected scenario's success rule. A missing target-application-insertion field means “unrecorded,” not “unsupported.” No Cue model was evaluated in this example.
With Node.js installed, open the root of the complete landscape source at f65fe8f. Keep scripts/query.mjs, scripts/data.mjs, data/schema.json and data/benchmarks.jsonl at those relative paths, then run:
node scripts/query.mjs --where audio_input=yes --where tool_use=yes --fields id,goal_completion --format csv
node scripts/query.mjs --where audio_input=yes --where tool_use=yes --where goal_completion=yes --fields id,goal_completion --format csv
| Task | Starting point | Evidence still needed |
|---|---|---|
| Dictate into a target application | OpenBench: audio input and real-time coverage; no tool or goal coverage in this snapshot | Final target-field text, focus and placement checks |
| Speak a tool task | NeMo: audio, tools and goal coverage; inspect scenario assertions and state checks | Tool arguments/results and scenario-specific success evidence; not just valid JSON |
| Measure conversational delay | OpenBenchmarks latency: audio input/output and real-time coverage; no goal coverage | Timing boundaries, excluded turns, recordings; separate answer/task evaluation |
Source snapshots (scope checked 2026-10-20; landscape row dates remain unchanged):
A single composite score hides failure modes. Keep these four layers visible and publish the result for each one.
Did the user get the intended result in the target system?
Was the exchange timely, understandable, interruptible, and recoverable?
Measure ASR, diarization, entities, tool arguments, or generated content without treating them as the final outcome.
Test consent, confirmation, reversibility, privacy, and whether the agent reports completion truthfully.
Record the benchmark commit, model and provider versions, prompts, tool schemas, audio preprocessing, trial count, sampling settings, timing boundaries, judge rubric, timeouts, refusals, and every dataset license. Failed runs should remain in the report rather than disappear from the denominator.
The underlying landscape is maintained as an independent community resource by Sophon LLC. It does not rank Cue against the listed projects, and no benchmark owner has sponsored inclusion.
Start with the open landscape if you are designing an evaluation. If you are evaluating a desktop voice agent in practice, explore Cue's voice typing, meeting, and cross-app workflows separately.