Open voice agent research

Choose a voice agent benchmark by the job it measures.

“Voice AI” is not one evaluation problem. A benchmark for speech recognition may say nothing about task completion; a tool-call suite may never execute the action; and a transcript benchmark may start after the microphone work is already done.

This guide maps 13 public starting points reviewed from first-party sources on September 1, 2026. It is a landscape, not a leaderboard, and inclusion does not imply endorsement.

Article updated:

Start with one of three product outcomes

Select the track that matches the user result you need to verify. Then use component metrics to explain why that result succeeded or failed.

1

Voice agents that act

Measure completed tasks, state changes, recovery, confirmation, reversibility, and truthful status—not just valid tool-call JSON.

2

Voice typing and dictation

Combine recognition accuracy with entity handling, formatting, latency, correction burden, and reliable insertion into the target app.

3

Meeting understanding

Keep capture, transcription, diarization, summary faithfulness, decisions, and action-item quality as separate layers.

Public benchmark starting points

The matrix below answers a narrow question: which public project is a reasonable first stop for a given evaluation need? “Partial” means the capability appears in only part of a suite or is measured indirectly.

BenchmarkBest starting point forSpoken I/OToolsTask completionComputer action
EVAEnd-to-end conversational agent accuracy and experienceInput + outputYesYesNo
tau2-bench / tau-VoiceDynamic user-agent-tool interaction in realistic domainsInput + outputYesYesNo
VoiceComputerBench / TalkActReal-time voice conversation plus browser actionInput + outputYesYesYes
NVIDIA voice-agent evaluationReproducible live harness over EVA and tau2 scenariosInput + outputYesYesNo
VoiceAgentBenchSpeech-based tool selection, orchestration, and safetyInput onlyYesPartialNo
Audio2ToolSpoken tool-call selection across intent complexityInput onlyYesNoNo
Voice Agent LatencyCaller-perceived time to first agent audioInput + outputNoNoNo
OpenBenchReproducible ASR, diarization, and streaming testsInput onlyNoNoNo
ELITR-BenchLong-context reasoning over meeting transcriptsTranscript textNoPartialNo

Capability coverage, evidence notes, licenses, and links for all 13 reviewed projects are available in the machine-readable JSONL dataset.

Worked example: audit the success condition

Start with the outcome, then inspect the evidence behind a capability label. In the frozen landscape, filtering audio_input=yes and tool_use=yes returns seven records. Adding goal_completion=yes leaves four. The other three have partial or no goal-completion coverage; they are not necessarily worse systems. This is a metadata shortlist, not a benchmark run or a ranking.

NeMo provides a useful next check. Its airline scenario source adds natural-language assertions for cases where the expected database remains unchanged. Scenario 0 requires the agent to explain that the reservation cannot be cancelled and not claim cancellation. Its success signals combine database-state matching, those assertions and clean exit. An unchanged database alone would not establish that the user received the required answer. These are simulated scenario checks, not evidence of a production booking change.

Choose the remaining measurement by task. For dictation, OpenBench supplies speech-recognition and streaming components; separately check the final text and its insertion into the intended application. For conversational delay, OpenBenchmarks measures caller-audio stop to agent-audio start from recordings. Report excluded turns alongside latency, and assess answer correctness separately.

Reproduce the filters below, then inspect the selected scenario's success rule. A missing target-application-insertion field means “unrecorded,” not “unsupported.” No Cue model was evaluated in this example.

With Node.js installed, open the root of the complete landscape source at f65fe8f. Keep scripts/query.mjs, scripts/data.mjs, data/schema.json and data/benchmarks.jsonl at those relative paths, then run:

node scripts/query.mjs --where audio_input=yes --where tool_use=yes --fields id,goal_completion --format csv
node scripts/query.mjs --where audio_input=yes --where tool_use=yes --where goal_completion=yes --fields id,goal_completion --format csv
TaskStarting pointEvidence still needed
Dictate into a target applicationOpenBench: audio input and real-time coverage; no tool or goal coverage in this snapshotFinal target-field text, focus and placement checks
Speak a tool taskNeMo: audio, tools and goal coverage; inspect scenario assertions and state checksTool arguments/results and scenario-specific success evidence; not just valid JSON
Measure conversational delayOpenBenchmarks latency: audio input/output and real-time coverage; no goal coverageTiming boundaries, excluded turns, recordings; separate answer/task evaluation

Source snapshots (scope checked 2026-10-20; landscape row dates remain unchanged):

Use a layered evaluation plan

A single composite score hides failure modes. Keep these four layers visible and publish the result for each one.

User outcome

Did the user get the intended result in the target system?

Interaction quality

Was the exchange timely, understandable, interruptible, and recoverable?

Component quality

Measure ASR, diarization, entities, tool arguments, or generated content without treating them as the final outcome.

Safety and trust

Test consent, confirmation, reversibility, privacy, and whether the agent reports completion truthfully.

Important gap: no public benchmark in this reviewed landscape fully measures continuous desktop dictation inside arbitrary applications. Teams should combine a public ASR baseline with a private, consented corpus that reflects their real microphones, languages, apps, and vocabulary.

Do not compare scores that mean different things

  • Tool-call accuracy does not prove that the downstream action succeeded or changed the right object.
  • Word error rate does not capture punctuation, formatting, entity accuracy, editing effort, or insertion reliability.
  • Transcript-only meeting tests cannot reveal microphone, overlap, diarization, reconnect, or device failures.
  • Latency measurements need explicit start and end boundaries; caller audio, first token, final text, and visible task progress are different clocks.

Record the benchmark commit, model and provider versions, prompts, tool schemas, audio preprocessing, trial count, sampling settings, timing boundaries, judge rubric, timeouts, refusals, and every dataset license. Failed runs should remain in the report rather than disappear from the denominator.

Open data and reproducibility

The underlying landscape is maintained as an independent community resource by Sophon LLC. It does not rank Cue against the listed projects, and no benchmark owner has sponsored inclusion.

Separate evidence: this landscape does not reproduce Cue's private 227-sample production benchmark. That narrower Apple Silicon measurement is documented in the Gemma 4 dictation case-study guide, with Google DeepMind's original Gemmaverse page as its primary source.

Evaluate the job, then choose the tool.

Start with the open landscape if you are designing an evaluation. If you are evaluating a desktop voice agent in practice, explore Cue's voice typing, meeting, and cross-app workflows separately.