The Next Agent Benchmark Is Silence

The Short Version

Most AI benchmarks reward the model for producing an answer.

The next generation of agents also needs to be rewarded for staying quiet.

A group of researchers from Google and six universities has published StreamVibe, a benchmark, model, and memory harness for proactive video agents. These systems watch someone cook, assemble an object, or use software, then decide every second whether to help or remain silent.

That sounds like a small interface detail.

It is the product.

The benchmark includes 4,100 evaluation samples across 183.4 hours of video and 20,546 annotated moments when help may be appropriate. Some requests are explicit: tell me when the water boils. Others are implicit: help me if I get stuck. Others change during the task: stop giving progress updates and only tell me when I make a mistake.

The researchers report that their trained model improves timing and answer quality over several baselines, and that a separate state-tracking harness reduces stale and redundant responses.

The more important result is less flattering.

Models that look capable when tested on isolated moments deteriorate when they must run through a full video using their own previous responses as history. Timing gets worse as the interaction grows longer. Unspoken and changing intentions remain especially difficult. More recall often means more unnecessary interruptions.

That is the real proactive-agent problem.

An agent does not become helpful by answering more often.

It becomes helpful by knowing when the answer is worth the interruption.

The hardest action is no action

Reactive assistants have a convenient contract.

The user asks. The model answers.

Even when the answer is bad, the system at least knows when it was invited to speak.

Proactive agents lose that certainty. A request like “help me finish this task” does not define the trigger. The agent has to infer whether the user is confused, whether the current step is complete, whether earlier advice is still relevant, and whether saying something now will improve the outcome.

Silence becomes an action.

Speak too early and the agent is distracting. Speak too late and it is useless. Repeat advice and it feels broken. Follow an old instruction after the user changes direction and it becomes actively annoying. Stay silent through a real mistake and the product fails at the exact moment it was supposed to help.

This is not only a video-assistant problem.

It applies to coding agents that watch a repository, security agents that monitor activity, office agents that follow projects, wearable assistants that see the physical world, and customer-support systems that decide whether to escalate.

Every proactive product is making two predictions:

What does the user need?

And is now the right time to intervene?

Most agent demos emphasize the first prediction.

The second one decides whether anyone keeps the feature enabled.

Pointwise tests flatter the agent

StreamVibe makes a useful distinction between pointwise and trajectory evaluation.

In a pointwise test, the model is dropped near a candidate help moment and asked whether it should respond. The history can be supplied. Earlier mistakes do not necessarily contaminate the next decision.

That is useful for isolating capability.

It is also a generous version of the product.

In the trajectory test, the model runs continuously over the full stream. It has to live with what it already said, track which requests remain active, update the task state, and avoid repeating itself. One premature response changes the history used for later decisions. One missed update can keep an obsolete intention alive.

The authors say timing quality declines steadily with interaction length. Their analysis also finds that agents struggle to balance recall against false triggers: systems that catch more useful moments often interrupt more when they should stay quiet.

The Decoder's review highlights the gap sharply. When models had to process full videos using their own interaction history, one content metric fell by as much as 61.5% compared with the cleaner evaluation setting.

This is why single-turn agent benchmarks can produce false confidence.

A correct local decision is not the same thing as stable behavior.

The agent has to survive its own past.

The harness is part of the intelligence

The researchers did not try to solve the whole problem inside one model.

StreamVibe-Harness maintains the active requests, task progress, prior replies, and relevant history. The model still decides when and what to say, but it receives a cleaner representation of the interaction state.

That separation matters.

Long context is not the same thing as current state. A transcript can contain every previous instruction and still fail to identify which one was replaced, which task step is finished, and which promise remains open.

The paper reports that the harness suppresses unnecessary responses and improves content quality in most settings, with the clearest gains when intentions change during the task.

That is another useful correction to the model race.

The proactive product is not only the vision model.

It is the state machine around the model.

Builders need explicit representations for active goals, superseded instructions, completed steps, unresolved requests, confidence, and the conditions that permit an interruption. Otherwise the agent keeps rereading a conversation and guessing which parts still matter.

Very intelligent.

The assistant has invented meeting amnesia at machine speed.

This is a research result, not a finished assistant

The paper is promising.

It is not a product launch or independent validation.

The authors built the benchmark, trained the model, and reported the improvements. Gemini 3.8 Flash proposed candidate help points before four reviewers checked them, while Claude Sonnet 4.6 scored answer quality in the main evaluation. The benchmark is carefully constructed, but model-generated annotation and model judging remain part of the measurement pipeline.

The proprietary models are also not reported in the paper's full trajectory comparison, which limits the cleanest cross-model conclusion. The headline 10.7% model gain and 12.3% harness gain are author-reported results on this evaluation setup, not independently reproduced production outcomes.

The project page says the code and Hugging Face release are coming soon and remain under internal approval. At the time of writing, the public artifact is the paper and demonstration site, not a released system that outside teams can run end to end.

The authors acknowledge the larger boundary themselves: real interactions may have richer intent changes and tighter feedback between the user's needs and the environment than this benchmark captures.

In other words, the controlled test is already difficult.

Reality will be less polite.

What builders should measure

If you are building a proactive agent, do not evaluate only whether it can produce the right answer after being asked.

Measure the intervention policy.

At minimum, track:

• useful interventions caught
• unnecessary interruptions
• missed moments that mattered
• early and late responses
• repeated or stale advice
• behavior after the user revises a request
• performance over full sessions, not sampled moments
• degradation as the session gets longer
• whether an interruption changed the user's outcome
• how often users mute, disable, or correct the assistant

Test with the agent's own history, not only a clean transcript assembled by the evaluator. Include users who change their minds, abandon a task, introduce a second goal, or do something the benchmark did not anticipate.

And give the product a conservative default.

For a proactive assistant, false positives are not merely metric noise. They are product debt charged directly to the user's attention.

The model should earn the right to interrupt.

The bottom line

StreamVibe is useful because it measures a part of agency that normal benchmarks mostly ignore.

Not what the model knows.

Not only what it can do.

When it should enter the moment.

The research shows why that cannot be reduced to another question-answer score. Intent can be implicit. Instructions can change. State can drift. Previous answers can poison later decisions. A model that succeeds at isolated moments can become unreliable across a real trajectory.

The next personal agent will always be one second away from saying something.

That does not make it helpful.

The best proactive system may be the one that understands more, tracks more, and speaks less.

Silence is not the absence of intelligence.

It is one of the outputs.