Freshness changes the failure mode
Benchmarks that rely only on static corpora miss a core production problem: the answer may have changed yesterday. Fresh retrieval makes that failure visible.
A useful benchmark should reward finding current primary sources and penalize stale but semantically similar pages.
Citation pressure improves diagnosis
When every answer must cite sources, retrieval mistakes become easier to see. Missing URLs, weak snippets, and duplicated domains show up before the model has a chance to smooth them over.
This makes citation-aware evaluation valuable even when your end-user UI does not display citations directly.
Latency is part of quality
An agent that waits three seconds for each search call will avoid searching, search less often, or hit task timeouts. Retrieval quality and latency have to be evaluated together.
For iterative agents, p95 latency is usually a better production metric than average latency because the slow tail compounds over a multi-step task.