Evaluate retrieval before generation
Most RAG quality problems are diagnosed too late. Teams look at the final answer, then try to guess whether the model, prompt, reranker, or search API caused the failure.
The faster path is to score retrieval separately. Before the LLM writes a token, inspect whether the candidate set contains current, relevant, sourceable material for the question.
The minimum scorecard
Start with four numbers: recall at top 10, citation coverage, freshness drift, and p95 retrieval latency. These metrics are simple enough to run on every build and direct enough to explain to product teams.
For agent workloads, add repeated-call behavior. A research agent may call search dozens of times in one task, so stable latency and deduplication matter more than a single best-case query.
What to log
Log the original query, normalized scope, returned URLs, scores, snippet length, and downstream citation usage. Do not log secrets or full private prompts unless your retention policy allows it.
When a final answer is wrong, this trace lets you distinguish missing evidence from poor synthesis. That separation is the difference between improving retrieval and endlessly rewriting prompts.