ORIGINAL TECHNICAL PERSPECTIVE
How should a RAG system be evaluated before release?
Separate retrieval quality, groundedness, and usefulness. Build a reviewable release decision from representative tests, source evidence, and explicit limits.
THE SHORT ANSWER
Evidence before confidence.
Evaluate retrieval and the generated answer separately, then test the complete user task. A useful release decision shows which evidence was retrieved, whether the response stays within that evidence, whether it answers the question, and how failures are handled. A groundedness score alone does not establish that a source is correct, current, or authorized for the user. Acceptance criteria should reflect the consequences of failure, not a universal benchmark.
Architectural judgment
What decision is the system allowed to support?
Start with the user and the consequence of a wrong answer. An assistant that helps someone find a policy has a different decision boundary from a system proposing an operational change. Define what it may answer, what it must escalate, and which questions require it to say that the available evidence is insufficient.
Build a versioned test set around those boundaries. Include common questions, ambiguous requests, absent evidence, conflicting documents, and information the requesting user is not allowed to access. Keep a held-out set for the release decision so that repeatedly tuning against familiar examples does not become the definition of success.
Supporting reading: Microsoft Learn: Design and develop a RAG solution.
Microsoft Learn guidance
Is the failure in retrieval or in the answer?
Microsoft Learn distinguishes process evaluation of retrieved documents from evaluation of the final response. Its Document Retrieval evaluator uses relevance labels to assess search quality. The Retrieval evaluator instead uses a model judge to assess the relevance of retrieved context when those labels are unavailable. These are different evidence sources, not interchangeable measures of truth.
For the answer, groundedness asks whether the response is supported by its supplied context. Relevance asks whether it addresses the query. Response completeness compares coverage with an expected answer. In the documentation checked for this perspective, Response Completeness and Groundedness Pro are marked preview; do not assume every evaluator has the same availability or operational requirements.
Supporting reading: Microsoft Learn: RAG evaluators.
Architectural judgment
Can a grounded answer still be wrong?
Yes. An answer can faithfully repeat an obsolete policy or a mistake in a retrieved document. Groundedness is a relationship between an answer and its context; source quality, freshness, and authorization still need their own controls. A citation is useful only if a reviewer can inspect the actual supporting passage and understand its applicability.
Treat automated judgments as evidence to review, not an independent certification. Examine disagreements between evaluators and human reviewers, particularly for consequential questions. Record why a failure happened: missing evidence, poor ranking, unsupported synthesis, stale content, or an access-control problem. Those diagnoses imply different fixes.
Supporting reading: Microsoft Learn: RAG evaluators; Microsoft Learn: Design and develop a RAG solution.
Decision checklist
What should a release reviewer be able to inspect?
The release artifact should make a decision reproducible rather than merely display a favorable average. Choose and document thresholds before the final evaluation. A business owner, domain reviewer, and technical owner should be able to understand the failures being accepted and the ones that block release.
- The dataset version, intended users, question categories, and known coverage gaps.
- The model, prompt, retrieval configuration, document versions, and evaluator configuration used in the run.
- Retrieved passages, response citations, failure examples, and human review of consequential cases.
- Results by scenario, alongside latency and operating cost for the intended workload, rather than quality averages alone.
- Explicit abstention and escalation behavior, an accountable release decision, and a rollback path.
Supporting reading: Microsoft Learn: RAG evaluators; Microsoft Learn: Observability in generative AI.
Architectural judgment
What changes after the initial release?
A release test is a bounded observation, not a promise about every future interaction. New documents, changed user behavior, model updates, and retrieval changes can invalidate earlier assumptions. Re-run the relevant evaluations when those dependencies change, and use production observations to discover missing test cases without exposing private user data.
Microsoft Foundry's observability guidance connects evaluation, tracing, and monitoring across development and production. The leadership responsibility is to decide who reviews those signals and can stop or reverse a release. A dashboard without an owner and a response procedure does not close that loop.
Supporting reading: Microsoft Learn: Observability in generative AI.
Sources, scope, and currency.
Original technical synthesis, not a customer deployment report. Recommendations are distinguished from documented platform capabilities. No customer results, adoption figures, or universal performance thresholds are claimed.
Microsoft Learn sources checked on . This is a dated review, not a live documentation feed or a claim that every capability is generally available. Recheck the current documentation, preview status, and deployment requirements before implementation.