RAG Evaluation — Measure Retrieval Quality in Production
Score faithfulness, context precision, answer relevancy and hallucination rate across your RAG pipelines in real-time — not just in offline test suites.
99.9%
Uptime SLA
< 5ms
Trace overhead
SOC 2
Certified
OTel
Native
RAG evaluation
RAG evaluation measures how accurately a retrieval-augmented generation pipeline answers questions — tracking whether retrieved context is faithful to the source, relevant to the query, and free of hallucinated claims. Perimattic AI Suite runs RAGAS-aligned metrics inline during inference and stores scored traces for regression tracking across model and retrieval changes.
Why RAG pipelines need continuous evaluation
Instrument once, observe everything
One OTel SDK, one OTLP exporter. No proprietary agents or middleware sitting in the critical request path.
Eval scores on production traffic
Faithfulness, answer relevancy, and hallucination rates measured on live requests — not just curated test sets.
Compliance evidence built in
Structured audit logs formatted for HIPAA, EU AI Act, DORA, and MAS FEAT. No manual export, no post-processing.
Why monitoring alone misses RAG failures
RAG pipelines fail in ways that standard monitoring cannot detect. A pipeline can return 200 OK, complete in 800ms, and still hallucinate an answer that contradicts every retrieved document. Latency and error rate tell you nothing about faithfulness — you need evaluation scores that measure the semantic relationship between the retrieved context, the generated answer, and ground truth.
The RAGAS metric set explained
The RAGAS framework (Retrieval Augmented Generation Assessment) defines the canonical metric set: faithfulness (does the answer follow from the retrieved context?), answer relevancy (does the answer address the question?), context precision (were the retrieved chunks actually needed?), and context recall (did retrieval find everything needed to answer correctly?). Running these metrics in production — not just in offline evals — gives you the feedback loop needed to catch retrieval degradation as your corpus evolves.
Span-level RAGAS scores on every trace
Perimattic AI Suite instruments your RAG pipeline at the span level, attaches RAGAS scores to every trace, and surfaces regressions when a new embedding model, re-ranking strategy, or chunk size change degrades downstream answer quality. It closes the gap between offline eval runs and production performance.
RAG evaluation capabilities in Perimattic AI Suite
Everything your team needs to instrument, evaluate, and audit AI systems in production — with evidence that satisfies your compliance requirements.
RAGAS Metric Scoring
Faithfulness, answer relevancy, context precision and context recall computed inline or as async batch eval against stored RAG traces.
LLM-as-Judge Evaluation
Configurable judge prompts that evaluate answer quality, factual grounding, and citation accuracy using a separate judge model from the generator.
Retrieval Span Tracing
OTel spans for every retrieval step — embedding call, vector search, re-ranking, chunk selection — with latency and score attached per chunk.
Hallucination Detection
Per-answer hallucination flags when generated claims cannot be traced to any retrieved document, surfaced in real-time during inference.
Regression Tracking
Compare RAGAS scores across model versions, chunk sizes, and retrieval strategies. Catch quality regressions before they reach end users.
Common questions, answered
Answers to the most common questions about this regulation, what it requires, and how AI observability helps you meet it.
What is RAG evaluation?
RAG evaluation is the practice of measuring the quality of retrieval-augmented generation pipelines — specifically whether the retrieved documents were relevant, whether the generated answer was faithful to those documents, and whether the answer correctly addressed the user question. The RAGAS framework defines the standard metric set: faithfulness, answer relevancy, context precision, and context recall.
What is the RAGAS framework?
RAGAS (Retrieval Augmented Generation Assessment) is an open-source framework for evaluating RAG pipelines without human-annotated ground truth. It uses an LLM to judge faithfulness (does the answer follow from the context?), answer relevancy (does the answer address the question?), context precision (are the retrieved chunks useful?) and context recall (did retrieval surface all necessary information?). RAGAS scores range from 0 to 1; production systems typically target faithfulness > 0.85.
Can RAG evaluation run in production, not just offline?
Yes. Perimattic AI Suite runs RAGAS-aligned evaluation inline during inference — after the answer is generated but before it is returned — and attaches evaluation scores to the OTel span. You can also run async batch evaluation against stored traces on a schedule, which is lower-latency impact but catches regressions with a delay. For regulated use cases (HIPAA, EU AI Act), inline evaluation that produces an immutable scored trace is often required as compliance evidence.
How does LLM-as-judge evaluation differ from RAGAS?
RAGAS is a structured metric framework with fixed dimensions and a defined computation method. LLM-as-judge is a flexible evaluation approach where you write a judge prompt that asks a separate LLM to evaluate answer quality — useful when you have domain-specific quality criteria (citation format, legal language accuracy, medical precision) that RAGAS metrics do not capture. Perimattic AI Suite supports both: RAGAS for standardised benchmarking and LLM-as-judge for custom domain quality gates.
What causes RAG pipeline hallucinations?
RAG hallucinations occur when the generator produces claims that are not supported by any retrieved document — either because retrieval failed (context recall failure) or because the model extrapolated beyond the retrieved evidence. Common causes: chunk size too large (model attends to irrelevant context), poor re-ranking (low-relevance chunks ranked above high-relevance chunks), or a generator model that 'fills in' missing information from parametric memory when retrieval evidence is thin. Faithfulness scoring detects this by checking every answer claim against retrieved chunks.
How do I improve low faithfulness scores?
Low faithfulness (< 0.75) usually indicates one of: the generator is hallucinating beyond retrieved context (try a lower temperature or a model with stronger instruction following), the retrieved chunks do not contain the answer (context recall problem — improve retrieval with re-ranking or hybrid search), or the chunk size is too large and the generator loses the relevant passage in a sea of context (reduce chunk size or use late chunking). Perimattic AI Suite shows faithfulness scores alongside retrieval span metadata so you can correlate low faithfulness with specific retrieval patterns.
Related pages
Dig deeper into the topics that matter most for your AI observability stack and compliance posture.
Ready to add observability to your AI systems?
Join the waitlist and we'll show you how Perimattic AI Suite traces your agents, catches hallucinations, and proves compliance.