A model that passed every evaluation before launch can still fail quietly in production, week after week, without a single error in your logs. That is the uncomfortable truth about running large language models at scale. Traditional software either works or throws an exception. LLMs do neither. They produce a confident, well-formatted, completely wrong answer, and nothing in a standard uptime dashboard will tell you that happened.
This is the problem LLM observability exists to solve. It is not just "logging for AI." It is a discipline built around two specific failure modes that do not exist in traditional software monitoring: hallucination, where the model generates content that is not grounded in fact or in the provided context, and drift, where the model's behavior or accuracy degrades over time even though nothing about your code changed.
This guide covers what actually needs to be monitored, how hallucination and drift detection work mechanically, what tooling categories exist, and how to build an observability stack that catches problems before your users do.
Why Traditional Monitoring Doesn't Work for LLMs
Standard application monitoring tracks latency, error rate, and uptime. All three matter for LLM systems too, but none of them catch the failure mode that damages trust: a fluent, grammatically correct, factually wrong response returned with a 200 status code.
Legal research is a well-documented example of this gap. A Stanford RegLab study testing public-facing LLMs on verifiable questions about real federal court cases found hallucination rates ranging from 58 percent with GPT-4 up to 88 percent with Llama 2, and the same research showed models frequently fail to flag or correct a user's incorrect legal premise instead of catching it. None of that shows up as a system error. It shows up as a wrong answer delivered with total confidence, which is exactly why hallucination requires its own monitoring layer rather than being caught by infrastructure metrics.
The Two Core Problems: Hallucination and Drift
Hallucination
Hallucination is when a model generates output that is not supported by its training data, the provided context, or verifiable fact. It comes in a few recognizable forms:
- Factual hallucination: stating something false as if it were true
- Context hallucination: contradicting or ignoring the source documents provided in a RAG pipeline
- Citation hallucination: inventing sources, links, case law, or references that do not exist
- Instruction hallucination: fabricating capabilities or performing an action it was never asked to do
Hallucination rates vary enormously by domain and task. Research on clinical case summarization found large-language-model-generated summaries hallucinated at high rates without mitigation, and structured prompting was able to meaningfully reduce, though not eliminate, that rate. A frontier-model benchmark focused specifically on package name hallucination in code generation, testing five current models, found much lower but still nontrivial rates, in the mid-single-digit percentages, with statistically significant differences between models. The range across tasks is the point: a single "our hallucination rate is X percent" number is close to meaningless without specifying the task, domain, and model being measured.
Drift
Drift is a change in model behavior or output quality over time. In production LLM systems it typically comes from one of three sources:
- Data drift: the distribution of incoming user queries changes from what the system was designed or evaluated against
- Concept drift: the real-world meaning of "correct" shifts, for example a policy, pricing structure, or regulation the model relies on changes
- Model drift: the underlying hosted model itself is updated or replaced by the provider, silently changing behavior on your existing prompts
Unlike hallucination, drift is not visible in any single interaction. It only shows up as a trend across many requests over time, which means it requires continuous statistical monitoring rather than one-off evaluation.
What to Actually Monitor
| Signal Category | What It Captures | Example Metric |
|---|---|---|
| Output groundedness | Whether claims in the output are supported by retrieved context or source documents. | Percentage of unsupported claims per response. |
| Factual accuracy | Whether verifiable statements in the output are true. | Accuracy rate against a labeled evaluation set. |
| Semantic drift | Whether the meaning of outputs for similar inputs is shifting over time. | Embedding cosine similarity compared with a rolling baseline. |
| Query distribution drift | Whether incoming user queries are moving away from the patterns the system was originally designed for. | Statistical distance between current and baseline query embedding clusters. |
| Retrieval quality | Whether a RAG pipeline is retrieving genuinely relevant context. | Retrieval precision or hit rate measured against a probe query set. |
| Consistency | Whether the model produces materially different answers to the same or similar prompts across repeated calls. | Output variance across repeated runs. |
| Refusal and failure rate | How often the model declines to answer, encounters errors, or returns malformed output. | Refusal rate and schema validation failure rate. |
| User feedback signals | Explicit and implicit indicators that an answer was incorrect or unhelpful. | Thumbs-down rate, correction rate, and escalation rate. |
How Hallucination Detection Actually Works
Reference-based checking. If the model's answer should be grounded in a specific document or context, you can programmatically check whether the claims in the output are actually supported by that source. This is the most reliable detection method available, but it only works when you have a ground-truth reference to check against, which limits it mostly to RAG-based systems.
LLM-as-judge evaluation. A second model (often a different model than the one generating the output) reviews production responses against a rubric and flags likely hallucinations or unsupported claims. This scales well and catches issues reference-based checking cannot, but it introduces its own error rate, since the judge model can also be wrong, so it should be validated periodically against human review rather than trusted blindly.
Consistency and self-verification checks. Running the same query multiple times, or asking the model to verify its own claims against retrieved evidence, and flagging responses that vary significantly. High variance across repeated runs on the same input is a strong signal that the answer is not well grounded.
Span-level claim verification. Rather than judging an entire response as hallucinated or not, this approach breaks the output into individual factual claims and checks each one independently against retrieved evidence, flagging only the unsupported spans. This gives far more actionable output than a single pass or fail judgment on the whole response.
Human review sampling. No automated method fully replaces spot-checking a sample of real production outputs by hand, especially for high-stakes domains. This is the calibration layer that keeps your automated detectors honest.
How Drift Detection Actually Works
Embedding-based distribution tracking. Query and output embeddings are tracked over time, and the distance between the current distribution and a stored baseline distribution is measured using statistical distance metrics. A sustained shift beyond a defined threshold indicates the input or output distribution has moved. Academic research on this specific technique found that embeddings generated by large language models offered notably higher sensitivity to detecting distributional shift in text compared to classical embedding methods, which is part of why embedding-based drift detection has become a standard approach for text data specifically.
Golden dataset regression testing. A fixed, version-controlled set of representative test cases is run against the production system on a regular cadence (or before every deployment), and results are compared against a known baseline to catch regressions before they reach real users.
Rolling accuracy and quality metrics. Core quality metrics, accuracy against labeled samples, groundedness scores, user feedback rates, are tracked on a rolling window (daily, weekly) rather than as a single point-in-time number, so gradual degradation shows up as a trend rather than being lost in the noise of day-to-day variance.
Retrieval health monitoring. In RAG systems specifically, retrieval quality is tracked independently of generation quality, since retrieval can degrade (through embedding drift, index staleness, or content changes) even when the underlying generator model has not changed at all. Isolating which layer is drifting is critical for knowing what to fix.
**Provider and version tracking. ** For teams using hosted, third-party models, the model version, API version, and any documented changes are logged alongside every request, since providers can update or deprecate model checkpoints in ways that change behavior on your exact prompts without any change on your end.
Building the Observability Stack: A Layered Approach
Synchronous checks should stay cheap and deterministic: schema validation, output format checks, basic safety filters. Anything that requires model-based judgment (hallucination scoring, groundedness evaluation) is generally too slow and too expensive to run on every single request, so it runs asynchronously on a sample, commonly in the single-digit percentage range of daily traffic, with the sample size scaled up for higher-stakes use cases.
Common Mistakes Teams Make
Treating evaluation as a one-time launch gate. A model that passed evaluation at launch can still degrade through data drift, provider-side model updates, or a shift in what users are actually asking. Evaluation needs to run continuously, not once.
Only measuring what is easy to measure. Latency and uptime are easy to track and get tracked religiously. Groundedness and factual accuracy are harder to measure and often get skipped entirely, even though they are the metrics most directly tied to whether users can trust the output.
No baseline to compare against. Drift is a relative concept. Without a stored baseline (of query distribution, output embeddings, or accuracy on a fixed test set) there is nothing to measure drift against, and teams end up reacting to complaints instead of catching degradation early.
Ignoring the retrieval layer in RAG systems. When a RAG-based system starts hallucinating more, the instinct is often to blame the generator model. Frequently the actual cause is retrieval drift, stale or poorly indexed content, or an embedding model mismatch, and that requires monitoring the retrieval step independently.
No feedback loop back into evaluation data. User corrections, downvotes, and escalations are the highest-signal data a team has for what is actually going wrong, and they are commonly collected but never routed back into the evaluation dataset used for regression testing.
