LLMOps in Practice: Evaluate, Trace and Monitor LLM Applications

Make quality changes explainable

An LLM application can return successful HTTP responses while giving worse answers. A prompt change may improve one task and break another, a retrieval update can remove critical evidence, and a provider timeout can trigger an expensive fallback that hides behind an apparently healthy uptime chart.

This guide is for engineers and technical decision makers building an operating process around LLM applications. Evaluation should describe what good behavior means before a release. Observability should show how the deployed system behaves and provide enough evidence to connect a production failure to the configuration that produced it.

Key takeaways

• Version evaluation data, prompts, tools, retrieval settings and model configuration together.
• Separate deterministic checks, human judgments and model-assisted review.
• Trace meaningful application stages without collecting every sensitive payload.
• Monitor quality, cost, latency and failure modes by task segment.
• Use explicit release gates and a tested rollback path.

Turn product expectations into evaluation cases

Start with tasks the application is supposed to complete. For a document assistant, that may mean answering from accessible sources with supported citations. For a structured extraction service, it may mean producing a valid schema with correct fields and a clear missing-value policy. A generic instruction to be helpful cannot serve as a release criterion.

Write cases that include the input, relevant context, expected behavior and prohibited outcomes. Some cases have an exact answer. Others need a rubric describing required facts, acceptable alternatives and when the application should ask for clarification. Keep those distinctions explicit so reviewers do not penalize harmless wording changes as factual errors.

Include normal requests, difficult requests and boundary cases. Empty input, conflicting evidence, unsupported languages, malicious instructions inside documents and expired tool permissions can expose failures that a collection of polished examples misses. Label the cases by task type and risk so aggregate results can be broken down meaningfully.

+------------------+     +-------------------+
| Versioned cases  | --> | Candidate pipeline|
+------------------+     +---------+---------+
                                   |
                         +---------v---------+
                         | Checks + reviewers|
                         +---------+---------+
                                   |
                         +---------v---------+
                         | Release decision  |
                         +---------+---------+
                                   |
+------------------+     +---------v---------+
| Reviewed failures| <-- | Production traces |
+--------+---------+     +-------------------+
         |
         +----> New regression cases

Separate development cases from held-out release cases. Repeatedly rewriting prompts against the same small test set can make progress look larger than it is. Keep a record of why a case exists and when it was last reviewed, since business policies and authoritative answers can change.

A useful first collection is modest but diverse. Its exact size depends on the task and the consequences of failure. Rather than claiming a universal minimum, start with the main behavior categories and expand when production review reveals a new class of error. Coverage of important failure modes matters more than a large count of near-duplicate prompts.

Use the right evaluator for each property

Deterministic checks are strong for schema validity, required fields, allowed tool names, citation identifiers and prohibited output patterns. They are fast and reproducible, but they do not establish that an answer is factually correct. A JSON object can be perfectly valid and still contain the wrong amount or customer identifier.

Human review is valuable for nuanced correctness, ambiguity and usefulness. Provide a rubric and examples of each score. Ask reviewers to identify unsupported claims separately from incomplete answers. When reviewers disagree, inspect whether the case is ambiguous before concluding that one reviewer is careless.

Model-assisted review can help triage larger samples, but its outputs should be calibrated against human judgments. Record the reviewing model, prompt and rubric version. Avoid giving the reviewer unnecessary identifying information about which candidate produced an answer, and vary presentation order when comparing two candidates to detect positional bias.

Evaluation method	Best use	Important limitation
Deterministic check	Format and explicit constraints	Cannot establish general factual accuracy
Reference comparison	Exact or constrained answers	May reject valid alternative wording
Human rubric	Nuance and business meaning	Requires reviewer time and calibration
Model-assisted rubric	Scalable triage	Can be inconsistent or share model biases

The following runnable Python example checks citation identifiers in a structured response. It validates only the reference relationship. It does not prove that the cited text supports the answer, which requires a separate grounding check.

def check_response(response, supplied_ids):
    if not isinstance(response, dict):
        return False, "response must be an object"
    answer = response.get("answer")
    citations = response.get("citations")
    if not isinstance(answer, str) or not answer.strip():
        return False, "answer required"
    if not isinstance(citations, list) or not citations:
        return False, "citations required"
    if any(not isinstance(item, str) for item in citations):
        return False, "citation ids must be strings"
    if not set(citations).issubset(supplied_ids):
        return False, "unknown citation"
    return True, "reference structure valid"

print(check_response(
    {"answer": "Use the approved process.", "citations": ["d1"]},
    {"d1", "d2"},
))

Use repeated runs when output variability can change the release decision. Compare cases pairwise between the current and candidate versions, and inspect regressions even if the mean improves. A candidate that helps many low-impact cases but breaks a critical authorization boundary should not pass because of its average score.

Trace the application as a sequence of decisions

A useful trace follows the request through retrieval, context assembly, model generation, validation and tool execution. Each stage should record start and end time, status, configuration version and relevant artifact identifiers. Distinguish a provider failure from an application validation failure and from a deliberate refusal.

The OpenTelemetry trace concepts describe traces and spans for following work across components. In an LLM application, choose span boundaries around operations that an engineer can investigate independently. A single span called AI hides too much; a span for every small formatting function creates noise.

Record model identifiers and parameters as configured, plus the prompt template version and retrieval index version. If a provider exposes a resolved model version, retain it where available. A friendly alias alone may not be enough to explain a behavioral change. Record fallbacks as explicit child operations rather than silently replacing the original attempt.

Capture less content by default

Trace metadata can still be sensitive. Tenant identifiers, document names and tool arguments may reveal business information even without full prompts. Define an allowlist of fields, redact at the instrumentation boundary and restrict access to the observability store. Redacting only in a dashboard leaves the original payload available elsewhere.

When full payload capture is necessary for investigation, make it controlled, time-limited and auditable. Store references to protected artifacts where practical. Retention should reflect both debugging needs and the sensitivity of the information, and deletion procedures should include traces, caches and exported evaluation fixtures.

The following runnable Python example emits a minimal JSON event around a simulated operation. It uses only the standard library and deliberately records no prompt or answer text. Production instrumentation would also connect parent and child spans and send events to your chosen telemetry backend.

import json
import time
import uuid

trace_id = str(uuid.uuid4())
started = time.perf_counter()
status = "ok"
try:
    result = sum([1, 2, 3])
    assert result == 6
except Exception:
    status = "error"
    raise
finally:
    print(json.dumps({
        "trace_id": trace_id,
        "operation": "validate_result",
        "release": "candidate-1",
        "status": status,
        "duration_ms": round(
            (time.perf_counter() - started) * 1000, 3
        ),
    }))

Monitor signals that lead to action

Operational monitoring should cover availability, latency, rate limits, retries and queue depth. LLM-specific monitoring should add input and output token usage where reported, estimated cost with a versioned price basis, invalid structured outputs, unsupported citations and fallback frequency. Keep estimated cost distinct from the final provider invoice.

Quality monitoring often depends on delayed or sampled feedback. User ratings can help locate issues, but they are not a representative ground truth because many users never rate an answer. Combine reviewed samples, deterministic checks and observed task outcomes. Track the sampling policy so a change in collection does not masquerade as a change in quality.

Segment metrics by task type, language, corpus, release and other relevant dimensions. Avoid identifiers with unbounded cardinality in metric labels. Keep request-level details in traces and use a link or exemplar to connect a metric spike to specific requests. This preserves aggregate usefulness without exploding storage costs.

Every alert should have an owner and a likely response. A spike in invalid output may require reverting a prompt. Rising retrieval misses may require inspecting ingestion. A token increase may reflect a changed context assembly policy. An alert saying quality is low without evidence or a runbook mostly creates interruption.

Release changes with a reproducible decision record

A release record should identify the candidate code, prompt templates, model configuration, tool schemas, policy rules and evaluation dataset. Store the resulting metrics and a short explanation of why the release passed or failed. This makes a later comparison possible even when multiple configuration files changed together.

Use hard gates for properties that cannot be traded away, such as unauthorized tool execution. Use reviewed tolerances for softer outcomes such as answer completeness. Thresholds should come from product needs and observed variation, not from an attractive round number selected after seeing the results.

Start with a controlled rollout and watch both operational and quality signals. Shadow execution can compare outputs without exposing them to users, but shadow calls must not execute real write actions. A canary can reveal live behavior on a limited share of traffic, provided routing is stable and rollback is practical.

The production RAG guide shows why retrieval and generation need separate measurements. Apply that same decomposition to release decisions: a model upgrade should not hide an unrelated corpus regression, and an index update should not require guessing whether the answer prompt changed.

Close the incident loop

After a failure, write a minimal replay case using approved or synthetic data. Preserve the failure mechanism without copying unnecessary customer information. Add the case to the relevant regression group and explain the expected behavior. An incident is not fully resolved when the dashboard turns green if the same mistake remains untested.

Run a staging rollback exercise before relying on rollback during an incident. Confirm that the previous prompt, model configuration and index version are still available and compatible with current application state. If an old version cannot read a new state schema, document a safe alternative such as disabling writes or routing to human review.

Common mistakes

• Treating successful requests as evidence that answers are correct.
• Using one model-generated score without human calibration or failure analysis.
• Capturing full prompts and tool outputs in every trace by default.
• Comparing releases whose datasets or source versions changed silently.
• Creating alerts without a specific owner, evidence link or response.

Checklist

• Define task-specific success criteria and prohibited outcomes.
• Version cases, rubrics and the complete application configuration.
• Separate schema checks from correctness and grounding review.
• Trace meaningful stages with privacy-aware field allowlists.
• Segment quality, latency, cost and fallback metrics.
• Establish release gates, controlled rollout and rollback procedures.
• Convert reviewed incidents into reproducible regression cases.

FAQ

What is the difference between evaluation and observability?

Evaluation judges behavior against explicit expectations. Observability records what the running system did, helping engineers investigate and identify new cases that should become evaluations.

Can an LLM judge another LLM reliably?

It can assist with review when its rubric and behavior are calibrated against people. Treat it as one measurement method with limitations, especially for factual disputes, subtle policy decisions and adversarial content.

Should production prompts always be logged?

No. Many investigations can begin with metadata, versions and protected artifact references. Capture full content only when there is a clear need and an appropriate access, retention and redaction policy.

How often should evaluations run?

Run relevant cases when prompts, models, tools, policies or retrieval behavior change. Use scheduled and incident-driven reviews to detect changes outside your codebase, with frequency based on product risk and traffic.

Final thoughts

LLMOps works when quality expectations and production evidence connect. Version the system, test the behaviors that matter and make every alert point toward a decision an operator can take.

For help building that process, explore my LLMOps service and book a conversation.