How is LLM observability different from traditional APM?

APM still covers infrastructure, errors, and latency. LLM observability adds model inputs and outputs, token and cost data, prompt and model versions, tool trajectories, retrieval context, and quality scores because a successful HTTP response does not prove a useful answer.

What should one Langfuse trace represent?

Use one trace for one user-visible task or transaction. Put retrieval, tool calls, model generations, parsing, and validation underneath it as observations. This makes end-to-end latency, cost, and success attributable to the same task.

Do I need to store raw prompts and responses?

No. Store only what your debugging and evaluation workflows require. Mask or omit sensitive input, output, and metadata before export; use stable internal IDs where deletion or aggregation is needed; configure access and retention explicitly.

Should I self-host Langfuse?

Self-host when data residency, network isolation, or infrastructure control justifies operating Postgres, ClickHouse, Redis or Valkey, blob storage, web, and worker services. Use Langfuse Cloud when you want the managed path. Docker Compose is suitable for local or low-scale use, not a default high-availability production architecture.

LLM Observability in Production: Metrics, Tracing, and Alerts

An LLM endpoint can return 200 OK, stay under its latency target, and still give the user a confident wrong answer. That is the observability gap: infrastructure telemetry shows that the request completed, but not whether the task succeeded.

Production LLM observability connects the whole path:

request → retrieval → tool calls → model generations → parser → guardrail → outcome

For each task, you should be able to answer:

what happened and in what order;
which prompt, model, tools, and retrieval release were used;
where time and money were spent;
whether the output passed its contract and helped the user;
whether a new release changed any of those signals.

Langfuse is one practical implementation. It is open source, uses OpenTelemetry-based SDKs, and combines tracing, prompt management, evaluation, and metrics. This article is about operating that loop. For installation details, use the separate Langfuse step-by-step tutorial. For the contracts, routing, reliability, and release controls around the same loop, use the broader production LLM stack guide.

Start with the task, not the model call

The most useful trace boundary is one user-visible task: classify a ticket, answer a support question, build an itinerary, or review a pull request. A trace that represents only one provider call loses the retrieval and tool steps that often caused the failure.

A useful hierarchy looks like this:

root: answer-support-question
├── span: load-account-policy
├── span: retrieve-help-center
├── generation: draft-answer
├── tool: check-subscription-status
├── generation: revise-answer
└── span: validate-citations

In Langfuse, observations are OpenTelemetry spans. A generation is a specialized observation with model, parameters, token usage, cost, and timing. Tool and retrieval observations keep non-model work in the same causal tree. The current SDK overview documents the OpenTelemetry mapping and context propagation.

Do not create a flat list of generations and try to reconstruct parentage later. Propagate trace context through async jobs and service boundaries from the beginning.

A minimal current Python trace

The current Python SDK uses the OpenTelemetry-based observation API:

from langfuse import get_client

langfuse = get_client()

with langfuse.start_as_current_observation(
    as_type="span",
    name="answer-support-question",
    input={"ticket_id": ticket_id},
) as root:
    with langfuse.start_as_current_observation(
        as_type="generation",
        name="draft-answer",
        model=model_id,
        input=messages,
        prompt=prompt,
    ) as generation:
        response = call_model(model_id=model_id, messages=messages)
        generation.update(output=response)

    root.update(output={"status": "completed"})

Set credentials through environment variables rather than source code. In long-running services, the SDK exports asynchronously. In short-lived jobs, flush before the process exits.

The prompt=prompt relationship matters. It links the exact managed prompt version to the generation, which makes prompt-level metrics and rollback analysis possible. The full release workflow is covered in prompt engineering in production.

Design a telemetry contract

Instrumentation drifts when every team invents names and metadata. Define a small schema before adding dashboards.

Names

Use stable task names rather than route paths or model names:

support.answer
travel.itinerary.create
code_review.pull_request

The model is a dimension, not the task identity. If the model changes, the time series should continue.

Dimensions

Attach only dimensions you will filter or group by:

Dimension	Example	Question it answers
`environment`	`production`	Is the issue isolated to one environment?
`release`	Git SHA or app version	Which application release changed behavior?
`feature`	`itinerary`	Which product surface owns the cost?
prompt version	linked prompt object	Did a prompt release cause the regression?
`model`	runtime model ID	Did routing or provider behavior change?
`session_id`	stable internal ID	Did a multi-turn flow fail midway?
tags	`canary`, `paid`	Does the issue affect one cohort?

Avoid email addresses and raw external account IDs in tags. High-cardinality dimensions are useful for debugging but expensive and awkward for dashboards. Use stable internal IDs only when you need session reconstruction or deletion workflows.

Inputs and outputs

Logging everything is not the default. Decide separately for:

production input and output;
retrieved documents;
tool arguments and results;
model configuration;
error bodies;
evaluation reasoning.

You can preserve operational metrics while omitting or masking content. Treat traces as a production dataset with the same access and retention discipline as the source data.

Measure five layers, not one dashboard

1. Contract correctness

These signals are deterministic and should be cheap:

JSON or schema validation failures;
missing required fields;
invalid or unauthorized tool calls;
citation targets that do not exist;
retry exhaustion and fallback activation;
empty or truncated output.

A contract error is not “low quality.” It is a broken interface and should usually page or block a release sooner than a subjective score.

2. Task quality

Langfuse stores evaluation results as scores. A score can come from user feedback, human annotation, deterministic code, an LLM judge, or an experiment. The current scores model supports numeric, categorical, boolean, and text values.

Choose scores that match the task:

Task	Better signal than “helpfulness”
Classification	label correctness by class and failure slice
Extraction	schema validity and field-level precision/recall
RAG answer	citation validity, groundedness, answer completeness
Agent workflow	task completion, tool errors, unnecessary steps
Customer support	accepted answer, correction, reopen, escalation

Use an LLM-as-a-judge for outputs that cannot be scored deterministically, but calibrate it against human decisions. A judge is a measurement instrument, not ground truth.

3. Operations

Track end-to-end and per-observation values:

request volume and error rate;
p50, p95, and p99 task latency;
provider latency and time to first token where available;
retries, rate limits, timeouts, and fallback usage;
queue delay and evaluator lag.

A slow tool call and a slow model call need different owners. That is why the trace tree matters.

4. Economics

Tokens are an input to cost, not the product metric. Track:

cost per task;
cost per successful task;
cost by feature, release, prompt version, and model route;
retry and fallback cost;
evaluator cost as a separate line item.

A cheaper response that causes more reopens or human corrections may be more expensive at the workflow level.

5. Product outcome

Join the trace ID to the downstream event that defines success: accepted suggestion, completed booking, resolved ticket, merged pull request, or retained user. Without that link, observability can optimize a proxy while the product gets worse.

Sampling: keep rare failures, sample routine success

Uniformly keeping 10% of all traces is simple but often wrong. Preserve:

every contract failure and provider error;
every canary request during a small rollout;
sessions with explicit negative feedback;
high-cost and high-latency outliers;
a representative sample of ordinary success.

Run expensive evaluators on a controlled sample. Langfuse uses deterministic evaluator sampling, so evaluators with the same filters and rate can inspect the same subset. That makes score comparisons less noisy than independent random samples.

Record the sampling policy beside the dashboard. A quality rate calculated on flagged failures cannot be compared with one calculated on random traffic.

Alerts: connect symptoms to actions

Do not alert on every raw metric. Each alert needs an owner, a comparison window, a minimum sample, and a response.

Alert	Compare	First action
Contract failures	current release vs recent baseline	stop rollout; inspect parser/tool schema
Quality regression	prompt/model version and failure slice	pause canary; review scored examples
Cost per success	feature and model route	inspect retries, context size, routing
p95 task latency	trace and child observation	find the slow span before tuning the model
Provider errors	provider, region, error class	trigger or inspect fallback policy
Telemetry delay	ingestion timestamp vs event time	verify exporter, queue, worker, storage

Set thresholds from observed variance and business impact. A universal “10% regression” rule produces false alarms on low traffic and misses absolute failures in high-risk workflows.

Dashboards should make version changes visible. An alert without the application release, prompt version, model route, and environment sends the on-call engineer back to manual archaeology.

Privacy and retention are part of instrumentation

Prompts and tool results often contain personal or confidential data. Redact before the trace leaves your process; post-ingestion cleanup is too late for strict data boundaries. Langfuse recommends mask_otel_spans for new Python SDK setups in its masking guide.

At minimum:

classify which fields may be exported;
mask secrets, tokens, emails, phone numbers, and document content as required;
separate production and non-production projects and keys;
restrict project access and export permissions;
define deletion and retention procedures;
test observability with representative redacted fixtures.

Langfuse does not automatically delete self-hosted event data by default. Retention is a configured product or infrastructure decision; the retention documentation explains plan availability, nightly deletion, and blob-storage implications.

Cloud or self-hosted: decide based on operations

Self-hosting is not a two-container PostgreSQL stack anymore. Current Langfuse v4 uses web and worker services plus Postgres, ClickHouse, Redis or Valkey, and S3-compatible blob storage. Docker Compose is supported for local and low-scale deployments; the official self-hosting guide recommends managed or orchestrated options for production scale and high availability.

Choose self-hosting when the organization needs data locality, network isolation, or infrastructure control and can own:

backups and restore drills;
schema and version upgrades;
ClickHouse capacity and retention;
queue and worker health;
object storage lifecycle;
authentication, TLS, and secrets;
alerting for the observability system itself.

Choose Cloud when the managed operational burden is worth more than running that stack. Do not claim self-hosting is cheaper without including engineering time and recovery.

Migrating old Langfuse instrumentation

If your code still uses langfuse.trace(...), trace.generation(...), or legacy batch ingestion, do not copy old examples into new services. Langfuse v4 is observations-first and OpenTelemetry-based. Python SDK v4 and JS/TS SDK v5 use the current data APIs by default.

The compatibility guide lists server and SDK requirements, deprecated endpoints, and migration deadlines. It also notes that older SDKs or OTLP exporters without the current ingestion header can show delayed data in v2 APIs. Verify freshness before treating a quiet dashboard as a quiet system.

A practical rollout

Day 1: one task. Trace one user-visible path end to end. Include retrieval, tools, model calls, parsing, and the final status.

Day 2: contract signals. Add schema failures, retries, fallback usage, latency, and cost. Verify that telemetry failure does not fail the user request.

Day 3: one quality score. Choose a signal tied to the task, attach it to the same trace, and inspect examples at both ends of the distribution.

Day 4: versions and privacy. Attach release and prompt versions. Add masking, access, and retention rules before expanding coverage.

Day 5: one actionable alert. Start with the failure that has a clear owner and response. Test the alert by creating the condition intentionally in a safe environment.

Then repeat for the next important workflow. The goal is not the maximum number of traces. It is the shortest path from “users say the AI got worse” to the exact release, prompt, model route, tool call, and failed example that explains why.