Langfuse vs LangSmith: Production Evaluation and Client Handover

Choose Langfuse or LangSmith around the evidence your customer's operator needs: what the application did, whether an answer met agreed criteria, and how to investigate the next failure. A trace dashboard is useful only when its data collection, evaluation rules and operating ownership fit the deployment.

For an independent implementation consultant, start with the customer environment and handover. Decide whether a managed service is acceptable, who can review application data, and who will maintain a self-hosted installation. This comparison covers documented capabilities and suggested evaluation steps. It offers no generic best-tool ranking or measured performance result.

Separate tracing from quality acceptance

Langfuse's observability documentation describes traces containing prompts, responses, token usage, timing and intermediate operations. LangSmith's observability documentation describes production traces, metrics, debugging and using traces to build evaluation datasets. Both provide evidence about recorded application behavior; that description is not proof of answer correctness. Sources checked October 3, 2026.

Our editorial distinction is simple: a trace helps you investigate what happened; an evaluation applies criteria to what happened. A successful model call can still give an unacceptable answer. Define that difference in the customer handover so an operator does not read a green infrastructure status as approval of the output.

For a hypothetical retrieval application, ask whether the answer cites the approved document and whether a reviewer can identify unsupported statements. For an action-taking application, separately ask whether the proposed action was authorized. These examples are planning exercises, not customer stories or benchmarks.

Both document online and offline evaluation

Langfuse's evaluation overview describes evaluations of live traces and applications against predefined datasets, with manual or automated methods and experiments. LangSmith's evaluation overview also distinguishes offline dataset testing from online production evaluation, including human, code and LLM-as-judge evaluators. Sources checked October 3, 2026.

Use offline tests to compare a proposed change against agreed cases. Use production review to find situations those cases missed. The evaluation criteria still belong to the customer problem: valid format, supported answer, correct escalation or permitted action. The presence of an evaluator in a product does not tell you which criterion is appropriate.

Ask a domain reviewer to label examples before accepting an automated judge's output. Retain disagreements rather than collapsing them into a single impressive-looking score. Keep the rubric and application version with the result so the next engineer can explain what a passing evaluation meant at the time.

Hosting changes the purchase and support boundary

Langfuse's current v4 architecture includes application containers, Postgres, ClickHouse, a Redis/Valkey cache and object storage. Its local Docker Compose option is described without high availability, scaling or backups. Do not present the ability to start containers as evidence that a production service is ready. Source checked October 3, 2026.

LangSmith's self-hosting documentation likewise lists application services and storage dependencies, including ClickHouse, PostgreSQL and Redis, with optional blob storage recommended for production. Its documented self-hosting entitlement is Enterprise, not an inference from the availability of an SDK. Source checked October 3, 2026.

Langfuse's self-hosting FAQ identifies its MIT-licensed core and separate commercial Enterprise features. The current license-key guide gates project-level RBAC, audit logs, retention policies and server-side masking behind a license key. Client-side masking and enterprise server-side masking are different controls. Check the component, version and required governance feature before calling a self-hosted deployment eligible. Sources checked October 3, 2026.

Our recommendation: investigate Langfuse first when customer-operated hosting is a hard requirement and you need to assess its available self-hosted feature set. Keep LangSmith on the shortlist when the customer can obtain its required Enterprise arrangement or approves a managed deployment. Neither recommendation establishes a lower total cost. Include maintenance, restoration and vendor support in the comparison.

Unresolved procurement questions remain visible: the customer's exact regional and contractual requirements, the required access controls on the purchased tier, and the cost and support scope of a negotiated agreement. Confirm them against the selected product version and contract before committing to delivery.

Approve trace collection before sending customer data

Langfuse's masking documentation describes redacting inputs, outputs and metadata before export. It also warns that its export masking does not modify copies sent through a separate OpenTelemetry exporter. A masking setting in one client does not establish that every telemetry destination receives redacted data. Source checked October 3, 2026.

LangSmith's sensitive-data guide documents hiding inputs/outputs, transforming metadata, masking and conditional tracing. Its examples distinguish hidden content from metadata that remains present. Do not assume that hiding the main prompt removes every identifying field. Source checked October 3, 2026.

Before your evaluation, make a collection map: prompt and answer fields, retrieved text, tool inputs, identifiers and exception details. Ask the customer which fields may be stored and which must be removed. Use synthetic records to inspect the exported data at each destination. Record what you intentionally excluded, because a trace with less content may support a different debugging procedure.

Self-hosting does not answer every data-path question. Check model endpoints, evaluators, gateways and additional telemetry exporters in your proposed architecture. This article does not certify legal compliance or confirm that a customer policy is satisfied.

Budget for telemetry and evaluation separately

Langfuse's official pricing page expresses usage in billable units. LangSmith's official pricing page distinguishes base and extended traces with different retention terms. The labels describe different billing structures; they are not directly comparable event counts. Sources checked October 3, 2026.

For a useful quote, identify the application paths you will instrument, the fields collected and the traces selected for evaluation. Ask which activity changes retention or incurs additional charges. Record evaluator model calls separately from the platform subscription and include infrastructure if self-hosting. We publish no numeric price here because your chosen tier, usage and negotiated features need a current check.

Keep unknowns on the cost sheet instead of substituting an estimate that looks like a verified price. Do not assume a stored trace, an application request, an observation and a judge invocation consume the same unit. Confirm the applicable billing definition before multiplying your proposed workload.

Turn production failures into a release decision

When a permitted production trace exposes a failure, save only the approved fields into a candidate dataset. Remove identifying data, obtain permission for any retained customer content and ask a domain reviewer to label the expected behavior. Separate cases used to tune the prompt from cases reserved to evaluate the change. Otherwise the next passing run may only show that the application remembers examples you used to improve it.

Record the dataset revision, application and prompt versions, model identifier, retrieval settings and evaluator configuration. Define critical failures that block release even if the average score improves, such as an unauthorized action. Preserve human disagreements and judge failures. An unavailable evaluator or missing trace should leave the decision unresolved, rather than becoming a passing result.

Agree what happens when telemetry ingestion fails: alert the operator, identify missing records and follow the application's approved degradation policy. Monitoring loss does not prove the application failed, but it reduces the evidence available to approve it. Keep release approval with the named customer owner.

Use one agreed evaluation procedure for both candidates

Prepare a customer-approved dataset with ordinary cases, ambiguous requests, missing evidence and disallowed actions. Give each case an expected behavior and a written review rule. Keep a reference version of the application so a tool comparison is not silently comparing different prompts or retrieval settings.

In your own evaluation, check whether the operator can connect an output to its application version, inspect the permitted trace content, apply the rubric and identify a disagreement. Try the failure investigation the operator will actually perform. Save the observed result and limitations; do not replace that evidence with a vendor feature checklist.

For handover, ask the customer operator to rerun the agreed cases, explain a failed result and locate the relevant production trace. Record who can change the rubric, who approves a release, how data is retained or removed, and what happens when consultant access is withdrawn. If self-hosting, add a customer-run restore exercise appropriate to the chosen architecture.

These are proposed acceptance checks, not StackScored test results. A documented export or access feature is only a starting point: verify that the required artifacts and permissions work in the purchased environment. Keep the dataset, evaluation rules and operating instructions available to the client under the agreed data policy.

Before closing the engagement, rehearse a permitted export of datasets, rubrics, trace references and evaluation results, then verify what another operator can actually import or read. Record any missing associations or history. Revoke consultant keys, rotate affected service credentials and verify ingestion still reaches the client's project. For self-hosting, agree backup and deletion coverage across databases, object storage and replicas; removing one visible trace does not establish removal of every stored copy.

Review the collection records for Langfuse and LangSmith before finalizing the hosting and entitlement decision.

Use the FDE toolkit guide to place tracing and evaluation in the wider delivery lifecycle. For the workflow runtime and connection handover, read n8n vs Make vs Zapier for client deployments.