Evals and Observability for AI Agents: The 2026 Tooling Map

If you can't replay what your agent did and score whether it did it well, you don't have a product — you have a demo. The tooling splits into three jobs: tracing (what happened, step by step), evals (is it good, measured against a dataset), and monitoring (is it getting worse in production). As of this writing, LangSmith is the default in LangChain and LangGraph shops, Langfuse is the open-source standard you can self-host, Braintrust has the best eval workflow for mixed PM-and-engineer teams, Arize Phoenix is the OpenTelemetry purist's pick, and Promptfoo owns config-as-code evals and red-teaming in CI. The full map, with pricing and skip-if notes, is below.

Three jobs, one bill

Most vendors here now sell all three jobs in one platform, which makes comparison shopping confusing on purpose. Untangle it by asking what you need this quarter. Pre-launch, you need tracing (to debug) and offline evals (to iterate against a golden dataset). Post-launch, you add online monitoring — sampled scoring of live traffic, cost and latency alerting, drift detection. Agents raise the stakes on all three because a single request can span dozens of model calls and tool invocations: without step-level traces, a failure is a haystack, and without trajectory-level evals, you're grading essays by their last sentence.

The 2026 directory

Tool-by-tool verdicts

LangSmith

LangSmith offers the deepest integration with LangGraph — traces mirror your graph, and datasets, annotation queues, and online evaluators live in one place.

Verdict: the path of least resistance for LangChain shops, and genuinely good beyond them.

Skip if: self-hosting on a budget is a requirement — that capability is gated to enterprise plans, as of this writing.

Langfuse

Langfuse became the open-source default by doing the fundamentals well: solid SDKs across frameworks, prompt management, eval scores, and a self-host story that actually works.

Verdict: the neutral choice — no framework lock-in, no hostage pricing, MIT-licensed core.

Skip if: you want the slickest managed eval-authoring UX — commercial rivals still lead there, though the gap narrows each quarter.

Braintrust

Braintrust treats evals as the primary object, not a tab: datasets, experiments, side-by-side diffs, and a playground PMs genuinely use without engineering hand-holding.

Verdict: the best eval loop in the business for mixed teams that iterate daily.

Skip if: you mainly need cheap production tracing — you'd be paying for an eval workflow you don't run.

Arize Phoenix

Phoenix is fully open source and built natively on OpenTelemetry, which makes it the least lock-in option on the page — instrument once, keep your options.

Verdict: the OTel purist's pick, and free is hard to argue with.

Skip if: you need managed alerting, SSO, and support without stepping up to Arize's commercial platform.

W&B Weave

Weave extends Weights & Biases — now part of CoreWeave — into LLM tracing and evals, with the experiment-tracking DNA that ML teams already trust.

Verdict: natural for teams already logging to W&B; little pull for anyone else.

Skip if: you're not a W&B shop — adopting it standalone buys you nothing special.

Helicone

Helicone is a proxy: change your base URL, get logging, cost tracking, caching, and rate limiting on every model call across providers.

Verdict: the fastest observability win available — ten minutes to full cost visibility.

Skip if: you need step-level agent traces and trajectory evals — a gateway sees requests, not reasoning.

Promptfoo

Promptfoo does declarative eval configs that run in CI and a red-teaming suite that probes for injection, jailbreaks, and data leakage — evals as code review, not dashboard.

Verdict: the best guardrail against silent regressions, and the red-team tooling is the quiet star.

Skip if: you want a hosted trace store and team UI — that's not the shape of this tool.

DeepEval / Confident AI

DeepEval gives Python teams pytest-style LLM tests with a large metrics library — hallucination, RAG faithfulness, task completion — and a cloud dashboard when you want one.

Verdict: the natural on-ramp for engineers who want evals inside their existing test suite.

Skip if: your team is not Python-first — the pytest ergonomics are the product.

AgentOps

AgentOps specializes in agent session replay — waterfall views of tool calls, costs per run, and integrations with CrewAI, AutoGen, and friends out of the box.

Verdict: purpose-built agent debugging with minimal setup; a focused tool, not a platform.

Skip if: you need a full eval suite in the same product — pair it with Promptfoo or DeepEval instead.

Galileo

Galileo targets enterprise AI programs: eval metrics at scale, guardrailing in production, and the compliance posture large buyers require.

Verdict: built for the procurement-and-governance buyer; strong, but sized for enterprises.

Skip if: you're a small team — the value density starts at enterprise scale.

Custom evals: the part you cannot buy

Every platform above hosts evals; none can write yours. The working recipe, in order:

1. Read 50 real traces manually before buying anything. Most teams discover their actual failure modes — wrong tool, premature stop, silent context loss — in the first twenty.
2. Build a golden dataset of 20-200 cases from those traces, weighted toward failures. Real cases beat synthetic ones every time.
3. Score trajectories, not final strings: task completion, tool-call correctness, steps-to-success, and cost per solved task are the metrics that predict user experience.
4. Use LLM-as-judge with discipline — calibrate against 30-50 human labels, keep each rubric narrow, and re-check the judge whenever you change models.
5. Wire the suite into CI and track pass rates across model versions; the week a provider silently updates a model is the week this pays for itself.

The OpenTelemetry note

OTel's generative-AI semantic conventions keep maturing, and at last check most tools above can ingest OTLP. Practical consequence: instrument with OpenTelemetry where you can, and your traces stay portable when you outgrow — or fall out of love with — your first vendor.

How to choose

• LangGraph stack: LangSmith, or Langfuse if self-hosting matters.
• Framework-agnostic, budget-conscious: Langfuse or Phoenix, plus Promptfoo in CI.
• Product team iterating on quality daily: Braintrust.
• Just need cost visibility this week: Helicone now, a real eval stack next quarter.
• Enterprise program with governance requirements: Galileo or Arize's commercial tier.

Your framework decision upstream shapes all of this — see the agent frameworks directory — and memory bugs are among the most common things traces catch, so read the memory directory alongside. The full decision tree lives in how to choose your agent stack.

This directory is re-checked monthly. Get the next update by email — new tools, pricing changes, zero filler.

Keep reading