Best AI Agent and LLM Evaluation Tools in 2026

Five jobs hide behind the word evals

Evaluation means five different jobs depending on who is asking. An engineer means an offline test suite that runs before a change merges. A team lead means replaying last month's real runs through the new prompt. A product manager means a model grading outputs against a rubric, known as LLM-as-judge. An operations lead means a queue where a domain expert marks answers right or wrong. A platform owner means a CI gate that blocks a prompt or model change when scores drop. Few tools do all five well, so know which job you are hiring for.

The market also moved this year. OpenAI bought Promptfoo, Cisco bought Galileo, and OpenAI is shutting down its own Evals platform on 30 November 2026, so older lists send readers to products that have changed hands or are closing.

I build and run production AI agents for operational teams, so my bias is towards whatever makes regression testing against real past runs cheap and routine. I have declared it, and I have tried to be fair to the others. No vendor has paid to be here, and every price was checked on the vendor's own page on 25 September 2026, so check again before you budget.

1. LangSmith

LangSmith is LangChain's tracing and evaluation platform, and it works outside LangChain too. You build datasets from curated cases or historical production traces, run experiments, and score results with code checks, LLM-as-judge, pairwise comparisons or human reviewers working through annotation queues. A pytest or Vitest integration turns eval cases into tests that can fail the build.

It wins for teams already on LangChain or LangGraph, where the whole loop sits in one place with no extra wiring. It is the wrong buy when many occasional reviewers each need a seat, or when trace volume is high and cost matters. The Developer plan is free for one seat and 5,000 base traces a month, and Plus is 39 dollars per seat per month with 10,000 included. Beyond that, LangSmith's own calculator works out at 5 dollars per 1,000 base traces kept for 14 days, against 50 cents per 1,000 on its 2025 price list. The new Engine feature, which scans traces and proposes fixes, is metered on top.

2. Braintrust

Braintrust is built around the experiment. You run a task over a versioned dataset, score each output with code, LLM-as-judge or human review, and compare it with any earlier experiment to see which cases improved and which regressed. Datasets can be built straight from production logs, and its GitHub Action runs evals on every pull request, posts the scores as a comment and can gate the merge.

It wins for product teams where engineers and product managers both need to see whether a change helped. It is the wrong buy for a solo developer, for whom a free command-line tool does the job, and the step from free to paid is steep. Starter is free with unlimited users, 10,000 scores a month, one human review score per project and 14-day retention. Pro is 249 dollars a month with 50,000 scores and 30-day retention, then usage charges, and self-hosting needs an Enterprise quote.

3. Promptfoo

Promptfoo is an open-source command-line tool under the MIT licence. Prompts, providers and test cases live in a config file next to your code, assertions range from exact matches to LLM-graded rubrics, and its GitHub Action runs a before and after comparison on any pull request that edits a prompt. It also does automated red teaming, the security work OpenAI said it was buying.

OpenAI announced the deal on 9 March 2026, and Promptfoo now describes itself as part of OpenAI. For users, the code stays MIT licensed and runs locally, Promptfoo says it will keep supporting other providers, and OpenAI names it as the migration path for its own Evals customers. It wins for engineering teams who want evals in the repository at no licence cost, and the free Community edition includes 10,000 red-team probes a month. It is the wrong buy if non-engineers need a hosted review workspace, which is Enterprise on a custom quote, or if procurement requires a harness no model vendor owns.

4. DeepEval and Confident AI

DeepEval is an open-source framework under the Apache 2.0 licence that plugs into pytest, with TypeScript support, so evals run with one command locally or in CI. It ships dozens of metrics for RAG, agents and conversations, plus G-Eval for criteria written in plain language. Confident AI, from the same company, is the hosted layer for test reports, run-to-run regression comparisons, datasets curated from production traces and annotation queues.

It wins for Python teams who want unit-test habits for LLM output without touching a vendor. It is the wrong buy if you would trust the default judge metrics as they come, because any LLM-graded metric needs checking against human labels before it gates a release. Confident AI swapped per-user plans, from 19.99 dollars per user per month as recently as March 2026, for flat organisation pricing. Free covers two seats, one project and five test runs a week, Starter is 200 dollars a month and Team is 2,000 dollars a month.

5. Arize Phoenix

Phoenix is Arize's self-hostable tracing and evaluation tool, built on OpenTelemetry. You group traces into datasets, rerun them through a new version of your application and compare scores from LLM judges, code checks or human labels, which is replay regression by design. Arize AX is the managed platform on the same standards.

It wins for teams that want tracing and evals without sending data to a vendor. Its licence is Elastic's ELv2, which makes it source-available rather than open source in the strict sense, and the main limit is that you may not offer Phoenix to others as a hosted service. It is the wrong buy if nobody owns the deployment or your legal team only approves OSI licences. AX Free includes 25,000 spans a month with 15-day retention, AX Pro is 50 dollars a month for 50,000 spans and 30 days, and Enterprise, including self-hosted AX, is a custom quote.

6. Galileo

Galileo's pitch is that today's evals become tomorrow's guardrails. You build datasets from development and production data, capture expert annotations, tune LLM-as-judge evaluators and distil them into its own small Luna models, which it says are cheap enough to score all live traffic. On Enterprise they also run as real-time guardrails.

Cisco announced it would buy Galileo on 9 April 2026, completed the deal on 22 May and is adding it to its Splunk Observability portfolio. Standalone pricing is still published. Free covers 5,000 traces a month, Pro starts at 100 dollars a month billed annually for 50,000 traces, and Enterprise with VPC or on-premises deployment is a custom quote. It wins for larger organisations that want offline evals and production guardrails from one vendor, especially Splunk users. It is the wrong buy if you want an independent specialist, because the roadmap now belongs to a much larger product line.

7. Inspect

Inspect is a free, MIT-licensed Python framework from the UK AI Security Institute and Meridian Labs, built for frontier model evaluations. It gives you datasets, solvers, scorers and agents as building blocks, runs untrusted model code in sandboxes such as Docker, supports more than 20 model providers, and ships over 200 ready-made evaluations and a log viewer.

It wins when the question is whether a model or agent can do a task well and safely, for example comparing two models on your own agentic tasks before you switch. Your only cost is model calls. It is the wrong buy for a product team that needs production tracing, dashboards for non-engineers or review queues, because it is not built for those.

8. Patronus AI

Patronus sells evaluation as an API. You call its hosted evaluator models to grade outputs, track experiments and traces on its platform, and use Percival, an agent that reads agent traces, detects more than 20 failure modes and suggests fixes.

It wins for teams that want hosted judge models without calibrating their own. The Developer tier is free with 10 dollars of credits and two weeks of history, then 10 dollars per 1,000 small evaluator calls, 20 dollars per 1,000 large ones and 10 dollars per 1,000 explanations. Enterprise is a custom quote. It is the wrong buy if you want a broad platform with flat pricing. Note the direction, too. Its 50 million dollar Series B in June 2026 went into simulated environments for training and testing agents, and it says it works with most of the leading frontier AI labs.

Replay regression, the test that catches what unit tests miss

Replay regression means testing a new prompt or model against a set of real past runs before it ships. You store the inputs and outputs of production runs, feed the same inputs through the candidate version, and diff the new outputs against the old. Every divergence is either a change you intended or a regression you have just caught.

This is how I work on every agent I run. On a payroll pipeline, a replay suite over real past runs caught a rule divergence before it could mis-pay a live run, after every unit test had passed, because the failing case only existed in real history. I log every run with its full input and the version that produced it, replay the whole set in a sandbox on every change, and have a domain expert sign off each divergence against a written list of intended changes before anything is promoted.

Two limits matter. Replay only covers cases you have already seen, and the old outputs may themselves be wrong, so pair it with a periodic check where an expert marks a sample of past outputs right or wrong. And mask personal data before stored runs leave the production boundary.

Tools make this cheaper, not unnecessary. LangSmith, Braintrust, Phoenix, Confident AI and Galileo can build datasets from production traces and compare runs, and Promptfoo can load exported runs from a file and compare before and after in CI. Several free tiers keep traces for only about two weeks, so keep your own copy of the runs you want to replay.

What changed in 2025 and 2026

OpenAI is retiring its Evals platform. It gave notice on 3 June 2026, existing evals become read-only on 31 October, and the Evals dashboard and API are scheduled to shut down on 30 November 2026. OpenAI's own migration guide points to Promptfoo, and the original open-source OpenAI Evals repository has had little more than maintenance commits since late 2024.

ClickHouse bought Langfuse, a widely used open-source tracing and eval platform, in January 2026 and says it stays open source and self-hostable. Humanloop, once a regular on these lists, sunset its platform in 2025 after its team joined Anthropic.

Ragas, the RAG metrics library many older guides recommend, is still Apache 2.0 licensed, but its last release was in January 2026 and its repository has had no commits since February. Its makers now run Vibrant Labs, which builds post-training data for AI agents, so use its metrics inside other tools rather than building a practice around it.

Prices moved too. LangSmith's per-trace overage is now ten times its 2025 list price, and Confident AI swapped per-user plans for flat monthly ones.

Comparison at a glance

How to pick

For a solo developer or a team of two or three, start free and in the repository, with DeepEval if you write Python and like pytest, or Promptfoo if you want config files and red teaming. Write 20 real cases with a domain expert before you add a single metric, because the cases are worth more than the tool.

For a product team of five to fifty where product managers need to see results, pay for a shared workspace. Braintrust fits if comparing experiments is the daily work, LangSmith if you already build on LangChain. If data must stay in-house, self-host Phoenix or Langfuse.

For large organisations, Galileo fits if you want evals that become guardrails and already run Splunk, Inspect fits research and model-selection work, and Patronus suits teams that want hosted judges billed per call.

Whatever you buy, the gate that protects a live operation is the same. Replay real past runs through every change, route each divergence to a person who knows the work, and let nothing promote without sign-off.