Tools · Agent observability
AI agent and LLM observability tools
These tools record what an agent actually did: each prompt and response, each tool call and its result, and the data it retrieved, and some add the latency, tokens and cost of each step. Most also run evaluations against those traces to catch failures and regressions before and after a release. A team needs one when agents run in production, because without a trace there is no record to check when an agent does something nobody asked for, and incident review and audit both start from that record.
Agent observability 9
-
Arize AX (vendor site, opens in a new tab)
generally availableCollects production traces of agent behavior and runs evaluations and experiments on them, surfacing failures and root causes for review.
-
Arize Phoenix (vendor site, opens in a new tab)
generally availableOpen-source tool that traces LLM and agent runs with OpenTelemetry instrumentation and evaluates them with datasets and experiments. It runs locally or self-hosted.
-
Braintrust (vendor site, opens in a new tab)
generally availableTraces each step of production agent calls, including prompts, responses, tool calls and retrieved context, with latency, tokens and cost per step, and scores live traffic with the same scorers used in offline experiments.
-
Datadog Agent Observability (vendor site, opens in a new tab)
generally availableDatadog
Traces AI agent requests across prompts, retrieval steps, tool calls and agent decisions, evaluates agents in production, and tests changes against datasets built from annotated traces before release.
-
Galileo (vendor site, opens in a new tab)
generally availableGalileo
Evaluates AI apps and agents with tuned evaluators, then runs those evaluators as guardrails that monitor production traffic.
-
Langfuse (vendor site, opens in a new tab)
generally availableOpen-source platform that records traces of LLM calls, retrieval and agent actions, runs evaluations against them, and manages prompts. It can be self-hosted.
-
LangSmith Observability (vendor site, opens in a new tab)
generally availableTraces each step an agent takes, from framework integrations or OpenTelemetry SDKs for Python, TypeScript, Go and Java, and monitors cost, latency and failures in production.
-
Opik (vendor site, opens in a new tab)
generally availableComet
Open-source platform that logs each step an agent takes, including user interactions, retrieval and tool calls, with automated evaluations, prompt management and production monitoring.
-
W&B Weave (vendor site, opens in a new tab)
generally availableWeights & Biases on the vendor map
Monitors production agent behavior end to end, surfaces failure modes, and runs an evaluation framework to catch regressions.
Recent launches
- Kong Advanced AI Observability · announced
- Claude Compliance API: local session transcripts (Anthropic) · generally available
Announced in the last 90 days. Each launch on New tools links the vendor's own announcement.
The briefing
New AI agent governance tools, by email
The briefing's New Tools section covers the latest governance launches, each linked to the vendor's announcement, next to the news that decides which ones matter. At least once a week, five-minute read, free.
You confirm by email. Unsubscribe in one click.
Questions to ask vendors
Neutral questions for a request for proposal (RFP) or a security questionnaire. No product here is named as the answer.
-
Does the log capture the full tool call and the full response, and can we export it to our existing security tooling?
-
When an agent acts for a user, does the log capture both the agent and the user it acted on behalf of?
-
Can an agent, or code it writes, change or switch off its own tracing, and would anyone be told?
-
How is sensitive data in prompts, tool results and retrieved documents redacted before traces are stored?
-
Where are traces stored and for how long, and can we host the tool ourselves?
-
Are traces in OpenTelemetry format, so they can move with us if we change tools?
More questions, grouped by topic: Questions to ask AI agent security vendors.
Related incidents
- OpenAI said two of its coding agents weakened misalignment monitoring during routine work, and one denied it
- UK AI Security Institute found agents taking unsanctioned real-world action in 10 of 122 test runs
- OpenAI agents posted 53 user images to public image hosts without the lab knowing
- CERT-In said AI agents under evaluation read personal and confidential records at Indian organizations
Every record in the AI agent incident database links the sources that reported it.
Missing a product, or found an entry that is out of date? Write to [email protected]. A submission is read, never paid for, and does not guarantee a listing. The glossary defines the terms used here.