Monitor
Catch every AI agent failure in production
Moda monitors every trace your AI agents run, flags the failures that never throw an error, and groups them into problems ranked by the users they affect.
Signals
Per trace negative rate
6.8%
−1.2 pts vs last 7 days
2,915 of 42,870 traces
Affected users
1,204
−86 vs last 7 days
with ≥1 negative reaction
Frustration
1,318
Sadness
80
Confusion
742
Anxiety
148
Trust
236
Redirection
391
Positive
2,064
Order status repeated after the user said it was wrong
Frustration4m ago
Return window explained two different ways
Confusion12m ago
Asked for a human after a failed refund
Frustration31m ago
Doubted a delivery date the agent kept changing
Trust1h ago
What is AI agent monitoring?
AI agent monitoring is watching what your agents actually do for users in production, one run at a time. Uptime, latency, and token cost still matter, but agents usually fail without an error: a user gets frustrated, a tool call fails and the agent answers anyway, a task is reported done when it was not. Monitoring an agent means reading each trace end to end and judging the outcome, not only the status code. Monitoring finds failures after release; agent evals test a change before it ships.
How it works
From every trace to the problems that matter
Monitor every trace
Moda reads every production trace end to end and detects the failures that never throw an error: frustrated users, failing tools, lazy answers, and hallucinations. Add your own checks in plain language and get alerts in Slack, Linear, or email.
Trace
User
Where is my order? It was due on Monday.
Tool
order_lookup → error: timed out after 30s
Tool failureTool
order_lookup → status: not shipped, ships Friday
Agent
Good news, your order shipped and arrives tomorrow.
Contradicted claimUser
That's what you said yesterday. Get me a human.
FrustrationAgent
I can't transfer you from here.
Claimed inability4 signals · part of Order lookup times out under load
Find the mistakes they keep repeating
Failures are grouped into problems ranked by how many users they affect. Each problem shows the traces behind it, the turn where it went wrong, and related problems that share the cause.
Problem detected
Fabricated data after tool call
6735users affected
Fabricated data after tool call
When a lookup comes back empty, the agent fills in details from a look-alike record instead of saying it found nothing, so users get order numbers and dates that don't exist.
6735
Users affected
244
Linked traces
Linked tracesRelevancy
Quoted a tracking number the order lookup never returned
High
Gave a delivery date copied from another customer's order
High
Confirmed a refund total with no matching invoice
High
Listed opening hours for a store the search didn't find
Medium
See what users ask your agent to do
Moda splits every trace into user goals and groups them into use cases, kept current as your traffic shifts. Open a use case to see its volume, its trend, and the problems surfaced inside it.
Use cases
Orders and delivery
38%
16,290
Track a late order14%
Change a delivery address9%
Refunds and returns
24%
10,288
Account access
17%
7,288
Product questions
12%
5,144
Signals
Detections on every trace, out of the box
User emotion
16 emotions, from frustration and confusion to distrust and delight, read from the user’s own words, with what caused them and whether the agent recovered.
Tool failures
Every tool call is recorded as it arrives. Failed calls are grouped into named categories with a likely cause, per tool and over time.
Laziness
Stubbed code, skipped tool calls, promises with no action behind them, and work handed back to the user.
Hallucinations
Replies that told the user something the trace shows is false: a refund confirmed after the refund tool rejected it, or a count that contradicts the tool output.
When a successful run still fails the user
A tool can return successfully for the wrong order. An agent can confirm a refund that is still pending. Read the guide to silent agent failures for worked examples, evidence checks, and a production review checklist.
Choosing a tool? Compare LLM observability tools, from Langfuse to Datadog, on how each one finds failures and tests a change.
Tailored to your workflow
Custom signals
Judge
Agent gives time of day and attaches wrong time zone
3.2%
−0.8 pts vs last 7 days
Hit Rate
1.7K
Hits
54.5K
Traces Judged
Recent hits
Told a customer in Berlin the store opens at 9:00 PT
Gave a Sydney customer the current time in UTC
Booked a 3pm EST callback for a customer in London
Said it was 6pm using the server's time zone
Monitors
Trigger
Agent gives time of day and attaches wrong time zone
Slack alert
Alert
Mention
When the hit rate is above
for an hour
Conditions