Observability Kit

The Observability Kit makes agent work measurable and debuggable. It covers usage, spend, feedback, replay, errors, traces, run diagnostics, and the app UI that operators use to understand what happened.

Pieces

Piece Purpose
<ObservabilityDashboard> Standard operator dashboard.
Usage and spend Workspace/app-level model usage and cost views.
Feedback User thumbs, comments, and product feedback loops.
Replay Inspectable agent run and UI/event timelines.
Errors Surfaced runtime, provider, and action failures.
Run diagnostics Tool calls, model choice, context, approvals, and recovery state.
Tracking Server-side analytics events with pluggable providers.

Usage

Mount the dashboard on its own route, and drop <ThumbsFeedback> next to any assistant message you want feedback on — both write into the same usage/spend/feedback data the dashboard reads back:

import {
  ObservabilityDashboard,
  ThumbsFeedback,
} from "@agent-native/toolkit/app/observability";
export default function ObservabilityPage() {
  return (
    <div className="min-h-screen bg-background p-6">
      <ObservabilityDashboard />
    </div>
  );
}

export function AssistantMessageFooter({
  threadId,
  runId,
  messageSeq,
}: {
  threadId: string;
  runId: string;
  messageSeq: number;
}) {
  return (
    <ThumbsFeedback threadId={threadId} runId={runId} messageSeq={messageSeq} />
  );
}

The dashboard has 5 tabs (overview, conversations, evals, experiments, feedback) and scopes all data to the signed-in user.

UX Standard

  • Every background or agent run should be traceable to a user-visible outcome.
  • Feedback should connect to the run, resource, or page that caused it.
  • Spend and usage should distinguish app, organization, user, and provider.
  • Error states should link back to setup when the fix is a missing connection.

What's next

  • Observability — the full data model and routes behind this dashboard.
  • Tracking — server-side analytics events with pluggable providers.
  • Progress — the run-progress surface <RunsTray> also reads.
  • Agent UX Kit — the composer and runs tray this kit's diagnostics tie back to.
Edit this page on GitHub