Arize AI

How do you know if an agent change actually made things better? Join us on Oct 8 at 11am PT / 2pm ET for a webinar on benchmarking AI agents and tool use with Harbor and Arize Phoenix. We’ll cover how to compare agents or models over the same task set, track results across repeated benchmark jobs, separate behavioral scores from infrastructure failures, and inspect ATIF traces for scored runs. Register here: https://lnkd.in/g4Nbt9Xn

OpenAI just shipped the Decisions API (beta) and it's now traced in our OpenInference instrumentors in Py and TS • predicate → probability • choice → one of your labels • score → rubric level $0.10/1M input tokens. No output charges. https://lnkd.in/dpS2d7Jh

We benchmarked 8 decision models across hallucination detection, 13 Phoenix evaluator suites, and agent routing. 3 weeks after Jev’s release, the top decision models are a commodity on accuracy. Jev, Clef 27B and Liquid sit within a point of each other on hallucination, and within the noise of an LLM judge that costs 60 to 400 times more. Even the cloud final couldn’t separate Jev and Liquid. Kev, an open clone whose first training run cost about $95 in rented GPUs, ties Jev in the agent challenge. Full benchmark: https://lnkd.in/gaSmDNDJ

Your agent failed. Make that your next test. Join Fuad A., Staff AI Product Manager at Arize AI, on October 12 for "Agents that Self Improve While You Sleep." Start with production traces and use Arize's Signal agent to find failures. Turn a failure into a dataset and an eval, then use the eval as a regression test for the fix. Fuad will also cover where to add human review. AI Engineer New York, October 12 to 14 Sheraton New York Times Square Workshops are an add-on to a conference pass. Tickets: https://ai.engineer/nyc

As agents gain capabilities due to skills, MCPs, and codemode, evaluating agents becomes increasingly more difficult because you need to test across multiple environments and harnesses. Arize Phoenix your benchmarking Harbor runs so pinpointing improvements and regressions becomes tractable. Come join Elizabeth in our free webinar: https://lnkd.in/gwnwdbFA

An agent experimentation loop is a repeatable way to answer one question: did this change actually improve the agent enough to ship? The workflow starts with a fixed, versioned test set. Run the control and candidate on the same cases with the same evaluator, then use a separate CI suite to catch regressions outside the behavior you meant to change. Arize AI If the candidate survives, test it in production. Keep assignment stable for the session and propagate experiment.id and experiment.variant through the trace so you can compare quality, latency, cost, tool use, and failures between arms. Then feed reproducible production failures into the next test-set version. Get the full walkthrough: https://lnkd.in/gSDzyusy

Here's a useful rule for evaluating agents: use deterministic checks when the trace can prove the failure. Reach for an LLM judge when the question actually requires judgment. Ashwin Govind Ugale built tracelint around that idea. It catches tool errors, invalid arguments, repeated calls, loops, and other structural failures directly from agent telemetry. It's also available on GitHub. We were glad to work with Ashwin to bring this Phoenix community contribution to the Arize blog. Read more: https://lnkd.in/gWKinMyE

Ask an LLM judge a yes/no question and it often writes you a paragraph. Your pipeline parses it, hopes the label landed where expected, then makes another call for the next question. That is a lot of machinery for what's really a classification task. Jev-as-a-Judge is a new evaluator type in Arize AX, built on TypeSafe AI's Jev decision model. Jev classifies and scores. It doesn't generate text, so it answers in types: - Boolean questions return true/false with a confidence score - Choice questions pick one of 2 to 255 options, with confidence - Score questions place a trace on a rubric of 2 to 10 levels Several typed questions run against a trace in one call, with no free-text explanation to parse. Keep LLM judges for the evals that need reasoning in words. Use Jev where the answer is a label. It runs on your own TypeSafe AI account, connected in AX settings. Set it up: https://lnkd.in/gJ6tAaiQ