What the Agent Test measures
The Agent Test is tool-selection accuracy — for each real-world site-owner request, did the agent call the correct purpose-built tool? It does not score answer quality, writing quality, or how the request was phrased back to the user — only whether the agent reached for the right tool to do the job.
The prompt catalog is the plugin's bundled “most popular prompts” list — 54 real owner requests, one per common ask, spanning the free bundled agents:
What owners actually ask
These are real requests straight from the Agent Test catalog — the kind of plain-English asks a site owner types in. Each one is what the benchmark actually measures: does the agent reach for the right purpose-built tool to handle it?
“Give me a plain-English health check of my whole site.”
“Is my site secure? I'm worried it might have been hacked.”
“My site doesn't show up on Google at all. What's wrong?”
“My media library is huge. What's taking up all the space?”
“Who has admin access to my site? I want to check nobody shouldn't be there.”
“I've got loads of plugins and no idea what half of them do. Which ones can I get rid of?”
Why a “wrong” tool call rarely means a mistake reached your site
Agent Builder runs in supervised mode by default: medium- and high-risk tools stop and wait for the site owner's approval before anything executes. In the Agent Test, a call that correctly stops for approval is recorded as a gated pass, not a failure — that pause is the safety feature working as designed, not the agent getting it wrong.
Methodology & test conditions
| Plugin tested | Agent Builder v4.0.0 + current main fixes |
|---|---|
| Test date | September 26, 2026 |
| Model | Gemini 2.5 Flash (hosted provider) |
| Environment | A real production WordPress site |
| Prompt catalog | 54 real-world, bundled “most popular prompts” — one per common owner request |
| Runs | 4 independent full runs (pass counts of 54: 50, 49, 48, 48) |
The model's tool choice is non-deterministic — the same prompt can select a different tool on different runs. That's why we report multiple runs and a stability view below rather than a single pass/fail number.
Stability across 4 runs
Of the 54 prompts, here is how consistently each one resolved to the correct tool:
- Always pass (0/4 fail) — 43 of 54
- Flappy (1–3/4 fail) — 9 of 54
- Frequent (4–5/4 fail) — 1 of 54
- Always fail (4/4 fail) — 1 of 54
43 of 54 prompts pass in every run. One — a content-writer step that doesn't reliably reach the featured-image tool — fails consistently and is on our fix list; about 9 more sit at a genuine tool-choice boundary and flap between runs, which is inherent LLM non-determinism at a close call.
Read this number honestly
- This is tool-selection accuracy, not an overall quality or satisfaction score.
- N=4 runs, range 88.9%–92.6% — the model is non-deterministic, so a single run can land anywhere in that range.
- We do not claim 100%, publish per-agent scores we haven't verified against the raw run data, or compare against other products.
- This page updates as the plugin and its prompts improve — see the history below.
History
Like the changelog, this benchmark is re-run and republished as Agent Builder improves.
90.3% mean accuracy · range 88.9% – 92.6%
Rounds 3–4 tuning: sharpened tool selection for form submissions, account lockout, thin-content detection, and meta-description follow-through. 43 of 54 prompts now pass every run.
88.6% mean accuracy · range 81.5% – 92.6%
First published run of the Agent Test — baseline for future releases.
See it for yourself
Download the free Agent Builder plugin and run your own Prompt Tests against the same catalog, right from your dashboard.
Download free Agent Builder