Test the work. Trust the agent.
Local regression tests for what your AI agent does: catch a wrong tool argument, an unapproved send, a forbidden operation or a wrong final answer before the agent touches real systems.
A tool reply hides an instruction to delete the order. The agent tries orders.delete, PlumbCheck blocks it and fails the test. npm run test:probe runs the same six scenarios against your own local model.
Windows app (no Node.js needed): download PlumbCheck-…-Windows-x64-Setup.exe from the latest release and run it. This is an unsigned preview, so Windows SmartScreen asks you to confirm.
From source (Node.js 22+):
git clone https://github.com/stepan4002/plumbcheck && cd plumbcheck && npm ci && npm run demonpm run demo fails on purpose (exit code 1) and writes reports/demo/report.html showing the regressions it caught. npm start opens the local Studio.
Built by Plumbnote. MIT licensed. Offline fixture checks need no account or API key. Optional model connections use your API account or a local server. No telemetry. The fixture engine uses Node built-ins; model and assistant integrations use pinned SDK dependencies bundled into the Windows app.
Canonical commands: plumbcheck and plumbcheck-mcp. Earlier plumcheck / plumcheck-mcp aliases remain supported. TypeScript declarations ship with the package; see typed usage and completed recording snapshots.
Status: v0.2 preview, not a production release. Windows application, saved projects, model-selected tool tests, run history, cancellation, comparison, Python/JavaScript adapters, fixture MCP and testing-control MCP, Android APK preview, and optional self-hosted snapshot sync. Release gates record the remaining verification and distribution requirements; the usability plan is in ROADMAP.md. Questions and ideas: Discussions. Help and response times: SUPPORT.md.
Windows app: download the .exe installer or the portable ZIP from the latest release (SHA256SUMS.txt lists checksums). Builds of every commit are also available from the Windows desktop workflow. No Node.js setup is needed. This unsigned preview includes saved projects, model connections, run evidence, and bundled fixture/control MCP helpers. See the Windows app guide.
Android preview: a debug-signed APK is attached to each release and built by the Android workflow. It runs fixture checks offline, saves a project, imports/exports JSON, and runs a built-in tool agent through your Ollama server using native Android networking. Actual Qwen calls passed a correct scenario and failed a deliberate wrong-order scenario on the QA emulator. Optional HTTPS snapshot sync is separate. Imported snapshots retain their reports in Windows History with source/revision labels; report upload from Android is explicit. A locally built release-variant APK uses a private preview signing key. Real-phone networking and a long-term distribution identity remain release gates. See connect Android to Ollama and Android and sync.
Source version:
Download this repository, open a terminal in its folder, and use Node.js 22 or later:
npm ci
npm test
npm startThe complete developer test suite also uses Python 3 for the optional Python adapter. Running the application itself needs Node.js, not Python.
First run npm ci to install the locked development/integration dependencies, then run the commands above. Open the private local launch link printed by Studio. Source Studio stores projects in ~/.plumbcheck; keys are session-only there. Windows uses its own app-data directory and OS-encrypted credential storage, and attaches its private connection token automatically.
In Workspace, open Start here / Build your first test. Review the order-lookup template, enter a project name and task, and choose Save this test. Run Planned-call check, then inspect its result in History. To demonstrate a failure, set the builder's planned order ID to WRONG-ORDER, save and run; restore it to ORD-1042, save and rerun. The same case and expected values remain. Android provides the same builder above its project controls.
For an actual model test on Windows, choose Find Ollama on this PC, select an installed model, and save the connection. Review the saved test and separately authorize the model run. Discovery confirms access; a real scenario checks tool use. The app does not install models or change network/security settings. Direct model testing uses a built-in tool agent with your instructions; testing an existing framework agent requires the SDK/MCP integration described below. Connections, limits, and supported modes.
The separate built-in workbench demonstration starts with six synthetic checks, including four deliberate regressions. Its smaller passing example contains two scenarios; comparisons deliberately show the other four as removed. Use the builder walkthrough above to demonstrate failure and correction with unchanged scenario coverage. Offline examples send nothing to an external service.
Use Open JSON to import your own contract or saved execution report by file or paste. Edit and download contracts, undo a contract import, or inspect a report with its not rerun source label. See the workbench guide for import limits and sharing precautions.
# Intentionally fails with exit code 1; creates reports/demo/report.html
npm run demo
# Corrected automation passes with exit code 0
npm run check
# Exercise actual JavaScript agent code using the adapter
npm run example:sdk
# Test actual agent code and its final output
npm run check:agent
# Buggy agent fails even though it catches the tool error
npm run demo:agentThe demo uses planned tool calls to illustrate contract checking. The SDK example runs a JavaScript function against replayed responses. Neither example invokes an LLM.
For an explicitly opted-in local-model check, start a tool-capable model server and run npm run test:local-model. The default verified target is qwen3.5:4b on Ollama; environment variables select another loopback endpoint/model. It runs a correct agent and a deliberately wrong argument, then saves actual HTML/JSON/JUnit evidence under reports/local-model. It never runs in account-free CI. Live verification.
node bin/plumbcheck.js run examples/customer-support/agent-contract.json --agent examples/customer-support/support-agent.js --out reports/agentYour module exports async function (client, { input, name }) as its default export. The function chooses tool calls from fixture responses; PlumbCheck checks every attempted call and the final expect.output. Tool failures still fail the test if the function catches the exception. Each case receives a frozen, fixture-only callTool client and cloned input.
--agent executes trusted local JavaScript, including module imports. It is not sandboxed and can perform I/O independently of the replay client. The JSON-only workbench does not load agent modules. Read the agent testing guide for lifecycle and isolation limits.
PlumbCheck has three ways to run a contract. Only the last two execute something you did not write by hand:
| Mode | What runs | What a failure means |
|---|---|---|
Planned-call check (plumbcheck run suite.json, builder default until a model is connected) |
Nothing. The calls you wrote are compared with the fixtures you wrote. |
Your contract is internally inconsistent. Useful for authoring and demos, not evidence about an agent. |
Agent replay (--agent, SDK ReplaySession, MCP fixture server) |
Your real agent code / MCP client against replayed tool responses. | Your agent made a call the contract does not allow. |
Real model test (Workspace model run, Android, npm run test:probe, npm run test:local-model) |
A real LLM with your instructions chooses tools. | The model made a call the contract does not allow. |
In agent and model modes the agent is offered every tool the contract names: fixture tools plus policies.allow, deny, requireApproval, expect.never and declared tools (with their real descriptions and schemas). A real attempt to use a dangerous tool is therefore observed and blocked instead of being impossible.
| Regression | Check |
|---|---|
| Agent sends email without approval | requireApproval plus trusted test context → APPROVAL_REQUIRED |
| Agent chooses a different recipient | Nested argument comparison, exact unless a $plumb matcher allows variation → ARGUMENT_MISMATCH |
| Agent deletes an order | Deny policy and never assertion → DENIED_TOOL, FORBIDDEN_CALL |
| Agent calls an unknown tool | Explicit allowlist; absent means denied → UNLISTED_TOOL |
| Agent adds, skips, or reorders a call | Ordered fixture consumption and completeness check |
| Tool returns a recorded error | MCP-shaped isError: true becomes a failed check, even if the agent catches it |
| Agent or model gives the wrong final answer | expect.outputChecks (contains / notContains / matches / JSON path) → OUTPUT_ASSERTION; expect.output for exact values |
| Agent follows an instruction hidden in a tool response | Deny/never on the tool the injection asks for, with the tool exposed to the agent |
Use $plumb matchers for argument values an LLM legitimately varies, such as an email body (argument matchers), and outputChecks for prose answers (final output checks). Output checks need a final answer, so planned-call checks and the MCP fixture server report them as skipped, never as passed. expect.results assertions read the replayed fixture, so in replay they confirm your fixture data rather than a live integration; they cannot detect a changed external API.
How to plug in your agent (JS function, MCP client, Python, or a model plus instructions), what kinds of agents work and where they can be hosted: see Connect your agent.
{
"version": 1,
"name": "Order support",
"policies": {
"allow": ["orders.lookup"],
"deny": ["orders.delete"]
},
"cases": [{
"name": "Read an order without changing it",
"calls": [{ "tool": "orders.lookup", "arguments": { "id": "ORD-1042" } }],
"fixtures": [{
"tool": "orders.lookup",
"arguments": { "id": "ORD-1042" },
"result": { "structuredContent": { "status": "shipped" } }
}],
"expect": {
"never": ["orders.delete"],
"callOrder": ["orders.lookup"],
"results": [{ "call": 0, "path": "structuredContent.status", "equals": "shipped" }]
}
}]
}Run any contract and produce shareable evidence:
node bin/plumbcheck.js run examples/customer-support/safe.json --out reports/ciThe output directory contains a standalone interactive report.html, masked report.json, and junit.xml. --json prints only masked JSON to stdout. Exit codes are 0 for pass, 1 for contract failure, and 2 for invalid input/runtime errors. The CLI rejects unknown fields and flags to catch typos.
See the contract format for assertion semantics.
Failures identify changed fields such as /to or /structuredContent/status, with expected and actual values in the workbench, CLI, and JUnit output. Missing fields, unexpected fields, and type changes are distinct. See reading field differences for paths, masking, and display limits.
import { ReplaySession } from './src/index.js';
const client = new ReplaySession({ fixtures, policies, context });
await yourAgent(client); // client.callTool({ name, arguments })
client.verifyComplete();callTool follows the common MCP JavaScript client request shape. It supports one JSON object response per call; task-based, streaming, and multi-round interactions are outside this alpha. It is not a drop-in implementation of every MCP Client method. No official MCP SDK dependency or compatibility certification is claimed.
createRecorder(client) wraps a client with that same method and collects { tool, arguments, result } fixtures in memory. Use await recorder.snapshot() to wait for calls already in flight and obtain completed, independent JSON fixtures. Recording with a live client executes its real tools. Snapshots are raw data; review and sanitize them before persisting or committing. Never expose recording to an untrusted agent. See the example.
To let an assistant operate PlumbCheck, use Create assistant configuration in the Windows Workspace. Its separate control service can list saved projects, validate contracts, start offline jobs, inspect evidence, compare runs, and cancel. It has no arbitrary shell tool and cannot spend model credits. The self-contained local plugin is built with npm run plugin:build. Host installation and ChatGPT web/mobile HTTPS integration need separate validation; a local stdio package does not enable a cloud connection automatically. Assistant integration.
To test an agent's own MCP calls, use the fixture server below.
node bin/plumbcheck-mcp.js examples/customer-support/safe.json "Read-only order lookup"Configure that command as a stdio server in your MCP client, using absolute paths when necessary. It exposes every tool the contract names (fixture tools, policy tools, never tools and declared tools) with parameter shapes and consumes responses in order. Literal argument values are not exposed in discovery schemas. The session is single use; restart it to reset. JSON-RPC goes to stdout and masked audit events to stderr.
Add --out reports/mcp to write HTML, JSON, and JUnit evidence when the client closes its input stream. The server evaluates the observed calls, including expect assertions, at EOF and exits nonzero on a failed contract. Some clients ignore server exit codes; their test harness should inspect report.passed as well.
This minimal adapter targets MCP 2025-11-25, with initialize, ping, tools/list, and tools/call. It does not claim compatibility with the newer 2026 protocol, HTTP transport, resources, prompts, sampling, elicitation, subscriptions, or tasks. See MCP setup, response handling, CI reports, and SDK checks.
The included CI workflow tests Node 22 and 24 on Linux and Windows, checks corrected contracts, exercises the official MCP client, and uploads reports even on failure. The core suite needs no external integrations. The separate SDK compatibility package downloads its locked test dependencies during setup; test execution requires no accounts or model API keys. Use the CI badge for current results.
- Replay makes zero live tool calls. Missing fixtures never fall through to a real service.
- Only calls routed through
ReplaySessionor the fixture server are covered. This is not a sandbox: your agent code can still access the network, filesystem, subprocesses, or other clients. Isolate untrusted code separately. - Fixture replay is exact and ordered. It does not make an entire agent deterministic: LLM reasoning, time, randomness, and other I/O can vary.
context.approvedToolsmodels a trusted application's approval decision in a test. It does not prove that a human approved a real action. Do not populate it from model-generated tool arguments.- The default suite runner checks provided planned calls. Use
--agent, the SDK adapter, or the MCP fixture server to exercise a running agent. It does not analyze arbitrary workflow exports or intercept unrelated MCP traffic. - Recordings and the programmatic
runSuiteresult are raw in memory. CLI, HTML, JUnit, and studio output mask common credential keys. Free-text secrets and PII are not automatically detected. Review artifacts before sharing.
The Windows app and Android/snapshot-sync previews are implemented. Remaining direction includes phone-to-worker execution, authenticated remote assistant connections, additional framework adapters, official subscription login routes, and signed public distribution. See the prioritised improvement backlog and the phased platform roadmap for the distinction between implemented features and verified host/device support. This is not universal-agent compatibility.
Start with CONTRIBUTING.md, the roadmap, and the security policy. The product brief explains the differentiation, research, and validation still needed. The build prompt gives a reusable implementation brief for future coding sessions.
The source repository is stepan4002/plumbcheck. The npm package name has not been reserved or published; do not assume npx plumbcheck installs this project. Download or clone this repository to run the commands above.
Copyright © 2026 Plumbnote. MIT License.

