tests/test.sh script is the entrypoint for the verifier. Harbor uploads tests/ to /tests/ after the agent runs, unless a dedicated verifier image or build definition provides the tests. tests/test.sh is executed in the task’s working directory and is responsible for verifying the task’s completion. It must produce a numerical reward at /logs/verifier/reward.txt or /logs/verifier/reward.json.
Required script
Install test dependencies in
tests/Dockerfile (or the agent image for shared verification). The script should:
- Verify that the agent satisfied the instruction
- Write a reward under
/logs/verifier/
cwd surprises.
Reward files
If both are present, Harbor prefers
reward.json.
Example
tests/test.sh
reward.json file.
Separate verifier environment
Tasks created withharbor task init or harbor init --task configure a separate sandbox for verification. Omitting both verifier.environment_mode and [verifier.environment] uses shared verification.
To opt into shared verification, set environment_mode = "shared" under [verifier] in task.toml, omit [verifier.environment], and install verifier dependencies in the agent image.
Harbor prefers [verifier.environment].docker_image, then a build definition in tests/ (Dockerfile or docker-compose.yaml), over the agent environment. These dedicated verifier images must include /tests/test.sh or /tests/test.bat; Harbor does not upload tests into them at runtime.
Without a dedicated verifier image or build definition, Harbor starts a fresh copy of the agent environment and uploads tests/ to /tests/. The agent’s filesystem changes are not inherited. For multi-step tasks, step verifier definitions take precedence over task-level definitions.
Declare artifacts in the top-level artifacts list in task.toml, before any section headers. Harbor collects them before verification and copies them into the separate verifier sandbox at their original source paths.
See Separate verifier for more details.
Example
task.toml
tests/Dockerfile
Regrading
Harbor can rerun verifiers for existing trials if the task uses a separate verifier environment. This is useful when iterating on a verifier. See Regrade for details.Passing environment variables to the verifier
Environment variables can be passed to the verifier using the[verifier.env] section in task.toml.
task.toml
${VAR} syntax is used to read from the host environment. Harbor will ask the user to confirm before passing the environment variables to the verifier.
LLM- or agent-as-a-judge
Task authors can implementtest.sh to use whatever verification approach they like, including LLM- or agent-as-a-judge.
To avoid boilerplate, consider using RewardKit to define your judge or programmatic criteria.
rewardkit
The Harbor team maintains a package called harbor-rewardkit. It is the easiest way to define and run common verifiers, including programmatic criteria and LLM- or agent-as-a-judge.