rewardkit: add REWARDKIT_MODEL override for agent judges#1778
Merged
alexgshaw merged 4 commits intoJun 8, 2026
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Contributor
|
Enjoy a better diff viewing experience by clicking one of these URLs: |
Collaborator
|
We only want |
Kurry
added a commit
to Kurry/harbor
that referenced
this pull request
Jun 1, 2026
…or-framework#1778 harbor-framework#1778 (benediktstroebl) is a minimal, isolated fix for the dropped-model-pin issue (Bug 2 of harbor-framework#1771), so remove that change here to keep this PR focused on the two Anthropic auth bugs: - Bug 1 — litellm LLM judge can now use the Claude subscription OAuth token (judges._resolve_anthropic_api_key), free-first for anthropic/* models. - Bug 3 — the claude-code agent judge drops a stray ANTHROPIC_API_KEY from the CLI subprocess env when an OAuth token is present (judges._agent_subprocess_env), so a container API key no longer shadows the subscription (Console bill / 401). Reverts the runner.py REWARDKIT_MODEL handling + its test, and trims the CLAUDE.md auth section to the credential-resolution behavior only. Both auth fixes remain behind the REWARDKIT_FORCE_API_KEY=1 opt-out. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Collaborator
Author
|
@alexgshaw please review |
alexgshaw
approved these changes
Jun 8, 2026
Messimeimei
added a commit
to Messimeimei/bitfun-harbor
that referenced
this pull request
Jul 7, 2026
700972a fix(gemini_cli): pre-select auth method for headless API-key/Vertex runs (harbor-framework#1794) 590f4f7 Fix installed agent ATIF exports (harbor-framework#1812) 9f65643 Update Islo SDK integration (harbor-framework#1824) 6cd496d fix(job): disable Rich stdout/stderr redirect to avoid deadlocks (harbor-framework#1829) 66cffed Suppress litellm debug output unless --debug is enabled (harbor-framework#1828) 48a577a Open PR diff links in a new tab. (harbor-framework#1831) ae938b8 feat: GKE multi-container support via privileged DinD compose (harbor-framework#1773) e1066b5 Add use.computer environment (harbor-framework#1834) 0f08f40 fix(pi): Reduce logging footprint (harbor-framework#1826) 9801da1 Update mini_swe_agent.py (harbor-framework#1837) 918a791 fix(check): accept all Claude auth methods, preferring the free subscription (harbor-framework#1766) 13f9692 fix(env): tar-based directory transfers for Modal, Daytona, and GKE (harbor-framework#1838) 8cfac6a Add Docker Compose support to Novita (harbor-framework#1830) c978013 harbor-langsmith: persist run tags (top-level) and dataset metadata (under extra) (harbor-framework#1808) a56546f Support wildcard network allowlist hosts (harbor-framework#1840) 6cd263a feat: add LangSmith sandbox environment (harbor-framework#1610) e75ecc0 rewardkit: add REWARDKIT_MODEL override for agent judges (harbor-framework#1778) 9c78b85 Update Claude Code bootstrap installer URL (harbor-framework#1852) 6a21701 Added cline v2 and ATIF Trajectory support (harbor-framework#1495) 6996f65 feat(claude_code): support CLAUDE_FORCE_OAUTH to bill the subscription on harbor run (harbor-framework#1846) 904b323 Add built-in LangGraph agent (harbor-framework#1807) b8976c9 Add GPU support to the Daytona environment (harbor-framework#1731) 2922220 Preserve tagged Docker images on delete (harbor-framework#1861) c2c5796 Improve use.computer environment (harbor-framework#1853) 3de07a0 add session-id header (harbor-framework#1855) 3f941bd ci: fix ai-review fork-PR checkout for claude-code-action@v1 (harbor-framework#1874) cdee965 Add network policy docs page under Tasks 253c546 Document environment support for network modes, phases, and capabilities 8d3384a Add Novita network policy support (harbor-framework#1867) 5270111 add procps to prevent process crashes when claude-code call tree-kill (harbor-framework#1864) 4195631 Add generic ACP registry agent support (harbor-framework#1464) 96eaf6b v0.13.2 Co-authored-by: Cursor <cursoragent@cursor.com>
vivek-vashistha
added a commit
to vivek-vashistha/harbor
that referenced
this pull request
Jul 9, 2026
* fix(env): tar-based directory transfers for Modal, Daytona, and GKE (#1838)
* fix(modal): tar-based directory transfers to stop silent file drops
Modal's _sdk_upload_dir/_sdk_download_dir transferred directories one
file at a time via Sandbox.filesystem exec-streamed calls. On large
multi-file uploads (e.g. DinD compose tasks) files were observed to be
silently skipped, and the per-file path also loses exec bits, symlinks,
and empty directories — unlike docker compose cp on the local Docker
environment (terminal-bench-3#846).
Pack the directory into a single tar.gz, transfer it with one SDK call,
and extract on the other side. gzip CRC + tar exit status make partial
transfers fail loudly. Falls back to the previous per-file path when
tar/gzip is unavailable on the sandbox.
https://claude.ai/code/session_01XmMGntgUhjovVk3LKBavzU
* refactor(env): shared tar_transfer lib; full-fidelity dir transfers for Modal, Daytona, GKE
Extract the tar-based directory packing/extraction introduced for Modal
into harbor.environments.tar_transfer and apply it uniformly:
- Modal (direct + DinD): unchanged behavior, now built on the shared lib.
- Daytona (direct + DinD): pack/upload/extract as a single archive,
preserving exec bits, symlinks, and empty dirs that the batched
fs.upload_files staging dropped; falls back to the batched per-file
path when tar/gzip is unavailable. Missing source dirs now raise
instead of silently uploading nothing.
- GKE (direct + DinD): already tar-streamed on the wire, but packed
only regular files; now packs the whole tree so symlinks and empty
dirs survive, and extraction uses the safe 'data' filter everywhere.
All dir transfers now match docker compose cp fidelity on the local
Docker environment.
https://claude.ai/code/session_01XmMGntgUhjovVk3LKBavzU
* refactor(env): require tar/gzip on sandboxes, drop per-file transfer fallbacks
Modal and Daytona directory transfers are now tar-only, matching GKE,
which has always required tar in the container for its exec-streamed
transfers. Any image Harbor can run (it already requires a shell)
ships tar and gzip, and the fallback paths silently degraded fidelity
(no exec bits, symlinks, or empty dirs), which is exactly the bug
class being fixed. A missing tar now fails loudly with a clear error
instead of falling back.
https://claude.ai/code/session_01XmMGntgUhjovVk3LKBavzU
* test(env): skip POSIX-fidelity transfer tests on Windows
The new dir-transfer tests create symlinks (WinError 1314 without
elevation on Windows runners) and assert exec bits (chmod is a no-op
on NTFS). They verify POSIX semantics of transfers into Linux
sandboxes, which a Windows filesystem cannot represent, so skip them
on win32. The truncated-archive test now uses a plain tree so it
keeps running cross-platform.
https://claude.ai/code/session_01XmMGntgUhjovVk3LKBavzU
---------
Co-authored-by: Claude <noreply@anthropic.com>
* Add Docker Compose support to Novita (#1830)
* Add Novita environment support to Harbor
- Introduced NovitaEnvironment class for integration with Novita's cloud sandbox service.
- Implemented end-to-end and unit tests for NovitaEnvironment functionality.
* Fix CI failures: type errors, lint, and pytest collection crash
- Add type: ignore comments for novita_sandbox SDK type issues
- Move sys.exit() guard into __main__ block so pytest collection doesn't crash
- Add template reuse test phase to e2e integration test
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Fix COPY instruction parsing and timeout_sec=0 handling
- Skip COPY --from=... instructions (multi-stage builds)
- Filter out COPY flags (--chown, --chmod) before extracting source path
- Use explicit None check for timeout_sec to allow timeout_sec=0
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Address Devin review: internet flag, default timeout, multi-source COPY
- Set can_disable_internet to False (not yet supported by Novita SDK)
- Change default exec timeout from 60s to 0 (no timeout), matching e2b
- Handle multi-source COPY instructions (COPY a.py b.py /dest/)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Fix Windows path separator in upload_dir remote paths
Use PurePosixPath for remote sandbox paths to ensure forward slashes
on all platforms.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Change default exec timeout from 0 to 300s
The novita_sandbox SDK defaults to 60s internally when 0 is passed.
Use 300s (5 minutes) to avoid premature termination of long-running
agent and verifier commands.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Fix build error log index and defer API base URL resolution
- Use logs[-1] instead of logs[-2] for build failure error message
- Move NOVITA_BASE_URL lookup from class definition to __init__,
consistent with NOVITA_API_KEY handling
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Handle null logs in build failure error reporting
Use `status.get("logs") or []` instead of `status.get("logs", [])`
to handle API returning `"logs": null`.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Wrap _http_client.aclose() in try/except in stop()
Prevent transport-level errors during HTTP client cleanup from
propagating out of stop() and masking the trial outcome.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Preserve sandbox when delete=False for debugging
When stop(delete=False) is called, skip killing the sandbox and closing
the HTTP client so the sandbox remains running for debugging purposes.
This aligns with how other environments (e.g. GKE) handle the delete flag.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* novita: use alias endpoint for template lookup and fix stale alias recovery
- Replace _api_list_templates + iteration with direct GET /templates/aliases/{alias}
endpoint for O(1) template lookup instead of scanning all templates
- Add stale alias recovery in _api_create_template: on 403 "Alias already used",
look up the stale template via alias endpoint, delete it, then retry creation
- Include API key suffix in template alias to avoid cross-account conflicts
- Increase build timeout from 600s to 1200s for heavy Dockerfiles
- Add _MIN_MEMORY_MB_PER_CPU constant (512 MB/CPU)
- Update tests to cover new alias endpoint behavior (44 tests passing)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* novita: auto-recover from stale cached templates on sandbox creation
When _find_template_by_alias returns a template ID that no longer exists
in the backend (alias registered but build failed/incomplete), AsyncSandbox
would raise a SandboxException("404: template not found"). Now start()
catches this case, deletes the stale template via REST API, and triggers
a fresh build before retrying sandbox creation.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* novita: include last 5 log lines in build failure error message
Previously only the last log line was shown, which was often just
"Postprocessing finished. Cleaning up..." instead of the actual error.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* feat(novita): upload COPY files via S3 pre-signed URL to fix 413 errors
* chore: update parity_summary.csv [skip ci]
* Fix review issues and CI failures in Novita environment
- Add _merge_env(env) call in exec() so persistent env vars (--ae flags,
task [environment.env] config) are correctly forwarded to sandbox commands
- Add user parameter to exec(), is_dir(), is_file() to match BaseEnvironment
interface (fixes type-check invalid-method-override errors)
- Close HTTP client in stop(delete=False) to prevent resource leak; update
test to assert aclose is called
- Fix uv.lock: missing [[package]] header before networkx entry caused TOML
parse errors that broke all CI checks; regenerate lockfile cleanly
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* Fix exec() to respect user parameter via _resolve_user
The user parameter was accepted but never used — all commands ran as
root. Now calls _resolve_user(user) to honour the orchestrator-set
default_user (e.g. task agent.user / verifier.user from task.toml).
Novita SDK's user parameter is Literal["root", "user"], so map any
non-root resolved user to "user"; add Literal import accordingly.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* Add preflight() and chmod 777 on log dirs in Novita environment
- Add preflight() classmethod to validate NOVITA_API_KEY before any
trials are queued, giving immediate feedback instead of failing mid-job
- chmod 777 agent/verifier log directories after creation in start() so
non-root agent/verifier users can write reward files and logs
- Update start() test mocks to handle both foreground (healthcheck) and
background (exec) sandbox.commands.run call patterns
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* style: ruff format test_novita.py
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* Fix template name slash escaping and cwd quoting in exec
- Replace '/' with '__' in template alias construction so org/name task
names (e.g. harbor/hello-world) don't break REST API URL paths
- Use shlex.quote(effective_cwd) in exec() to handle paths with spaces
or shell metacharacters safely
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* Use timeout=0 (no limit) as default in exec, aligning with E2B
timeout_sec or 0 matches E2B and the Novita SDK docs where 0 means
no connection time limit, avoiding premature 300s cutoffs on long-running
agent setup or verifier scripts.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* Update src/harbor/environments/novita.py
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix: deal with build conflict error and enhance Dockerfile handling in NovitaEnvironment
* refactor: move novita-sandbox to optional extra, matching other cloud providers
- Move `novita-sandbox` from main deps to `[novita]` optional extra
- Add `dockerfile-parse` to `novita` extra (was only in `e2b`, but novita.py needs it)
- Include `harbor[novita]` in the `cloud` bundle
- Wrap SDK imports in try/except with `_HAS_NOVITA` flag, following the same
lazy-import pattern introduced for daytona/e2b/modal in the upstream refactor
- Raise `MissingExtraError` in `preflight()` when novita-sandbox is not installed
- Regenerate uv.lock
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* fix: add _HAS_NOVITA guard in __init__ for clear MissingExtraError
Without this guard, instantiating NovitaEnvironment when novita-sandbox
is not installed raises a raw NameError (on DockerfileParser) instead of
a helpful MissingExtraError with install instructions. Follows the same
pattern as E2BEnvironment and RunloopEnvironment.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* Update src/harbor/environments/novita.py
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* Update src/harbor/environments/novita.py
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix: import EnvironmentCapabilities in Novita environment
Add the missing capabilities import after migrating NovitaEnvironment to the new capabilities API so ruff and ty can resolve the type.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* fix: update Novita capability tests
Update Novita environment tests to assert the new capabilities API after migrating away from deprecated properties.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* fix: fix file upload endpoint
* fix: integrate Novita SDK template builds
Use the Novita SDK template builder directly while preserving Harbor's Dockerfile COPY handling, and pin the alpha SDK version without enabling global prerelease resolution.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* fix: pin Novita sandbox domain
Use the regional Novita sandbox endpoint consistently so local domain overrides cannot route template operations to the wrong API host.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* fix: avoid Novita SDK import during test collection
Load Novita SDK modules only when the Novita environment actually needs them so pytest can collect E2B and Novita tests in the same process without duplicate protobuf descriptor registration.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* fix: fix novita sandbox default domain
* feat: Add docker-compose support to Novita (in-sandbox docker compose) and extend unit tests for env-driven domain/API settings
Signed-off-by: muskmo <yunming.mo@novita.ai>
* fix: fix dependencies
* fix(novita): restore bridge DNS for DinD compose on prod kernels
Signed-off-by: muskmo <yunming.mo@novita.ai>
---------
Signed-off-by: muskmo <yunming.mo@novita.ai>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: mac <mac@macdeMacBook-Pro.local>
Co-authored-by: muskmo <yunming.mo@novita.ai>
* harbor-langsmith: persist run tags (top-level) and dataset metadata (under extra) (#1808)
* Support wildcard network allowlist hosts (#1840)
* Support wildcard network allowlist hosts
* docs: clarify allowlist subdomain matching
---------
Co-authored-by: Boxuan Li <boxuanli@microsoft.com>
* feat: add LangSmith sandbox environment (#1610)
* rewardkit: add REWARDKIT_MODEL override for agent judges (#1778)
* rewardkit: add REWARDKIT_MODEL override for agent judges
* rewardkit: clean up CLI short-flag aliases to follow -/-- convention
* Update Claude Code bootstrap installer URL (#1852)
Replace the claude.ai/install.sh endpoint with the downloads.claude.ai Claude Code release bootstrap script.
The old claude.ai URL now returns a Cloudflare managed challenge to non-browser curl requests, which breaks agent setup. The downloads endpoint returns the installer script directly and preserves the existing version flag flow.
* Added cline v2 and ATIF Trajectory support (#1495)
* Adding Cline V2
* Adding Cline Core which is just V2 with just bash tool
* Adding Cline Core extra logic
* Remove cline core and keep cline v2
* Revert modal changes from cline v2 branch
* feat(cline-v2): ATIF trajectories + output cleanup (#5)
* feat(cline): emit ATIF trajectories
Adds ATIF-v1.6 trajectory support for the Cline CLI agent. The agent now
copies its session directory from ~/.cline/data/sessions/ into
/logs/agent/cline-sessions/ at the end of a run, then converts the
native messages.json into a Trajectory and writes it to
logs_dir/trajectory.json.
Tool-result user messages are folded into the preceding agent step's
observation so tool_call <-> result correlation matches the spec.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* fix(cline): preserve tool_results in mixed-content user messages
A user message containing both text and tool_result blocks silently
dropped the tool_results: the first guard required `not text_parts`
and the second required `not message_text`, so neither branch fired
and the step was emitted with only the text.
Always attempt to attach tool_results to the prior agent step's
observation, fold any unmatched orphans into the message text, and
skip emitting the step only when nothing is left.
Adds tests for both the mixed-content and orphan-with-text cases.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* cleanup(cline): drop redundant sort and scope session copy to latest only
- _find_session_messages_file: remove sorted() before max(); max alone does
the selection.
- run() cleanup: copy only the most recently modified session subdirectory
into /logs/agent/cline-sessions/ instead of the entire sessions/ tree,
so long-lived or reused environments don't bloat the log artifact.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* feat(cline-v2): emit ATIF trajectories and scope session copy
The cline-v2 agent is a sibling class to cline-cli and was missing the
same trajectory emission + scoped session copy applied in earlier
commits. Mirror the changes here so runs of cline-v2 produce
trajectory.json and a cline-sessions/ dir under logs.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* revert(cline-cli): drop trajectory changes from cline.py
cline-v2 is the agent that actually runs; keep the PR focused on v2.py
and the shared trajectory.py module. cline-cli can be wired up in a
follow-up if needed.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* cleanup(cline-v2): drop redundant agent output artifacts
Harbor only consumes trajectory.json from an agent's logs_dir. The rest
of what cline-v2 was emitting was cline-specific observability with no
downstream reader:
- prompt-meta.pre.json (full content already logged verbatim in job.log
via the heredoc command)
- prompt.debug.txt (one int; duplicated in meta)
- system_prompt.latest.md, prompt_manifest.latest.json
(copies of the sole task-*.{system_prompt.md,
manifest.json} files — trivially redundant since
each trial runs exactly one task)
- taskHistory.json (superseded by cline-sessions/*.messages.json)
- api_history/ (legacy pre-v2 task format; messages.json covers it)
- artifacts.index.txt (ls -lah dump; reproducible)
Cleanup command is now just the scoped cline-sessions/ copy. Also drop
_build_pre_run_metadata and _infer_task_id_from_trial_id which became
dead after the prompt-meta write went away.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* feat(cline-v2): persist install step stdout/stderr to setup/<label>.log
Harbor's BaseInstalledAgent creates an empty logs_dir/setup/ for every
installed agent; the directory is meant for install-time artifacts but
cline-v2 was leaving it empty. Write one log file per setup step keyed
by retry_label (install-root-prereqs.log, install-agent-runtime.log)
with the command's captured stdout/stderr, so failures and version
output are persisted to disk instead of only reaching the trial-level
debug logger.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* feat(cline-v2): bookend setup logs with start/end timestamps
Add ISO-8601 start/end timestamps and duration to each setup/<label>.log
header. Makes it easy to see which install step hung or took longest
without grepping the trial-level debug logger.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* cleanup(cline-v2): rename cline-sessions/ to sessions/
Match the source path name (~/.cline/data/sessions). The defensive
"cline-" prefix guarded against a namespace collision that doesn't
exist.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* refactor(cline-v2): drop --json, decouple usage extraction from trajectory
Stop passing --json to cline so cline.txt becomes a human-readable
summary instead of a 380K machine-parseable event stream. Usage
(tokens/cost) now comes from messages.json directly via a dedicated
_populate_usage_from_session method, decoupled from the trajectory
converter so observability keeps working if conversion trips on an
edge case.
Removes ~240 lines of now-dead cline.txt parsers:
_load_cline_json_records, _extract_task_id_from_records,
_load_task_history_entries, _extract_usage_from_run_result,
_extract_usage_from_agent_events, _extract_usage_from_task_history,
_coerce_int, _coerce_float.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* address gemini review: write setup logs on failure; harden glob race
- _exec_with_setup_retries now writes a per-attempt setup log on
timeout and generic exception paths, not only on success, so
failed install attempts are diagnosable under logs_dir/setup/.
- _find_session_messages_file wraps the mtime max() in try/except
OSError to handle the theoretical case where a globbed file is
removed before stat().
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
* Fix cline-v2 flag handling
* Update src/harbor/agents/installed/cline/v2.py
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* Fix committed modal conflict markers
* Replace cline-v2 with cline-cli
* Handle Cline CLI local binary fallback
---------
Co-authored-by: Robin Newhouse <robin@cline.bot>
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(claude_code): support CLAUDE_FORCE_OAUTH to bill the subscription on harbor run (#1846)
Default is unchanged (ANTHROPIC_API_KEY wins). With CLAUDE_FORCE_OAUTH=<truthy>,
the API key and ANTHROPIC_AUTH_TOKEN are dropped so the CLI authenticates with
CLAUDE_CODE_OAUTH_TOKEN; raises clearly if no token is present. The flag is read
via _get_env, so it works from a config env: block, --ae, or the shell, mirroring
codex's CODEX_FORCE_AUTH_JSON and gemini's GEMINI_FORCE_OAUTH.
Scope: token env var only; ~/.claude/.credentials.json and macOS Keychain are a
planned follow-up. Like codex, the key is dropped from the per-call env only;
guarding a key set in the agent env: block (_extra_env) is left as an open
question for review.
Co-authored-by: Kobe Chen <xiaokunchen0@gmail.com>
* Add built-in LangGraph agent (#1807)
* Add built-in LangGraph agent
* nest langgraph rollout under harbor-provided parent trace context
* Clean up langgraph agent: drop unused langgraph-cli, capture answer and token usage in AgentContext
* Document langgraph agent model contract and output grading
* Support langgraph graph factories so the model can be set at runtime via config
* cr
* Drop in-memory file materialization; document filesystem backend requirement for file tasks
* docs: remove per-agent LangGraph page
We are not documenting individual agents for now; may add per-agent
docs in the future.
---------
Co-authored-by: Harrison Chase <hw.chase.17@gmail.com>
* Add GPU support to the Daytona environment (#1731)
* Add GPU support to the Daytona environment
Daytona exposes GPUs as a count on its sandbox Resources object. Wire that
through so GPU tasks (e.g. examples/tasks/hello-cuda) can run on Daytona:
- Declare gpus=True in capabilities and pass the requested GPU count into
Resources(gpu=...).
- Validate eagerly at construction. Daytona only provisions H100 GPUs and its
API takes a count, not a type, so gpu_types is validated (the request is
satisfiable iff H100 is among the acceptable types -- the common alias or
GKE's canonical nvidia-h100-80gb label) but not forwarded to the SDK.
- Reject GPU requests on docker-compose (DinD) tasks: the stock docker:*-dind
image lacks the nvidia container toolkit, so a GPU on the VM never reaches the
inner task container. Supporting that is a follow-up.
Verified end-to-end: hello-cuda on Daytona provisions a real H100 and the CUDA
verifier passes (reward 1.0).
Signed-off-by: rovle <lovre.pesut@gmail.com>
* Reject non-ephemeral Daytona GPU sandboxes at construction
Daytona requires GPU sandboxes to set autoDeleteInterval=0. Surface a
harbor-level error when a GPU is requested with auto_delete_interval_mins != 0,
instead of deferring to the SDK's later validation failure.
Signed-off-by: rovle <lovre.pesut@gmail.com>
* Clarify Daytona GPU ephemerality check comment
Signed-off-by: rovle <lovre.pesut@gmail.com>
* Forward GPU type to the Daytona environment
Daytona 0.184.0 adds a `gpu_type` field to `Resources` (a `GpuType` enum
whose provisionable values are H100 and RTX-PRO-6000), so the environment
can now request a specific GPU rather than only a count.
`_sandbox_resources` resolves the task's acceptable `gpu_types` through
`DAYTONA_GPU_TYPE_MAP` (short names plus GKE canonical labels) to the
matching Daytona wire values and forwards them as a list, in task order.
Validation broadens from H100-only to "at least one acceptable type is
provisionable", and unmatched types are dropped rather than rejected.
Bumps the daytona pin to >=0.184.0.
Signed-off-by: rovle <lovre.pesut@gmail.com>
* Document Daytona GPU support in managing-resources
Signed-off-by: rovle <lovre.pesut@gmail.com>
---------
Signed-off-by: rovle <lovre.pesut@gmail.com>
* Preserve tagged Docker images on delete (#1861)
* Improve use.computer environment (#1853)
* Improve use.computer environment
* Remove Ubuntu runtime desktop prep
* Add use.computer OSWorld environment
* Avoid shell setup for OSWorld images
* Upload OSWorld files through OSWorld API
* Skip local OSWorld tests upload
* Refine use.computer OSWorld mode
* Clean up use.computer OSWorld startup
* Simplify use.computer service profiles
* Fix use.computer OSWorld uploads
* Fix use.computer CI
---------
Co-authored-by: Use.Computer <joan@use.computer>
* add session-id header (#1855)
Signed-off-by: SumanthRH <sumanthrh99@gmail.com>
* ci: fix ai-review fork-PR checkout for claude-code-action@v1 (#1874)
The ai-review job checked out twice into the same workspace; the second
checkout (repository: <fork>) repointed origin to the fork. claude-code-action@v1
fetches the PR via `git fetch origin pull/<N>/head` for fork PRs, but pull refs
only exist on the base repo, so the fetch failed with "couldn't find remote ref
pull/<N>/head".
Remove the redundant manual PR checkout and let claude-code-action@v1 do its own
PR checkout against origin (= base repo), where pull/<N>/head exists. This is the
action's intended single-checkout usage.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Add network policy docs page under Tasks
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Document environment support for network modes, phases, and capabilities
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Add Novita network policy support (#1867)
* Add Novita environment support to Harbor
- Introduced NovitaEnvironment class for integration with Novita's cloud sandbox service.
- Implemented end-to-end and unit tests for NovitaEnvironment functionality.
* Fix CI failures: type errors, lint, and pytest collection crash
- Add type: ignore comments for novita_sandbox SDK type issues
- Move sys.exit() guard into __main__ block so pytest collection doesn't crash
- Add template reuse test phase to e2e integration test
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Fix COPY instruction parsing and timeout_sec=0 handling
- Skip COPY --from=... instructions (multi-stage builds)
- Filter out COPY flags (--chown, --chmod) before extracting source path
- Use explicit None check for timeout_sec to allow timeout_sec=0
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Address Devin review: internet flag, default timeout, multi-source COPY
- Set can_disable_internet to False (not yet supported by Novita SDK)
- Change default exec timeout from 60s to 0 (no timeout), matching e2b
- Handle multi-source COPY instructions (COPY a.py b.py /dest/)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Fix Windows path separator in upload_dir remote paths
Use PurePosixPath for remote sandbox paths to ensure forward slashes
on all platforms.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Change default exec timeout from 0 to 300s
The novita_sandbox SDK defaults to 60s internally when 0 is passed.
Use 300s (5 minutes) to avoid premature termination of long-running
agent and verifier commands.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Fix build error log index and defer API base URL resolution
- Use logs[-1] instead of logs[-2] for build failure error message
- Move NOVITA_BASE_URL lookup from class definition to __init__,
consistent with NOVITA_API_KEY handling
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Handle null logs in build failure error reporting
Use `status.get("logs") or []` instead of `status.get("logs", [])`
to handle API returning `"logs": null`.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Wrap _http_client.aclose() in try/except in stop()
Prevent transport-level errors during HTTP client cleanup from
propagating out of stop() and masking the trial outcome.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Preserve sandbox when delete=False for debugging
When stop(delete=False) is called, skip killing the sandbox and closing
the HTTP client so the sandbox remains running for debugging purposes.
This aligns with how other environments (e.g. GKE) handle the delete flag.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* novita: use alias endpoint for template lookup and fix stale alias recovery
- Replace _api_list_templates + iteration with direct GET /templates/aliases/{alias}
endpoint for O(1) template lookup instead of scanning all templates
- Add stale alias recovery in _api_create_template: on 403 "Alias already used",
look up the stale template via alias endpoint, delete it, then retry creation
- Include API key suffix in template alias to avoid cross-account conflicts
- Increase build timeout from 600s to 1200s for heavy Dockerfiles
- Add _MIN_MEMORY_MB_PER_CPU constant (512 MB/CPU)
- Update tests to cover new alias endpoint behavior (44 tests passing)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* novita: auto-recover from stale cached templates on sandbox creation
When _find_template_by_alias returns a template ID that no longer exists
in the backend (alias registered but build failed/incomplete), AsyncSandbox
would raise a SandboxException("404: template not found"). Now start()
catches this case, deletes the stale template via REST API, and triggers
a fresh build before retrying sandbox creation.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* novita: include last 5 log lines in build failure error message
Previously only the last log line was shown, which was often just
"Postprocessing finished. Cleaning up..." instead of the actual error.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* feat(novita): upload COPY files via S3 pre-signed URL to fix 413 errors
* chore: update parity_summary.csv [skip ci]
* Fix review issues and CI failures in Novita environment
- Add _merge_env(env) call in exec() so persistent env vars (--ae flags,
task [environment.env] config) are correctly forwarded to sandbox commands
- Add user parameter to exec(), is_dir(), is_file() to match BaseEnvironment
interface (fixes type-check invalid-method-override errors)
- Close HTTP client in stop(delete=False) to prevent resource leak; update
test to assert aclose is called
- Fix uv.lock: missing [[package]] header before networkx entry caused TOML
parse errors that broke all CI checks; regenerate lockfile cleanly
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* Fix exec() to respect user parameter via _resolve_user
The user parameter was accepted but never used — all commands ran as
root. Now calls _resolve_user(user) to honour the orchestrator-set
default_user (e.g. task agent.user / verifier.user from task.toml).
Novita SDK's user parameter is Literal["root", "user"], so map any
non-root resolved user to "user"; add Literal import accordingly.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* Add preflight() and chmod 777 on log dirs in Novita environment
- Add preflight() classmethod to validate NOVITA_API_KEY before any
trials are queued, giving immediate feedback instead of failing mid-job
- chmod 777 agent/verifier log directories after creation in start() so
non-root agent/verifier users can write reward files and logs
- Update start() test mocks to handle both foreground (healthcheck) and
background (exec) sandbox.commands.run call patterns
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* style: ruff format test_novita.py
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* Fix template name slash escaping and cwd quoting in exec
- Replace '/' with '__' in template alias construction so org/name task
names (e.g. harbor/hello-world) don't break REST API URL paths
- Use shlex.quote(effective_cwd) in exec() to handle paths with spaces
or shell metacharacters safely
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* Use timeout=0 (no limit) as default in exec, aligning with E2B
timeout_sec or 0 matches E2B and the Novita SDK docs where 0 means
no connection time limit, avoiding premature 300s cutoffs on long-running
agent setup or verifier scripts.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* Update src/harbor/environments/novita.py
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix: deal with build conflict error and enhance Dockerfile handling in NovitaEnvironment
* refactor: move novita-sandbox to optional extra, matching other cloud providers
- Move `novita-sandbox` from main deps to `[novita]` optional extra
- Add `dockerfile-parse` to `novita` extra (was only in `e2b`, but novita.py needs it)
- Include `harbor[novita]` in the `cloud` bundle
- Wrap SDK imports in try/except with `_HAS_NOVITA` flag, following the same
lazy-import pattern introduced for daytona/e2b/modal in the upstream refactor
- Raise `MissingExtraError` in `preflight()` when novita-sandbox is not installed
- Regenerate uv.lock
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* fix: add _HAS_NOVITA guard in __init__ for clear MissingExtraError
Without this guard, instantiating NovitaEnvironment when novita-sandbox
is not installed raises a raw NameError (on DockerfileParser) instead of
a helpful MissingExtraError with install instructions. Follows the same
pattern as E2BEnvironment and RunloopEnvironment.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* Update src/harbor/environments/novita.py
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* Update src/harbor/environments/novita.py
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix: import EnvironmentCapabilities in Novita environment
Add the missing capabilities import after migrating NovitaEnvironment to the new capabilities API so ruff and ty can resolve the type.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* fix: update Novita capability tests
Update Novita environment tests to assert the new capabilities API after migrating away from deprecated properties.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* fix: fix file upload endpoint
* fix: integrate Novita SDK template builds
Use the Novita SDK template builder directly while preserving Harbor's Dockerfile COPY handling, and pin the alpha SDK version without enabling global prerelease resolution.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* fix: pin Novita sandbox domain
Use the regional Novita sandbox endpoint consistently so local domain overrides cannot route template operations to the wrong API host.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* fix: avoid Novita SDK import during test collection
Load Novita SDK modules only when the Novita environment actually needs them so pytest can collect E2B and Novita tests in the same process without duplicate protobuf descriptor registration.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* fix: fix novita sandbox default domain
* feat: Add docker-compose support to Novita (in-sandbox docker compose) and extend unit tests for env-driven domain/API settings
Signed-off-by: muskmo <yunming.mo@novita.ai>
* fix: fix dependencies
* fix(novita): restore bridge DNS for DinD compose on prod kernels
Signed-off-by: muskmo <yunming.mo@novita.ai>
* Add network allowlist support to Novita direct mode
Direct mode now reports disable_internet and network_allowlist
capabilities, enforcing them via the Novita SDK at create time:
allow_internet_access toggles no-network, and ALLOWLIST mode sets
network.allow_out (allowed hosts) + deny_out (0.0.0.0/0). Switch
network decisions from task_env_config.allow_internet to the
_network_disabled / _network_is_allowlist policy helpers. Compose
(DinD) mode rejects ALLOWLIST tasks as fail-safe since it can only
enforce all-or-nothing network via network_mode: none.
Also upload the environment/ dir after start in both direct and DinD
strategies to fix prebuilt-image uploads (regression #1830), and only
pass cpu_count/memory_mb to template builds when explicitly configured
instead of injecting hardcoded defaults.
Signed-off-by: muskmo <yunming.mo@novita.ai>
---------
Signed-off-by: muskmo <yunming.mo@novita.ai>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: mac <mac@macdeMacBook-Pro.local>
Co-authored-by: muskmo <yunming.mo@novita.ai>
* add procps to prevent process crashes when claude-code call tree-kill (#1864)
* Add generic ACP registry agent support (#1464)
* Add generic ACP registry agent support
* Drop unneeded reset_dirs mock from multi-step trial test
The mocked environment already satisfies the multi-step flow; the test
passes identically without it and the file now matches main.
---------
Co-authored-by: Kobe Chen <xiaokunchen0@gmail.com>
* v0.13.2
* fix(codex): skip install when codex is already available (#1848)
* Clarify wildcard allowlist depth (#1854)
* refactor: reduce memory usage (#1882)
* rewardkit: support Claude subscription auth for judges (#1770)
* \#1850 updated default package download timeout from 20s to 120s (#1883)
Co-authored-by: Sam Vance <sam@harborframework.com>
* Allow WebFetch and WebSearch tool for review bot (#1887)
* feat: add --repo flag for git-based dataset registries (#1884)
* feat: add --repo flag for git-based dataset registries
Support resolving datasets from any git repository (GitHub, Hugging Face,
GitLab, or arbitrary git hosts) via a new --repo flag on harbor run,
harbor download, and harbor datasets list.
Two resolution paths:
- Path A (registry): --repo org/name --dataset name@ver reads registry.json
- Path B (implicit): --repo org/name scans tasks/ for harbor-format task dirs
Source grammar supports: bare org/name (GitHub default), full URLs (HTTPS,
SSH, git://), scheme-less host URLs, /tree/<ref>/<subdir> parsing, and
@ref pinning with SHA resolution for reproducibility.
Adds GitRepoRegistryClient with sparse checkout for efficient cloning.
17 unit tests covering parser, config validation, and client helpers.
Design doc: docs/design/registry-repo.md
* fix: --registry-path accepts both file path and directory
--registry-path /some/path/my-registry.json # uses file directly
--registry-path /some/path # assumes registry.json in dir
Also unified help text for --registry-path across all CLI commands.
* fix: add missing RegistryClientFactory import and remove unused test imports
* chore: remove design doc from PR (moved to gist)
* fix(terminus-2): deliver oversized keys via tmux paste buffers (#1873) (#1904)
Long literal payloads (e.g. heredoc answer writes) previously went
through chunked `tmux send-keys` commands, which fail with "command too
long" when they exceed the tmux client's message size limit — losing
the whole trial. The limit also varies across tmux builds, so the
hardcoded ~16 KB margin alone is not reliable.
TmuxSession now:
- stages keys whose quoted form cannot fit in a single send-keys
command into a file inside the environment (base64-chunked through
environment.exec, which is not subject to tmux limits) and pastes
them with `tmux load-buffer`/`paste-buffer`, which handle arbitrarily
large content;
- falls back at runtime when a tmux build rejects a within-margin
send-keys command as too long, resending keys individually and
pasting any that are still rejected;
- removes the old key-splitting path, which spread one literal key
across many send-keys commands and could leave half-typed input
behind on failure.
Fixes #1873
https://claude.ai/code/session_01Ww6iVMYhU7WuBcvDE8phYY
Co-authored-by: Claude <noreply@anthropic.com>
* feat: support include/exclude patterns for agent and verifier log downloads (#1889)
* Support include/exclude patterns for agent and verifier log downloads
* cleanest fix for commet1: 🔴 Reward files not protected from exclude_logs when include_logs is unset
* Add Runloop network policy launch support (#1885)
* Update network modes and capabilities in documentation
* Add Runloop network policy launch support
* Fix Runloop network policy creation races
* Fix Runloop startup workdir creation
* Expand Runloop startup workdir coverage
* Fix Codex nvm install under Runloop allowlists
* Fix Runloop startup tests without credentials
* [WIP] Computer-1 (#1572)
* harness only
* cleanup
* fixes, +example
* update impl, add providers
* pr fixes
* ruff reformatingg
* computer-1 pure litellm
* self-contained harness, providers involve deps again
* fail fast on non-vision models
* minimal computer-1 dockerfile example, remove default models, minimal deps
* fix compaction, cleanup provider classes
* reorganize, minimize provider classes
* +fable
* cleanup
* devin feedback updates
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
* Classify rate-limit agent failures as ApiRateLimitError (#1798) (#1886)
* Classify rate-limit agent failures as ApiRateLimitError (#1798)
* Lock retry-policy behavior for ApiRateLimitError (#1798)
---------
Co-authored-by: Kobe Chen <xiaokunchen0@gmail.com>
* Add sidecar artifact collection and verifier collect hooks (#1775)
* Add sidecar artifact collection and verifier collect hooks
Multi-container (compose) tasks often hold their score signal inside a
sidecar service: a database the agent wrote to, an API server that
logged the agent's requests, a load generator with in-memory counters.
In separate verifier mode all containers are torn down before
verification, so that evidence was unreachable (#1694).
Sidecars are now first-class artifact sources:
- ArtifactConfig gains a `service` field. Sidecar entries are pulled
from the named compose service's filesystem and re-materialize at
their original absolute paths in the verifier environment.
- New [[verifier.collect]] hooks run snapshot commands inside services
after the agent finishes (e.g. pg_dump), so runtime state can be
captured as files before teardown.
- In separate verifier mode the main service is stopped before sidecar
evidence is collected, so leftover agent processes cannot interfere
with collection.
- BaseEnvironment gains per-service operations (service_exec,
service_download_file, service_download_dir, stop_service),
implemented by every compose-capable provider: docker, daytona,
modal, islo.
- The host artifacts layout becomes canonical per-service:
artifacts/services/<name>/<abs path>, with the conventional publish
dir at services/main/logs/artifacts/. Verifier-side placement is
unchanged ("no translation").
- Artifact source/destination paths are validated (no '..' components,
relative-only destinations, reserved names), fixing a path traversal
where a crafted path could write outside the trial directory on the
controller host.
- Cross-service source collisions are rejected at task load so one
service's content can never masquerade as another's in the verifier.
New example task: examples/tasks/sidecar-artifacts, verified end-to-end
with the oracle agent on local Docker.
Closes #1694
* Update multi-step integration tests for per-service artifact layout
The multi-step artifact tests asserted the old flat host layout and the
old main-only download API:
- test_multi_step_downloads_convention_artifacts_per_step_non_mounted and
test_multi_step_merges_task_and_step_artifacts now assert the canonical
services/<name>/<abs path> host layout and the service-scoped download
calls.
- Add test_multi_step_collects_sidecar_artifacts_per_step covering sidecar
artifacts and step-scoped collect hooks in multi-step compose tasks:
task-level sidecar entries collected after every step, step-level entries
and hooks scoped to their step, and main never stopped mid-trial.
* artifacts: flat shared base dir instead of per-service subtree
Per review: don't segregate collected artifacts by service. All services now
share one flat artifacts/ base dir, keyed only by source path (artifacts/<abs
source>), instead of artifacts/services/<service>/<abs source>.
- paths.host_artifact_path + artifact_handler._host_path drop the services/<svc>
prefix; explicit `destination` still honored (host-only, unchanged). Verifier
upload is unaffected (it was always keyed on the artifact's source path, not
the host layout), so the copy into the verifier still works.
- Collisions are handled at collection time instead of being rejected at load:
a per-handler claim map (persisting across the main + sidecar passes) detects
exact and nested host-path overlaps; the first claimant is kept and later ones
log a warning and are skipped (never overwritten), recorded as status
"skipped" in the manifest.
- validate_artifact_entries: overlapping sources/destinations now warn instead
of raising; "services/" is no longer a reserved destination prefix (only
manifest.json remains reserved). Absolute-sidecar-source guard kept.
- Removed now-dead paths helpers (artifacts_services_dir/service_artifacts_dir)
and RESERVED_ARTIFACTS_SUBTREE.
- Updated unit + multi-step tests and the artifacts doc for the flat layout.
Verified: full test suite (2771 passed), and the kv-live-surgery sidecar oracle
runs end-to-end on this harbor (separate verifier + sidecar collect) with
reward 1.0 on docker.
* Add GKE per-service compose support; extract shared service-ops mixin
GKE was the only compose-capable provider without the per-service
operations (service_exec / service_download_file / service_download_dir
/ stop_service) that sidecar artifact collection and verifier collect
hooks require. Implement them on _GKEDinDCompose and GKEEnvironment,
mirroring the Modal/Daytona DinD pattern: sidecar execs do not inherit
main-specific defaults (workdir, default user, persistent env), and
sidecar transfers compose-cp via the pod before tarring out.
With three structurally identical env-level dispatchers, extract them
into ComposeServiceOpsMixin (environments.compose_service_ops): main
service delegates to the environment's regular methods, sidecars route
to the provider's DinD helper via the ComposeServiceTransport protocol.
Modal, Daytona, and GKE now share one implementation.
https://claude.ai/code/session_01XmMGntgUhjovVk3LKBavzU
* Add Novita per-service compose support; enforce compose-capability contract
Novita was the last compose-capable provider without the per-service
operations (service_exec / service_download_file / service_download_dir
/ stop_service) that sidecar artifact collection and verifier collect
hooks require. Implement them on _NovitaDinD (mirroring the other DinD
providers) and adopt ComposeServiceOpsMixin on NovitaEnvironment.
Add a contract test (test_compose_contract.py) that statically verifies
every environment class claiming the docker_compose capability provides
its own per-service operations instead of inheriting BaseEnvironment's
raising defaults, so new compose-capable providers cannot ship without
sidecar support again.
https://claude.ai/code/session_01XmMGntgUhjovVk3LKBavzU
* docs: align sidecar-artifact docs with flat layout; harden compose contract test
The artifact host layout was changed mid-PR to a single flat artifacts/ base
dir (no per-service subtree), but the CHANGELOG, tasks/index.mdx, and a
source docstring still described the abandoned services/<name>/ tree and the
old "collisions rejected at load" behavior. Update them to match the shipped
behavior: flat artifacts/<abs source path>, convention dir at
artifacts/logs/artifacts/, only manifest.json reserved, and overlap handling
that warns + keeps-first instead of erroring. Also list gke/novita among the
compose-capable providers.
Add test_detection_heuristic_flags_known_compose_providers so a regression in
the compose-capability detection heuristic fails loudly instead of silently
skipping a provider in the contract test.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* langsmith: implement per-service compose ops for sidecar artifacts
LangSmithEnvironment (merged from main in parallel) claims the
docker_compose capability but inherited BaseEnvironment's raising
per-service stubs, so the new compose-contract test failed on it once the
PR was merged with main. Implement service_exec, service_download_file,
service_download_dir (via the generic tar downloader), and stop_service,
following the same main-delegates / sidecar-targets pattern as the other
DinD providers. service_download_dir_with_exclusions and service_is_dir
come from BaseEnvironment for free. Add unit tests covering sidecar
targeting, main delegation, and the non-compose-mode error path, and add
langsmith to the documented compose-capable provider list.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* Scope artifact collision claims to one collection pass
_claimed_targets persisted for the whole trial, but multi-step trials
vacate the shared host artifacts dir between steps (outputs archive to
steps/<name>/), so a prior step's claims could falsely skip a later
step's entries that no longer collide. Reset claims at the start of
each _collect_artifacts_phased pass; they still span that pass's main
and sidecar phases.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Wrap sidecar execs with sh -c; keep bash for main
Sidecar containers are arbitrary third-party images where bash is
frequently absent (e.g. the *-alpine variants of postgres, redis,
nginx), while POSIX sh is universal. Switch the sidecar branch of every
compose-capable provider (docker, daytona, modal, langsmith, gke, islo,
novita) to `sh -c`, and keep `bash`/`bash -lc` for the harbor-built main
container so existing tasks that rely on bash semantics are unaffected.
Authors needing bash on a sidecar can invoke it explicitly (bash -c '...')
on images that ship it. Documents the behavior and adds docker tests.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* Update sidecar exec tests to assert sh -c
Match the provider change: sidecar service_exec now wraps with `sh -c`
across docker, modal, gke, novita, langsmith, daytona, and islo. Main
container assertions remain `bash -lc`.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* Correct CHANGELOG to match shipped artifact validation
Two fixes where the changelog described an earlier iteration of this PR
rather than the behavior relative to main:
- Collision section: main had no overlap validation at all; basename
collisions silently overwrote (last write wins). Describe that as the
prior behavior instead of "warning rather than failing" (which implied
users had hard errors to lose).
- Document the new artifact `source` `..` restriction (previously
accepted) alongside the destination rules, and retitle the section to
cover both source and destination.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* Fix stale per-service artifact layout in comments
Two comments still described the abandoned services/<service>/<abs path>
subtree as the canonical layout. The shipped layout is a single flat
artifacts/ base dir mirroring each entry's absolute source path, with no
per-service level. Update the TrialPaths docstring tree and the
_agent_env_mounts comment to match.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* Document multi-step caveat for stop-main-before-sidecar
The anti-cheat section stated unconditionally that separate verifier mode
stops main before sidecar collection. That guarantee only holds for
single-step trials and the final step of a multi-step trial: earlier
steps keep main running (later steps need it), so their sidecar evidence
is collected with the agent container still live. Add a corollary so
authors put tamper-sensitive sidecar evidence on the final step.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: Ruiyang Wang <rynewang@users.noreply.github.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Alex Shaw <alexgshaw64@gmail.com>
* fix(cli): hide removed task check and debug commands from help (#1923)
The check and debug task subcommands were removed a while ago, but they
were still registered as normal commands, so harbor task --help listed
them and showed their arguments even though running them just prints a
removal notice. Mark them hidden so they no longer show up in help, but
keep them around so the old commands still point people to the new ones.
Also fixed the error text, which said harbor tasks (plural) even when you
ran the singular harbor task.
Fixes #1751
* Add Network Allowlist provider capability to Modal (#1932)
* Remove deterministic OpenHands test (#1936)
* fix(goose): keep token counts when trajectory write fails (#1933)
* fix(goose): keep token counts when trajectory write fails
In the Goose agent, the token count was set inside the same try block that writes trajectory.json. If that write failed, the error was logged at debug level and the token count was never set, so it stayed at 0. This made the trial look like it used no tokens, which throws off cost reporting and leaderboard scores.
Move the token count out of the write block: the write keeps its own try/except, and the token count is read from the in-memory metrics right after, so it is always set even when the write fails. This matches how codex.py, opencode.py, and openhands_sdk.py already work.
Fixes #1709.
* feat(goose): record input/output token split when goose reports it
goose 1.37 (block/goose#8870) added input_tokens and output_tokens to the stream-json complete event, flat alongside total_tokens. The adapter only read total_tokens and stored the combined total in n_input_tokens. Read the input/output split and set n_input_tokens and n_output_tokens, falling back to the combined total for older goose that reports only total_tokens.
Addresses review feedback on #1933 (Devin) about token semantics.
* refactor(goose): use standard FinalMetrics token fields
Populate FinalMetrics.total_prompt_tokens / total_completion_tokens and read them back in populate_context, matching codex.py and opencode.py, instead of storing token counts under custom extra keys. This keeps downstream trajectory.json consumers reading goose the same as every other agent. Older goose that reports only a combined total keeps it in total_prompt_tokens to preserve prior behaviour; the raw total is retained in extra.
Addresses Devin review feedback on #1933.
* docs: add documentation for --repo git repository datasets (#1906)
* feat: add MiMo agent — Xiaomi's MiMoCode CLI with ATIF trajectory support (#1899)
Add the MiMo agent (Xiaomi MiMoCode CLI) as a built-in Harbor agent.
- Register `mimo` in `AgentName` enum and `AgentFactory`
- Implement `MiMo` class extending `BaseInstalledAgent` with:
- Installation via `curl -fsSL https://mimo.xiaomi.com/install | bash`
- ATIF trajectory generation from `mimo run --format=json` stdout
- MCP server and skills directory registration
- Provider-specific environment variable passthrough
- Configurable CLI flags (e.g. `--variant`)
- Deep-merge config support via `mimo_config` kwargs
* feat(islo): support basic no-network policy (#1935)
* feat(islo): support basic no-network policy
Map Harbor's portable no-network mode to a static Islo gateway profile so Islo can provide basic allow/deny behavior without exposing gateway-specific task schema.
Co-authored-by: Cursor <cursoragent@cursor.com>
* feat(islo): support dynamic network policy
Co-authored-by: Cursor <cursoragent@cursor.com>
* docs(islo): add network policy example
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(islo): avoid compose no-network gateway overlay
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(islo): stabilize compose policy switching
Co-authored-by: Cursor <cursoragent@cursor.com>
* docs(tasks): update Islo network policy support
Co-authored-by: Cursor <cursoragent@cursor.com>
* refactor(islo): minimize PR1 review surface
Restore existing Islo naming and docs footnote ordering while keeping
portable network policy behavior unchanged.
Co-authored-by: Cursor <cursoragent@cursor.com>
* docs(tasks): drop redundant Islo network policy footnotes
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(islo): use gateway for compose network policy
Use Islo gateway profiles as the single enforcement layer for portable
network policy, including Docker Compose tasks, so runtime phase switching
has one mutable source of truth.
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Alex Shaw <alexgshaw64@gmail.com>
* refactor(env): unify DinD compose providers on shared service-ops/transfer layer (includes #1775 sidecar collection) (#1843)
Extracts the duplicated DinD compose operations layer into a shared
DinDComposeOps class and ports all five remote DinD providers onto it.
- environments/dind_compose.py: compose exec arg building, two-hop
uploads/downloads staged through the DinD host via `docker compose cp`,
self-bind log-dir fast path, `compose stop`, ComposeServiceTransport
adapters, is_dir/is_file -- implemented once. Providers supply six
primitives (host exec + local<->host file moves) and may override the
cp timeouts.
- Modal, Daytona, GKE, Novita: drop their copy-pasted ops; GKE keeps
transfer retries via thin wrappers; Novita preserves slower cp timeouts.
- Islo: thin _IsloComposeOps adapter maps the shared layer onto its
sandbox/compose helpers.
- Local Docker stays separate by design (single-hop compose cp, Windows).
Sidecar service_exec wraps commands with POSIX `sh -c` (not `bash -lc`)
so they run on minimal images such as the *-alpine variants; the main
service keeps bash. Rebased onto current main: retains the finalized flat
artifact layout, the include/exclude log filtering, and the post-branch
provider features (#1861 docker --rmi local, #1867 novita network policy,
#1731 daytona GPU).
Net ~ -400 lines. Contract test enforces that any environment claiming
docker_compose capability implements the per-service operations.
Co-authored-by: Ruiyang Wang <rynewang@users.noreply.github.com>
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
Co-authored-by: Alex Shaw <alexgshaw64@gmail.com>
* Tighten verbose comments in rewardkit judges (#1863)
* Tighten verbose comments in rewardkit judges
* Potential fix for pull request finding
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
* Potential fix for pull request finding
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
* Computer-1 Caching (#1939)
* +caching for computer-1
* devin comment
* formatting
* Network policy: allow empty allowlists; clarify wildcard for apex domain (#1940)
* Allow empty network allowlists
* Document wildcard allowlist apex behavior
---------
Co-authored-by: Boxuan Li <boxuanli@microsoft.com>
* Add Daytona sandbox labels (#1930)
* feat(daytona): add opt-in sandbox labels
Attach harbor.managed / harbor.environment_name / harbor.session_id to
each Daytona sandbox when auto_labels=true (default off), plus arbitrary
user labels via the labels kwarg (applied independently of the gate).
Set at the single create choke point, so the image, snapshot, and DinD
paths are all covered.
Signed-off-by: rovle <lovre.pesut@gmail.com>
* Simplify Daytona sandbox labels
---------
Signed-off-by: rovle <lovre.pesut@gmail.com>
Co-authored-by: Alex Shaw <alexgshaw64@gmail.com>
* fix(upload): avoid ambiguous primary rewards (#1813)
Co-authored-by: sudolinzekun <sudolinzekun@local>
Co-authored-by: benediktstroebl <50178209+benediktstroebl@users.noreply.github.com>
* Fix E2B exec retry safety (#1941)
- retry E2B command dispatch only for failures that indicate the command did not start
- stop retrying post-dispatch `handle.wait()` failures to avoid double-running non-idempotent agent commands
- add unit coverage for safe retry, non-retry, and nonzero-exit behavior
* Add strict type check (#1947)
* Update ty from 0.0.19 to 0.0.49 (#1948)
* Publications
* Remove ruff from dependencies (#1951)
* Add Blaxel cloud sandbox provider (#1643)
* Add Blaxel cloud sandbox provider
* Remove internal Blaxel wording
* Put Blaxel first in provider lists
* Address review feedback: shared helpers, image caching, faithful images
- Reuse definition.py helpers; move multi-stage WORKDIR parsing into
parse_dockerfile_workdir
- Cache sandbox images by content + build-config hash and honor
force_build; reap the builder sandbox with a short TTL
- Declare resource_capabilities (memory requests honored)
- Preflight auth via the SDK's get_credentials
- Run exec through bash -c to match the Docker provider
- Opt out of Blaxel rootfs slimming via blaxel.toml so sandbox
filesystems match the task Dockerfile byte-for-byte
- Derive image build timeout from the task's build_timeout_sec
* Upload environment files after Blaxel start
Ensure prebuilt-image Blaxel tasks receive supplementary environment files, matching other environment providers.
* Use debug log for retained Blaxel sandbox
Match environment logging conventions for delete=False cleanup paths.
* Update Blaxel provider for current type checks
* Append Blaxel to provider lists and remove dedicated example
Move Blaxel to the end of the cloud sandbox provider lists in the docs, pyproject extras, environment enum, and factory registry, and drop the Blaxel-specific example from the sandboxes doc.
* add support for modal vm runtime (#1907)
* Add override.
* docs: add Novita sandbox provider references (#1811)
* docs: add Novita sandbox provider references
* docs: mark Novita sidecar artifacts supported
* docs: align compose provider support
* docs: address Novita provider list feedback
* docs: limit Novita provider list updates
* docs: only append Novita to network allowlist
* docs: restore network policy wording
* docs: fix Novita no-network annotation
---------
Co-authored-by: Alex Shaw <alexgshaw64@gmail.com>
* Add configurable Claude Code permission mode (#1950)
* Add configurable Claude Code permission mode
* Handle null Claude Code permission mode
* Document null permission mode opt-out
* Remove Claude Code permission docs update
* Update Claude Code permission modes
---------
Co-authored-by: Kobe Chen <xiaokunchen0@gmail.com>
* Add Terminus viewer support for JSON payloads and split views (#1958)
* Extract viewer updates from Terminus branch
* Address Devin viewer review feedback
* Address follow-up Devin viewer feedback
* v0.14.0
* fix(daytona): create workdir in sandbox if specified (#1953)
* perf(registry): enumerate --repo tasks via git ls-tree (#1920)
* perf(registry): enumerate --repo tasks via git ls-tree
Implicit dataset enumeration sparse-checked-out the entire tasks/
directory just to read task directory names, materializing all blob
content (e.g. ~932MB / ~4k files for frontier-swe) into a throwaway
temp dir on every run. This dominated --repo startup (~64s before the
environment even began building).
Read the git tree directly instead: a new _tree_only_clone does a
blobless --no-checkout clone + fetch of the target sha, and
_get_implicit_metadata uses `git ls-tree` to find top-level task dirs
containing a task.toml. No blob content is downloaded; per-trial task
downloads (already lazy and cached) are unchanged.
Enumeration of frontier-swe drops from ~64s to ~3s with identical
results (17 tasks). Adds regression tests asserting enumeration returns
the right tasks and never issues a checkout/sparse-checkout.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(registry): handle repo-root subdir in ls-tree enumeration
When tasks live at the repo root (--path .), _effective_subdir returns
"." and the prefix became "./", but `git ls-tree` emits root-relative
paths with no leading "./" — so every line was filtered out and
enumeration spuriously found zero tasks.
Special-case the root: skip the subdir-existence check (root always
exists), omit the path arg from `git ls-tree` so it lists from the root,
and use …
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Addresses Bug 2 in #1771 — adds a parallel
REWARDKIT_MODELenv var (and matching--modelflag) so a globalREWARDKIT_JUDGE=claude-codeno longer silently drops the rubric's pinned model. Precedence: env var > TOML > none. Also adds single-dash aliases (-j,-je,-mcp, etc.) alongside the existing--short forms — purely additive.Overlaps with the Bug 2 portion of #1770 (which fixes all three auth bugs); this PR is a minimal, isolated fix for just the model-pin issue.