feat: harborize Generate Analysis on UI job-level#2002
Merged
alexgshaw merged 3 commits intoJun 19, 2026
Conversation
Contributor
|
Enjoy a better diff viewing experience by clicking one of these URLs: |
Make the job-level analyze flow match the working trial-level one. The job summarize endpoint now runs harbor analyze (run_analyze) in a chosen environment instead of the legacy local Analyzer, writing the aggregated analysis.json back to the job dir for the Analysis tab to render. - add an Environment selector to the job Generate Analysis dialog - reword Concurrent Claude Codes to Concurrent analyses - default Only analyze failed trials to off - render structured analysis.json (per-trial results) instead of the no-longer-written analysis.md; drop the dead summary endpoint - extract shared AnalysisContent component used by trial and job pages Adds a test covering the harborized summarize_job endpoint.
Collaborator
|
How hard do you think it would be to enable multiple agent options in the analysis form? |
8e01b26 to
8b6357b
Compare
Let the Generate Analysis dialogs (job and trial) pick the agent that runs the analysis, mirroring the existing environment selector. The summarize endpoints forward the chosen agent to run_analyze; /api/config exposes the available agents. Model selection is unchanged, so for now only claude-code is guaranteed to work end-to-end.
Collaborator
Author
it's a lightweighted change. The thing is the model options are much more. The previous approach is only list <haiku, sonnet, opus>. Wdyt about the models here, only openai and anthropic models? especially the mini and haiku? inspired by this - probably someone raised this idea before - but if there's a way to run harbor in ui to see & tweak the configs, it would be awesome! 
|
Collaborator
|
ohh i love that idea! Yeah lets do that. But maybe for now we constrain to just claude-code, codex, and cursor-cli from the form? And then for each of those we can hard code the basic model options? Bc someone can always run from the CLI if they want full configurability. |
Collaborator
Author
|
done. 
|
Constrain the Generate Analysis form to a supported subset (claude-code, codex, cursor-cli) with a per-agent model dropdown instead of exposing the full agent enum. Model options are hardcoded per agent, smallest first and used as the default; picking an agent resets the model. Field order is now agent then model. Other agents/models remain available via harbor analyze. Drops the /api/config agents field added for the selector; the form sources agents from the frontend map.
alexgshaw
approved these changes
Jun 19, 2026
Messimeimei
added a commit
to Messimeimei/bitfun-harbor
that referenced
this pull request
Jul 7, 2026
e077080 fix(daytona): create workdir in sandbox if specified (harbor-framework#1953) 5352049 perf(registry): enumerate --repo tasks via git ls-tree (harbor-framework#1920) b279e2d Use Modal filesystem list_files API (harbor-framework#1978) 6737ce2 fix(langgraph): harden dependency installation (harbor-framework#1973) 4fef16f fix(langgraph): preserve summary cancellation propagation (harbor-framework#1974) 1242b5f fix(langsmith): retry transient requests (harbor-framework#1975) bda0f6e feat: harborize harbor check (harbor-framework#1924) 721ffa4 Remove overlapping network policy artifacts (harbor-framework#1979) 4ce26cb fix(langsmith,langgraph): tag traces with harbor runner (harbor-framework#1976) f1cb1e6 fix(modal): keep direct-mode sandbox alive with sleep infinity (harbor-framework#1545) 24f8262 fix(modal): create workdir in sandbox if specified (harbor-framework#1981) 7075d73 fix(copilot-cli): parse Copilot's native session-event JSONL schema (harbor-framework#1985) 7a9ae96 docs(adapters): fix stale parity_experiment.json filename in READMEs (harbor-framework#1964) 11cdb3a feat(agents): add dspy.RLM agent (harbor-framework#1965) 05b1279 Retry Daytona process session creation, treating duplicate as success (harbor-framework#1954) 511a6c1 feat: harborize harbor analyze (harbor-framework#1984) 4001800 Tighten analysis section header spacing (harbor-framework#1987) 29ee9fa fix(mini-swe-agent): honor configured agent env for credentials (harbor-framework#1968) 623da23 feat(trial): scoped log streaming via Trial.add_log_callback (harbor-framework#1966) db0c467 rewardkit: honor individual mode for agent judges (harbor-framework#1793) ab171f3 Add separate verifier network mode matrix demo (harbor-framework#1995) a99d87c Fix network-policy matrix artifact warnings (harbor-framework#1994) 446db38 feat(islo): auto-delete sandboxes after 1h and shield create on cancel (harbor-framework#1990) 38c2848 Add execute permissions to the executables of all example tasks. (harbor-framework#1996) c26dc75 feat(tensorlake): prebuilt docker_image import, OCI build by default, and boot-path hardening (harbor-framework#1989) d5e9e67 Add sandbox labels for Modal environments (harbor-framework#1999) 3c5a69d For Modal - don't use host networking for vm runtime sandboxes (harbor-framework#1998) bd96b55 changelog for harborized check and analyze (harbor-framework#2001) a9148a9 feat: harborize Generate Analysis on UI job-level (harbor-framework#2002) c2d4411 Add Openshift Environment (harbor-framework#1875) 307534f fix(modal): reuse env secrets across sandbox operations (harbor-framework#1969) 4c2c2d1 Show cursor model names in viewer analysis (harbor-framework#2007) 975106d fix(rewardkit): codex judge auto-logs-in from OPENAI_API_KEY (harbor-framework#2009) fd8ee76 Add Eve agent implementation and task configs (harbor-framework#1982) 68daf28 Fix dev typecheck extras (harbor-framework#2012) 8aba950 Fix auth credentials storage writes (harbor-framework#2013) 50ece5b v0.15.0 Co-authored-by: Cursor <cursoragent@cursor.com>
danielpeng2
added a commit
to warpdotdev/harbor
that referenced
this pull request
Jul 10, 2026
* ci: fix ai-review fork-PR checkout for claude-code-action@v1 (#1874)
The ai-review job checked out twice into the same workspace; the second
checkout (repository: <fork>) repointed origin to the fork. claude-code-action@v1
fetches the PR via `git fetch origin pull/<N>/head` for fork PRs, but pull refs
only exist on the base repo, so the fetch failed with "couldn't find remote ref
pull/<N>/head".
Remove the redundant manual PR checkout and let claude-code-action@v1 do its own
PR checkout against origin (= base repo), where pull/<N>/head exists. This is the
action's intended single-checkout usage.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Add network policy docs page under Tasks
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Document environment support for network modes, phases, and capabilities
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Add Novita network policy support (#1867)
* Add Novita environment support to Harbor
- Introduced NovitaEnvironment class for integration with Novita's cloud sandbox service.
- Implemented end-to-end and unit tests for NovitaEnvironment functionality.
* Fix CI failures: type errors, lint, and pytest collection crash
- Add type: ignore comments for novita_sandbox SDK type issues
- Move sys.exit() guard into __main__ block so pytest collection doesn't crash
- Add template reuse test phase to e2e integration test
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Fix COPY instruction parsing and timeout_sec=0 handling
- Skip COPY --from=... instructions (multi-stage builds)
- Filter out COPY flags (--chown, --chmod) before extracting source path
- Use explicit None check for timeout_sec to allow timeout_sec=0
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Address Devin review: internet flag, default timeout, multi-source COPY
- Set can_disable_internet to False (not yet supported by Novita SDK)
- Change default exec timeout from 60s to 0 (no timeout), matching e2b
- Handle multi-source COPY instructions (COPY a.py b.py /dest/)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Fix Windows path separator in upload_dir remote paths
Use PurePosixPath for remote sandbox paths to ensure forward slashes
on all platforms.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Change default exec timeout from 0 to 300s
The novita_sandbox SDK defaults to 60s internally when 0 is passed.
Use 300s (5 minutes) to avoid premature termination of long-running
agent and verifier commands.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Fix build error log index and defer API base URL resolution
- Use logs[-1] instead of logs[-2] for build failure error message
- Move NOVITA_BASE_URL lookup from class definition to __init__,
consistent with NOVITA_API_KEY handling
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Handle null logs in build failure error reporting
Use `status.get("logs") or []` instead of `status.get("logs", [])`
to handle API returning `"logs": null`.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Wrap _http_client.aclose() in try/except in stop()
Prevent transport-level errors during HTTP client cleanup from
propagating out of stop() and masking the trial outcome.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Preserve sandbox when delete=False for debugging
When stop(delete=False) is called, skip killing the sandbox and closing
the HTTP client so the sandbox remains running for debugging purposes.
This aligns with how other environments (e.g. GKE) handle the delete flag.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* novita: use alias endpoint for template lookup and fix stale alias recovery
- Replace _api_list_templates + iteration with direct GET /templates/aliases/{alias}
endpoint for O(1) template lookup instead of scanning all templates
- Add stale alias recovery in _api_create_template: on 403 "Alias already used",
look up the stale template via alias endpoint, delete it, then retry creation
- Include API key suffix in template alias to avoid cross-account conflicts
- Increase build timeout from 600s to 1200s for heavy Dockerfiles
- Add _MIN_MEMORY_MB_PER_CPU constant (512 MB/CPU)
- Update tests to cover new alias endpoint behavior (44 tests passing)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* novita: auto-recover from stale cached templates on sandbox creation
When _find_template_by_alias returns a template ID that no longer exists
in the backend (alias registered but build failed/incomplete), AsyncSandbox
would raise a SandboxException("404: template not found"). Now start()
catches this case, deletes the stale template via REST API, and triggers
a fresh build before retrying sandbox creation.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* novita: include last 5 log lines in build failure error message
Previously only the last log line was shown, which was often just
"Postprocessing finished. Cleaning up..." instead of the actual error.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* feat(novita): upload COPY files via S3 pre-signed URL to fix 413 errors
* chore: update parity_summary.csv [skip ci]
* Fix review issues and CI failures in Novita environment
- Add _merge_env(env) call in exec() so persistent env vars (--ae flags,
task [environment.env] config) are correctly forwarded to sandbox commands
- Add user parameter to exec(), is_dir(), is_file() to match BaseEnvironment
interface (fixes type-check invalid-method-override errors)
- Close HTTP client in stop(delete=False) to prevent resource leak; update
test to assert aclose is called
- Fix uv.lock: missing [[package]] header before networkx entry caused TOML
parse errors that broke all CI checks; regenerate lockfile cleanly
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* Fix exec() to respect user parameter via _resolve_user
The user parameter was accepted but never used — all commands ran as
root. Now calls _resolve_user(user) to honour the orchestrator-set
default_user (e.g. task agent.user / verifier.user from task.toml).
Novita SDK's user parameter is Literal["root", "user"], so map any
non-root resolved user to "user"; add Literal import accordingly.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* Add preflight() and chmod 777 on log dirs in Novita environment
- Add preflight() classmethod to validate NOVITA_API_KEY before any
trials are queued, giving immediate feedback instead of failing mid-job
- chmod 777 agent/verifier log directories after creation in start() so
non-root agent/verifier users can write reward files and logs
- Update start() test mocks to handle both foreground (healthcheck) and
background (exec) sandbox.commands.run call patterns
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* style: ruff format test_novita.py
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* Fix template name slash escaping and cwd quoting in exec
- Replace '/' with '__' in template alias construction so org/name task
names (e.g. harbor/hello-world) don't break REST API URL paths
- Use shlex.quote(effective_cwd) in exec() to handle paths with spaces
or shell metacharacters safely
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* Use timeout=0 (no limit) as default in exec, aligning with E2B
timeout_sec or 0 matches E2B and the Novita SDK docs where 0 means
no connection time limit, avoiding premature 300s cutoffs on long-running
agent setup or verifier scripts.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* Update src/harbor/environments/novita.py
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix: deal with build conflict error and enhance Dockerfile handling in NovitaEnvironment
* refactor: move novita-sandbox to optional extra, matching other cloud providers
- Move `novita-sandbox` from main deps to `[novita]` optional extra
- Add `dockerfile-parse` to `novita` extra (was only in `e2b`, but novita.py needs it)
- Include `harbor[novita]` in the `cloud` bundle
- Wrap SDK imports in try/except with `_HAS_NOVITA` flag, following the same
lazy-import pattern introduced for daytona/e2b/modal in the upstream refactor
- Raise `MissingExtraError` in `preflight()` when novita-sandbox is not installed
- Regenerate uv.lock
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* fix: add _HAS_NOVITA guard in __init__ for clear MissingExtraError
Without this guard, instantiating NovitaEnvironment when novita-sandbox
is not installed raises a raw NameError (on DockerfileParser) instead of
a helpful MissingExtraError with install instructions. Follows the same
pattern as E2BEnvironment and RunloopEnvironment.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* Update src/harbor/environments/novita.py
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* Update src/harbor/environments/novita.py
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix: import EnvironmentCapabilities in Novita environment
Add the missing capabilities import after migrating NovitaEnvironment to the new capabilities API so ruff and ty can resolve the type.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* fix: update Novita capability tests
Update Novita environment tests to assert the new capabilities API after migrating away from deprecated properties.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* fix: fix file upload endpoint
* fix: integrate Novita SDK template builds
Use the Novita SDK template builder directly while preserving Harbor's Dockerfile COPY handling, and pin the alpha SDK version without enabling global prerelease resolution.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* fix: pin Novita sandbox domain
Use the regional Novita sandbox endpoint consistently so local domain overrides cannot route template operations to the wrong API host.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* fix: avoid Novita SDK import during test collection
Load Novita SDK modules only when the Novita environment actually needs them so pytest can collect E2B and Novita tests in the same process without duplicate protobuf descriptor registration.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* fix: fix novita sandbox default domain
* feat: Add docker-compose support to Novita (in-sandbox docker compose) and extend unit tests for env-driven domain/API settings
Signed-off-by: muskmo <yunming.mo@novita.ai>
* fix: fix dependencies
* fix(novita): restore bridge DNS for DinD compose on prod kernels
Signed-off-by: muskmo <yunming.mo@novita.ai>
* Add network allowlist support to Novita direct mode
Direct mode now reports disable_internet and network_allowlist
capabilities, enforcing them via the Novita SDK at create time:
allow_internet_access toggles no-network, and ALLOWLIST mode sets
network.allow_out (allowed hosts) + deny_out (0.0.0.0/0). Switch
network decisions from task_env_config.allow_internet to the
_network_disabled / _network_is_allowlist policy helpers. Compose
(DinD) mode rejects ALLOWLIST tasks as fail-safe since it can only
enforce all-or-nothing network via network_mode: none.
Also upload the environment/ dir after start in both direct and DinD
strategies to fix prebuilt-image uploads (regression #1830), and only
pass cpu_count/memory_mb to template builds when explicitly configured
instead of injecting hardcoded defaults.
Signed-off-by: muskmo <yunming.mo@novita.ai>
---------
Signed-off-by: muskmo <yunming.mo@novita.ai>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: mac <mac@macdeMacBook-Pro.local>
Co-authored-by: muskmo <yunming.mo@novita.ai>
* add procps to prevent process crashes when claude-code call tree-kill (#1864)
* Add generic ACP registry agent support (#1464)
* Add generic ACP registry agent support
* Drop unneeded reset_dirs mock from multi-step trial test
The mocked environment already satisfies the multi-step flow; the test
passes identically without it and the file now matches main.
---------
Co-authored-by: Kobe Chen <xiaokunchen0@gmail.com>
* v0.13.2
* fix(codex): skip install when codex is already available (#1848)
* Clarify wildcard allowlist depth (#1854)
* refactor: reduce memory usage (#1882)
* rewardkit: support Claude subscription auth for judges (#1770)
* \#1850 updated default package download timeout from 20s to 120s (#1883)
Co-authored-by: Sam Vance <sam@harborframework.com>
* Allow WebFetch and WebSearch tool for review bot (#1887)
* feat: add --repo flag for git-based dataset registries (#1884)
* feat: add --repo flag for git-based dataset registries
Support resolving datasets from any git repository (GitHub, Hugging Face,
GitLab, or arbitrary git hosts) via a new --repo flag on harbor run,
harbor download, and harbor datasets list.
Two resolution paths:
- Path A (registry): --repo org/name --dataset name@ver reads registry.json
- Path B (implicit): --repo org/name scans tasks/ for harbor-format task dirs
Source grammar supports: bare org/name (GitHub default), full URLs (HTTPS,
SSH, git://), scheme-less host URLs, /tree/<ref>/<subdir> parsing, and
@ref pinning with SHA resolution for reproducibility.
Adds GitRepoRegistryClient with sparse checkout for efficient cloning.
17 unit tests covering parser, config validation, and client helpers.
Design doc: docs/design/registry-repo.md
* fix: --registry-path accepts both file path and directory
--registry-path /some/path/my-registry.json # uses file directly
--registry-path /some/path # assumes registry.json in dir
Also unified help text for --registry-path across all CLI commands.
* fix: add missing RegistryClientFactory import and remove unused test imports
* chore: remove design doc from PR (moved to gist)
* fix(terminus-2): deliver oversized keys via tmux paste buffers (#1873) (#1904)
Long literal payloads (e.g. heredoc answer writes) previously went
through chunked `tmux send-keys` commands, which fail with "command too
long" when they exceed the tmux client's message size limit — losing
the whole trial. The limit also varies across tmux builds, so the
hardcoded ~16 KB margin alone is not reliable.
TmuxSession now:
- stages keys whose quoted form cannot fit in a single send-keys
command into a file inside the environment (base64-chunked through
environment.exec, which is not subject to tmux limits) and pastes
them with `tmux load-buffer`/`paste-buffer`, which handle arbitrarily
large content;
- falls back at runtime when a tmux build rejects a within-margin
send-keys command as too long, resending keys individually and
pasting any that are still rejected;
- removes the old key-splitting path, which spread one literal key
across many send-keys commands and could leave half-typed input
behind on failure.
Fixes #1873
https://claude.ai/code/session_01Ww6iVMYhU7WuBcvDE8phYY
Co-authored-by: Claude <noreply@anthropic.com>
* feat: support include/exclude patterns for agent and verifier log downloads (#1889)
* Support include/exclude patterns for agent and verifier log downloads
* cleanest fix for commet1: 🔴 Reward files not protected from exclude_logs when include_logs is unset
* Add Runloop network policy launch support (#1885)
* Update network modes and capabilities in documentation
* Add Runloop network policy launch support
* Fix Runloop network policy creation races
* Fix Runloop startup workdir creation
* Expand Runloop startup workdir coverage
* Fix Codex nvm install under Runloop allowlists
* Fix Runloop startup tests without credentials
* [WIP] Computer-1 (#1572)
* harness only
* cleanup
* fixes, +example
* update impl, add providers
* pr fixes
* ruff reformatingg
* computer-1 pure litellm
* self-contained harness, providers involve deps again
* fail fast on non-vision models
* minimal computer-1 dockerfile example, remove default models, minimal deps
* fix compaction, cleanup provider classes
* reorganize, minimize provider classes
* +fable
* cleanup
* devin feedback updates
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
* Classify rate-limit agent failures as ApiRateLimitError (#1798) (#1886)
* Classify rate-limit agent failures as ApiRateLimitError (#1798)
* Lock retry-policy behavior for ApiRateLimitError (#1798)
---------
Co-authored-by: Kobe Chen <xiaokunchen0@gmail.com>
* Add sidecar artifact collection and verifier collect hooks (#1775)
* Add sidecar artifact collection and verifier collect hooks
Multi-container (compose) tasks often hold their score signal inside a
sidecar service: a database the agent wrote to, an API server that
logged the agent's requests, a load generator with in-memory counters.
In separate verifier mode all containers are torn down before
verification, so that evidence was unreachable (#1694).
Sidecars are now first-class artifact sources:
- ArtifactConfig gains a `service` field. Sidecar entries are pulled
from the named compose service's filesystem and re-materialize at
their original absolute paths in the verifier environment.
- New [[verifier.collect]] hooks run snapshot commands inside services
after the agent finishes (e.g. pg_dump), so runtime state can be
captured as files before teardown.
- In separate verifier mode the main service is stopped before sidecar
evidence is collected, so leftover agent processes cannot interfere
with collection.
- BaseEnvironment gains per-service operations (service_exec,
service_download_file, service_download_dir, stop_service),
implemented by every compose-capable provider: docker, daytona,
modal, islo.
- The host artifacts layout becomes canonical per-service:
artifacts/services/<name>/<abs path>, with the conventional publish
dir at services/main/logs/artifacts/. Verifier-side placement is
unchanged ("no translation").
- Artifact source/destination paths are validated (no '..' components,
relative-only destinations, reserved names), fixing a path traversal
where a crafted path could write outside the trial directory on the
controller host.
- Cross-service source collisions are rejected at task load so one
service's content can never masquerade as another's in the verifier.
New example task: examples/tasks/sidecar-artifacts, verified end-to-end
with the oracle agent on local Docker.
Closes #1694
* Update multi-step integration tests for per-service artifact layout
The multi-step artifact tests asserted the old flat host layout and the
old main-only download API:
- test_multi_step_downloads_convention_artifacts_per_step_non_mounted and
test_multi_step_merges_task_and_step_artifacts now assert the canonical
services/<name>/<abs path> host layout and the service-scoped download
calls.
- Add test_multi_step_collects_sidecar_artifacts_per_step covering sidecar
artifacts and step-scoped collect hooks in multi-step compose tasks:
task-level sidecar entries collected after every step, step-level entries
and hooks scoped to their step, and main never stopped mid-trial.
* artifacts: flat shared base dir instead of per-service subtree
Per review: don't segregate collected artifacts by service. All services now
share one flat artifacts/ base dir, keyed only by source path (artifacts/<abs
source>), instead of artifacts/services/<service>/<abs source>.
- paths.host_artifact_path + artifact_handler._host_path drop the services/<svc>
prefix; explicit `destination` still honored (host-only, unchanged). Verifier
upload is unaffected (it was always keyed on the artifact's source path, not
the host layout), so the copy into the verifier still works.
- Collisions are handled at collection time instead of being rejected at load:
a per-handler claim map (persisting across the main + sidecar passes) detects
exact and nested host-path overlaps; the first claimant is kept and later ones
log a warning and are skipped (never overwritten), recorded as status
"skipped" in the manifest.
- validate_artifact_entries: overlapping sources/destinations now warn instead
of raising; "services/" is no longer a reserved destination prefix (only
manifest.json remains reserved). Absolute-sidecar-source guard kept.
- Removed now-dead paths helpers (artifacts_services_dir/service_artifacts_dir)
and RESERVED_ARTIFACTS_SUBTREE.
- Updated unit + multi-step tests and the artifacts doc for the flat layout.
Verified: full test suite (2771 passed), and the kv-live-surgery sidecar oracle
runs end-to-end on this harbor (separate verifier + sidecar collect) with
reward 1.0 on docker.
* Add GKE per-service compose support; extract shared service-ops mixin
GKE was the only compose-capable provider without the per-service
operations (service_exec / service_download_file / service_download_dir
/ stop_service) that sidecar artifact collection and verifier collect
hooks require. Implement them on _GKEDinDCompose and GKEEnvironment,
mirroring the Modal/Daytona DinD pattern: sidecar execs do not inherit
main-specific defaults (workdir, default user, persistent env), and
sidecar transfers compose-cp via the pod before tarring out.
With three structurally identical env-level dispatchers, extract them
into ComposeServiceOpsMixin (environments.compose_service_ops): main
service delegates to the environment's regular methods, sidecars route
to the provider's DinD helper via the ComposeServiceTransport protocol.
Modal, Daytona, and GKE now share one implementation.
https://claude.ai/code/session_01XmMGntgUhjovVk3LKBavzU
* Add Novita per-service compose support; enforce compose-capability contract
Novita was the last compose-capable provider without the per-service
operations (service_exec / service_download_file / service_download_dir
/ stop_service) that sidecar artifact collection and verifier collect
hooks require. Implement them on _NovitaDinD (mirroring the other DinD
providers) and adopt ComposeServiceOpsMixin on NovitaEnvironment.
Add a contract test (test_compose_contract.py) that statically verifies
every environment class claiming the docker_compose capability provides
its own per-service operations instead of inheriting BaseEnvironment's
raising defaults, so new compose-capable providers cannot ship without
sidecar support again.
https://claude.ai/code/session_01XmMGntgUhjovVk3LKBavzU
* docs: align sidecar-artifact docs with flat layout; harden compose contract test
The artifact host layout was changed mid-PR to a single flat artifacts/ base
dir (no per-service subtree), but the CHANGELOG, tasks/index.mdx, and a
source docstring still described the abandoned services/<name>/ tree and the
old "collisions rejected at load" behavior. Update them to match the shipped
behavior: flat artifacts/<abs source path>, convention dir at
artifacts/logs/artifacts/, only manifest.json reserved, and overlap handling
that warns + keeps-first instead of erroring. Also list gke/novita among the
compose-capable providers.
Add test_detection_heuristic_flags_known_compose_providers so a regression in
the compose-capability detection heuristic fails loudly instead of silently
skipping a provider in the contract test.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* langsmith: implement per-service compose ops for sidecar artifacts
LangSmithEnvironment (merged from main in parallel) claims the
docker_compose capability but inherited BaseEnvironment's raising
per-service stubs, so the new compose-contract test failed on it once the
PR was merged with main. Implement service_exec, service_download_file,
service_download_dir (via the generic tar downloader), and stop_service,
following the same main-delegates / sidecar-targets pattern as the other
DinD providers. service_download_dir_with_exclusions and service_is_dir
come from BaseEnvironment for free. Add unit tests covering sidecar
targeting, main delegation, and the non-compose-mode error path, and add
langsmith to the documented compose-capable provider list.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* Scope artifact collision claims to one collection pass
_claimed_targets persisted for the whole trial, but multi-step trials
vacate the shared host artifacts dir between steps (outputs archive to
steps/<name>/), so a prior step's claims could falsely skip a later
step's entries that no longer collide. Reset claims at the start of
each _collect_artifacts_phased pass; they still span that pass's main
and sidecar phases.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Wrap sidecar execs with sh -c; keep bash for main
Sidecar containers are arbitrary third-party images where bash is
frequently absent (e.g. the *-alpine variants of postgres, redis,
nginx), while POSIX sh is universal. Switch the sidecar branch of every
compose-capable provider (docker, daytona, modal, langsmith, gke, islo,
novita) to `sh -c`, and keep `bash`/`bash -lc` for the harbor-built main
container so existing tasks that rely on bash semantics are unaffected.
Authors needing bash on a sidecar can invoke it explicitly (bash -c '...')
on images that ship it. Documents the behavior and adds docker tests.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* Update sidecar exec tests to assert sh -c
Match the provider change: sidecar service_exec now wraps with `sh -c`
across docker, modal, gke, novita, langsmith, daytona, and islo. Main
container assertions remain `bash -lc`.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* Correct CHANGELOG to match shipped artifact validation
Two fixes where the changelog described an earlier iteration of this PR
rather than the behavior relative to main:
- Collision section: main had no overlap validation at all; basename
collisions silently overwrote (last write wins). Describe that as the
prior behavior instead of "warning rather than failing" (which implied
users had hard errors to lose).
- Document the new artifact `source` `..` restriction (previously
accepted) alongside the destination rules, and retitle the section to
cover both source and destination.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* Fix stale per-service artifact layout in comments
Two comments still described the abandoned services/<service>/<abs path>
subtree as the canonical layout. The shipped layout is a single flat
artifacts/ base dir mirroring each entry's absolute source path, with no
per-service level. Update the TrialPaths docstring tree and the
_agent_env_mounts comment to match.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* Document multi-step caveat for stop-main-before-sidecar
The anti-cheat section stated unconditionally that separate verifier mode
stops main before sidecar collection. That guarantee only holds for
single-step trials and the final step of a multi-step trial: earlier
steps keep main running (later steps need it), so their sidecar evidence
is collected with the agent container still live. Add a corollary so
authors put tamper-sensitive sidecar evidence on the final step.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: Ruiyang Wang <rynewang@users.noreply.github.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Alex Shaw <alexgshaw64@gmail.com>
* fix(cli): hide removed task check and debug commands from help (#1923)
The check and debug task subcommands were removed a while ago, but they
were still registered as normal commands, so harbor task --help listed
them and showed their arguments even though running them just prints a
removal notice. Mark them hidden so they no longer show up in help, but
keep them around so the old commands still point people to the new ones.
Also fixed the error text, which said harbor tasks (plural) even when you
ran the singular harbor task.
Fixes #1751
* Add Network Allowlist provider capability to Modal (#1932)
* Remove deterministic OpenHands test (#1936)
* fix(goose): keep token counts when trajectory write fails (#1933)
* fix(goose): keep token counts when trajectory write fails
In the Goose agent, the token count was set inside the same try block that writes trajectory.json. If that write failed, the error was logged at debug level and the token count was never set, so it stayed at 0. This made the trial look like it used no tokens, which throws off cost reporting and leaderboard scores.
Move the token count out of the write block: the write keeps its own try/except, and the token count is read from the in-memory metrics right after, so it is always set even when the write fails. This matches how codex.py, opencode.py, and openhands_sdk.py already work.
Fixes #1709.
* feat(goose): record input/output token split when goose reports it
goose 1.37 (block/goose#8870) added input_tokens and output_tokens to the stream-json complete event, flat alongside total_tokens. The adapter only read total_tokens and stored the combined total in n_input_tokens. Read the input/output split and set n_input_tokens and n_output_tokens, falling back to the combined total for older goose that reports only total_tokens.
Addresses review feedback on #1933 (Devin) about token semantics.
* refactor(goose): use standard FinalMetrics token fields
Populate FinalMetrics.total_prompt_tokens / total_completion_tokens and read them back in populate_context, matching codex.py and opencode.py, instead of storing token counts under custom extra keys. This keeps downstream trajectory.json consumers reading goose the same as every other agent. Older goose that reports only a combined total keeps it in total_prompt_tokens to preserve prior behaviour; the raw total is retained in extra.
Addresses Devin review feedback on #1933.
* docs: add documentation for --repo git repository datasets (#1906)
* feat: add MiMo agent — Xiaomi's MiMoCode CLI with ATIF trajectory support (#1899)
Add the MiMo agent (Xiaomi MiMoCode CLI) as a built-in Harbor agent.
- Register `mimo` in `AgentName` enum and `AgentFactory`
- Implement `MiMo` class extending `BaseInstalledAgent` with:
- Installation via `curl -fsSL https://mimo.xiaomi.com/install | bash`
- ATIF trajectory generation from `mimo run --format=json` stdout
- MCP server and skills directory registration
- Provider-specific environment variable passthrough
- Configurable CLI flags (e.g. `--variant`)
- Deep-merge config support via `mimo_config` kwargs
* feat(islo): support basic no-network policy (#1935)
* feat(islo): support basic no-network policy
Map Harbor's portable no-network mode to a static Islo gateway profile so Islo can provide basic allow/deny behavior without exposing gateway-specific task schema.
Co-authored-by: Cursor <cursoragent@cursor.com>
* feat(islo): support dynamic network policy
Co-authored-by: Cursor <cursoragent@cursor.com>
* docs(islo): add network policy example
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(islo): avoid compose no-network gateway overlay
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(islo): stabilize compose policy switching
Co-authored-by: Cursor <cursoragent@cursor.com>
* docs(tasks): update Islo network policy support
Co-authored-by: Cursor <cursoragent@cursor.com>
* refactor(islo): minimize PR1 review surface
Restore existing Islo naming and docs footnote ordering while keeping
portable network policy behavior unchanged.
Co-authored-by: Cursor <cursoragent@cursor.com>
* docs(tasks): drop redundant Islo network policy footnotes
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(islo): use gateway for compose network policy
Use Islo gateway profiles as the single enforcement layer for portable
network policy, including Docker Compose tasks, so runtime phase switching
has one mutable source of truth.
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Alex Shaw <alexgshaw64@gmail.com>
* refactor(env): unify DinD compose providers on shared service-ops/transfer layer (includes #1775 sidecar collection) (#1843)
Extracts the duplicated DinD compose operations layer into a shared
DinDComposeOps class and ports all five remote DinD providers onto it.
- environments/dind_compose.py: compose exec arg building, two-hop
uploads/downloads staged through the DinD host via `docker compose cp`,
self-bind log-dir fast path, `compose stop`, ComposeServiceTransport
adapters, is_dir/is_file -- implemented once. Providers supply six
primitives (host exec + local<->host file moves) and may override the
cp timeouts.
- Modal, Daytona, GKE, Novita: drop their copy-pasted ops; GKE keeps
transfer retries via thin wrappers; Novita preserves slower cp timeouts.
- Islo: thin _IsloComposeOps adapter maps the shared layer onto its
sandbox/compose helpers.
- Local Docker stays separate by design (single-hop compose cp, Windows).
Sidecar service_exec wraps commands with POSIX `sh -c` (not `bash -lc`)
so they run on minimal images such as the *-alpine variants; the main
service keeps bash. Rebased onto current main: retains the finalized flat
artifact layout, the include/exclude log filtering, and the post-branch
provider features (#1861 docker --rmi local, #1867 novita network policy,
#1731 daytona GPU).
Net ~ -400 lines. Contract test enforces that any environment claiming
docker_compose capability implements the per-service operations.
Co-authored-by: Ruiyang Wang <rynewang@users.noreply.github.com>
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
Co-authored-by: Alex Shaw <alexgshaw64@gmail.com>
* Tighten verbose comments in rewardkit judges (#1863)
* Tighten verbose comments in rewardkit judges
* Potential fix for pull request finding
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
* Potential fix for pull request finding
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
* Computer-1 Caching (#1939)
* +caching for computer-1
* devin comment
* formatting
* Network policy: allow empty allowlists; clarify wildcard for apex domain (#1940)
* Allow empty network allowlists
* Document wildcard allowlist apex behavior
---------
Co-authored-by: Boxuan Li <boxuanli@microsoft.com>
* Add Daytona sandbox labels (#1930)
* feat(daytona): add opt-in sandbox labels
Attach harbor.managed / harbor.environment_name / harbor.session_id to
each Daytona sandbox when auto_labels=true (default off), plus arbitrary
user labels via the labels kwarg (applied independently of the gate).
Set at the single create choke point, so the image, snapshot, and DinD
paths are all covered.
Signed-off-by: rovle <lovre.pesut@gmail.com>
* Simplify Daytona sandbox labels
---------
Signed-off-by: rovle <lovre.pesut@gmail.com>
Co-authored-by: Alex Shaw <alexgshaw64@gmail.com>
* fix(upload): avoid ambiguous primary rewards (#1813)
Co-authored-by: sudolinzekun <sudolinzekun@local>
Co-authored-by: benediktstroebl <50178209+benediktstroebl@users.noreply.github.com>
* Fix E2B exec retry safety (#1941)
- retry E2B command dispatch only for failures that indicate the command did not start
- stop retrying post-dispatch `handle.wait()` failures to avoid double-running non-idempotent agent commands
- add unit coverage for safe retry, non-retry, and nonzero-exit behavior
* Add strict type check (#1947)
* Update ty from 0.0.19 to 0.0.49 (#1948)
* Publications
* Remove ruff from dependencies (#1951)
* Add Blaxel cloud sandbox provider (#1643)
* Add Blaxel cloud sandbox provider
* Remove internal Blaxel wording
* Put Blaxel first in provider lists
* Address review feedback: shared helpers, image caching, faithful images
- Reuse definition.py helpers; move multi-stage WORKDIR parsing into
parse_dockerfile_workdir
- Cache sandbox images by content + build-config hash and honor
force_build; reap the builder sandbox with a short TTL
- Declare resource_capabilities (memory requests honored)
- Preflight auth via the SDK's get_credentials
- Run exec through bash -c to match the Docker provider
- Opt out of Blaxel rootfs slimming via blaxel.toml so sandbox
filesystems match the task Dockerfile byte-for-byte
- Derive image build timeout from the task's build_timeout_sec
* Upload environment files after Blaxel start
Ensure prebuilt-image Blaxel tasks receive supplementary environment files, matching other environment providers.
* Use debug log for retained Blaxel sandbox
Match environment logging conventions for delete=False cleanup paths.
* Update Blaxel provider for current type checks
* Append Blaxel to provider lists and remove dedicated example
Move Blaxel to the end of the cloud sandbox provider lists in the docs, pyproject extras, environment enum, and factory registry, and drop the Blaxel-specific example from the sandboxes doc.
* add support for modal vm runtime (#1907)
* Add override.
* docs: add Novita sandbox provider references (#1811)
* docs: add Novita sandbox provider references
* docs: mark Novita sidecar artifacts supported
* docs: align compose provider support
* docs: address Novita provider list feedback
* docs: limit Novita provider list updates
* docs: only append Novita to network allowlist
* docs: restore network policy wording
* docs: fix Novita no-network annotation
---------
Co-authored-by: Alex Shaw <alexgshaw64@gmail.com>
* Add configurable Claude Code permission mode (#1950)
* Add configurable Claude Code permission mode
* Handle null Claude Code permission mode
* Document null permission mode opt-out
* Remove Claude Code permission docs update
* Update Claude Code permission modes
---------
Co-authored-by: Kobe Chen <xiaokunchen0@gmail.com>
* Add Terminus viewer support for JSON payloads and split views (#1958)
* Extract viewer updates from Terminus branch
* Address Devin viewer review feedback
* Address follow-up Devin viewer feedback
* v0.14.0
* fix(daytona): create workdir in sandbox if specified (#1953)
* perf(registry): enumerate --repo tasks via git ls-tree (#1920)
* perf(registry): enumerate --repo tasks via git ls-tree
Implicit dataset enumeration sparse-checked-out the entire tasks/
directory just to read task directory names, materializing all blob
content (e.g. ~932MB / ~4k files for frontier-swe) into a throwaway
temp dir on every run. This dominated --repo startup (~64s before the
environment even began building).
Read the git tree directly instead: a new _tree_only_clone does a
blobless --no-checkout clone + fetch of the target sha, and
_get_implicit_metadata uses `git ls-tree` to find top-level task dirs
containing a task.toml. No blob content is downloaded; per-trial task
downloads (already lazy and cached) are unchanged.
Enumeration of frontier-swe drops from ~64s to ~3s with identical
results (17 tasks). Adds regression tests asserting enumeration returns
the right tasks and never issues a checkout/sparse-checkout.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(registry): handle repo-root subdir in ls-tree enumeration
When tasks live at the repo root (--path .), _effective_subdir returns
"." and the prefix became "./", but `git ls-tree` emits root-relative
paths with no leading "./" — so every line was filtered out and
enumeration spuriously found zero tasks.
Special-case the root: skip the subdir-existence check (root always
exists), omit the path arg from `git ls-tree` so it lists from the root,
and use an empty match prefix. Adds a regression test for the root case.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* Use Modal filesystem list_files API (#1978)
* fix(langgraph): harden dependency installation (#1973)
* fix(langgraph): allow prerelease dependencies
* fix(langgraph): scope prerelease installs
* fix: detect missing ensurepip for langgraph agent
* fix: generalize langgraph prerelease installs
---------
Co-authored-by: Kobe Chen <xiaokunchen0@gmail.com>
* fix(langgraph): preserve summary cancellation propagation (#1974)
* fix: ignore cancelled langgraph summary downloads
* fix(langgraph): preserve cancellation propagation
---------
Co-authored-by: Kobe Chen <xiaokunchen0@gmail.com>
* fix(langsmith): retry transient requests (#1975)
* fix: retry transient langsmith requests
* fix(langsmith): return unexpected successful responses
---------
Co-authored-by: Kobe Chen <xiaokunchen0@gmail.com>
* feat: harborize harbor check (#1924)
* feat: harborize harbor check
* fix majority of the problems
* refactor: derive check template paths from TaskPaths
Restructure check_task_template/ into a task skeleton (task.toml at root, tests/test.sh, tests/validate.py) so assemble_check_task reads the source side through TaskPaths, removing the hardcoded "test.sh" and "task.toml" literals.
* fix: add dict type argument to output_schema for ty check
* refactor: update task path handling to use dynamic workdir in check templates
* fix: kebab-case for check-result
* --env
* remove hardware resource
* make bot happy
* fix: align check --model default to claude-sonnet-4-6
* last fix
* refactor: write check result to workdir, collect as artifact
* Remove overlapping network policy artifacts (#1979)
* fix(langsmith,langgraph): tag traces with harbor runner (#1976)
* fix: tag langsmith traces with harbor runner
* chore(langsmith): rerun checks
* style: fix formatting in langsmith plugin test
---------
Co-authored-by: Kobe Chen <xiaokunchen0@gmail.com>
* fix(modal): keep direct-mode sandbox alive with sleep infinity (#1545)
Modal's direct strategy ran the image's ENTRYPOINT/CMD as the sandbox's
main process.
Task images that reset ENTRYPOINT and rely on an external
keepalive (e.g. SWE-Bench Pro) terminated immediately, causing
follow-up mkdir/exec calls to fail with "request cancelled due to
internal error".
Direct mode now passes ["sh", "-c", "sleep infinity"]
by default — matching the convention in docker, apple_container, and
islo — and exposes a `keepalive` env kwarg for task authors who need
to override or opt out. DinD is unchanged so dockerd still starts.
Signed-off-by: James Kunstle <j5@notdiamond.ai>
* fix(modal): create workdir in sandbox if specified (#1981)
* fix(copilot-cli): parse Copilot's native session-event JSONL schema (#1985)
* fix(copilot-cli): parse Copilot's native session-event JSONL schema
`copilot --output-format json` emits a namespaced session-event stream
(`assistant.message` / `tool.execution_*` / `result`) for OpenAI/GPT
models that `_convert_jsonl_to_trajectory` didn't recognize, so GPT runs
matched zero events and wrote no trajectory or token counts. Parse it
alongside the existing flat Anthropic schema (unchanged), matching tool
results to calls by id and summing `outputTokens`.
Parsing is per-event, so a malformed event is salvaged rather than
discarding the whole run, and unmapped fields, the `result` summary, and
sibling tool-result keys are preserved rather than dropped. Add tests
for both schemas.
* copilot-cli: keep call id on orphan tool-result steps
* docs(adapters): fix stale parity_experiment.json filename in READMEs (#1964)
The adapter validator and wizard reference `parity_experiment.json`
(singular), but the deveval and ineqmath READMEs still pointed at the
plural `parity_experiments.json`. Align the docs with the contract.
Co-authored-by: EazyReal <8047065+EazyReal@users.noreply.github.com>
Co-authored-by: Ivan Bercovich <ibercovich@gmail.com>
* feat(agents): add dspy.RLM agent (#1965)
Adds `DspyRlmAgent` (`dspy-rlm`), a host-side agent wrapping `dspy.RLM`.
RLM lets the model explore large contexts through a sandboxed Python REPL
on demand instead of putting the whole workspace in the prompt.
An `EnvironmentToolBridge` exposes the Harbor environment to RLM's
synchronous tool calls (exec, read/write file, list, find, search, patch)
by bridging them onto the running event loop via
`asyncio.run_coroutine_threadsafe`. `dspy` is an optional extra
(`harbor[dspy]`) and is lazy-imported, so normal Harbor imports are
unaffected. The agent is registered in `AgentName`/`AgentFactory`.
Testing follows the existing deterministic pattern: a `runtime` integration
test drives RLM with a scripted `DummyLM` (no model API key) through the
real Deno REPL and a real Docker environment, comparing the RLM trajectory
to a golden. CI installs Deno (no secret) so this runs upstream. A separate
real-model e2e remains available locally/manually.
Co-authored-by: EazyReal <8047065+EazyReal@users.noreply.github.com>
Co-authored-by: Ivan Bercovich <ibercovich@gmail.com>
* Retry Daytona process session creation, treating duplicate as success (#1954)
create_session can fail with a transient error after the session was
actually created server-side; the retry then sees a 'session already
exists' conflict. Wrap creation in a retry and treat that specific
conflict (DaytonaConflictError with a 'session already exists' message)
as success so retries are idempotent.
Signed-off-by: rovle <lovre.pesut@gmail.com>
* feat: harborize harbor analyze (#1984)
* feat: harborize harbor analyze
* feat: render harbor analyze json as viewer ui
* fix: always include analysis key in agent-logs response
* Tighten analysis section header spacing (#1987)
* fix(mini-swe-agent): honor configured agent env for credentials (#1968)
* fix(mini-swe-agent): honor configured agent env for credentials
mini-swe-agent resolved its model credentials and API base directly from
`os.environ`, so values supplied through `AgentConfig.env` (CLI `--ae`) /
`extra_env` were ignored — and the agent would raise "Unset API variable"
even when the key was configured.
Resolve `MSWEA_API_KEY`, the model's API key vars, and the API base
through `BaseInstalledAgent._get_env`, which prefers `extra_env` and falls
back to `os.environ`. Also forward `OPENAI_BASE_URL` alongside
`OPENAI_API_BASE` for OpenAI-compatible providers.
* fix(mini-swe-agent): alias OPENAI_API_BASE and OPENAI_BASE_URL
These are two names for the same endpoint (LiteLLM vs the OpenAI SDK).
Forwarding each only under its own name leaves a user who sets one but
whose downstream tool reads the other empty-handed. Resolve whichever is
set and forward both names. Adds a regression test.
---------
Co-authored-by: EazyReal <8047065+EazyReal@users.noreply.github.com>
Co-authored-by: Ivan Bercovich <ibercovich@gmail.com>
* feat(trial): scoped log streaming via Trial.add_log_callback (#1966)
* feat(trial): scoped log streaming via Trial.add_log_callback
Trials can run for a long time, but callers only see command output after
each command completes (buffered ExecResult) or by scraping provider log
files. This adds an opt-in, Harbor-native way to observe stdout/stderr as
it is produced.
`Trial.add_log_callback(cb)` subscribes a `LogCallback` (list semantics,
mirroring `add_hook`). While agent setup, agent execution, or verification
run, the trial scopes the callbacks on the real environment via
`BaseEnvironment.scoped_output_callback(...)` (a contextvar, for async-task
isolation across awaited exec calls). Raw `(text, stream)` chunks become
structured `LogEntry` values (a Pydantic model, like `TrialHookEvent`)
carrying trial id, phase, stream, text, timestamp, and optional step name.
Docker streams for real; the environment object is never wrapped.
Zero cost by default: with nothing subscribed, `exec` uses the existing
buffered path unchanged.
* fix(docker): bound streamed exec wait() by timeout_sec
In _collect_streamed_output, process.wait() ran after the timed read loop
with no deadline, so a process that closed stdout but failed to exit could
block past timeout_sec — unlike the buffered path, where wait_for wraps the
whole communicate(). Move the wait inside the timed coroutine so the
streamed path honors timeout_sec end-to-end. Adds a regression test.
---------
Co-authored-by: EazyReal <8047065+EazyReal@users.noreply.github.com>
Co-authored-by: Ivan Bercovich <ibercovich@gmail.com>
* rewardkit: honor individual mode for agent judges (#1793)
* rewardkit: support Claude subscription auth for judges
* simplify OAuth auth: drop CLAUDE_FORCE_OAUTH and rely on litellm's sk-ant-oat detection
* add auth docs reference and setup-token hint
* only pass api_key to litellm when non-None
* generalize credential resolution: _resolve_credentials step with anthropic oauth as the one impl
* trim _resolve_credentials docstring
* add REWARDKIT_FORCE_OAUTH to prefer subscription token over API key when both set
* codex agent judge: log in with CODEX_ACCESS_TOKEN subscription token when set
* drop internal codex login command from docs
* simplify Anthropic judge model wording in docs
* docs: 'has priority' instead of 'wins'
* codex: OPENAI_API_KEY has priority over subscription token unless REWARDKIT_FORCE_OAUTH; share force_oauth helper
* docs: clarify CODEX_ACCESS_TOKEN is an enterprise access token, note where to create it
* fix(rewardkit): record judge timeouts instead of crashing the whole run
A single judge call that times out aborted the entire verifier run and wrote
no reward file: the timeout propagated through the per-criterion TaskGroup
(individual mode), Reward.arun, and runner._run_all (all fail-fast), and
_write_outputs runs only after _run_all succeeds — so the process exited with
neither reward.json nor reward-details.json, surfacing to Harbor as an opaque
RewardFileNotFoundError rather than a timeout.
Catch the timeout at the two judge call seams and convert it to errored 0.0
Scores instead of propagating:
- _arun_llm_call: catch litellm.Timeout (covers batched and individual LLM).
- arun_agent: on asyncio.TimeoutError, return errored scores (covers agent
judges, which are always a single call).
Each timed-out criterion is recorded as Score(value=0.0, error="judge timed
out after Ns") via the existing-but-previously-unused Score.error field, so a
valid reward file is always written, the timeout is attributed per criterion
in reward-details.json, and sibling criteria/rewards keep their scores. Only
timeouts are caught; other errors (parse failures, missing-extra ImportError)
still surface loudly.
Closes #1790
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* clean up timeout comment in agent judge path
* remove timeout non-retry comment
* fix ty: narrow str | None to str in _timeout_scores
* remove redundant timeout comment in LLM judge path
* validate criterion names against provider key constraint at construction time
* fix(rewardkit): honor mode="individual" for agent judges
Agent judges silently ignored [judge].mode = "individual": AgentJudge had no
`mode` field, so an agent judge always graded every criterion in a single
batched call regardless of the rubric — no warning, no error. A rubric
authored for per-criterion grading just ran batched.
Honor it: add `mode` to AgentJudge and, in individual mode, grade each
criterion in its own agent call. Agent calls are heavy local CLI subprocesses,
so the individual path runs them sequentially (one per criterion) rather than
fanning out concurrently like the LLM path — N parallel agent processes would
exhaust a small verifier container. arun_agent is split into a dispatcher +
_arun_agent_call (single call) + _arun_agent_individual (sequential per-criterion);
batched behavior and the public arun_agent signature are unchanged.
Closes #1792
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* tighten _arun_agent_call: remove unused defaults on private function
* fix test: use criterion names as response keys now that c0/c1 is dropped
* rewardkit: drop redundant docstrings on private helpers
* rewardkit: align timeout score names with parse path
* docs(rewardkit): drop change-framing from judge timeout note
* chore: remove lengthy docstring
---------
Co-authored-by: benediktstroebl <stroebl@princeton.edu>
Co-authored-by: benediktstroebl <50178209+benediktstroebl@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* Add separate verifier network mode matrix demo (#1995)
* Fix network-policy matrix artifact warnings (#1994)
* feat(islo): auto-delete sandboxes after 1h and shield create on cancel (#1990)
* feat(islo): auto-delete sandboxes after 1h and shield create on cancel
Set lifecycle.delete_after on sandbox creation (default 3600s, configurable
via delete_after_seconds) so leaked Islo VMs are reclaimed server-side.
Shield create from cancellation using the Daytona pattern so Ctrl+C can still
route cleanup through trial stop().
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(trial): await environment stop on cancel before loop exits
asyncio.shield alone detached stop() on Ctrl+C; the trial caught
CancelledError without waiting, so IsloEnvironment.stop() was destroyed
pending and sandboxes leaked. Uncancel the trial task and await stop()
to completion after cancellation.
Co-authored-by: Cursor <cursoragent@cursor.com>
* revert: drop trial.py cancel cleanup changes
Keep the Islo auto-delete lifecycle scoped to the islo environment only.
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
* Add execute permissions to the executables of all example tasks. (#1996)
* feat(tensorlake): prebuilt docker_image import, OCI build by default, and boot-path hardening (#1989)
* cleanup
* add docker image import
* honor force build and fix name collosion
* update
* stop sandbox when tasks are done
* fix
* type check
* fix(tensorlake): create /run/lock on every MicroVM boot
Ubuntu/Debian's /var/lock is a symlink to /run/lock, which
systemd-tmpfiles normally creates at boot but a MicroVM boot does not,
leaving /var/lock a dangling symlink. Scripts then fail at
`mkdir -p /var/lock/<x>` with "cannot create directory '/var/lock':
File exists", breaking services that lock there (e.g. mailman never
starts, so its SMTP/LMTP ports never bind, and the verifier fails with
ConnectionRefused -> reward 0).
Add an idempotent shim in _install_persistent_shims() alongside the
/dev/fd shim. It runs on every boot before the registered-image
early-return, so it covers the docker_image -> registered-image path too.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Add sandbox labels for Modal environments (#1999)
* For Modal - don't use host networking for vm runtime sandboxes (#1998)
* changelog for harborized check and analyze (#2001)
* feat: harborize Generate Analysis on UI job-level (#2002)
* feat: harborize Generate Analysis on UI job-level
Make the job-level analyze flow match the working trial-level one. The
job summarize endpoint now runs harbor analyze (run_analyze) in a chosen
environment instead of the legacy local Analyzer, writing the aggregated
analysis.json back to the job dir for the Analysis tab to render.
- add an Environment selector to the job Generate Analysis dialog
- reword Concurrent Claude Codes to Concurrent analyses
- default Only analyze failed trials to off
- render structured analysis.json (per-trial results) instead of the
no-longer-written analysis.md; drop the dead summary endpoint
- extract shared AnalysisContent component used by trial and job pages
Adds a test covering the harborized summarize_job endpoint.
* feat: add agent selector to analysis form
Let the Generate Analysis dialogs (job and trial) pick the agent that
runs the analysis, mirroring the existing environment selector. The
summarize endpoints forward the chosen agent to run_analyze; /api/config
exposes the available agents. Model selection is unchanged, so for now
only claude-code is guaranteed to work end-to-end.
* feat: curate agent and model dropdowns in analysis form
Constrain the Generate Analysis form to a supported subset (claude-code,
codex, cursor-cli) with a per-agent model dropdown instead of exposing the
full agent enum. Model options are hardcoded per agent, smallest first and
used as the default; picking an agent resets the model. Field order is now
agent then model. Other agents/models remain available via harbor analyze.
Drops the /api/config agents field added for the selector; the form sources
agents from the frontend map.
* Add Openshift Environment (#1875)
* Add Openshift Environment
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Fix file mounts
* Surface actionable error when OpenShift ServiceAccount is missing
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Add custom SCC for openshift pods
Co-Authored-By: Claude <noreply@anthropic.com>
* Fix type checker errors in OpenShift environment
Co-Authored-By: Claude <noreply@anthropic.com>
* Apply suggestions from Devin code review
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* Fix failing test
* Assign resources only when defined
* Handle None cpus/memory_mb in OpenShift pod spec
Co-Authored-By: Claude <noreply@anthropic.com>
* Use resource enforcement policy helpers in OpenShift environment
The OpenShift environment was hardcoding resource requests/limits directly
from task_env_config, bypassing the standard policy helpers that GKE uses.
This meant --cpu-enforcement-policy / --memory-enforcement-policy flags had
no effect. Now resource_capabilities() returns all True and _pod_spec() uses
_resource_request_value/_resource_limit_value, bringing OpenShift in line
with GKE's resource handling.
Co-Authored-By: Claude <noreply@anthropic.com>
* Apply fixes from Devin
* Fix stop() deleting shared build resources used by concurrent trials
The BuildConfig and ImageStream are keyed by environment_name (shared
across trials of the same task), but stop(delete=True) was removing them.
This caused concurrent trials to rebuild from scratch or fail with
ImagePullBackOff. Now only the per-trial pod is deleted, preserving the
cached image for reuse.
Co-Authored-By: Claude <noreply@anthropic.com>
* Fix review issues in OpenShift environment
- Simplify _sanitize_k8s_name: remove dead branch after strip("-")
- Handle OSError in _start_log_streaming to keep file handle and
streamer process in sync
- Use time.monotonic() in _wait_for_pod_ready for accurate wall-clock
timeouts instead of range()-based iteration count
- Map None returncode to -1 instead of 0 in _run_oc_command to surface
unexpected process state as an error
Co-Authored-By: Claude <noreply@anthropic.com>
* Pre-create directories for all bind mounts, not just writable ones
Ensures read-only file mount uploads don't fail due to missing parent
directories on oc versions where oc cp doesn't implicitly create them.
Co-Authored-By: Claude <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Kobe Chen <xiaokunchen0@gmail.com>
* fix(modal): reuse env secrets across sandbox operations (#1969)
* fix(modal): reuse env secrets across sandbox operations
Modal creates an ephemeral Secret for every Secret.from_dict object.
Under high concurrency, identical env payloads produced a fresh Secret
per sandbox startup and per exec, so a startup/setup burst could hit
Modal's workspace-wide secret create-rate limit even though the env
payload never changed.
Cache Secret.from_dict objects by an order-independent env-payload key
and reuse them for both sandbox creation (_secrets_config) and per-exec
(_sdk_exec). Named secrets are unchanged.
* fix(modal): bound env secret cache as an LRU
Address review feedback on #1969: the module-level _ENV_SECRET_CACHE
could grow without limit in a long-running process that sees many unique
per-exec env payloads, retaining a Secret object for every payload ever
observed.
Back the cache with an OrderedDict and evict the least-recently-used
entry once it exceeds _ENV_SECRET_CACHE_MAXSIZE (256). Cache hits move
the key to the most-recently-used position. This keeps memory bounded
while still collapsing the bursts of identical payloads that motivate
the cache.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BT4bKRHP5ePMZGDnUuHbBU
---------
Co-authored-by: EazyReal <8047065+EazyReal@users.noreply.github.com>
Co-authored-by: Kobe Chen <xiaokunchen0@gmail.com>
* Show cursor model names in viewer analysis (#2007)
* fix(rewardkit): codex judge auto-logs-in from OPENAI_API_KEY (#2009)
* Add Eve agent implementation and task configs (#1982)
* Add Eve agent support
* Make Eve info diagnostics non-fatal
* Allow recoverable Eve step failures
* Fix dev typecheck extras (#2012)
* Fix auth credentials storage writes (#2013)
* v0.15.0
* Add per-agent concurrency limits and agent-end hooks (#2014)
* Add agent concurrency limits and hook events
* Fix Windows CLI error output assertion
* Address agent concurrency CLI feedback
* Add Blaxel network policy and DinD support (#1980)
* Add Blaxel network policy and DinD support
* Fix Blaxel compose host exec defaults
* Format Blaxel regression tests
* Keep Blaxel compose staging on host defaults
* Keep cloud sandbox docs scoped to Blaxel
* Clarify Blaxel compose sandbox behavior
* Forward referenced compose env vars on Blaxel
---------
Co-authored-by: Ivan Bercovich <ibercovich@gmail.com>
* feat(agents): propagate AgentConfig.env to all agent load paths (#1967)
* feat(agents): propagate AgentConfig.env to all agent load paths
`AgentConfig.env` (CLI `--ae KEY=VALUE`) only reached the sandbox for
`BaseInstalledAgent` subclasses, which merged it into their own per-exec
calls. Arbitrary `BaseAgent` subclasses — including import-path agents
(`--agent module:Class`) — silently dropped it.
Carry the resolved env on `BaseAgent` for every load path, and apply it
as a scoped overlay (via a contextvar) only while agent setup and agent
run execute. `_merge_env` layers it as persistent < per-exec < agent env,
matching prior installed-agent precedence. The overlay is deliberately
scoped off verifier, build, and artifact commands so agent credentials
never leak into verification.
* fix(oracle): keep solution.env authoritative over agent env overlay
Removing oracle's inline extra_env merge let the trial's agent-env overlay
(highest precedence in _merge_env) win over the task's solution.env, flipping
the prior precedence. Scope solution.env around the solution exec so it layers
above the overlay and stays authoritative for a faithful oracle run.
Also documents why scoped_exec_env uses contextvars rather than
with_default_user's attribute save/restore (async-task isolation), and adds a
nested-overlay precedence test.
* fix(environments): scope exec-env overlay per environment instance
Devin flagged that scoped_exec_env held its overlay in a module-level
ContextVar shared by every BaseEnvironment, so the "this environment"
contract was misleading and the overlay could leak across instances in
the same async task.
Move it to a per-instance ContextVar, mirroring the already-merged
scoped_output_callback (self._output_callbacks): the instance attribute
scopes the overlay to one environment while the ContextVar keeps it
isolated per asyncio task across await boundaries. This makes the
contract literally true and settles the file on one scoped-context
pattern, as requested in review.
Add a regression test asserting the overlay does not leak to a second
environment instance.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FCC75BKFCNSiNcJGzuhsNb
---------
Co-authored-by: EazyReal <8047065+EazyReal@users.noreply.github.com>
Co-authored-by: Ivan Bercovich <ibercovich@gmail.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat: mvp for run launcher directly in viewer (#2017)
* feat: mvp for run launcher directly in viewer
* feat: add stop button to cancel launcher-spawned runs
* feat: implement git repo skills (#1909)
* docs: add design doc for git-based skills support
Design for extending git repo resolution (from datasets PR #1884) to skills.
Resolves git repos to local cache paths early, feeding into existing
list[Path] pipeline unchanged.
Key decisions:
- --skill accepts strings, not just paths
- Git sources resolve to cached local dirs before entering pipeline
- AgentSkillLock gets optional git_url + git_commit_id fields
- No new CLI flags, models, or registry formats
* feat: implement git repo skills
Add support for specifying skills from git repositories via CLI:
--skill org/name[@ref]
--ski…
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up work of #1984
Previous behavior:
Current behavior:
job level rendering
trial level rendering after clicking on job level generate analysis
Make the job-level analyze flow match the working trial-level one. The job summarize endpoint now runs harbor analyze (run_analyze) in a chosen environment instead of the legacy local Analyzer, writing the aggregated analysis.json back to the job dir for the Analysis tab to render.
analysis.mdbut render in uiAdds a test covering the harborized summarize_job endpoint.