test(pass-at-k): add unit tests for pass@k metric utility#2046
Merged
Conversation
`src/harbor/utils/pass_at_k.py` computes the user-facing pass@k metric
(consumed by `harbor.job`, stored on `JobStats`, rendered by the `harbor
jobs` CLI) but had no test coverage.
Add `tests/unit/test_pass_at_k.py`:
- Validate `_pass_at_k_for_task` against an independent closed-form
reference (`1 - C(n-c, k)/C(n, k)`) over a grid of (n, c, k), plus
boundary cases (c==0, c==n, n-c<k).
- Cover `_eligible_k_values` (powers of two ∪ multiples of five, sorted
and deduplicated; empty below 2).
- Exercise the public `compute_pass_at_k_by_evals`: grouping by eval key,
averaging across tasks, missing-verifier-as-failure, and the guards
that return `{}` for non-binary/out-of-range/multi-key rewards and
single-attempt inputs.
Test-only; no production code changes.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
@xyuzh is attempting to deploy a commit to the Harbor Framework Team on Vercel. A member of the Team first needs to authorize it. |
Contributor
|
Enjoy a better diff viewing experience by clicking one of these URLs: |
Contributor
Author
alexgshaw
approved these changes
Jun 23, 2026
Messimeimei
added a commit
to Messimeimei/bitfun-harbor
that referenced
this pull request
Jul 7, 2026
338b7d0 Add per-agent concurrency limits and agent-end hooks (harbor-framework#2014) a264cef Add Blaxel network policy and DinD support (harbor-framework#1980) d0e0869 feat(agents): propagate AgentConfig.env to all agent load paths (harbor-framework#1967) 4b09d97 feat: mvp for run launcher directly in viewer (harbor-framework#2017) e266675 feat: implement git repo skills (harbor-framework#1909) 31b38dd Skip uv download if already installed for mini swe agent (harbor-framework#1952) c6f4898 Refactor Harbor task handling and viewer UI (harbor-framework#2023) 19585c3 feat(atif): bring installed-agent trajectory converters to full ATIF v1.7 fidelity (harbor-framework#1888) 6485949 Add EC2 sandbox environment (harbor-framework#2031) c004b05 [CI] Post adapter validation as a fresh timestamped comment (harbor-framework#2040) 6671d38 Handle API usage limit errors (harbor-framework#2037) 34e545b add `harbor run --install-only` flag (harbor-framework#1983) c2e11d6 ci: update to windows server 2025 (harbor-framework#2033) 20160fc fix(swe-agent): don't let testbed-conda.sh abort under set -u (harbor-framework#2039) 7657d45 feat: combine --agent-import-path into --agent (harbor-framework#1988) 496293b test(pass-at-k): add unit tests for pass@k metric utility (harbor-framework#2046) 472bf7e fix(docker): fall back to tar upload when compose cp fails (harbor-framework#1944) 1cd8438 feat(cli): add agent/verifier log-filter flags to `harbor trial start` (harbor-framework#2047) 8cf5fdf fix(openhands-sdk): use uv to pin Python >=3.12 for SDK venv (harbor-framework#2041) 84b2a96 Api key auth (harbor-framework#2062) 52a31c3 Network mode support for docker environment (harbor-framework#1799) 3751070 Preview task definition image files (harbor-framework#2055) 0e8ae01 Add bash action, start_url, and configurable IO dir to computer-1 (harbor-framework#2042) 87c3c47 Minor updates. f3e7488 Merge branch 'main' of github.com:harbor-framework/harbor c0cecbb fix(resume): tolerate empty/truncated result.json on resume (harbor-framework#2053) dcaaebe default internet (harbor-framework#2066) 629f8ae Fix Gemini CLI JSONL session handling (harbor-framework#2069) 86d3bc0 fix(llms): cap Anthropic cache_control breakpoints at 4 (harbor-framework#2072) 4e6dfd6 Harden Kimi CLI runtime configuration (harbor-framework#2070) 0bc7e18 deps: pin transformers>=5.3.0 to fix CVE-2026-4372 (harbor-framework#2004) 6e2d9f5 Fix mini-swe-agent ATIF steps for responses API (harbor-framework#2073) f7110f1 feat(launcher): multi-agent/multi-dataset run launcher w/ configs (harbor-framework#2057) 053c05c feat(env): add Cua Cloud environment for desktop VMs (harbor-framework#1957) 5afda90 Add Codex web search CLI flag (harbor-framework#2032) c390779 Reorganize example configs by category (harbor-framework#2083) 0cdf5cf Add max_tokens support to MiniSweAgent (harbor-framework#2082) bd5005c Refine trial viewer typography and step detail interactions (harbor-framework#2077) a7667a0 fix(langgraph): forward task-environment MCP servers to the graph (harbor-framework#2085) 7ae9ee0 feat(agents): skip install for pre-built images (auto-detect + guarded mkdir) (harbor-framework#1970) 6fca45a feat: reload past run configs (harbor-framework#2076) 3fb4a92 Link observation results to source tool calls (harbor-framework#2087) e90f959 feat(viewer): save launcher config as YAML (harbor-framework#2089) 575a102 Remove job plugins from config files (harbor-framework#2074) 4340b0c Move verifier job options into a dedicated help panel (harbor-framework#2019) 701c90a feat(modal): add dynamic network policy switching (harbor-framework#2003) 703747d Toggle modal dynamic network configuration based on phase_network_policies (harbor-framework#2092) ff7e1a5 feat(rewardkit): opt-in cross-dimension aggregation for reward.json (harbor-framework#2010) 6712637 feat: launcher browse dataset path (harbor-framework#2093) 0de1cd4 Fix mswea reasoning. Co-authored-by: Cursor <cursoragent@cursor.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds
tests/unit/test_pass_at_k.py, coveringsrc/harbor/utils/pass_at_k.py, which previously had no test coverage.Why
pass@kis a user-facing metric — it's computed inharbor.jobviacompute_pass_at_k_by_evals, stored onJobStats(models/job/result.py), and rendered by theharbor jobsCLI (cli/jobs.py). Despite sitting on a user-facing path, the module had no direct tests, so a regression in the estimator or its guard logic would pass silently.This is test-only — no production code changes — and the implementation is already correct, so the PR is zero-risk.
What's covered
_pass_at_k_for_task— validated against an independent closed-form reference (1 - C(n-c, k)/C(n, k), viamath.comb) over a grid of(n, c, k), plus boundaries:c == 0→ 0,c == n→ 1,n - c < k→ 1._eligible_k_values— powers of two ∪ multiples of five, sorted and deduplicated; empty belowk = 2.compute_pass_at_k_by_evals(public API, over realTrialResultmodels) — grouping by eval key, averaging across tasks, missing-verifier-counts-as-failure, single-attempt →{}, and the guards that bail to{}on non-binary / out-of-range / multi-key rewards.Verification
🤖 Generated with Claude Code