RewardKit: 3 auth bugs block grading on a Claude subscription

[Kurry]

RewardKit can't grade on a Claude subscription end to end. Three independent bugs force a subscriber onto the paid Console API key — or crash the verifier outright.

Background

• CLAUDE_CODE_OAUTH_TOKEN (sk-ant-oat01…) authenticates on a Claude subscription (Pro/Max) — free.
• ANTHROPIC_API_KEY (sk-ant-api03…) authenticates on the Console account — metered/paid.

These are different Anthropic orgs. The OAuth token only works as an OAuth bearer (litellm auto-detects the sk-ant-oat prefix and routes it correctly); it 401s if sent as x-api-key.

Environment: harbor-rewardkit==0.1.4, litellm 1.86.2.

What main already supports (and where it stops half-way)

Subscription grading is already partly wired in main — each bug is the point where that support stops short, not a net-new feature:

Already in main	Where it stops short (the bug)
The claude-code agent judge already grades on the subscription — it shells out to the claude CLI, which inherits CLAUDE_CODE_OAUTH_TOKEN. This is the working free path today.	A stray container ANTHROPIC_API_KEY silently shadows that OAuth (CLI auth precedence), breaking the working path — Console bill, or 401 crash on an invalid key. → Bug 3
litellm itself already authenticates a subscription via ANTHROPIC_AUTH_TOKEN (OAuth bearer).	The litellm LLM judge never passes the OAuth token to litellm — it calls acompletion() with no api_key, so the subscription is unreachable from that judge. → Bug 1
REWARDKIT_JUDGE already globalizes judge selection (runner.py:79).	There is no parallel REWARDKIT_MODEL, so a global REWARDKIT_JUDGE=claude-code silently drops the rubric's pinned model. → Bug 2

So this is finishing half-built support, not adding a new capability.

---

Bug 1 — litellm LLM judge ignores the subscription token

Expected: with only CLAUDE_CODE_OAUTH_TOKEN set, an anthropic/* LLM judge grades on the subscription.
Actual: it fails — judges._arun_llm_call calls litellm.acompletion(model=…) with no api_key, so litellm reads ANTHROPIC_API_KEY / ANTHROPIC_AUTH_TOKEN from the env and never the OAuth token.

unset ANTHROPIC_API_KEY
export CLAUDE_CODE_OAUTH_TOKEN=sk-ant-oat01-...     # a real subscription token
# rubric:  [judge] judge = "anthropic/claude-sonnet-4-6"
uvx --from 'harbor-rewardkit==0.1.4' rewardkit tests/

litellm.exceptions.AuthenticationError: AnthropicException - Missing Anthropic API Key -
 A call is being made to anthropic but no key is set ... Please set `ANTHROPIC_API_KEY` or `ANTHROPIC_AUTH_TOKEN`

Bug 2 — REWARDKIT_JUDGE=claude-code silently drops the pinned model

Expected: a rubric that pins its model survives a global judge override.
Actual: REWARDKIT_JUDGE overrides the whole [judge].judge field. When a rubric encodes its model there (judge = "anthropic/claude-opus-4-8"), overriding judge to the agent name claude-code leaves the agent's model = None, so the claude CLI runs its built-in default model — the pinned model vanishes with no error. There is no global REWARDKIT_MODEL to set the model once.

# rubric:  [judge] judge = "anthropic/claude-opus-4-8"
export REWARDKIT_JUDGE=claude-code
uvx --from 'harbor-rewardkit==0.1.4' rewardkit tests/
# → agent judge runs the claude CLI's DEFAULT model, not opus-4-8 (silent)

Bug 3 — stray ANTHROPIC_API_KEY shadows OAuth in the agent judge

Expected: the claude-code agent judge grades on the subscription via CLAUDE_CODE_OAUTH_TOKEN.
Actual: the CLI's auth precedence puts ANTHROPIC_API_KEY above CLAUDE_CODE_OAUTH_TOKEN, so a key present in the verifier container ([verifier.env]) wins: a valid key silently grades on Console (paid); an invalid key 401s and the judge crashes with no reward written.

export CLAUDE_CODE_OAUTH_TOKEN=sk-ant-oat01-...
export ANTHROPIC_API_KEY=sk-ant-api03-invalid       # stray / invalid key in the container
# rubric:  [judge] judge = "claude-code"
uvx --from 'harbor-rewardkit==0.1.4' rewardkit tests/

ValueError: Agent CLI 'claude' exited with code 1:
  {"is_error":true,"api_error_status":401, ... "result":"Invalid API key · Fix external API key"}
→ RewardFileNotFoundError   (no reward.json written)

Note: this only bites when ANTHROPIC_API_KEY reaches the verifier container ([verifier.env] is an explicit allowlist). A host-only .env value is harmless; an empty value is ignored by the CLI; only a non-empty invalid key crashes, and a valid one silently bills Console.

---

Fix

Proposed in #1770 — resolves each path free-first behind one opt-out flag (REWARDKIT_FORCE_API_KEY=1 keeps the paid API key for bugs 1 & 3), and adds REWARDKIT_MODEL to pin the model globally. Companion to #1766 (same auth for harbor check / harbor analyze).

[alexgshaw]

@benediktstroebl

[kobe0938]

@alexgshaw this issue paired with the following 2 prs are the ones I mentioned about rewardkit.
#1778
#1770

I'll push back for the design choices we discussed on #1766 and make a new pr about configuring CLAUDE_FORCE_OAUTH in harbor run

[benediktstroebl]

@kobe0938 I am looking into this over to add Claude subscription support. Feel free to close this and make a new PR. I will then give feedback on that on what the simplest design could be

[kobe0938]

@kobe0938 I am looking into this over to add Claude subscription support. Feel free to close this and make a new PR. I will then give feedback on that on what the simplest design could be

Hi Ben, I left some comments here:
#1766 (comment)

Principles could be transferred here. Alex said that someone is in charge of the reward kit(I suppose that's you) so feel free to modify/suggest anything on adding Claude subscription support.

[Kurry]

@benediktstroebl @kobe0938 @alexgshaw quick status update for this issue: #1770 has been realigned to the #1766 auth principles and now includes #1778's RewardKit model override/flag cleanup.

Final pushed head: afc738a0.

Current behavior in #1770:

• ANTHROPIC_API_KEY remains the default when both credentials are present.
• A lone CLAUDE_CODE_OAUTH_TOKEN works for Anthropic-direct RewardKit LLM judges.
• CLAUDE_FORCE_OAUTH=<truthy> is the explicit opt-in to subscription auth.
• Invalid CLAUDE_FORCE_OAUTH values raise ValueError.
• Forced OAuth without CLAUDE_CODE_OAUTH_TOKEN raises RuntimeError.
• RewardKit keeps the surface minimal: no keychain probing, no new modules, and no auth CLI flag.

Latest verification on the final head:

uv run ruff check --fix .
uv run ruff format .
uv run ty check
uv run pytest tests/unit/ packages/rewardkit/tests -q

Result: 3257 passed, 1 skipped.

I also requested re-review on #1770 via comment because GitHub blocked the formal review request from this fork with RequestReviewsByLogin permissions.

[kobe0938]

@benediktstroebl I prefer to just let you decide with your own discretion as you are the owner of RewardKit and it is fairly standalone.

Feel free to modify/merge it, the principle is there. and then if you want any help from me, let me know. thanks.

[Kurry]

@benediktstroebl @kobe0938 @alexgshaw updated #1770 again after applying the PR hygiene checklist from this issue/PR cluster.

Current #1770 head: 72394831.

Status:

• Rebased onto current main (c978013a8) with a clean 3-commit stack.
• Preserves rewardkit: add REWARDKIT_MODEL override for agent judges #1778's REWARDKIT_MODEL / flag cleanup as Benedikt-authored commits.
• Keeps ANTHROPIC_API_KEY as the default when both credentials exist.
• Uses CLAUDE_FORCE_OAUTH=<truthy> as the explicit subscription opt-in.
• Fails loudly on invalid force values and on forced OAuth without CLAUDE_CODE_OAUTH_TOKEN.
• Keeps RewardKit surface minimal: no keychain probing, no new modules, no auth CLI flag.
• PR title/body now use the RewardKit title convention, Fixes #1771, and separate CI gate results.

Verification on the final head:

uv run ruff check --fix .
uv run ruff format .
uv run ty check
uv run pytest tests/unit/
uv run pytest packages/rewardkit/tests/

Result: tests/unit/: 2812 passed, 1 skipped; packages/rewardkit/tests/: 445 passed; ruff and ty passed.