feat(models): add prompt caching for the Anthropic provider (backport to release/v0.10.x)

[teemow]

Backport of #2788 to release/v0.10.x. Refs #2787 (the feature request, with measurements) and #1866 (the earlier request whose Bedrock half shipped in #1940).

Problem

A declarative agent re-sends its full history on every model call. With the Anthropic provider nothing is marked cacheable today: promptCaching exists only on BedrockConfig, and neither runtime sends cache_control, so the whole prefix (system prompt, tool definitions, previous turns) is billed at full input price on every call.

Measured on kagent 0.10.1 on an internal installation, through a gateway that logs gen_ai.usage.* per model call (162 calls over 24 h, three agents, claude-sonnet-4-6 via the Anthropic provider): cache_read.input_tokens and cache_creation.input_tokens were 0 on every call. One incident-investigation session of a Python (kagent-app) agent made 119 model calls whose input grew from 4,423 to 190,809 tokens per call (average 93 k), i.e. roughly 11 M input tokens billed for one session, most of it the same prefix over and over; a Go (golang-adk) agent in the same window went 3.6 k → 93 k over 41 calls. Anthropic's prompt caching turns that repeat into cache reads at a fraction of the input price. Full numbers in #2787.

Change

Same shape as #2788, on the v1alpha2 API:

• spec.anthropic.promptCaching (bool, default false) and spec.anthropic.cacheTTL ("5m" | "1h", default "5m") on ModelConfig, with the same names, defaults and doc shape as the Bedrock fields; adk_api_translator.go wires them into adk.Anthropic (prompt_caching, cache_ttl). CRDs regenerated (make controller-manifests).
• When enabled, every Messages API request carries three cache_control breakpoints: the last tool definition, the last system prompt block, and the last content block of the latest conversation turn. Anthropic renders tools → system → messages and caches the prefix up to each breakpoint, so each call of an agent loop reads its whole previous history from the cache and writes only the new turn. Three of the four breakpoints Anthropic allows; the conversation breakpoint skips thinking blocks, which Anthropic refuses to cache.
• Go runtime (go/adk/pkg/models/anthropic_adk.go): request shaping extracted into buildAnthropicParams, markAnthropicCacheBreakpoints sets the markers, and usage maps cache_read_input_tokens to CachedContentTokenCount and folds cache read/creation tokens into PromptTokenCount, so the UI's usage view shows the effect. Streaming and non-streaming.
• Python runtime (KAgentAnthropicLlm): wraps the SDK client's messages resource with PromptCachingMessages, which marks the same three breakpoints before each create call — the one seam google-adk's AnthropicLlm hands its request through on both paths.
• Difference from main: this branch pins google-adk 1.x, which maps only input_tokens for Anthropic and drops the cache breakdown, whereas google-adk 2.x on main already folds it. Here the wrapper also records the raw response usage per request (a ContextVar, so concurrent requests on one model stay apart; for streaming it reads message_start) and generate_content_async folds it into google-adk's usage metadata the same way google-adk 2.x does (fold_cache_usage). With promptCaching off nothing changes.
• Helm: providers.anthropic.config.promptCaching / cacheTTL documented in values.yaml; the chart already renders config into the provider block.

The Claude-harness compiler change from #2788 does not apply here (no harness compilers on 0.10.x).

Testing

• Go: go vet + go test ./adk/... ./api/... ./core/internal/controller/translator/... (new anthropic_adk_test.go — wire body carries exactly three markers at the expected positions for default/5m/1h, none when disabled, trailing-tool-result and no-tools/no-system cases, usage fold, and the SSE streaming + non-streaming paths through an httptest server; agent wiring and config deserialization; Test_AdkApiTranslator_AnthropicPromptCaching). golangci-lint: the full make -C go lint crashes locally inside staticcheck's buildir on this branch with my toolchain (unrelated to the change); the remaining linters report 0 issues on the changed packages, CI runs the full set.

• Python: pytest packages/kagent-adk/tests/unittests (443 passed) — new tests for mark_prompt_cache_breakpoints, the client wrapping, fold_cache_usage, and end-to-end through google-adk's request builder with the SDK client mocked, for both the non-streaming and the streaming path.

• Against the Anthropic API with the same runtime code on main (see feat(models): add prompt caching for the Anthropic provider (promptCaching, cacheTTL) #2788): after the first call ~99 % of every prompt is served from the cache and reported as cached tokens, in both runtimes.

• kind end-to-end on this branch (controller, golang-adk and app images built from this commit; chart installed with providers.default=anthropic, providers.anthropic.config.promptCaching=true, model claude-sonnet-4-6; the rendered ModelConfig was accepted with promptCaching: true / cacheTTL: 5m). Two declarative agents, runtime: go and runtime: python, with the built-in kagent-tool-server Kubernetes tools (~14 k tokens of tool definitions + system prompt), driven through the controller's A2A endpoint (/api/a2a/kagent/<agent>, message/send with one contextId) for three turns. Values are prompt_token_count / cached_content_token_count from the usage metadata the runtime attaches to the A2A response, i.e. what the UI shows:

  runtime	turn 1, first model call	turn 1, call after the tool result	turn 2	turn 3
  Go	14260 / 0 (cache write)	15665 / 14257	16269 / 15664	16337 / 16266
  Python	14321 / 0 (cache write)	15481 / 14318	16237 / 15480	16305 / 16234

  Every model call after the first, including the tool-result follow-up inside the same turn, reads ~99.5 % of its prompt from the cache.

[Copilot]

🟡 Changes recommended

Two newly added tests intended to assert “no caller object mutation” currently use shallow copies and can miss nested in-place mutations, weakening the regression coverage for this change.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

This PR backports Anthropic prompt caching support to release/v0.10.x, adding per-ModelConfig knobs and wiring them through the controller translator into both the Go and Python runtimes so Anthropic Messages requests include cache_control breakpoints and cache usage is reflected in token accounting.

Changes:

• Add spec.anthropic.promptCaching (default false) and spec.anthropic.cacheTTL ("5m"|"1h", default "5m") to the v1alpha2 ModelConfig API + regenerate CRDs.
• Implement request shaping for Anthropic prompt caching in both runtimes (mark 3 breakpoints) and fold Anthropic cache usage into the GenAI usage shape.
• Add/extend unit tests for translator wiring, runtime request shaping, and usage folding; document Helm values.

File summaries

File	Description
python/packages/kagent-adk/tests/unittests/models/test_anthropic.py	Adds unit tests covering Anthropic prompt caching request shaping and usage folding (plus translator/config wiring).
python/packages/kagent-adk/src/kagent/adk/types.py	Extends the Python Anthropic model config with prompt_caching/cache_ttl and forwards them into LLM construction.
python/packages/kagent-adk/src/kagent/adk/models/_anthropic.py	Implements prompt-caching request shaping via an AsyncMessages wrapper + folds cache usage into response usage metadata.
helm/kagent/values.yaml	Documents Anthropic provider config values for prompt caching.
helm/kagent-crds/templates/kagent.dev_modelconfigs.yaml	Helm-rendered CRD schema update for the new Anthropic fields.
go/core/internal/controller/translator/agent/adk_api_translator.go	Wires promptCaching/cacheTTL from ModelConfig into the internal ADK model config.
go/core/internal/controller/translator/agent/adk_api_translator_test.go	Adds a translator test ensuring Anthropic prompt caching fields are passed through.
go/api/v1alpha2/modelconfig_types.go	Adds the new Anthropic CRD fields and docs to the Go API types.
go/api/config/crd/bases/kagent.dev_modelconfigs.yaml	Regenerated base CRD YAML reflecting the new schema fields.
go/api/adk/types.go	Extends the internal adk.Anthropic config with prompt_caching/cache_ttl JSON fields.
go/adk/pkg/models/anthropic.go	Threads the prompt caching knobs into the Go Anthropic runtime config struct.
go/adk/pkg/models/anthropic_adk.go	Implements breakpoint marking and cache-usage folding for Anthropic in the Go runtime (streaming + non-streaming).
go/adk/pkg/models/anthropic_adk_test.go	New Go unit tests asserting wire-level markers and usage folding, including streaming via httptest.
go/adk/pkg/agent/agent.go	Passes new Anthropic prompt caching config through agent LLM creation.
go/adk/pkg/agent/agent_test.go	Extends deserialization and creation tests to validate Anthropic prompt caching fields flow into the runtime.

Review details

Suppressed comments (1)

python/packages/kagent-adk/tests/unittests/models/test_anthropic.py:251

• test_does_not_mutate_caller_objects snapshots last_tool/last_message with shallow dict(...) copies; last_message still shares nested content lists/dicts with the original, so in-place mutations to nested structures wouldn’t be caught. Use deep copies for these snapshots to make the test robust.

        tools, messages = kwargs["tools"], kwargs["messages"]
        last_tool, last_message = dict(tools[-1]), dict(messages[-1])
        mark_prompt_cache_breakpoints(kwargs, self.CC)
        assert tools[-1] == last_tool
        assert messages[-1] == last_message

• Files reviewed: 15/15 changed files
• Comments generated: 1
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[teemow]

Fixed in 71daedf: both tests now compare against copy.deepcopy snapshots of the whole kwargs / the tools and messages lists, so a nested in-place mutation would fail them. Applied the same fix to #2788 (f5679c2).

[github-actions]

This pull request has been marked as stale because of no activity in the last 15 days. It will be closed in the next 5 days unless it is tagged "no stalebot" or other activity occurs.

[github-actions]

This pull request has been closed after 15 days of inactivity and an additional 5 days after being marked as stale.

[ashsha]

Could this backport be reopened? It was closed by the stale bot, not by review, and the 0.10.x line is what the current Helm chart installs.

We needed it this week and can confirm it works as described:

• The diff applies cleanly on both v0.10.1 and v0.10.3 (git apply --check, no offsets).
• go build and go test ./adk/... ./core/internal/controller/translator/... pass on v0.10.1 with it applied.
• Verified end to end on kagent 0.10.1 with the Go runtime rebuilt from this diff, the Anthropic provider pointed at agentgateway 1.5.0 as baseUrl, and Phoenix reading the usage from the ADK traces. Four k8s-agent calls on claude-haiku-4-5: 4623 prompt / 0 cached (cache write), then 4918 / 4620, and in a fresh session 4623 / 4620 and 4918 / 4913, so 94 to 99.9 percent of the prompt served from the cache from the second call on. Before the patch every call reported cache_read_input_tokens: 0, and Anthropic's console estimated 49 percent of the organisation's direct API spend as cacheable.

Two notes from that run, in case they are useful for the docs that #2788 mentions as a follow-up:

• The usage mapping in this PR matters as much as the breakpoints: without it the 0.10.x Go runtime reports only input_tokens, so traces show no cache effect even when the cache is working.
• Behind agentgateway the route needs an AgentgatewayPolicy with backend.ai.routes: {"/v1/messages": Messages}; the gateway's default treats everything as chat completions and drops the system blocks, breakpoints included.

Happy to test a rebased version if that helps get it in.