Repository navigation
Keep 64 prefix checkpoints so new sessions reuse a shared preamble - #24
Open
pedro-pscunha wants to merge 1 commit into
Open
pedro-pscunha wants to merge 1 commit into
pedro-pscunha wants to merge 1 commit into
Conversation
A new prompt that shares only its head with the cached prompt (a new client session with the same preamble) found no checkpoint inside the shared head and prefilled from zero. Keep one checkpoint per prefill chunk, up to 64 (~31 MB each on Flint), so select_prefix can roll back to the last chunk boundary inside the shared head. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
PromptCache.checkpoint()keeps only the two newest checkpoints. When a new prompt matches the cached prompt only at the start,select_prefixfinds no checkpoint inside the shared part and prefills the whole prompt again.This happens on every new session of a client that sends a large fixed preamble. With Claude Code on
/v1/messages, a new session in the same directory shares the first 19k of its 23k prompt tokens with the previous session. Today it reuses 0 of them.Change
MAX_CHECKPOINTS = 64inprefix.py.checkpoint()keeps the newest 64 instead of 2. That is one per prefill chunk, so it covers 256K tokens at chunk 4096.select_prefixalready takes the newest checkpoint at or before the common prefix. Now it usually finds one.prefix_checkpoint_bytesreports about 31 MB per checkpoint for Flint, so 64 come to about 2 GB at full context. If that is too much for 48 GB machines, a smaller cap or keeping only some of the older checkpoints would also work. I'm happy to change it.Measurements
Apple M5 Max, 128 GB, macOS 27,
flint-int4,rec-mlx servewith--context-limit 262144. Claude Code 2.1.309 in-pmode, two sessions in a row in the same directory, each with a different first message. The numbers come from--metrics.main, small preambleWith the larger preamble, TTFT for the second session was 5.9 s. A cold session took 17.6 s in the same period. The machine had other CPU load during these runs, so absolute times vary. Prefill speed is the same as on
main. I ran a cold 28K prompt onmainand on this branch, alternating two rounds each: prefill 20.0–22.6 s vs 19.4–24.5 s, decode 71–88 vs 84–90 tok/s.Tests
test_new_prompt_sharing_only_the_head_reuses_an_early_checkpoint. It fails with the old cap of 2 and passes with this change.test_checkpoints_are_bounded.test_shortening_exact_hit_early_edit_and_snapshot_boundno longer asserts<= 2.make testpasses. Engine suite: 68 passed.I wrote this change with an AI coding agent (Claude Code) and ran the measurements on my machine.
🤖 Generated with Claude Code