Skip to content

Speed up kv-test's streamedPut and remove blobGc's sleeps - #8373

Merged
danlapid merged 2 commits into
mainfrom
dlapid/kv-test-streamed-put
Nov 12, 2026
Merged

danlapid merged 2 commits into
mainfrom
dlapid/kv-test-streamed-put

Conversation

@danlapid

@danlapid danlapid commented Nov 11, 2026 •

Copy link
Copy Markdown
Collaborator

kv-test's streamedPut case took 21–24 s of the test's 25 s on x64 debug builds, and 45 s on arm64, where all three kv-test variants timed out at 60 s (edgeworker's arm64 job runs workerd's tests locally on one machine).

Nearly all of that time went to building and checking the 2 MiB value, not to the streamed put itself. Phase timings on a debug build:

Phase (2 MiB) Time
Uint8Array.from({ length }, (_, i) => i % 251) 2.6 s
put + get 0.1 s
assert.deepStrictEqual(new Uint8Array(got), value) 13.6–17.2 s
Buffer.compare() of the same arrays 37 ms
util.isDeepStrictEqual() of the same arrays 15.8 s

isDeepStrictEqual() compares typed arrays with Buffer.compare(), so its time goes to the two getOwnNonIndexProperties() calls that follow it. Those are slow on large typed arrays in release builds as well: 1.8 s for 2 MiB, against 6 ms for Buffer.compare(). That may be worth a separate look, since applications can call assert.deepStrictEqual() and util.isDeepStrictEqual() on large buffers too.

This change fills the value in a loop and compares it with Buffer.equals() after a length check. It doesn't change the sizes, the 16 KiB chunks, or the split between inline values and blobs.

Before After
streamedPut, debug 21–24 s 0.3 s
kv-test@, debug 25.6 s 5.4 s (avg of 5)
kv-test variants, ASan 4.6–5.1 s (avg of 5)

I updated the comment on size = "large" to match. The test still needs more than 15 s on a loaded arm64 machine.

Verified with bazel test --config=bazelflare --nocache_test_results --runs_per_test=5 on //src/workerd/server/tests/kv:all, and with --config=bazelflare-asan on the three kv-test variants. All runs passed.

blobGc without sleeps

The second commit removes blobGc's three 500 ms sleeps. They waited for the alarm to delete blobs after a 100 ms grace period under TEST_CLOCK, so a slow machine could fail the test either way: the alarm could fire after a sleep ended, or it could delete a blob before the test counted it. threshold, which also counts blobs, had the second race.

Under TEST_CLOCK the grace period is now ten minutes, so the alarm never deletes a blob during a test. A new test-only POST /gc runs the alarm's work as though every unnamed blob's grace period were over, and returns once the deletes finish. Reserved blobs that haven't been claimed yet still use the real clock. blobGc uses the new endpoint. So does cluster-test's KV failover case, which used to poll the bucket for up to 10 s waiting for the replaced blob to go. Production behavior is unchanged: the grace period stays 2 s, and alarm() runs the same sweep.

I ran this with --runs_per_test=5 on bazelflare for //src/workerd/server/tests/kv/... and //src/workerd/server/tests/cluster:cluster-test, and on bazelflare-asan for the kv tests. All runs passed.

🤖 Generated with Claude Code

streamedPut took 21-24 s of kv-test's 25 s on x64 debug builds and
45 s on arm64, where kv-test timed out at 60 s. Nearly all of it was
building and checking the 2 MiB value, not the streamed put:

- assert.deepStrictEqual() on the 2 MiB Uint8Array: about 16 s.
  Buffer.compare() of the same arrays takes 37 ms, so the time goes
  to isDeepStrictEqual()'s getOwnNonIndexProperties() calls, which
  are slow on large typed arrays (1.8 s in a release build too).
- Uint8Array.from() with a callback per byte: 2.6 s.

Fill the value in a loop and compare with Buffer.equals(). The sizes,
chunking and inline/blob split are unchanged. streamedPut now takes
0.3 s, and kv-test 5-6 s, both on debug and ASan builds.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@danlapid
danlapid requested review from a team as code owners November 11, 2026 21:14

@clanky-the-clanker clanky-the-clanker Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Superseded by the review of f6d626fc.

This review, of 47b13704

✅ Approved — This change only affects the test and has no findings. The new fill and comparison check the same bytes and sizes as before, and the comments follow the repository's comment guidelines.

The decision is advisory: this review is a comment and never approves, blocks or requests changes on the pull request.


✅ All checks passed

📋 Code Quality: ✅ LGTM

The fill loop writes the same bytes as the old Uint8Array.from(..., i % 251). The new check still catches a wrong length (strictEqual on byteLength) and wrong content (Buffer.from(got).equals(value)). Buffer.from(ArrayBuffer) shares memory with got rather than copying it. The node:buffer import works because the test enables nodejs_compat. The sizes (1000, 70,000 and 2 MiB) are unchanged.


📝 Documentation: ✅ LGTM

The updated comments describe the code as it stands now: the run-time note on size = "large" and the reason for avoiding assert.deepStrictEqual() on large arrays.


Reviewed commit: 47b13704

kv-test's blobGc slept 500 ms three times for the alarm to delete
blobs after a 100 ms grace period under TEST_CLOCK. On a loaded
machine either side could lose: the alarm could be late past a
sleep, or could delete a blob before the test counted it, as it
could under threshold, which counts blobs too.

Under TEST_CLOCK the grace period is now ten minutes, so the alarm
never deletes a blob during a test, and POST /gc runs the alarm's
work with every unnamed blob's grace period over, waiting for the
deletes. blobGc and cluster-test's KV failover case, which polled
the bucket for up to 10 s, use it instead of waiting.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@danlapid danlapid changed the title Speed up kv-test's streamedPut Speed up kv-test's streamedPut and remove blobGc's sleeps Nov 11, 2026

@clanky-the-clanker clanky-the-clanker Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Approved — No findings. The test-only POST /gc endpoint only works under TEST_CLOCK, and the alarm's sweep in production behaves as before. The new fill and comparison in streamedPut check the same bytes as the old code.

The decision is advisory: this review is a comment and never approves, blocks or requests changes on the pull request.


✅ All checks passed

📋 Code Quality: ✅ LGTM

I found no problems in this pull request.

kv.js sweep

  • #sweep(dueBy) selects rows where due <= now OR (key IS NULL AND due <= dueBy).
  • alarm() passes Date.now(), so the second clause adds nothing and production behavior is unchanged.
  • collectAll() passes Date.now() + grace, so it sweeps every unnamed blob.
  • Reserved rows keep using the real clock.
  • A blob whose delete fails is re-queued as unnamed, so a later /gc retries it.

/gc route

  • It only works when TEST_CLOCK is set, and it sits under POST, so it can't collide with a key named gc.
  • Without TEST_CLOCK the request falls through to 405.

Tests

  • With a ten-minute grace period under TEST_CLOCK, the alarm can't delete a blob during a test. That removes both of the races the description mentions.
  • blobGc reads its baseline after an initial gc(), so blobs left over from earlier tests don't throw off the counts.
  • In the cluster KV failover case, the delete is awaited before the request returns, so the bucket check no longer has to poll.

streamedPut

  • The fill loop writes the same bytes as the old Uint8Array.from(..., i % 251).
  • The length check followed by Buffer.equals() still catches a wrong length and wrong content.

📝 Documentation: ✅ LGTM

The new and updated comments describe the code as it is now:

  • the TEST_CLOCK grace period and the POST /gc endpoint in kv.js;
  • the run-time note on size = "large";
  • the reason the test avoids assert.deepStrictEqual() on large arrays.

Reviewed commit: f6d626fc

@danlapid
danlapid merged commit 227f434 into main Nov 12, 2026
24 checks passed
@danlapid
danlapid deleted the dlapid/kv-test-streamed-put branch November 12, 2026 04:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants