Repository navigation
Speed up kv-test's streamedPut and remove blobGc's sleeps - #8373
Conversation
streamedPut took 21-24 s of kv-test's 25 s on x64 debug builds and 45 s on arm64, where kv-test timed out at 60 s. Nearly all of it was building and checking the 2 MiB value, not the streamed put: - assert.deepStrictEqual() on the 2 MiB Uint8Array: about 16 s. Buffer.compare() of the same arrays takes 37 ms, so the time goes to isDeepStrictEqual()'s getOwnNonIndexProperties() calls, which are slow on large typed arrays (1.8 s in a release build too). - Uint8Array.from() with a callback per byte: 2.6 s. Fill the value in a loop and compare with Buffer.equals(). The sizes, chunking and inline/blob split are unchanged. streamedPut now takes 0.3 s, and kv-test 5-6 s, both on debug and ASan builds. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
There was a problem hiding this comment.
Superseded by the review of
f6d626fc.
This review, of 47b13704
✅ Approved — This change only affects the test and has no findings. The new fill and comparison check the same bytes and sizes as before, and the comments follow the repository's comment guidelines.
The decision is advisory: this review is a comment and never approves, blocks or requests changes on the pull request.
✅ All checks passed
📋 Code Quality: ✅ LGTM
The fill loop writes the same bytes as the old Uint8Array.from(..., i % 251). The new check still catches a wrong length (strictEqual on byteLength) and wrong content (Buffer.from(got).equals(value)). Buffer.from(ArrayBuffer) shares memory with got rather than copying it. The node:buffer import works because the test enables nodejs_compat. The sizes (1000, 70,000 and 2 MiB) are unchanged.
📝 Documentation: ✅ LGTM
The updated comments describe the code as it stands now: the run-time note on size = "large" and the reason for avoiding assert.deepStrictEqual() on large arrays.
Reviewed commit: 47b13704
kv-test's blobGc slept 500 ms three times for the alarm to delete blobs after a 100 ms grace period under TEST_CLOCK. On a loaded machine either side could lose: the alarm could be late past a sleep, or could delete a blob before the test counted it, as it could under threshold, which counts blobs too. Under TEST_CLOCK the grace period is now ten minutes, so the alarm never deletes a blob during a test, and POST /gc runs the alarm's work with every unnamed blob's grace period over, waiting for the deletes. blobGc and cluster-test's KV failover case, which polled the bucket for up to 10 s, use it instead of waiting. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
There was a problem hiding this comment.
✅ Approved — No findings. The test-only POST /gc endpoint only works under TEST_CLOCK, and the alarm's sweep in production behaves as before. The new fill and comparison in streamedPut check the same bytes as the old code.
The decision is advisory: this review is a comment and never approves, blocks or requests changes on the pull request.
✅ All checks passed
📋 Code Quality: ✅ LGTM
I found no problems in this pull request.
kv.js sweep
#sweep(dueBy)selects rows wheredue <= now OR (key IS NULL AND due <= dueBy).alarm()passesDate.now(), so the second clause adds nothing and production behavior is unchanged.collectAll()passesDate.now() + grace, so it sweeps every unnamed blob.- Reserved rows keep using the real clock.
- A blob whose delete fails is re-queued as unnamed, so a later
/gcretries it.
/gc route
- It only works when
TEST_CLOCKis set, and it sits underPOST, so it can't collide with a key namedgc. - Without
TEST_CLOCKthe request falls through to 405.
Tests
- With a ten-minute grace period under
TEST_CLOCK, the alarm can't delete a blob during a test. That removes both of the races the description mentions. blobGcreads its baseline after an initialgc(), so blobs left over from earlier tests don't throw off the counts.- In the cluster KV failover case, the delete is awaited before the request returns, so the bucket check no longer has to poll.
streamedPut
- The fill loop writes the same bytes as the old
Uint8Array.from(..., i % 251). - The length check followed by
Buffer.equals()still catches a wrong length and wrong content.
📝 Documentation: ✅ LGTM
The new and updated comments describe the code as it is now:
- the
TEST_CLOCKgrace period and thePOST /gcendpoint inkv.js; - the run-time note on
size = "large"; - the reason the test avoids
assert.deepStrictEqual()on large arrays.
Reviewed commit: f6d626fc
kv-test'sstreamedPutcase took 21–24 s of the test's 25 s on x64 debug builds, and 45 s on arm64, where all threekv-testvariants timed out at 60 s (edgeworker's arm64 job runs workerd's tests locally on one machine).Nearly all of that time went to building and checking the 2 MiB value, not to the streamed put itself. Phase timings on a debug build:
Uint8Array.from({ length }, (_, i) => i % 251)put+getassert.deepStrictEqual(new Uint8Array(got), value)Buffer.compare()of the same arraysutil.isDeepStrictEqual()of the same arraysisDeepStrictEqual()compares typed arrays withBuffer.compare(), so its time goes to the twogetOwnNonIndexProperties()calls that follow it. Those are slow on large typed arrays in release builds as well: 1.8 s for 2 MiB, against 6 ms forBuffer.compare(). That may be worth a separate look, since applications can callassert.deepStrictEqual()andutil.isDeepStrictEqual()on large buffers too.This change fills the value in a loop and compares it with
Buffer.equals()after a length check. It doesn't change the sizes, the 16 KiB chunks, or the split between inline values and blobs.streamedPut, debugkv-test@, debugkv-testvariants, ASanI updated the comment on
size = "large"to match. The test still needs more than 15 s on a loaded arm64 machine.Verified with
bazel test --config=bazelflare --nocache_test_results --runs_per_test=5on//src/workerd/server/tests/kv:all, and with--config=bazelflare-asanon the threekv-testvariants. All runs passed.blobGcwithout sleepsThe second commit removes
blobGc's three 500 ms sleeps. They waited for the alarm to delete blobs after a 100 ms grace period underTEST_CLOCK, so a slow machine could fail the test either way: the alarm could fire after a sleep ended, or it could delete a blob before the test counted it.threshold, which also counts blobs, had the second race.Under
TEST_CLOCKthe grace period is now ten minutes, so the alarm never deletes a blob during a test. A new test-onlyPOST /gcruns the alarm's work as though every unnamed blob's grace period were over, and returns once the deletes finish. Reserved blobs that haven't been claimed yet still use the real clock.blobGcuses the new endpoint. So doescluster-test's KV failover case, which used to poll the bucket for up to 10 s waiting for the replaced blob to go. Production behavior is unchanged: the grace period stays 2 s, andalarm()runs the same sweep.I ran this with
--runs_per_test=5onbazelflarefor//src/workerd/server/tests/kv/...and//src/workerd/server/tests/cluster:cluster-test, and onbazelflare-asanfor thekvtests. All runs passed.🤖 Generated with Claude Code