Repository navigation
Don't sync Durable Object storage in workerd test - #8375
Merged
Merged
Conversation
Each SQLite commit to a Durable Object's localDisk storage fsync()s its WAL. On an idle disk that costs little, but on a busy one every fsync() waits for the device. In edgeworker's arm64 CI job, which runs workerd's tests on one machine while Bazel writes gigabytes of outputs to the same disk, unsafe-module-test (217 syncs) went from 2 s to 60 s and timed out, and sql-restrict-names-test (14 syncs) hung for 15 s on the five syncs the alarm scheduler makes at startup, before its test even began. The tests slowed the most in those windows were the ones with on-disk SQLite storage: sql-test, d1-api-test and its variants, and sql-resizable-arraybuffer-test. A test's storage need not survive a crash of the machine, so add a VfsOptions::sync option, which makes xSync a no-op in both the native-file and KJ-file VFS paths, and turn it off for the storage that `workerd test` gives Durable Objects (and their alarm scheduler). Data still reaches the page cache, so a crash of the workerd process alone loses nothing. Everything else keeps syncing. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
There was a problem hiding this comment.
💬 Approved with comments — The change is correct and scoped to workerd test. The only note is an INFO about wording in the new VfsOptions::sync comment.
Findings: 1 info (1 is an inline comment on the diff)
The decision is advisory: this review is a comment and never approves, blocks or requests changes on the pull request.
📝 Documentation: 💬 1 info
The new VfsOptions::sync comment states the process-crash guarantee more broadly than it holds.
Recommendation: 💬 Optional Nits
- [INFO] Limit the process-crash guarantee in the
VfsOptions::synccomment to disk-backed directories. (src/workerd/util/sqlite.h:902)
View full review
src/workerd/util/sqlite.h:902: the page-cache reasoning applies only to disk-backed directories. A VFS over a non-disk (for example in-memory) KJ directory has no OS page cache behind it, so the claim should be limited to disk-backed storage.
✅ All other checks passed
📋 Code Quality: ✅ LGTM
No problems found in the code.
- Native-file wrapper:
xSyncskips the sync only on files the wrapper actually opened, and it skips the directory sync as well. - KJ-file implementation: it reads the option through
self.vfs.options, the same way the existingdeviceCharacteristicssetting is read. - Plumbing:
applyTestOptions()setssyncStoragewhile the factory options are being built, so the VFS thatActorStorage::Implcreates sees it. The alarm scheduler uses the same VFS. - Test: the new test checks both the default (at least one sync) and
.sync = false(no syncs).
Reviewed commit: b297f335
southpolesteve
approved these changes
Nov 12, 2026
csjh
approved these changes
Nov 12, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
workerd testno longer fsync()s the SQLite files of Durable Object storage onlocalDisk(or the alarm scheduler's database next to them). A test's storage doesn't need to survive a crash of the machine, and on a busy disk every fsync() can stall for seconds.Why
edgeworker's arm64 CI job runs workerd's tests locally, on one 32-vCPU machine, while Bazel writes gigabytes of build outputs to the same disk. In one such job, the machine went through IO stalls: for about 10 s, CPU use fell to 3–8 of 32 cores while the load average stayed at about 49. The tests hit hardest were the ones that keep Durable Object storage in SQLite on disk:
unsafe-module-test(all 3 variants) takes 2.1 s on an idle x64 machine. Here every case slowed down:test_delete_all_durable_objectstook 8–13 s. One variant passed at 59.98 s, and the other two timed out at 60 s after all 16 cases had passed.sql-restrict-names-test@all-compat-flagstakes 0.3 s. It timed out at 15 s before printing[ TEST ] main. Before that line, the only file writes it makes are the alarm scheduler's, which sets upmetadata.sqlitewith five syncs.sql-test(15x, 75 s), everyd1-api-testvariant (6–19x), andsql-resizable-arraybuffer-test(10x). Most other tests slowed by 2–6x.Syncs per run, counted with a temporary log in
xSync:unsafe-module-testsql-testkv-testd1-api-test-with-sessionssql-restrict-names-testChange
SqliteDatabase::VfsOptionsgetsbool sync = true. When it's false,xSyncreturnsSQLITE_OKwithout doing anything, both in the VFS that wraps SQLite's native files and in the one over KJ files. That also skips the directory sync that the nativexSyncdoes after creating a journal or WAL. Writes still reach the OS's page cache, so if only the workerd process crashes, nothing is lost.WorkerFactory::Options::syncStoragedefaults to true.applyTestOptions()turns it off, andActorStorage::Implpasses it to the VFS it opens Durable Object and alarm-scheduler databases through.workerd serveand every other user of the VFS keep syncing.Testing
sqlite-testcase: a commit through a VFS with default options syncs its files, and one with.sync = falsedoesn't. The count comes fromErrorInjectableFile.bazel test --config=bazelflare --runs_per_test=5onsqlite-test,unsafe-module-test,sql-restrict-names-test,sql-test,d1-api-test-with-sessionsandactor-kv-test: all pass. On an idle bazelflare executor the gain is small, as expected: averages went 2.2 → 1.9 s, 0.4 → 0.4 s, 5.3 → 5.0 s, 6.9 → 5.7 s and 0.5 → 0.5 s. The difference matters under IO contention, which I couldn't reproduce on a shared executor.bazel test --config=bazelflare @workerd//src/workerd/... @workerd//src/cloudflare/...: 1,905 pass, 5 skipped.🤖 Generated with Claude Code