Skip to content

Don't sync Durable Object storage in workerd test - #8375

Merged
danlapid merged 1 commit into
mainfrom
dlapid/test-sqlite-no-sync
Nov 12, 2026
Merged

danlapid merged 1 commit into
mainfrom
dlapid/test-sqlite-no-sync

Conversation

@danlapid

@danlapid danlapid commented Nov 11, 2026 •

Copy link
Copy Markdown
Collaborator

workerd test no longer fsync()s the SQLite files of Durable Object storage on localDisk (or the alarm scheduler's database next to them). A test's storage doesn't need to survive a crash of the machine, and on a busy disk every fsync() can stall for seconds.

Why

edgeworker's arm64 CI job runs workerd's tests locally, on one 32-vCPU machine, while Bazel writes gigabytes of build outputs to the same disk. In one such job, the machine went through IO stalls: for about 10 s, CPU use fell to 3–8 of 32 cores while the load average stayed at about 49. The tests hit hardest were the ones that keep Durable Object storage in SQLite on disk:

  • unsafe-module-test (all 3 variants) takes 2.1 s on an idle x64 machine. Here every case slowed down: test_delete_all_durable_objects took 8–13 s. One variant passed at 59.98 s, and the other two timed out at 60 s after all 16 cases had passed.
  • sql-restrict-names-test@all-compat-flags takes 0.3 s. It timed out at 15 s before printing [ TEST ] main. Before that line, the only file writes it makes are the alarm scheduler's, which sets up metadata.sqlite with five syncs.
  • In the same windows, 13 of the 20 most slowed tests (relative to their x64 durations) had on-disk storage: sql-test (15x, 75 s), every d1-api-test variant (6–19x), and sql-resizable-arraybuffer-test (10x). Most other tests slowed by 2–6x.

Syncs per run, counted with a temporary log in xSync:

Test Syncs
unsafe-module-test 217
sql-test 166
kv-test 102
d1-api-test-with-sessions 76
sql-restrict-names-test 14 (5 before the first test case)

Change

  • SqliteDatabase::VfsOptions gets bool sync = true. When it's false, xSync returns SQLITE_OK without doing anything, both in the VFS that wraps SQLite's native files and in the one over KJ files. That also skips the directory sync that the native xSync does after creating a journal or WAL. Writes still reach the OS's page cache, so if only the workerd process crashes, nothing is lost.
  • WorkerFactory::Options::syncStorage defaults to true. applyTestOptions() turns it off, and ActorStorage::Impl passes it to the VFS it opens Durable Object and alarm-scheduler databases through.

workerd serve and every other user of the VFS keep syncing.

Testing

  • New sqlite-test case: a commit through a VFS with default options syncs its files, and one with .sync = false doesn't. The count comes from ErrorInjectableFile.
  • bazel test --config=bazelflare --runs_per_test=5 on sqlite-test, unsafe-module-test, sql-restrict-names-test, sql-test, d1-api-test-with-sessions and actor-kv-test: all pass. On an idle bazelflare executor the gain is small, as expected: averages went 2.2 → 1.9 s, 0.4 → 0.4 s, 5.3 → 5.0 s, 6.9 → 5.7 s and 0.5 → 0.5 s. The difference matters under IO contention, which I couldn't reproduce on a shared executor.
  • bazel test --config=bazelflare @workerd//src/workerd/... @workerd//src/cloudflare/...: 1,905 pass, 5 skipped.

🤖 Generated with Claude Code

Each SQLite commit to a Durable Object's localDisk storage fsync()s
its WAL. On an idle disk that costs little, but on a busy one every
fsync() waits for the device. In edgeworker's arm64 CI job, which
runs workerd's tests on one machine while Bazel writes gigabytes of
outputs to the same disk, unsafe-module-test (217 syncs) went from
2 s to 60 s and timed out, and sql-restrict-names-test (14 syncs)
hung for 15 s on the five syncs the alarm scheduler makes at startup,
before its test even began. The tests slowed the most in those
windows were the ones with on-disk SQLite storage: sql-test,
d1-api-test and its variants, and sql-resizable-arraybuffer-test.

A test's storage need not survive a crash of the machine, so add a
VfsOptions::sync option, which makes xSync a no-op in both the
native-file and KJ-file VFS paths, and turn it off for the storage
that `workerd test` gives Durable Objects (and their alarm
scheduler). Data still reaches the page cache, so a crash of the
workerd process alone loses nothing. Everything else keeps syncing.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@danlapid
danlapid requested review from a team as code owners November 11, 2026 21:28

@clanky-the-clanker clanky-the-clanker Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💬 Approved with comments — The change is correct and scoped to workerd test. The only note is an INFO about wording in the new VfsOptions::sync comment.

Findings: 1 info (1 is an inline comment on the diff)

The decision is advisory: this review is a comment and never approves, blocks or requests changes on the pull request.


📝 Documentation: 💬 1 info

The new VfsOptions::sync comment states the process-crash guarantee more broadly than it holds.

Recommendation: 💬 Optional Nits

  • [INFO] Limit the process-crash guarantee in the VfsOptions::sync comment to disk-backed directories. (src/workerd/util/sqlite.h:902)
View full review
  • src/workerd/util/sqlite.h:902: the page-cache reasoning applies only to disk-backed directories. A VFS over a non-disk (for example in-memory) KJ directory has no OS page cache behind it, so the claim should be limited to disk-backed storage.

✅ All other checks passed

📋 Code Quality: ✅ LGTM

No problems found in the code.

  • Native-file wrapper: xSync skips the sync only on files the wrapper actually opened, and it skips the directory sync as well.
  • KJ-file implementation: it reads the option through self.vfs.options, the same way the existing deviceCharacteristics setting is read.
  • Plumbing: applyTestOptions() sets syncStorage while the factory options are being built, so the VFS that ActorStorage::Impl creates sees it. The alarm scheduler uses the same VFS.
  • Test: the new test checks both the default (at least one sync) and .sync = false (no syncs).

Reviewed commit: b297f335

Comment thread src/workerd/util/sqlite.h Resolved
@danlapid
danlapid merged commit 6713c32 into main Nov 12, 2026
24 checks passed
@danlapid
danlapid deleted the dlapid/test-sqlite-no-sync branch November 12, 2026 04:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants