Skip to content

Add R2 buckets to workerd - #8507

Merged
ry merged 16 commits into
mainfrom
ry/r2
Nov 19, 2026
Merged

ry merged 16 commits into
mainfrom
ry/r2

Conversation

@ry

@ry ry commented Nov 16, 2026 •

Copy link
Copy Markdown
Member

workerd has no R2, so a Worker with an r2Bucket binding only runs under Miniflare or an embedder's own implementation. This adds R2 to standalone workerd and to a cluster.

R2 is a bundled Worker, /workerd/r2.js, as KV is: it serves the protocol the existing R2 binding already speaks, so the binding is unchanged, and the bucket is named in the binding's props. It stores each R2 object as one object under its own key in a bucket service, which gains a small S3-like HTTP API (HEAD, ranged GET, list, conditional writes, multipart). In a cluster that is the cluster's S3 or Cloud Storage bucket, so S3 and GCS tools read a bucket directly; without one, a local store on disk that several processes can share. There is no index.

(name = "PHOTOS", r2Bucket = (name = "r2", props = (json = "{\"bucket\": \"photos\"}"))),
(name = "r2", worker = (
  modules = [(name = "r2.js", bundled = "/workerd/r2.js")], compatibilityDate = "2026-01-01",
  bindings = [(name = "BLOBS", service = "r2-blobs")])),
(name = "r2-blobs", bucket = (path = "r2")),
// wrangler.jsonc, then `workerd serve --experimental`
{
  "name": "app",
  "main": "index.js",
  "r2_buckets": [{ "binding": "PHOTOS", "bucket_name": "photos" }]
}

Values stream both ways. A put is hashed as it is read and written only if its checksums pass. Miniflare's R2 tests pass against it (cloudflare/workers-sdk#17019, draft). Where a plain object store can't be R2, listed at the top of r2.js:

  • objects over 8 MiB get a multipart etag, and their onlyIf isn't fenced
  • part etags are their MD5; aborting an unknown upload succeeds
  • on S3, customMetadata is limited by S3's 2 KB of user metadata
  • a listing that includes metadata is a HEAD per object
  • uploads cut off mid-stream are left for the bucket's lifecycle rule

Not yet: more than one part in flight (one stream tops out around 12-15 MB/s), and an S3-compatible endpoint.

Adds R2 as a bundled Worker, /workerd/r2.js, serving the HTTP protocol
the r2Bucket binding already speaks, so the binding is unchanged; the
only C++ change registers the module.

Each R2 object is one object in a `bucket` service at <bucket>/<key>:
in a cluster, <prefix>v1/services/<path>/<bucket>/<key> in its S3 or
Cloud Storage bucket; without one, a local store. Custom metadata,
checksums and storage class are JSON in one user metadata field.

The bucket service grows from get/put/delete to a small S3-like API:
HEAD, ranged streamed GET, streamed PUT with content headers, user
metadata and version guards, paged list, and multipart uploads. A PUT
over 8 MiB becomes the store's multipart upload. ClusterBucket gains
localDisk (a JSON file and a blob per object, for workerd without
a cluster) and minPartSize (for tests). The cluster crate's Store gains
head, ranged read, page, write and multipart, for S3 and Cloud Storage.

wrangler.json r2_buckets map to the local store under
.wrangler/state/workerd/r2/. Miniflare's R2 binding tests pass against
r2.js (in the workers-sdk change that runs Miniflare's R2 on it); four
assertions there mark where a plain object store differs from its
simulator. tests/r2 covers what they don't.
@ry
ry requested review from a team as code owners November 16, 2026 22:48
The JSON value r2.js stores beside each object, for what R2 has and a
store does not, is now in the user metadata field workerd-r2. The
header comments no longer refer to another implementation.

@clanky-the-clanker clanky-the-clanker Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Superseded by the review of 846eddd0.

This review, of 52add21b

💬 Approved with comments — Two WARNINGs, both in the local bucket store: an unchecked upload ID is used as a path component, and multipart completion runs a blocking copy on the event loop. The rest are optional suggestions.

Findings: 2 warnings · 3 suggestions (5 are inline comments on the diff)

The decision is advisory: this review is a comment and never approves, blocks or requests changes on the pull request.

🔧 Fix with your agent

Paste the following into your coding agent to investigate and fix these issues.

Fix the following code review issues found in pull request #8507.

Before making changes, review each issue and present a brief summary table:
- For each finding, state whether you agree, disagree, or need clarification
- If you disagree (e.g., the fix would require disproportionate changes
  for minimal ROI, or the finding is factually incorrect), explain why
- If you need to clarify something before deciding, ask those questions
- Then share your plan for which issues to tackle and in what order

## WARNING (2)

### [Security] The local store joins the Worker-supplied multipart upload ID into `uploads/` without checking it, so `../` or an absolute ID escapes the bucket directory.
- **File:** src/workerd/server/server/services/bucket/local.rs:249

### [Performance] `Local::complete` concatenates all parts with synchronous `std::io::copy` on the loop thread, stalling every request in the process for the length of a potentially multi-GiB copy.
- **File:** src/workerd/server/server/services/bucket/local.rs:501

## SUGGESTION (3)

### [Code Quality] `range()` computes `last + 1` on a client-supplied u64, which overflows for `bytes=0-18446744073709551615`.
- **File:** src/workerd/server/server/services/bucket.rs:287

### [Code Quality] The test helper resolves on the child's `exit` event, which can fire before stderr is fully read, so the stderr assertion can flake.
- **File:** src/workerd/server/tests/r2/r2-wrangler-test.mjs:68

### [Code Quality] `blob_services` was generalized with a `(module, binding, class)` parameter, but KV is its only caller.
- **File:** src/workerd/server/cli/wrangler.rs:727

📋 Code Quality: 💬 3 suggestions

No correctness bugs; three small robustness and cleanup suggestions.

Recommendation: 💬 Optional Nits

  • [SUGGESTION] range() computes last + 1 on a client-supplied u64, which overflows for bytes=0-18446744073709551615. (src/workerd/server/server/services/bucket.rs:287)
  • [SUGGESTION] The test helper resolves on the child's exit event, which can fire before stderr is fully read, so the stderr assertion can flake. (src/workerd/server/tests/r2/r2-wrangler-test.mjs:68)
  • [SUGGESTION] blob_services was generalized with a (module, binding, class) parameter, but KV is its only caller. (src/workerd/server/cli/wrangler.rs:727)
View full review
  • range() overflow (src/workerd/server/server/services/bucket.rs:287): last + 1 overflows for bytes=0-18446744073709551615. That panics in builds with overflow checks; release builds wrap and happen to reject it. Use checked_add.
  • Test flake (src/workerd/server/tests/r2/r2-wrangler-test.mjs:68): the helper resolves on exit, which can fire before stderr has been fully read, so the stderr match in the refusal test can flake. Listen for close instead.
  • One-caller helper (src/workerd/server/cli/wrangler.rs:727): blob_services was generalized with a (module, binding, class) tuple, but KV is its only caller, and its doc comment still says "The KV Worker".

🔒 Security: ⚠️ 1 warning

The local store uses the Worker-supplied multipart upload ID as a path component without checking it.

Recommendation: 📝 Minor Comments

  • [WARNING] The local store joins the Worker-supplied multipart upload ID into uploads/ without checking it, so ../ or an absolute ID escapes the bucket directory. (src/workerd/server/server/services/bucket/local.rs:249)
View full review

Upload ID joined into a path unchecked (src/workerd/server/server/services/bucket/local.rs:248-265)

A Worker controls the upload ID, for example through resumeMultipartUpload(key, uploadId). r2.js forwards it as ?upload=<id>, and the bucket service passes it to Local::upload, save_upload and forget unchanged. Those build the path with self.root.join("uploads").join(id). An ID containing ../, or an absolute path (which replaces the base path entirely), therefore reads, rewrites or removes files outside uploads/.

The practical risk is limited. The operation only goes ahead if the target parses as an Upload JSON whose key matches the request, and blob names are random. It is still a path-traversal primitive reachable from Worker input, and it is cheap to close: accept only IDs in the format random() produces (32 lowercase hex characters) and answer NoSuchUpload otherwise.

Object keys are hashed to file names, so keys cannot traverse the filesystem. Part, completion and list bodies are size-capped.


⚡ Performance: ⚠️ 1 warning

Local multipart completion copies the whole object with blocking I/O on the event-loop thread.

Recommendation: 📝 Minor Comments

  • [WARNING] Local::complete concatenates all parts with synchronous std::io::copy on the loop thread, stalling every request in the process for the length of a potentially multi-GiB copy. (src/workerd/server/server/services/bucket/local.rs:501)
View full review

Blocking copy in Local::complete (src/workerd/server/server/services/bucket/local.rs:496-508)

Completing an upload concatenates every part blob into a new blob using synchronous std::fs::File and std::io::copy, with no .await in the loop. Objects over 8 MiB go through this path, and they can be many GiB. For the whole copy, every other request in the process stalls, including requests to other Workers.

The module comment says disk I/O is synchronous on purpose. That is reasonable for small reads and writes, but not for a copy that grows with the object's size. Possible fixes:

  • Move the copy to a blocking thread pool.
  • Copy in bounded chunks and yield between them.
  • Have the object reference its part blobs, which avoids the copy entirely.

✅ All other checks passed

📖 AGENTS.md: ✅ LGTM

The src/workerd/server/AGENTS.md updates are accurate.


📝 Documentation: ✅ LGTM

The README and schema documentation changes match the new behaviour.


Reviewed commit: 52add21b

Comment thread src/workerd/server/cli/wrangler.rs Resolved
The local store copied files with fs::copy and io::copy, whose
copy_file_range does not link against the arm64 Linux sysroot; it reads
and writes instead.

The cluster test Worker built and compared multi-MiB bodies a byte at a
time in JS, which in debug and tsan builds held the node's thread past
its lease; it uses native copies and a digest. The client test took the
first chunk of a dribbled body to be one byte; on a busy runner two
arrive together.

@clanky-the-clanker clanky-the-clanker Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Superseded by the review of ce034841.

This review, of 846eddd0

💬 Approved with comments — Two WARNINGs, both still open from the last review and both in the local bucket store: an upload ID that goes into a file path unchecked, and a blocking part copy in multipart completion. The rest are optional suggestions.

Findings: 2 warnings · 3 suggestions (2 are inline comments on the diff)

The decision is advisory: this review is a comment and never approves, blocks or requests changes on the pull request.

🔧 Fix with your agent

Paste the following into your coding agent to investigate and fix these issues.

Fix the following code review issues found in pull request #8507.

Before making changes, review each issue and present a brief summary table:
- For each finding, state whether you agree, disagree, or need clarification
- If you disagree (e.g., the fix would require disproportionate changes
  for minimal ROI, or the finding is factually incorrect), explain why
- If you need to clarify something before deciding, ask those questions
- Then share your plan for which issues to tackle and in what order

## WARNING (2)

### [Security] A multipart upload ID supplied by the Worker is joined into the `uploads/` path unchecked, so a `../` or absolute ID reads, rewrites or removes files outside the bucket directory.
- **File:** src/workerd/server/server/services/bucket/local.rs:250

### [Performance] `Local::complete` concatenates every part with synchronous file I/O on the loop thread, and now reads each whole part into memory. Every request in the process stalls for the length of a large multipart completion.
- **File:** src/workerd/server/server/services/bucket/local.rs:504

## SUGGESTION (3)

### [Code Quality] `range()` computes `last + 1` on a client-supplied u64, which overflows for `bytes=0-18446744073709551615`.
- **File:** src/workerd/server/server/services/bucket.rs:287

### [Code Quality] The test helper resolves on the child's `exit` event, which can fire before stderr is fully read, so the stderr assertion can flake.
- **File:** src/workerd/server/tests/r2/r2-wrangler-test.mjs:68

### [Code Quality] `blob_services` was generalized with a `(module, binding, class)` parameter, but KV is its only caller, and its doc comment still describes the KV Worker.
- **File:** src/workerd/server/cli/wrangler.rs:727

📋 Code Quality: 💬 3 suggestions

The three optional suggestions from the last review are still open.

Recommendation: 💬 Optional Nits

  • [SUGGESTION] range() computes last + 1 on a client-supplied u64, which overflows for bytes=0-18446744073709551615. (src/workerd/server/server/services/bucket.rs:287)
  • [SUGGESTION] The test helper resolves on the child's exit event, which can fire before stderr is fully read, so the stderr assertion can flake. (src/workerd/server/tests/r2/r2-wrangler-test.mjs:68)
  • [SUGGESTION] blob_services was generalized with a (module, binding, class) parameter, but KV is its only caller, and its doc comment still describes the KV Worker. (src/workerd/server/cli/wrangler.rs:727)
View full review
  • range() overflow (src/workerd/server/server/services/bucket.rs:287): last + 1 overflows for bytes=0-18446744073709551615.
  • Test flake (src/workerd/server/tests/r2/r2-wrangler-test.mjs:68): the helper resolves on exit instead of close, so the stderr check can see partial output.
  • One-caller helper (src/workerd/server/cli/wrangler.rs:727): blob_services is generalized, but KV is its only caller.

🔒 Security: ⚠️ 1 warning

The path traversal through the upload ID in the local store is still present.

Recommendation: 📝 Minor Comments

  • [WARNING] A multipart upload ID supplied by the Worker is joined into the uploads/ path unchecked, so a ../ or absolute ID reads, rewrites or removes files outside the bucket directory. (src/workerd/server/server/services/bucket/local.rs:250)
View full review

Upload ID joined into a path unchecked (src/workerd/server/server/services/bucket/local.rs:184-186, 249-265)

The Worker controls the ?upload=<id> value. Local::upload, save_upload and forget all pass it to self.root.join("uploads").join(id) without checking it. An ID containing ../, or an absolute path, therefore reaches files outside uploads/.

upload() only goes ahead when the target file parses as an Upload record with a matching key, which limits the impact. It is still a filesystem path that Worker input controls. To fix it, accept only IDs in the format random() produces (32 lowercase hex characters) and answer NoSuchUpload for anything else.

Object keys are hashed into file names, so keys cannot traverse the filesystem.


⚡ Performance: ⚠️ 1 warning

Multipart completion still copies all parts synchronously on the event loop.

Recommendation: 📝 Minor Comments

  • [WARNING] Local::complete concatenates every part with synchronous file I/O on the loop thread, and now reads each whole part into memory. Every request in the process stalls for the length of a large multipart completion. (src/workerd/server/server/services/bucket/local.rs:504)
View full review

Blocking copy in Local::complete (src/workerd/server/server/services/bucket/local.rs:500-506)

The loop now runs out.write_all(&std::fs::read(...)) once per part, using synchronous std::fs calls with no .await. Every object over 8 MiB goes through this path. While the copy runs, every other request in the process waits, including requests to other Workers.

The new code also reads each part fully into memory before writing it. Part uploads are capped at 64 MiB, so each step of the copy can now hold up to 64 MiB.

Possible fixes:

  • Copy in bounded chunks with a reused buffer and yield between chunks.
  • Run the copy on a blocking thread.
  • Have the object reference its part blobs, which avoids the copy entirely.

✅ All other checks passed

📖 AGENTS.md: ✅ LGTM

The src/workerd/server/AGENTS.md updates are accurate.


📝 Documentation: ✅ LGTM

The README and schema documentation match the behaviour.


Reviewed commit: 846eddd0

Comment thread src/workerd/server/r2/r2.js Resolved
Comment thread src/workerd/server/r2/r2.js Resolved
Comment thread src/workerd/server/r2/r2.js Resolved
Comment thread src/workerd/server/cluster/client.rs Resolved
Comment thread src/workerd/server/r2/r2.js Resolved
ry added 2 commits November 16, 2026 21:06
- The local store refuses an upload ID it could not have made before it
  is a path, so a resumed upload cannot reach outside uploads/.
- SigV4 signs every x-amz-* header, metadata included, as S3 requires;
  a test pins the signature to botocore's for the same request.
- The bucket service refuses a key with a . or .. segment, which a store
  or proxy that resolves them could follow out of the binding's
  directory; r2.js answers it as InvalidObjectName.
- A listing carries each object's version (Cloud Storage's generation,
  S3's ETag, the local index's), so list and head agree.
- A local store's directory is locked by the process that opens it.
- A completed multipart object on the local store is its parts' blobs,
  so completion copies nothing; the store is documented as for
  development and tests.
- A Range header's last byte saturates, so bytes=N-<u64 max> reads to
  the end instead of overflowing.
- The wrangler test waits for close, so it sees all of stderr.
- kv_services is back as on main; R2 has its own helper.

@clanky-the-clanker clanky-the-clanker Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Superseded by the review of 9819c0eb.

This review, of ce034841

💬 Approved with comments — Earlier warnings are fixed. Three WARNINGs remain: a non-ASCII Content-Disposition (or other HTTP field) fails the put, customMetadata over S3's 2 KB user-metadata limit fails on S3-backed clusters, and the schema doc still claims dot-segment keys are allowed.

Findings: 3 warnings · 1 suggestion (4 are inline comments on the diff)

The decision is advisory: this review is a comment and never approves, blocks or requests changes on the pull request.

🔧 Fix with your agent

Paste the following into your coding agent to investigate and fix these issues.

Fix the following code review issues found in pull request #8507.

Before making changes, review each issue and present a brief summary table:
- For each finding, state whether you agree, disagree, or need clarification
- If you disagree (e.g., the fix would require disproportionate changes
  for minimal ROI, or the finding is factually incorrect), explain why
- If you need to clarify something before deciding, ask those questions
- Then share your plan for which issues to tackle and in what order

## WARNING (3)

### [Code Quality] httpMetadata values are sent as raw headers, and the bucket service refuses non-ASCII header values, so a put with a non-ASCII Content-Disposition (or another HTTP field) fails where R2 accepts it.
- **File:** src/workerd/server/r2/r2.js:373

### [Code Quality] The whole R2 envelope is stored as one user-metadata header. R2 allows 8 KiB of customMetadata, but S3 caps user metadata at 2 KB, so larger puts fail on an S3-backed cluster.
- **File:** src/workerd/server/r2/r2.js:379

### [Documentation] The ClusterBucket schema comment says keys may hold dot segments, but the bucket service refuses keys with `.` or `..` segments.
- **File:** src/workerd/server/workerd.capnp:467

## SUGGESTION (1)

### [Code Quality] `content_range` computes `last + 1` on a u64 parsed from the store's Content-Range header. It overflows when `last` is u64::MAX.
- **File:** src/workerd/server/cluster/client.rs:820

📋 Code Quality: ⚠️ 2 warnings · 1 suggestion

Earlier nits are fixed. Two put paths still fail where R2 succeeds, plus one overflow nit.

Recommendation: 📝 Minor Comments

  • [WARNING] httpMetadata values are sent as raw headers, and the bucket service refuses non-ASCII header values, so a put with a non-ASCII Content-Disposition (or another HTTP field) fails where R2 accepts it. (src/workerd/server/r2/r2.js:373)
  • [WARNING] The whole R2 envelope is stored as one user-metadata header. R2 allows 8 KiB of customMetadata, but S3 caps user metadata at 2 KB, so larger puts fail on an S3-backed cluster. (src/workerd/server/r2/r2.js:379)
  • [SUGGESTION] content_range computes last + 1 on a u64 parsed from the store's Content-Range header. It overflows when last is u64::MAX. (src/workerd/server/cluster/client.rs:820)
View full review

The three earlier suggestions are fixed: range() saturates, the wrangler test resolves on close, and kv_services is back.

Two correctness problems from the discussion remain in r2.js:

  • Non-ASCII HTTP metadata: headers() sends httpMetadata values as raw request headers. described()/ascii() in services/bucket.rs refuse any non-ASCII value, so a put with, say, contentDisposition: 'attachment; filename="résumé.pdf"' fails.
  • Envelope size on S3: the envelope goes into one x-meta-workerd-r2 header. R2 allows 8 KiB of customMetadata (MAX_METADATA), but S3 caps user metadata at 2 KB, so large-metadata puts fail on S3-backed clusters.

One minor point: content_range in cluster/client.rs still computes an unchecked last + 1.


📝 Documentation: ⚠️ 1 warning

The schema comment says keys may contain dot segments, but the service refuses them.

Recommendation: 📝 Minor Comments

  • [WARNING] The ClusterBucket schema comment says keys may hold dot segments, but the bucket service refuses keys with . or .. segments. (src/workerd/server/workerd.capnp:467)
View full review

workerd.capnp says ClusterBucket keys "may hold /, empty segments and dot segments". services/bucket.rs now answers 400 InvalidArgument for any key with a . or .. segment.


✅ All other checks passed

🔒 Security: ✅ LGTM

  • Upload IDs: Local::upload now accepts only 32 lowercase hex characters before an ID becomes a path. part, complete, abort and parts all go through it first.
  • Dot segments: BucketRequest::handle refuses keys with . or .. segments.
  • SigV4: signing now covers every x-amz-* header, metadata included. A test pins the signature to botocore's.
  • Shared directory: Local::open takes an exclusive lock on the store directory.

📖 AGENTS.md: ✅ LGTM

The src/workerd/server/AGENTS.md updates are accurate.


⚡ Performance: ✅ LGTM

Local::complete now builds the object from references to its parts' blobs, so it no longer copies or buffers part data on the loop thread.


Reviewed commit: ce034841

Comment thread src/workerd/server/r2/r2.js Resolved
Comment thread src/workerd/server/r2/r2.js Resolved
Comment thread src/workerd/server/workerd.capnp Resolved
Comment thread src/workerd/server/cluster/client.rs Resolved
ry added 2 commits November 16, 2026 21:51
A `bucket` service with `r2 = true` serves the R2 binding's protocol
itself (services/bucket/r2.rs): one service per bucket, named by the
binding, `r2Bucket = "photos"`, with no Worker or props. It replaces
r2.js and the S3-like HTTP API the bucket service grew for it; the
plain service is back to main's GET, PUT and DELETE, plus the local
store.

From review of #8507:
- A store's refusal of metadata (S3 keeps 2 KB) is R2's
  MetadataTooLarge, 10012; fake S3 enforces the limit, and a PUT's
  failure is now `Error::Refused` with its code, as other calls are.
- Content fields that are header-safe are stored as headers, others
  in the record.
- `uploaded` is the put's own time, kept in the record; a multipart
  object's, and a listing's without `include`, is the store's.
- A put without a Content-Length (or a known body length) is refused.

MD5 can be fed in pieces (`cluster::Md5`), so a put over one part is
hashed as it streams and completed only if its checksums pass.

wrangler.rs makes `<name>-r2-<bucket>` per bucket, stored in
`.wrangler/state/workerd/r2/<bucket>/`.
The error table is one line or two per code, unformatted. A reply is
its metadata and an optional body. r2-test.rs's cases are covered end
to end by r2-test.js and Miniflare's R2 tests.
@ry

ry commented Nov 17, 2026

Copy link
Copy Markdown
Member Author

[AI] Heads up: since your review I've replaced r2.js with an R2 service in Rust, so much of what you read is gone. Following the question of why we implement S3's API only to translate R2's onto it, a bucket service with r2 = true now speaks the R2 binding's protocol directly (services/bucket/r2.rs), one service per bucket, with no Worker or props. The bucket service's S3-like HTTP API is gone with it; the plain service is back to main's GET/PUT/DELETE plus the local store. The PR is about 800 lines smaller.

The decisions from your review are in: a store's metadata refusal maps to R2's 10012 (fake S3 now enforces S3's 2 KB), header-safe content fields are stored as headers, uploaded is the put's own time, and a put without a known length is a 400. The local store, the S3 and Cloud Storage client and the cluster changes you reviewed are unchanged, apart from MD5 now taking its input in pieces and a PUT failure carrying its store error code.

@clanky-the-clanker clanky-the-clanker Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Superseded by the review of 2a3bf5f9.

This review, of 9819c0eb

💬 Approved with comments — The non-ASCII metadata and dot-segment doc warnings are fixed. Three WARNINGs remain: a Range end near u64::MAX still overflows on the cluster read path, a wrangler bucket named "storage" collides with the shared disk service, and customMetadata over S3's 2 KB user-metadata limit fails on S3-backed clusters.

Findings: 3 warnings · 2 info · 1 suggestion (5 are inline comments on the diff)

The decision is advisory: this review is a comment and never approves, blocks or requests changes on the pull request.

🔧 Fix with your agent

Paste the following into your coding agent to investigate and fix these issues.

Fix the following code review issues found in pull request #8507.

Before making changes, review each issue and present a brief summary table:
- For each finding, state whether you agree, disagree, or need clarification
- If you disagree (e.g., the fix would require disproportionate changes
  for minimal ROI, or the finding is factually incorrect), explain why
- If you need to clarify something before deciding, ask those questions
- Then share your plan for which issues to tackle and in what order

## WARNING (3)

### [Code Quality] `Range::header` computes `offset + n` unchecked. For `bytes=N-<u64 max>`, `header_range` saturates the length to `u64::MAX - N + 1`, so a cluster-backed get panics (overflow checks on) or sends `bytes=N-0` and returns the whole object with no range (release).
- **File:** src/workerd/server/cluster/bucket.rs:146

### [Code Quality] A wrangler R2 bucket named `storage` gets the service name `<name>-r2-storage`, the same name as the shared disk service, so the generated config has two services with one name.
- **File:** src/workerd/server/cli/wrangler.rs:785

### [Code Quality] The whole R2 record is stored as one `x-amz-meta-workerd-r2` header, with non-ASCII escaped to `\uXXXX`. customMetadata that passes R2's 8 KiB check can exceed S3's 2 KB user-metadata limit and fail the put on an S3-backed cluster.
- **File:** src/workerd/server/server/services/bucket/r2.rs:386

## INFO (2)

### [Code Quality] `uploadPart` refuses parts over 64 MiB (`MAX_PART`), while R2 accepts up to 5 GiB. The constant's comment notes this, but the module's list of differences from R2 does not.
- **File:** src/workerd/server/server/services/bucket/r2.rs:814

### [Performance] Multipart completion finds each chosen part with a linear scan of the uploaded parts, which is quadratic in the part count.
- **File:** src/workerd/server/server/services/bucket/r2.rs:939

## SUGGESTION (1)

### [Code Quality] `content_range` still computes an unchecked `last + 1` on a u64 parsed from the store's Content-Range header.
- **File:** src/workerd/server/cluster/client.rs:820

📋 Code Quality: ⚠️ 3 warnings · 1 info · 1 suggestion

The range-overflow fix doesn't cover the cluster read path, a wrangler bucket named "storage" collides with the disk service, and the S3 2 KB metadata limit is still open.

Recommendation: 📝 Minor Comments

  • [WARNING] Range::header computes offset + n unchecked. For bytes=N-<u64 max>, header_range saturates the length to u64::MAX - N + 1, so a cluster-backed get panics (overflow checks on) or sends bytes=N-0 and returns the whole object with no range (release). (src/workerd/server/cluster/bucket.rs:146)
  • [WARNING] A wrangler R2 bucket named storage gets the service name <name>-r2-storage, the same name as the shared disk service, so the generated config has two services with one name. (src/workerd/server/cli/wrangler.rs:785)
  • [WARNING] The whole R2 record is stored as one x-amz-meta-workerd-r2 header, with non-ASCII escaped to \uXXXX. customMetadata that passes R2's 8 KiB check can exceed S3's 2 KB user-metadata limit and fail the put on an S3-backed cluster. (src/workerd/server/server/services/bucket/r2.rs:386)
  • [INFO] uploadPart refuses parts over 64 MiB (MAX_PART), while R2 accepts up to 5 GiB. The constant's comment notes this, but the module's list of differences from R2 does not. (src/workerd/server/server/services/bucket/r2.rs:814)
  • [SUGGESTION] content_range still computes an unchecked last + 1 on a u64 parsed from the store's Content-Range header. (src/workerd/server/cluster/client.rs:820)
View full review

Two earlier warnings are fixed. record() in services/bucket/r2.rs now keeps fields that are not header-safe in the JSON record, so a non-ASCII httpMetadata value no longer fails the put. The workerd.capnp comment no longer says keys may hold dot segments.

Range end overflow on the cluster path (cluster/bucket.rs:146)

  • header_range saturates the length for bytes=5-18446744073709551615, so it produces length = u64::MAX - 4.
  • Range::header then computes offset + n, which overflows for that input.
  • With overflow checks on, this panics. In release builds it wraps to bytes=5-0, which S3 ignores, so the reply carries the whole object with no range.
  • The local store never calls header(), so the tests don't catch this. The earlier fix covered only the local path.

Service-name collision (cli/wrangler.rs:785)

  • The shared disk is named <name>-r2-storage, and each bucket service is named <name>-r2-<bucket>.
  • A wrangler bucket named storage (by bucket_name, or by a binding name with no bucket_name) produces two services with the same name.

Envelope size on S3 (services/bucket/r2.rs:386)

  • The whole record goes into one x-amz-meta-workerd-r2 header, and non-ASCII characters are escaped as \uXXXX.
  • customMetadata that passes R2's 8 KiB check can therefore go over S3's 2 KB user-metadata limit, and the put fails with MetadataTooLarge.
  • Either list this as a difference or store large metadata some other way.

Smaller points

  • uploadPart refuses parts over 64 MiB (MAX_PART), while R2 takes up to 5 GiB. The constant's comment notes the limit, but the module's list of differences does not.
  • content_range in cluster/client.rs still computes an unchecked last + 1.

⚡ Performance: 💬 1 info

One minor quadratic lookup in multipart completion.

Recommendation: 💬 Optional Nits

  • [INFO] Multipart completion finds each chosen part with a linear scan of the uploaded parts, which is quadratic in the part count. (src/workerd/server/server/services/bucket/r2.rs:939)
View full review

complete in services/bucket/r2.rs matches each chosen part against the uploaded list with a linear scan. With 10,000 parts that is up to about 1e8 comparisons, run on the request task. Reads stream, put buffering is bounded, and listing fetches and bulk deletes are bounded by PARALLEL.


✅ All other checks passed

🔒 Security: ✅ LGTM

  • Upload IDs: Local::upload accepts only 32 lowercase hex characters before an ID becomes a path.
  • Dot segments: every R2 operation takes its key through key() in services/bucket/r2.rs, which refuses a key with a . or .. segment.
  • SigV4: sign() signs every x-amz-* header on the request.
  • Directory lock: Local::open takes an exclusive lock on the store directory.
  • Puts: a put without a valid Content-Length is refused, and a put is written only after its checksums pass.

📖 AGENTS.md: ✅ LGTM

The src/workerd/server/AGENTS.md updates are accurate.


📝 Documentation: ✅ LGTM

The workerd.capnp comment no longer says keys may hold dot segments. It now matches the bucket service, which refuses them.


Reviewed commit: 9819c0eb

Comment thread src/workerd/server/cluster/bucket.rs Resolved
Comment thread src/workerd/server/cli/wrangler.rs Resolved

@piscisaureus piscisaureus left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Two non-blocking nits.

Comment thread src/workerd/server/cluster/bucket.rs Resolved
ry added 2 commits November 17, 2026 15:18
- A list reply stops before its JSON passes 512 KiB (the binding
  refuses over 1 MiB), with a cursor, `k.<key>`, that resumes after its
  last key, as R2 returns fewer than `limit`.
- `Range::header` saturates its end, so `bytes=5-<u64 max>` on a
  cluster is a range, not an overflow.
- wrangler's R2 disk is `<name>-r2:storage`, which no bucket's service
  name can be.
- Completion finds parts in a map, not a scan per part.
- r2.rs lists the 64 MiB part limit and S3's 2 KB of user metadata
  among its differences from R2; `Version`'s doc drops `x-version`.

@clanky-the-clanker clanky-the-clanker Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Superseded by the review of 07354356.

This review, of 2a3bf5f9

✅ Approved — All three earlier warnings and the earlier INFO notes are fixed at this commit: the cluster Range end saturates, the shared disk service name can no longer collide with a bucket, and the S3 metadata cap and part-size limit are listed as differences from R2. No new issues found.

The decision is advisory: this review is a comment and never approves, blocks or requests changes on the pull request.


✅ All checks passed

🔒 Security: ✅ LGTM

  • SigV4: sign() signs every x-amz-* header on the request, metadata included.
  • Keys: key() caps keys at 1024 bytes and refuses any . or .. segment before a key reaches the store.
  • Local store: object files are named by the MD5 of the key. An upload ID must be exactly 32 lowercase hex characters before it is used as a path. An exclusive lock stops two processes from sharing a directory.
  • Puts: a put without a valid Content-Length is refused. Checksums are verified before a write or before a multipart upload completes, and a failed upload is aborted.
  • Streaming: only 2xx bodies stream. Any other body is still capped, and each wait for the next chunk has a timeout.

📖 AGENTS.md: ✅ LGTM

The src/workerd/server/AGENTS.md updates are accurate.


📋 Code Quality: ✅ LGTM

The earlier findings, checked against this commit:

  • Range end on the cluster path (cluster/bucket.rs:146): fixed. Range::header now computes offset.saturating_add(n).saturating_sub(1), so bytes=5-<u64 max> becomes bytes=5-18446744073709551614 and no longer overflows. bucket-test.rs covers this case.
  • Service-name collision (cli/wrangler.rs:786): fixed. The shared disk service is now named <name>-r2:storage. A bucket name cannot contain :, so no <name>-r2-<bucket> service can share that name. wrangler-test.rs checks the generated service names.
  • customMetadata over S3's 2 KB user-metadata limit: addressed as a documented difference. The module header in services/bucket/r2.rs lists it, a put that doesn't fit is refused as MetadataTooLarge, and the cluster test covers this against fake-s3.mjs.
  • MAX_PART: addressed as a documented difference. It is now in the module's list of differences from R2.

One earlier note is still open and was left as the author's choice: the unchecked last + 1 in content_range in cluster/client.rs.


⚡ Performance: ✅ LGTM

complete in services/bucket/r2.rs now builds a HashMap keyed by (part number, etag), so each part is found in O(1) instead of by a linear scan.


📝 Documentation: ✅ LGTM

The README and workerd.capnp changes match the code.


Reviewed commit: 2a3bf5f9

A Content-Range ending at u64::MAX overflowed on last + 1. Use
checked_add so a malformed header is rejected instead.

@clanky-the-clanker clanky-the-clanker Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Superseded by the review of ec8ace17.

This review, of 07354356

✅ Approved — The one change since the last review fixes the last open overflow note in content_range, and no new issues were found.

The decision is advisory: this review is a comment and never approves, blocks or requests changes on the pull request.


✅ All checks passed

🔒 Security: ✅ LGTM

  • A malformed Content-Range from the store is now an error, not an overflow.
  • Key validation, upload-ID checks, the put size limits and checksum verification behave as described in the previous review.

📖 AGENTS.md: ✅ LGTM

The src/workerd/server/AGENTS.md updates are accurate.


📋 Code Quality: ✅ LGTM

  • content_range (cluster/client.rs:816-820): fixed. last.checked_add(1)?.checked_sub(first)? rejects a range ending at u64::MAX and an inverted range. Neither causes a panic or a wrapped length any more.
  • The R2 service, the tests and the schema changes are consistent with each other and with the module's documented differences from R2.

⚡ Performance: ✅ LGTM

No performance concerns in the change since the last review.


📝 Documentation: ✅ LGTM

The README and workerd.capnp changes match the code.


Reviewed commit: 07354356

@clanky-the-clanker clanky-the-clanker Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Superseded by the review of c1565453.

This review, of ec8ace17

✅ Approved — The one change since the last review fixes the last open overflow note in content_range, and no new issues were found.

The decision is advisory: this review is a comment and never approves, blocks or requests changes on the pull request.


✅ All checks passed

🔒 Security: ✅ LGTM

  • A malformed Content-Range from the store is now an error, not an overflow.
  • Key validation, upload-ID checks, the put size limits and checksum verification behave as described in the previous review.

📖 AGENTS.md: ✅ LGTM

The src/workerd/server/AGENTS.md updates are accurate.


📋 Code Quality: ✅ LGTM

  • content_range (cluster/client.rs:816-820): fixed. last.checked_add(1)?.checked_sub(first)? rejects a range ending at u64::MAX and an inverted range. Neither causes a panic or a wrapped length any more.
  • The R2 service, the tests and the schema changes are consistent with each other and with the module's documented differences from R2.

⚡ Performance: ✅ LGTM

No performance concerns in the change since the last review.


📝 Documentation: ✅ LGTM

The README and workerd.capnp changes match the code.


Reviewed commit: ec8ace17 — same changes as 07354356, so this is that commit's review (no new review was run; line numbers are as of 07354356)

@danlapid danlapid left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for taking R2 on, it's great to see it come to standalone workerd. Same direction as on #8172, though: R2 should be "just a worker", with all of its logic in JS. The latest revision moves it the other way. r2.js became services/bucket/r2.rs, switched on by an r2 = true flag on the generic bucket service. That's the per-product native mode we just took out of KV. I'd go back to JS.

1. R2 as r2.js, shaped like kv.js

Load it through Module.builtin = "/workerd/r2.js", and point r2Bucket bindings at it with {"bucket": "<name>"} in props. That's how KV works now, and how production passes per-binding config. Today each bucket is its own native service with no props, so KV and R2 would have two different binding models.

2. Consider an index, as KV and Miniflare have

Most of the native code, and most of the gaps from real R2, come from mapping each R2 object 1:1 onto a store object with no index. The header of r2.rs lists where that falls short:

  • onlyIf isn't fenced for puts over 8 MiB.
  • Those puts also get a multipart etag.
  • A part's etag is its MD5.
  • Aborting an unknown upload succeeds.
  • uploadPart is capped at 64 MiB (R2 takes up to 5 GiB).
  • On S3, custom metadata is capped at S3's 2 KB, where R2 allows 8,192 bytes.

KV's design avoids all of these: one Durable Object per bucket, with a SQLite index of keys, etags, versions, metadata and multipart state, and the bytes stored as blobs through BLOBS (a disk locally, the existing plain GET/PUT/DELETE bucket service in a cluster).

  • The object is the single writer, so onlyIf is exact.
  • Metadata lives in SQLite, so the 8 KB limit holds.
  • Etags are computed in JS, with WebCrypto's MD5.
  • An alarm can collect uploads cut off mid-stream, as KV's collects blobs.

Miniflare's R2 already works this way (R2BucketObject over _mf_objects and _mf_multipart_uploads), so it could come over as KV's did. Then r2.rs, local.rs, the store API added to the cluster client (list, guards, multipart, ranges) and the hand-written MD5 in cluster/lib.rs aren't needed.

The trade-off: objects wouldn't be readable by S3 tools under their own names, and one object per bucket caps throughput, as it does for KV. If readable-by-S3-tools is a hard requirement, the first revision's shape still works: r2.js over a generic store API. The translation layer is the price of keeping the product logic in JS, and I think it's worth paying.

3. Local dev

local.rs takes an exclusive lock, so a second process on the same state fails with "in use by another process". That means wrangler dev and wrangler r2 object put --local can't run together, which KV supports and tests. It's also a second on-disk format next to disk, and it does synchronous std::fs I/O on the loop. With the index design, local R2 is just a disk plus SQLite, safe across processes like KV.

4. Self-hosting

Beyond the metadata cap above, uploads cut off mid-stream are never cleaned up, so their parts accumulate in the self-hoster's bucket. A single upload also tops out around 12-15 MB/s with one part in flight. The first goes away with the index design. The second is fine as a follow-up.

The path traversal fix for upload IDs looks good, and the wrangler r2_buckets handling (preview_bucket_name, jurisdiction, remote, defaulting to the binding name) is complete. Happy to talk through the index trade-off if I'm missing a reason the 1:1 mapping matters 🙂

ry added 2 commits November 18, 2026 07:59
Per review: R2's product logic belongs in a Worker, as KV's does, not in
the bucket service behind an `r2` flag. This reverts 8d79d52's Rust R2
service and keeps what came after it:

- r2.js again maps the binding's protocol onto the bucket service's
  S3-like HTTP API (HEAD, ranged GET, list, If-Match/If-None-Match,
  multipart). Each R2 object is the store object of its key, so S3 and
  GCS tools read a bucket directly. Bindings name the bucket in props,
  like KV's namespaces; one r2.js Worker serves every bucket.
- r2.js keeps a list reply under 512 KiB with a `k.<key>` cursor,
  refuses garbled metadata and unsized puts as 10029, and maps a store's
  refusal (S3's 2 KB of user metadata) to R2's error.
- The local store no longer keeps an index in memory or locks its
  directory: it reads object files on each call and holds a file lock
  only around each read-then-write, so `wrangler dev` and another
  process can share a directory.
- r2.js lists its differences from R2, now including the 64 MiB part
  limit, S3's metadata cap, a HEAD per listed object with `include`, and
  uploads left for a lifecycle rule to collect.
- `r2` is gone from ClusterBucket.
- A list reply's budget counts its JSON's UTF-8 bytes, prefixes
  included, walking objects and prefixes in the store's byte order; a
  page that stops at a prefix resumes after every key in it.
- A put's time is kept in the record, so put, head, get and onlyIf
  agree; a content field that can't be a header is only in the record
  (as before the Rust port, a non-ASCII Content-Disposition failed the
  put).
- The local store's complete and abort hold the lock throughout, so two
  processes can't both complete an upload or drop blobs an object
  holds; a part for an upload that is gone drops its blob.
- A store's refusal in a 2xx (S3's failed completion) answers 500.
- Listing XML decodes numeric entities (S3's `&#13;` in a key).
- r2.js lists two more differences: a listing without `include` says
  Standard, and `.`/`..` segments are refused.
@ry

ry commented Nov 18, 2026

Copy link
Copy Markdown
Member Author

[AI] @piscisaureus heads-up since you approved the Rust version: per Dan's review this is back to r2.js over the generic bucket service (b97a7a6, fixes in c156545). One change against your earlier ask: instead of an exclusive lock, the local store now reads object files per call and locks only around each read-then-write, so two processes can share .wrangler/state (Dan wanted wrangler dev plus --local CLI writes). Two of your earlier points that only the Rust version had fixed (non-ASCII content headers, one uploaded time) are fixed again in r2.js. Worth a look at local.rs and r2.js.

@clanky-the-clanker clanky-the-clanker Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Superseded by the review of 581944d7.

This review, of c1565453

💬 Approved with comments — The rework back to r2.js over the generic bucket service holds up: key, upload-ID, size and checksum handling are sound. One WARNING remains: the ClusterBucket schema doc says keys may contain dot segments, but the service refuses them. The other notes are optional edge cases and performance points.

Findings: 1 warning · 1 info · 3 suggestions (4 are inline comments on the diff)

The decision is advisory: this review is a comment and never approves, blocks or requests changes on the pull request.

🔧 Fix with your agent

Paste the following into your coding agent to investigate and fix these issues.

Fix the following code review issues found in pull request #8507.

Before making changes, review each issue and present a brief summary table:
- For each finding, state whether you agree, disagree, or need clarification
- If you disagree (e.g., the fix would require disproportionate changes
  for minimal ROI, or the finding is factually incorrect), explain why
- If you need to clarify something before deciding, ask those questions
- Then share your plan for which issues to tackle and in what order

## WARNING (1)

### [Documentation] `ClusterBucket` doc says keys may contain dot segments; the service refuses them (`bucket.rs:401`).
- **File:** src/workerd/server/workerd.capnp:468

## INFO (1)

### [Performance] Each local-store list page rebuilds the whole index from disk.
- **File:** src/workerd/server/server/services/bucket/local.rs:455

## SUGGESTION (3)

### [Code Quality] `toRange` can emit a Range end above u64::MAX for a huge range length, which the bucket service refuses with 400.
- **File:** src/workerd/server/r2/r2.js:548

### [Code Quality] A relative request target containing `://` (in the key or query) is misparsed as an absolute URL.
- **File:** src/workerd/server/server/services/bucket.rs:392

### [Code Quality] `fill` zero-fills about `len` bytes on every read, which is quadratic for bodies that arrive in small chunks.
- **File:** src/workerd/server/server/services/bucket.rs:344

📋 Code Quality: 💬 3 suggestions

Three optional edge cases in the reworked r2.js and bucket service.

Recommendation: 💬 Optional Nits

  • [SUGGESTION] toRange can emit a Range end above u64::MAX for a huge range length, which the bucket service refuses with 400. (src/workerd/server/r2/r2.js:548)
  • [SUGGESTION] A relative request target containing :// (in the key or query) is misparsed as an absolute URL. (src/workerd/server/server/services/bucket.rs:392)
  • [SUGGESTION] fill zero-fills about len bytes on every read, which is quadratic for bodies that arrive in small chunks. (src/workerd/server/server/services/bucket.rs:344)
View full review

🔄 Since last review: the earlier findings are still resolved; 5 new notes (1 WARNING, 4 optional).

The move back to r2.js over the generic bucket service is consistent and well tested. Points from the earlier discussion are handled in this commit: a missing Content-Length is refused, list replies have a size budget, and range ends saturate on the Rust side.

  • r2.js:548: toRange computes first + length - 1 as a JS Number. A range near 2^64 prints above u64::MAX, so the bucket service answers 400 instead of reading to the end.
  • bucket.rs:391-393: any :// in a relative request target is treated as the start of an absolute URL.
  • bucket.rs:344: fill zero-fills a doubled buffer on every read, so a body that arrives in small chunks costs quadratic zeroing per part.

⚡ Performance: 💬 1 info

Local-disk list pages re-read every object's metadata.

Recommendation: 💬 Optional Nits

  • [INFO] Each local-store list page rebuilds the whole index from disk. (src/workerd/server/server/services/bucket/local.rs:455)
View full review
  • bucket/local.rs:455: every list page calls self.index(), which reads and parses every object file synchronously. A paginated listing of N objects is therefore O(N²/page size) file reads. This is acceptable for a dev store, but it is worth caching if large local buckets matter.

📝 Documentation: ⚠️ 1 warning

The schema doc says keys may contain dot segments, but they are refused.

Recommendation: 📝 Minor Comments

  • [WARNING] ClusterBucket doc says keys may contain dot segments; the service refuses them (bucket.rs:401). (src/workerd/server/workerd.capnp:468)
View full review
  • workerd.capnp:468 says keys may hold dot segments. bucket.rs:401 refuses any key with a . or .. segment with a 400, so users following the doc will hit errors.

✅ All other checks passed

🔒 Security: ✅ LGTM

Checked in this commit:

  • Keys with . or .. segments are refused in both r2.js and bucket.rs. The local store names object files by the MD5 of the key, so a key never becomes a filesystem path.
  • Upload IDs must be 32 lowercase hex characters before they are joined into uploads/.
  • Put, part and completion bodies are size-capped. A missing Content-Length is refused.
  • The checksums are verified before the last chunk is released, so a mismatched put stores nothing.
  • SigV4 now signs every x-amz-* header.

📖 AGENTS.md: ✅ LGTM

The src/workerd/server/AGENTS.md updates match the code.


Reviewed commit: c1565453

Comment thread src/workerd/server/r2/r2.js Resolved
- r2.js reads to the end for a range whose end a Number can't hold
  exactly, rather than sending one past u64::MAX.
- The bucket service strips a scheme and host only from an absolute
  URL, so a key or query holding `://` keeps its path.
- `fill` zeroes its buffer only as it doubles, not on every read, so a
  body in small chunks costs linear time.

@clanky-the-clanker clanky-the-clanker Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Superseded by the review of e26ce06f.

This review, of 581944d7

💬 Approved with comments — The three notes from the last review (the toRange end, :// in a relative target, and fill's zeroing) are fixed at this commit. One WARNING is still open: the ClusterBucket schema doc says keys may hold dot segments, but the bucket service refuses them.

Findings: 1 warning

The decision is advisory: this review is a comment and never approves, blocks or requests changes on the pull request.

🔧 Fix with your agent

Paste the following into your coding agent to investigate and fix these issues.

Fix the following code review issues found in pull request #8507.

Before making changes, review each issue and present a brief summary table:
- For each finding, state whether you agree, disagree, or need clarification
- If you disagree (e.g., the fix would require disproportionate changes
  for minimal ROI, or the finding is factually incorrect), explain why
- If you need to clarify something before deciding, ask those questions
- Then share your plan for which issues to tackle and in what order

## WARNING (1)

### [Documentation] `ClusterBucket` doc says keys may hold dot segments, but `bucket.rs:408` refuses keys with a `.` or `..` segment with a 400.
- **File:** src/workerd/server/workerd.capnp:468

📝 Documentation: ⚠️ 1 warning

The ClusterBucket doc still says keys may hold dot segments, which the service refuses.

Recommendation: 📝 Minor Comments

  • [WARNING] ClusterBucket doc says keys may hold dot segments, but bucket.rs:408 refuses keys with a . or .. segment with a 400. (src/workerd/server/workerd.capnp:468)
View full review

workerd.capnp:467-468 says keys may hold /, empty segments and dot segments. server/services/bucket.rs:408 answers any key with a . or .. segment with a 400 InvalidArgument, and r2-test.js expects a/../b to be refused. A user who follows the schema doc and stores such a key gets an error.


✅ All other checks passed

🔒 Security: ✅ LGTM

Checked in this commit:

  • target(): it strips a scheme and host only when the target does not start with /. The query is split off before percent-decoding, so an encoded %3F stays in the key.
  • Dot segments: . and .. segments are refused after percent-decoding (bucket.rs:408), so %2e%2e and %2F encodings are caught too.
  • Local store: object files are named by the MD5 of the key, and upload IDs must be 32 lowercase hex characters before they are joined into uploads/.
  • Limits: part, completion-body and list-page sizes are capped.

📖 AGENTS.md: ✅ LGTM

The src/workerd/server/AGENTS.md updates match the code.


📋 Code Quality: ✅ LGTM

🔄 Since last review: 3 findings resolved, 1 preserved, 0 new.

The author's replies hold up against the code at this commit:

  • r2.js toRange: when the end of a range is too large for a Number to hold exactly, the range is sent open-ended and reads to the end. It can no longer print an end above u64::MAX.
  • bucket.rs target(): the scheme and host are stripped only from an absolute URL. bucket-test.rs covers /a/b://c and /?list&prefix=http://x.
  • bucket.rs fill: it reads into the spare space and zeroes the buffer only when the buffer doubles. The length bookkeeping in next_part, the ?part upload path and the completion body is consistent.

The cluster Range::header now saturates its end, and content_range uses checked_add. Wrangler's r2_buckets handling and the storage service name (<name>-r2:storage) are consistent with the tests.


⚡ Performance: ✅ LGTM

fill now zeroes its buffer only when the buffer doubles, so each byte is zeroed about once.

Rebuilding the local store's index on every list page is a deliberate trade-off. As the author explains, the local store serves workerd serve and tests, and this lets several processes share the directory without a cached index going stale.


Reviewed commit: 581944d7

@ry

ry commented Nov 18, 2026

Copy link
Copy Markdown
Member Author

[AI] @danlapid Agreed on dropping r2.rs and the r2 flag. b97a7a6 goes back to r2.js, bundled like kv.js with the bucket in props, over the bucket service's generic S3-like API; one r2.js Worker serves every bucket.

I'd rather not add the index. The backend is already a bucket, and keeping each R2 object as the store object of its key means S3/GCS tools and self-hosters can read it directly, with no second source of truth and no per-bucket throughput cap. That matters more to me than exact R2 fidelity; the gaps are listed at the top of r2.js.

The local store now has no in-memory index and no directory lock, so wrangler dev and wrangler r2 object put --local can share state. Miniflare side: cloudflare/workers-sdk#17019 (draft).

@danlapid danlapid left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the quick turnaround! With r2.js bundled like kv.js, the bucket in props, and the bucket service as a generic S3-like store, this is the shape I was hoping for. I'm happy with keeping objects 1:1 without an index. Readable by S3/GCS tools and no per-bucket cap are good reasons, and the gaps are documented at the top of r2.js.

Two small things:

  1. Interrupted uploads in the local store. r2.js leaves unfinished multipart uploads for the store's lifecycle rule, but the local store has none, so their parts pile up in .wrangler/state forever. Sweeping uploads/ entries older than a day or so when the store opens would cover it.
  2. The lifecycle rule for self-hosters. A cluster bucket without an "abort incomplete multipart uploads" rule keeps orphaned parts forever. A sentence in the README or the ClusterConfig docs telling operators to set one would save them a surprise bill.

Besides these, LGTM to land. Really nice work 🙂

Per review:
- The local store drops, when it opens, uploads no part has reached in
  a day, with their parts, and stale files in `tmp/`. It has no
  lifecycle rule, so an R2 put cut off mid-stream otherwise kept its
  parts in .wrangler/state forever.
- `ClusterConfig.bucket` and the README tell operators to give a
  cluster's bucket a rule that aborts incomplete multipart uploads.
- The `ClusterBucket` doc no longer says keys may hold dot segments,
  which the service refuses.
@ry

ry commented Nov 19, 2026

Copy link
Copy Markdown
Member Author

[AI] @danlapid Thanks. Both in e26ce06: the local store drops uploads no part has reached in a day (and their parts) when it opens, and ClusterConfig.bucket and the README tell operators to set an abort-incomplete-multipart-uploads lifecycle rule. The ClusterBucket doc also no longer claims keys may hold dot segments (the bot's warning).

@clanky-the-clanker clanky-the-clanker Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Superseded by the review of a478d226.

This review, of e26ce06f

💬 Approved with comments — The ClusterBucket dot-segment doc is now correct. One WARNING is left: the new startup sweep in the local store can make Local::open fail when another process is mid-write in tmp/. There is also one SUGGESTION about files with a future mtime.

Findings: 1 warning · 1 suggestion (2 are inline comments on the diff)

The decision is advisory: this review is a comment and never approves, blocks or requests changes on the pull request.

🔧 Fix with your agent

Paste the following into your coding agent to investigate and fix these issues.

Fix the following code review issues found in pull request #8507.

Before making changes, review each issue and present a brief summary table:
- For each finding, state whether you agree, disagree, or need clarification
- If you disagree (e.g., the fix would require disproportionate changes
  for minimal ROI, or the finding is factually incorrect), explain why
- If you need to clarify something before deciding, ask those questions
- Then share your plan for which issues to tackle and in what order

## WARNING (1)

### [Code Quality] `sweep` propagates NotFound from `file.metadata()` on a `tmp/` entry another process renamed away mid-scan, so `Local::open` fails and the bucket service does not start.
- **File:** src/workerd/server/server/services/bucket/local.rs:197

## SUGGESTION (1)

### [Code Quality] A file whose mtime is in the future makes `elapsed()` return `Err`, so `is_ok_and` treats it as stale and the sweep deletes it, a live upload included.
- **File:** src/workerd/server/server/services/bucket/local.rs:198

📋 Code Quality: ⚠️ 1 warning · 1 suggestion

The new open-time sweep in the local store can fail store startup on a race with another process's write, and it deletes files whose mtime is in the future.

Recommendation: 📝 Minor Comments

  • [WARNING] sweep propagates NotFound from file.metadata() on a tmp/ entry another process renamed away mid-scan, so Local::open fails and the bucket service does not start. (src/workerd/server/server/services/bucket/local.rs:197)
  • [SUGGESTION] A file whose mtime is in the future makes elapsed() return Err, so is_ok_and treats it as stale and the sweep deletes it, a live upload included. (src/workerd/server/server/services/bucket/local.rs:198)
View full review

🔄 Since last review: 1 finding resolved (the ClusterBucket dot-segment doc), 0 preserved, 2 new.

The changes since the last review are the abandoned-upload sweep in local.rs, its test, the r2.js limitations note, and the lifecycle-rule docs in workerd.capnp and the README. The docs agree with the code: STALE is one day, and the ClusterConfig.bucket note describes the parts an unfinished upload leaves behind.

1. The sweep can abort Local::open (local.rs:194-211)

  • sweep runs inside locked, but put_blob and begin write into tmp/ without the lock and then rename the file away.
  • If a second process opens the store while such a write is in flight, read_dir can list the entry before it is renamed away.
  • file.metadata()? then returns NotFound, and the error propagates out of open, so the bucket service fails to start.
  • Since this is cleanup, each per-entry error should be skipped rather than propagated.

2. A future mtime counts as stale (local.rs:198)

  • elapsed() returns Err when the mtime is in the future, and is_ok_and then treats the file as stale and removes it.
  • That covers a live upload too, together with its parts.
  • Separately, a stale tmp/ file is read and JSON-parsed before the dir == "uploads" filter is applied, even though it is deleted either way.

The sweep test covers uploads/ but not the tmp/ branch.


✅ All other checks passed

🔒 Security: ✅ LGTM

The startup sweep runs under the cross-process lock and touches only names it got from read_dir in uploads/ and tmp/. It removes only a stale upload's own part blobs and its own file.

Object files are named by the MD5 of the key, and upload IDs must be 32 lowercase hex characters before they are joined into a path. Keys with . or .. segments are refused after percent-decoding, so no key can reach outside the store root.


📖 AGENTS.md: ✅ LGTM

The src/workerd/server/AGENTS.md updates match the code.


⚡ Performance: ✅ LGTM

The open-time sweep checks each file's age from its metadata before reading it. On a normal open, that is one read_dir plus a stat per file in uploads/ and tmp/.


📝 Documentation: ✅ LGTM

The ClusterBucket doc now says keys may not hold a . or .. segment, which matches bucket.rs and r2-test.js. The lifecycle-rule notes in ClusterConfig.bucket and the README describe what an unfinished upload leaves behind.


Reviewed commit: e26ce06f

@ry
ry enabled auto-merge (squash) November 19, 2026 05:57
A file another process renames away mid-scan no longer fails the
store's open, and a future mtime counts as fresh. Only uploads/
entries are read.
@ry
ry disabled auto-merge November 19, 2026 07:09
@ry
ry enabled auto-merge (squash) November 19, 2026 07:09

@clanky-the-clanker clanky-the-clanker Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💬 Approved with comments — The two earlier sweep findings (NotFound aborting open, future mtime counted as stale) are fixed at this commit. One INFO remains: the tmp/ branch of the sweep is untested.

Findings: 1 info (1 is an inline comment on the diff)

The decision is advisory: this review is a comment and never approves, blocks or requests changes on the pull request.


📋 Code Quality: 💬 1 info

Earlier sweep findings fixed; one INFO about test coverage of the tmp/ sweep.

Recommendation: 💬 Optional Nits

  • [INFO] The sweep test does not exercise the tmp/ branch or an unparseable stale upload file. (src/workerd/server/server/services/bucket/local-test.rs:169)
View full review

The open-time sweep in local.rs:192-214 now skips entries it cannot age, treats a future mtime as fresh, and reads only uploads/ entries. The rest of the change agrees with the documented Store semantics: the cluster client, streaming, SigV4 signing, range saturation, wrangler r2_buckets translation and the fake S3 server.

1. The tmp/ branch of the sweep is untested (local-test.rs:169)

  • The sweep test covers a stale upload and a future-dated upload in uploads/.
  • It does not cover a stale file in tmp/ or an unparseable stale file in uploads/, and those are the paths the latest commit changed.

✅ All other checks passed

🔒 Security: ✅ LGTM

The startup sweep runs under the cross-process lock. It acts only on names returned by read_dir, and it skips entries it cannot age.

Paths built from untrusted input are guarded:

  • Upload IDs must be 32 lowercase hex characters before they are joined into a path.
  • Object files are named by the MD5 of their key.
  • Keys with . or .. segments are refused after percent-decoding.

📖 AGENTS.md: ✅ LGTM

The src/workerd/server/AGENTS.md updates match the code.


⚡ Performance: ✅ LGTM

No performance issues in the changes since the last review.


📝 Documentation: ✅ LGTM

The docs match the code.


Reviewed commit: a478d226

@ry
ry merged commit 0efc13c into main Nov 19, 2026
23 checks passed
@ry
ry deleted the ry/r2 branch November 19, 2026 15:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants