Repository navigation
Export Workers user spans over OTLP - #7686
Conversation
|
All contributors have signed the CLA ✍️ ✅ |
|
I have read the CLA Document and I hereby sign the CLA |
82bb3ec to
9faabe2
Compare
|
The generated output of |
122e1d1 to
f727d85
Compare
f727d85 to
4f06a7e
Compare
There was a problem hiding this comment.
Superseded by the review of
91d0101e.
This review, of 4f06a7ee
🛑 Significant concerns — 1 BLOCKING: UserSpanObserver::getTime() now uses the precise system clock for every user span, including the existing streaming-tail path. Tail workers, which are user code, now receive precise timestamps, and their spans mix a precise start time with a coarse I/O-clock end time. There are also 4 WARNINGs, among them an ungated change to the fetch spans tail workers receive and a Durable Object id tag that is never set.
Findings: 1 blocking · 4 warnings · 1 info · 1 suggestion (4 are inline comments on the diff)
The decision is advisory: this review is a comment and never approves, blocks or requests changes on the pull request.
🔧 Fix with your agent
Paste the following into your coding agent to investigate and fix these issues.
Fix the following code review issues found in pull request #7686.
Before making changes, review each issue and present a brief summary table:
- For each finding, state whether you agree, disagree, or need clarification
- If you disagree (e.g., the fix would require disproportionate changes
for minimal ROI, or the finding is factually incorrect), explain why
- If you need to clarify something before deciding, ask those questions
- Then share your plan for which issues to tackle and in what order
## BLOCKING (1)
### [Security] `UserSpanObserver::getTime()` now uses `systemPreciseCalendarClock()` for every user span, including the existing streaming-tail path. Tail workers (user code) receive precise start and exception timestamps, which gets around the coarse I/O clock. Tail spans also combine a precise start with an I/O-clock end set by `adjustSpanTime()`, so durations can be zero or negative. The change is not gated.
- **File:** src/workerd/io/tracer.c++:838
## WARNING (4)
### [Code Quality] The internal-binding fetch filter drops `fetch` spans for D1, Images, AI and Vectorize hosts from every streaming tail worker, not only when OTLP export is enabled. Because it matches by host only, a user's own fetches to those hosts lose their spans as well.
- **File:** src/workerd/api/http.c++:1922
### [Code Quality] `OtlpServiceInfo.durableObjectId` is never set, so exported Durable Object spans never carry the `cloudflare.durable_object.id` tag that the header documents.
- **File:** src/workerd/server/factory/worker-factory.c++:1128
### [Code Quality] `exportSpans()` starts a new POST task for every batch, with no limit on how many run at once and no timeout. When the collector is slow or down, memory grows without limit and an ERROR is logged for every batch.
- **File:** src/workerd/server/factory/worker-factory-otlp.c++:27
### [Documentation] `docs/async-trace.md` still says user spans use the Spectre-safe clock, but `UserSpanObserver::getTime()` now uses the precise system clock.
- **File:** docs/async-trace.md:231
## INFO (1)
### [Documentation] The new `tracing.otlp` setting is described only in the `workerd.capnp` schema comments. A short guide, such as how to point a local workerd at Jaeger, would help users find it.
## SUGGESTION (1)
### [Code Quality] Rust unit tests are written as inline `mod tests` blocks in `buffer.rs` and `rules.rs`. The repository convention puts them in `buffer-test.rs` and `rules-test.rs` beside the module.
- **File:** src/rust/otlp/buffer.rs:309
📋 Code Quality: ⚠️ 3 warnings · 1 suggestion
An ungated change to tail-visible fetch spans, a Durable Object id tag that is never set, unbounded in-flight OTLP exports, and inline Rust tests.
Recommendation: 📝 Minor Comments
- [WARNING] The internal-binding fetch filter drops
fetchspans for D1, Images, AI and Vectorize hosts from every streaming tail worker, not only when OTLP export is enabled. Because it matches by host only, a user's own fetches to those hosts lose their spans as well. (src/workerd/api/http.c++:1922) - [WARNING]
OtlpServiceInfo.durableObjectIdis never set, so exported Durable Object spans never carry thecloudflare.durable_object.idtag that the header documents. (src/workerd/server/factory/worker-factory.c++:1128) - [WARNING]
exportSpans()starts a new POST task for every batch, with no limit on how many run at once and no timeout. When the collector is slow or down, memory grows without limit and an ERROR is logged for every batch. (src/workerd/server/factory/worker-factory-otlp.c++:27) - [SUGGESTION] Rust unit tests are written as inline
mod testsblocks inbuffer.rsandrules.rs. The repository convention puts them inbuffer-test.rsandrules-test.rsbeside the module. (src/rust/otlp/buffer.rs:309)
View full review
Internal-binding fetch filter applies to everyone (src/workerd/api/http.c++:1616-1629, 1922)
fetchImplNoOutputLockAttempt now passes OmitUserSpan(isInternalBindingFetch(...)) for every worker, whether or not OTLP export is enabled. Every streaming tail worker stops receiving fetch spans for these hosts. The updated D1 and Images instrumentation tests confirm this. Matching is by host only, so a user's own fetch() to js.images.cloudflare.com or to http://d1/ also loses its span. Commit 837539b explains why the filter exists. Still, this is a visible change to tail output with no gate, and it contradicts the description's "nothing changes" claim. Consider gating it on OTLP export being active, or recording the compatibility decision explicitly.
Durable Object id is never exported (src/workerd/server/factory/worker-factory.c++:1128-1131)
The OtlpServiceInfo initializer sets only .serviceName and .entrypoint. The durableObjectId computed at lines 1086-1096 is moved into the Trace, so makeOtlpIdentityTags() never emits cloudflare.durable_object.id, although the doc comment in otlp-span-submitter.h promises it on every span.
Unbounded in-flight exports (src/workerd/server/factory/worker-factory-otlp.c++:25-49)
Each batch adds a TaskSet task that creates a new HTTP client and POSTs up to about 1 MiB. Nothing caps concurrency, and there is no timeout or drop policy. When the collector is slow or hung, tasks and their request bodies pile up for as long as the server runs. taskFailed also logs an ERROR for every failed batch. Consider capping the number of in-flight exports, dropping batches beyond the cap with spansDropped(), adding a timeout, and rate-limiting the log.
Inline Rust tests (src/rust/otlp/buffer.rs:309, src/rust/otlp/rules.rs:84)
The repository convention puts Rust unit tests in <module>-test.rs, never in an inline mod tests. wd_rust_crate already splits *-test.rs files into the test target.
📝 Documentation: ⚠️ 1 warning · 1 info
The timing description in docs/async-trace.md is now out of date, and tracing.otlp is not documented.
Recommendation: 📝 Minor Comments
- [WARNING]
docs/async-trace.mdstill says user spans use the Spectre-safe clock, butUserSpanObserver::getTime()now uses the precise system clock. (docs/async-trace.md:231) - [INFO] The new
tracing.otlpsetting is described only in theworkerd.capnpschema comments. A short guide, such as how to point a local workerd at Jaeger, would help users find it.
View full review
docs/async-trace.md:231-234says user-facing spans use the Spectre-safe clock. With this change,UserSpanObserver::getTime()uses the precise clock. Once the clock question in the security section is settled, update this text to match.- The new
tracing.otlpsetting has no guide indocs/. The schema comments inworkerd.capnpare its only documentation.
🔒 Security: 🛑 1 blocking
The precise clock change also applies to the existing streaming-tail path, so tail workers receive precise timestamps.
Recommendation: 🔴 Serious Concerns
- [BLOCKING]
UserSpanObserver::getTime()now usessystemPreciseCalendarClock()for every user span, including the existing streaming-tail path. Tail workers (user code) receive precise start and exception timestamps, which gets around the coarse I/O clock. Tail spans also combine a precise start with an I/O-clock end set byadjustSpanTime(), so durations can be zero or negative. The change is not gated. (src/workerd/io/tracer.c++:838)
View full review
Precise timestamps reach tail workers, and tail spans mix two clocks (src/workerd/io/tracer.c++:837-840)
UserSpanObserver::getTime() used to return IoContext::current().now(). It now returns kj::systemPreciseCalendarClock().now(). This observer is not specific to OTLP. worker-factory.c++ installs it with a SequentialSpanSubmitter for every streaming tail worker, and the production embedder's tail path uses it too.
SpanBuildertakes its start time fromobserver->getTime()(trace.c++:1965).WorkerTracer::addSpanOpenthen forwards that start time unchanged to the tail stream (tracer.c++:213-234). Exception timestamps (trace.c++:2089) are handled the same way.- A tail worker is user code. A worker can bracket any piece of its own code with
enterSpan()and read nanosecond start times in its tail worker. That gets around the coarse I/O clock that exists to deny Workers a high-resolution timer. It also contradicts the description's statement that precise timestamps "no longer reach user code directly". WorkerTracer::addSpanClosestill callsadjustSpanTime(), which overwritesendTimewithcontext.now()on the I/O clock (tracer.c++:534-560). Tail spans therefore have a precise start and a coarse end. The I/O clock does not advance while JavaScript runs, so the end time can be earlier than the start time. Existing tail consumers would then see zero or negative durations.- This change is not gated. It takes effect whether or not the OTLP hook or
tracing.otlpis configured, which also contradicts "Nothing changes unless an embedder installs the hook or the setting is configured".
Fix: use the precise clock only for observers created by the OTLP export path, for example with a flag or a dedicated observer, and keep IoContext::now() on the tail-stream path.
Apart from this, the untrusted inputs into the Rust encoder are handled safely: UTF-8 is converted lossily, the open-span and batch buffers have size limits, and query strings are redacted by rebuilding url.full.
✅ All other checks passed
📖 AGENTS.md: ✅ LGTM
The addition to src/rust/AGENTS.md is consistent with the rest of that file.
⚡ Performance: ✅ LGTM
The issue of unbounded in-flight OTLP exports is covered in the code quality section. src/rust/otlp/buffer.rs limits open spans to MAX_OPEN_SPANS and limits batches by span count and by bytes.
Reviewed commit: 4f06a7ee
4f06a7e to
91d0101
Compare
There was a problem hiding this comment.
Superseded by the review of
855b0965.
This review, of 91d0101e
🛑 Significant concerns — 1 BLOCKING: UserSpanObserver::getTime() now uses the precise system clock for every user span, including the existing streaming-tail path. Tail workers, which are user code, now receive precise timestamps, and their spans mix a precise start time with a coarse I/O-clock end time. There are also 4 WARNINGs, among them an ungated change to the fetch spans tail workers receive and a Durable Object id tag that is never set.
Findings: 1 blocking · 4 warnings · 1 info · 1 suggestion
The decision is advisory: this review is a comment and never approves, blocks or requests changes on the pull request.
🔧 Fix with your agent
Paste the following into your coding agent to investigate and fix these issues.
Fix the following code review issues found in pull request #7686.
Before making changes, review each issue and present a brief summary table:
- For each finding, state whether you agree, disagree, or need clarification
- If you disagree (e.g., the fix would require disproportionate changes
for minimal ROI, or the finding is factually incorrect), explain why
- If you need to clarify something before deciding, ask those questions
- Then share your plan for which issues to tackle and in what order
## BLOCKING (1)
### [Security] `UserSpanObserver::getTime()` now uses `systemPreciseCalendarClock()` for every user span, including the existing streaming-tail path. Tail workers (user code) receive precise start and exception timestamps, which gets around the coarse I/O clock. Tail spans also combine a precise start with an I/O-clock end set by `adjustSpanTime()`, so durations can be zero or negative. The change is not gated.
- **File:** src/workerd/io/tracer.c++:838
## WARNING (4)
### [Code Quality] The internal-binding fetch filter drops `fetch` spans for D1, Images, AI and Vectorize hosts from every streaming tail worker, not only when OTLP export is enabled. Because it matches by host only, a user's own fetches to those hosts lose their spans as well.
- **File:** src/workerd/api/http.c++:1922
### [Code Quality] `OtlpServiceInfo.durableObjectId` is never set, so exported Durable Object spans never carry the `cloudflare.durable_object.id` tag that the header documents.
- **File:** src/workerd/server/factory/worker-factory.c++:1128
### [Code Quality] `exportSpans()` starts a new POST task for every batch, with no limit on how many run at once and no timeout. When the collector is slow or down, memory grows without limit and an ERROR is logged for every batch.
- **File:** src/workerd/server/factory/worker-factory-otlp.c++:27
### [Documentation] `docs/async-trace.md` still says user spans use the Spectre-safe clock, but `UserSpanObserver::getTime()` now uses the precise system clock.
- **File:** docs/async-trace.md:231
## INFO (1)
### [Documentation] The new `tracing.otlp` setting is described only in the `workerd.capnp` schema comments. A short guide, such as how to point a local workerd at Jaeger, would help users find it.
## SUGGESTION (1)
### [Code Quality] Rust unit tests are written as inline `mod tests` blocks in `buffer.rs` and `rules.rs`. The repository convention puts them in `buffer-test.rs` and `rules-test.rs` beside the module.
- **File:** src/rust/otlp/buffer.rs:309
📋 Code Quality: ⚠️ 3 warnings · 1 suggestion
An ungated change to tail-visible fetch spans, a Durable Object id tag that is never set, unbounded in-flight OTLP exports, and inline Rust tests.
Recommendation: 📝 Minor Comments
- [WARNING] The internal-binding fetch filter drops
fetchspans for D1, Images, AI and Vectorize hosts from every streaming tail worker, not only when OTLP export is enabled. Because it matches by host only, a user's own fetches to those hosts lose their spans as well. (src/workerd/api/http.c++:1922) - [WARNING]
OtlpServiceInfo.durableObjectIdis never set, so exported Durable Object spans never carry thecloudflare.durable_object.idtag that the header documents. (src/workerd/server/factory/worker-factory.c++:1128) - [WARNING]
exportSpans()starts a new POST task for every batch, with no limit on how many run at once and no timeout. When the collector is slow or down, memory grows without limit and an ERROR is logged for every batch. (src/workerd/server/factory/worker-factory-otlp.c++:27) - [SUGGESTION] Rust unit tests are written as inline
mod testsblocks inbuffer.rsandrules.rs. The repository convention puts them inbuffer-test.rsandrules-test.rsbeside the module. (src/rust/otlp/buffer.rs:309)
View full review
Internal-binding fetch filter applies to everyone (src/workerd/api/http.c++:1616-1629, 1922)
fetchImplNoOutputLockAttempt now passes OmitUserSpan(isInternalBindingFetch(...)) for every worker, whether or not OTLP export is enabled. Every streaming tail worker stops receiving fetch spans for these hosts. The updated D1 and Images instrumentation tests confirm this. Matching is by host only, so a user's own fetch() to js.images.cloudflare.com or to http://d1/ also loses its span. Commit 837539b explains why the filter exists. Still, this is a visible change to tail output with no gate, and it contradicts the description's "nothing changes" claim. Consider gating it on OTLP export being active, or recording the compatibility decision explicitly.
Durable Object id is never exported (src/workerd/server/factory/worker-factory.c++:1128-1131)
The OtlpServiceInfo initializer sets only .serviceName and .entrypoint. The durableObjectId computed at lines 1086-1096 is moved into the Trace, so makeOtlpIdentityTags() never emits cloudflare.durable_object.id, although the doc comment in otlp-span-submitter.h promises it on every span.
Unbounded in-flight exports (src/workerd/server/factory/worker-factory-otlp.c++:25-49)
Each batch adds a TaskSet task that creates a new HTTP client and POSTs up to about 1 MiB. Nothing caps concurrency, and there is no timeout or drop policy. When the collector is slow or hung, tasks and their request bodies pile up for as long as the server runs. taskFailed also logs an ERROR for every failed batch. Consider capping the number of in-flight exports, dropping batches beyond the cap with spansDropped(), adding a timeout, and rate-limiting the log.
Inline Rust tests (src/rust/otlp/buffer.rs:309, src/rust/otlp/rules.rs:84)
The repository convention puts Rust unit tests in <module>-test.rs, never in an inline mod tests. wd_rust_crate already splits *-test.rs files into the test target.
📝 Documentation: ⚠️ 1 warning · 1 info
The timing description in docs/async-trace.md is now out of date, and tracing.otlp is not documented.
Recommendation: 📝 Minor Comments
- [WARNING]
docs/async-trace.mdstill says user spans use the Spectre-safe clock, butUserSpanObserver::getTime()now uses the precise system clock. (docs/async-trace.md:231) - [INFO] The new
tracing.otlpsetting is described only in theworkerd.capnpschema comments. A short guide, such as how to point a local workerd at Jaeger, would help users find it.
View full review
docs/async-trace.md:231-234says user-facing spans use the Spectre-safe clock. With this change,UserSpanObserver::getTime()uses the precise clock. Once the clock question in the security section is settled, update this text to match.- The new
tracing.otlpsetting has no guide indocs/. The schema comments inworkerd.capnpare its only documentation.
🔒 Security: 🛑 1 blocking
The precise clock change also applies to the existing streaming-tail path, so tail workers receive precise timestamps.
Recommendation: 🔴 Serious Concerns
- [BLOCKING]
UserSpanObserver::getTime()now usessystemPreciseCalendarClock()for every user span, including the existing streaming-tail path. Tail workers (user code) receive precise start and exception timestamps, which gets around the coarse I/O clock. Tail spans also combine a precise start with an I/O-clock end set byadjustSpanTime(), so durations can be zero or negative. The change is not gated. (src/workerd/io/tracer.c++:838)
View full review
Precise timestamps reach tail workers, and tail spans mix two clocks (src/workerd/io/tracer.c++:837-840)
UserSpanObserver::getTime() used to return IoContext::current().now(). It now returns kj::systemPreciseCalendarClock().now(). This observer is not specific to OTLP. worker-factory.c++ installs it with a SequentialSpanSubmitter for every streaming tail worker, and the production embedder's tail path uses it too.
SpanBuildertakes its start time fromobserver->getTime()(trace.c++:1965).WorkerTracer::addSpanOpenthen forwards that start time unchanged to the tail stream (tracer.c++:213-234). Exception timestamps (trace.c++:2089) are handled the same way.- A tail worker is user code. A worker can bracket any piece of its own code with
enterSpan()and read nanosecond start times in its tail worker. That gets around the coarse I/O clock that exists to deny Workers a high-resolution timer. It also contradicts the description's statement that precise timestamps "no longer reach user code directly". WorkerTracer::addSpanClosestill callsadjustSpanTime(), which overwritesendTimewithcontext.now()on the I/O clock (tracer.c++:534-560). Tail spans therefore have a precise start and a coarse end. The I/O clock does not advance while JavaScript runs, so the end time can be earlier than the start time. Existing tail consumers would then see zero or negative durations.- This change is not gated. It takes effect whether or not the OTLP hook or
tracing.otlpis configured, which also contradicts "Nothing changes unless an embedder installs the hook or the setting is configured".
Fix: use the precise clock only for observers created by the OTLP export path, for example with a flag or a dedicated observer, and keep IoContext::now() on the tail-stream path.
Apart from this, the untrusted inputs into the Rust encoder are handled safely: UTF-8 is converted lossily, the open-span and batch buffers have size limits, and query strings are redacted by rebuilding url.full.
✅ All other checks passed
📖 AGENTS.md: ✅ LGTM
The addition to src/rust/AGENTS.md is consistent with the rest of that file.
⚡ Performance: ✅ LGTM
The issue of unbounded in-flight OTLP exports is covered in the code quality section. src/rust/otlp/buffer.rs limits open spans to MAX_OPEN_SPANS and limits batches by span count and by bytes.
Reviewed commit: 91d0101e — same changes as 4f06a7ee, so this is that commit's review (no new review was run; line numbers are as of 4f06a7ee)
There was a problem hiding this comment.
Superseded by the review of
27fb47ae.
This review, of 855b0965
🛑 Significant concerns — 1 BLOCKING, not yet fixed: UserSpanObserver::getTime() uses the precise system clock for every user span. Streaming tail workers, which are user code, therefore receive precise timestamps, and their spans mix a precise start with a coarse I/O-clock end. There are also 3 code-quality WARNINGs (an ungated tail-span filter, a Durable Object id that is never exported, unbounded OTLP exports) and a stale clock statement in the docs.
Findings: 1 blocking · 4 warnings · 2 info · 1 suggestion
Not opened by the correctness review: 1 changed file it was meant to read
src/workerd/server/server/config.rs
The decision is advisory: this review is a comment and never approves, blocks or requests changes on the pull request.
🔧 Fix with your agent
Paste the following into your coding agent to investigate and fix these issues.
Fix the following code review issues found in pull request #7686.
Before making changes, review each issue and present a brief summary table:
- For each finding, state whether you agree, disagree, or need clarification
- If you disagree (e.g., the fix would require disproportionate changes
for minimal ROI, or the finding is factually incorrect), explain why
- If you need to clarify something before deciding, ask those questions
- Then share your plan for which issues to tackle and in what order
## BLOCKING (1)
### [Security] `UserSpanObserver::getTime()` uses `systemPreciseCalendarClock()` for every user span, including the streaming-tail path. Tail workers (user code) receive precise start and exception timestamps, which gets around the coarse I/O clock. Tail spans also combine a precise start with an I/O-clock end, so durations can be zero or negative. The change is not gated on OTLP export.
- **File:** src/workerd/io/tracer.c++:838
## WARNING (4)
### [Code Quality] The internal-binding fetch filter drops `fetch` spans for D1, Images, AI and Vectorize hosts from every streaming tail worker, not only when OTLP export is enabled. Because it matches by host only, a user's own fetches to those hosts lose their spans too.
- **File:** src/workerd/api/http.c++:1922
### [Code Quality] `OtlpServiceInfo.durableObjectId` is never set, so exported Durable Object spans never carry the `cloudflare.durable_object.id` tag that the header documents.
- **File:** src/workerd/server/factory/worker-factory.c++:1129
### [Code Quality] `exportSpans()` starts a new POST task for every batch, with no limit on how many run at once and no timeout. When the collector is slow or down, memory grows without bound and an ERROR is logged for every batch.
- **File:** src/workerd/server/factory/worker-factory-otlp.c++:27
### [Documentation] `docs/async-trace.md` still says user spans use the Spectre-safe clock, but `UserSpanObserver::getTime()` now uses the precise system clock.
- **File:** docs/async-trace.md:231
## INFO (2)
### [Documentation] `docs/async-trace.md:255` describes spans as always going to tail workers. When `tracing.otlp` is set, they are not delivered to streaming tail workers.
- **File:** docs/async-trace.md:255
### [Documentation] The new `tracing.otlp` setting is described only in the `workerd.capnp` schema comments. A short user guide, for example a local Jaeger setup, would help users find it.
## SUGGESTION (1)
### [Code Quality] The Rust unit tests in `buffer.rs` and `rules.rs` are inline `mod tests` blocks. The repository convention puts them in `buffer-test.rs` and `rules-test.rs` beside each module.
- **File:** src/rust/otlp/buffer.rs:309
📋 Code Quality: ⚠️ 3 warnings · 1 suggestion
All earlier findings are still present: an ungated internal-binding fetch filter, a Durable Object id that is never exported, unbounded OTLP exports, and inline Rust tests.
Recommendation: 📝 Minor Comments
- [WARNING] The internal-binding fetch filter drops
fetchspans for D1, Images, AI and Vectorize hosts from every streaming tail worker, not only when OTLP export is enabled. Because it matches by host only, a user's own fetches to those hosts lose their spans too. (src/workerd/api/http.c++:1922) - [WARNING]
OtlpServiceInfo.durableObjectIdis never set, so exported Durable Object spans never carry thecloudflare.durable_object.idtag that the header documents. (src/workerd/server/factory/worker-factory.c++:1129) - [WARNING]
exportSpans()starts a new POST task for every batch, with no limit on how many run at once and no timeout. When the collector is slow or down, memory grows without bound and an ERROR is logged for every batch. (src/workerd/server/factory/worker-factory-otlp.c++:27) - [SUGGESTION] The Rust unit tests in
buffer.rsandrules.rsare inlinemod testsblocks. The repository convention puts them inbuffer-test.rsandrules-test.rsbeside each module. (src/rust/otlp/buffer.rs:309)
View full review
Internal-binding fetch filter applies to every worker (src/workerd/api/http.c++:1616-1629, 1922)
OmitUserSpan(isInternalBindingFetch(...)) applies to every worker, whether or not OTLP export is enabled. Every streaming tail worker stops getting fetch spans for these hosts; the D1 and Images instrumentation tests were updated to match. The function's own comment says it also drops the span of a user's own fetch to one of these hosts. Commit bcceb70 calls the filter a stopgap. Even so, this is a visible change to tail output, and the repository asks for that compatibility decision to be recorded or put behind a gate. Consider applying it only when OTLP export is active.
Durable Object id is never exported (src/workerd/server/factory/worker-factory.c++:1128-1131)
The OtlpServiceInfo initializer sets only .serviceName and .entrypoint. The durableObjectId computed earlier in that function goes only to the tracer. As a result, cloudflare.durable_object.id, which otlp-span-submitter.h documents, is never emitted.
Unbounded in-flight exports (src/workerd/server/factory/worker-factory-otlp.c++:25-32)
Each batch adds a TaskSet task that opens a new client and POSTs up to about 1 MiB. Nothing limits how many run at once, and there is no timeout. When the collector is slow or hung, tasks and their request bodies pile up without bound, and each failure logs an ERROR. Add a cap on in-flight requests that drops batches beyond it, plus a timeout.
Inline Rust tests (src/rust/otlp/buffer.rs:309, src/rust/otlp/rules.rs:84)
The repository convention (src/rust/AGENTS.md) is a <module>-test.rs file beside each module, not an inline mod tests block.
📝 Documentation: ⚠️ 1 warning · 2 info
The docs still describe user spans as using the Spectre-safe clock, the export-path description leaves out the OTLP mode, and tracing.otlp has no guide.
Recommendation: 📝 Minor Comments
- [WARNING]
docs/async-trace.mdstill says user spans use the Spectre-safe clock, butUserSpanObserver::getTime()now uses the precise system clock. (docs/async-trace.md:231) - [INFO]
docs/async-trace.md:255describes spans as always going to tail workers. Whentracing.otlpis set, they are not delivered to streaming tail workers. (docs/async-trace.md:255) - [INFO] The new
tracing.otlpsetting is described only in theworkerd.capnpschema comments. A short user guide, for example a local Jaeger setup, would help users find it.
View full review
docs/async-trace.md:231-234says user-facing spans use the Spectre-safe clock.UserSpanObserver::getTime()now uses the precise clock. Update this text once the clock question in the security section is settled.docs/async-trace.md:255-256says spans go to tail workers and trace export. Whentracing.otlpis set, spans are not delivered to streaming tail workers (workerd.capnp:181-182). Consider qualifying the sentence.tracing.otlpis documented only in theworkerd.capnpschema comments. A short guide indocs/, such as pointing a local workerd at Jaeger, would help users find it.
🔒 Security: 🛑 1 blocking
Not yet fixed: precise timestamps reach streaming tail workers, and tail spans mix two clocks.
Recommendation: 🔴 Serious Concerns
- [BLOCKING]
UserSpanObserver::getTime()usessystemPreciseCalendarClock()for every user span, including the streaming-tail path. Tail workers (user code) receive precise start and exception timestamps, which gets around the coarse I/O clock. Tail spans also combine a precise start with an I/O-clock end, so durations can be zero or negative. The change is not gated on OTLP export. (src/workerd/io/tracer.c++:838)
View full review
🔄 Since last review: 0 findings resolved, all preserved. The only file changed since then is src/workerd/server/server/config.rs, which moves the OTLP setup into OtlpTracing::from_config. That change raises no new concerns.
Precise timestamps reach tail workers, and tail spans mix two clocks (src/workerd/io/tracer.c++:837-840)
UserSpanObserver::getTime()returnskj::systemPreciseCalendarClock().now(). This observer is not specific to OTLP: streaming tail workers use it too.WorkerTracer::addSpanOpen(tracer.c++:213-234) passesstartTimetotailStreamWriter->report(...)unchanged. A worker can wrap any code inenterSpan()and read nanosecond start times in its tail worker. That gets around the coarse I/O clock, which exists to keep high-resolution timers away from Workers. It also contradicts the description's statement that precise timestamps "no longer reach user code directly".addSpanClosestill callsadjustSpanTime(). That function overwritesendTimewithcontext.now()from the I/O clock (tracer.c++:545-547). A tail span therefore gets a precise start and a coarse end. The I/O clock does not advance while JavaScript runs, so the end can come before the start, giving zero or negative durations.- The change is not gated on OTLP export.
Fix: use the precise clock only for spans that go to the OTLP submitter (for example, let each submitter or observer choose its clock). Otherwise, coarsen timestamps before they reach the tail stream. Either way, start, end and exception times of one span should come from the same clock.
✅ All other checks passed
📖 AGENTS.md: ✅ LGTM
The new line in src/rust/AGENTS.md matches the rest of the guidance. The inline Rust tests that go against its test-file convention are reported in the code quality section.
⚡ Performance: ✅ LGTM
src/rust/otlp/buffer.rs caps open spans at 1024 and batches at 512 spans or 1 MiB, and tracks the encoded size as spans are added. The unbounded concurrency of OTLP POSTs is reported in the code quality section.
Reviewed commit: 855b0965
There was a problem hiding this comment.
Superseded by the review of
5708482f.
This review, of 27fb47ae
Findings: 4 warnings · 2 info (1 is an inline comment on the diff)
Not opened by the correctness review: 3 changed files it was meant to read
src/workerd/io/BUILD.bazelsrc/workerd/io/tracer.hsrc/workerd/server/server/config.rs
The decision is advisory: this review is a comment and never approves, blocks or requests changes on the pull request.
🔧 Fix with your agent
Paste the following into your coding agent to investigate and fix these issues.
Fix the following code review issues found in pull request #7686.
Before making changes, review each issue and present a brief summary table:
- For each finding, state whether you agree, disagree, or need clarification
- If you disagree (e.g., the fix would require disproportionate changes
for minimal ROI, or the finding is factually incorrect), explain why
- If you need to clarify something before deciding, ask those questions
- Then share your plan for which issues to tackle and in what order
## WARNING (4)
### [Security] When the embedder's OTLP invocation-span hook is installed and the worker has a streaming tail worker, a span exception's timestamp comes from the precise OTLP observer and is forwarded unchanged to the tail stream. The tail worker (user code) therefore sees a precise timestamp.
- **File:** src/workerd/io/tracer.c++:353
### [Code Quality] The internal-binding fetch filter drops `fetch` spans for the D1, AI, Images, Vectorize, Workflow and test hosts from every streaming tail worker, not only when OTLP export is on. Because it matches by host only, a user's own fetches to those hosts lose their spans too.
- **File:** src/workerd/api/http.c++:1922
### [Code Quality] `OtlpServiceInfo.durableObjectId` is never set, so exported Durable Object spans never carry `cloudflare.durable_object.id`.
- **File:** src/workerd/server/factory/worker-factory.c++:1129
### [Code Quality] `exportSpans()` starts a new POST task for every batch, with no limit on how many run at once and no timeout. A slow or hung collector makes memory grow without bound.
- **File:** src/workerd/server/factory/worker-factory-otlp.c++:27
## INFO (2)
### [Documentation] The Timing section should say that spans exported through native OTLP use precise timestamps, while streaming-tail spans keep the Spectre-safe I/O clock.
- **File:** docs/async-trace.md:231
### [Documentation] The Outputs section implies spans always go to tail workers. With `tracing.otlp` configured, they are not delivered to streaming tail workers.
- **File:** docs/async-trace.md:255
📋 Code Quality: ⚠️ 3 warnings
Three earlier warnings are still present: the binding fetch filter applies to every worker, the Durable Object id is never exported, and OTLP exports have no cap and no timeout.
Recommendation: 📝 Minor Comments
- [WARNING] The internal-binding fetch filter drops
fetchspans for the D1, AI, Images, Vectorize, Workflow and test hosts from every streaming tail worker, not only when OTLP export is on. Because it matches by host only, a user's own fetches to those hosts lose their spans too. (src/workerd/api/http.c++:1922) - [WARNING]
OtlpServiceInfo.durableObjectIdis never set, so exported Durable Object spans never carrycloudflare.durable_object.id. (src/workerd/server/factory/worker-factory.c++:1129) - [WARNING]
exportSpans()starts a new POST task for every batch, with no limit on how many run at once and no timeout. A slow or hung collector makes memory grow without bound. (src/workerd/server/factory/worker-factory-otlp.c++:27)
View full review
Internal-binding fetch filter applies to every worker (src/workerd/api/http.c++:1922). OmitUserSpan(isInternalBindingFetch(...)) runs whether or not OTLP export is on. Every streaming tail worker loses the fetch spans for these hosts. Because the filter matches by host only, a user's own fetch() to one of them (for example the bare hosts d1 or vector-search) loses its span too. The D1 and Images instrumentation tests were edited to match, so this is a visible change to tail output. The repository's compatibility rules ask for that decision to be recorded or put behind a flag. Consider applying the filter only when OTLP export is active, or only to the binding's own internal fetcher.
Durable Object id is never exported (src/workerd/server/factory/worker-factory.c++:1129-1132). The OtlpServiceInfo initializer sets only .serviceName and .entrypoint. The durableObjectId computed at line 1086 is moved into the tracer at line 1100. As a result, cloudflare.durable_object.id, which otlp-span-submitter.h:67 declares, is never emitted.
Unbounded in-flight exports (src/workerd/server/factory/worker-factory-otlp.c++:27-32). Every batch adds a TaskSet task that POSTs a body of up to about 1 MiB. Nothing limits how many run at once, and there is no timeout. If the collector is slow or hung, tasks and their bodies pile up without bound, and each failure logs an ERROR. Cap the number of in-flight requests, drop batches beyond the cap (reporting them as dropped spans), and add a timeout.
One earlier note is still open and unchanged: the inline Rust tests in buffer.rs and rules.rs.
📝 Documentation: 💬 2 info
Two passages in docs/async-trace.md should mention native OTLP export.
Recommendation: 💬 Optional Nits
- [INFO] The Timing section should say that spans exported through native OTLP use precise timestamps, while streaming-tail spans keep the Spectre-safe I/O clock. (
docs/async-trace.md:231) - [INFO] The Outputs section implies spans always go to tail workers. With
tracing.otlpconfigured, they are not delivered to streaming tail workers. (docs/async-trace.md:255)
View full review
docs/async-trace.md:231-234: streaming-tail spans use the Spectre-safe clock again, so the text is accurate for spans user code can see. Spans exported through native OTLP use precise timestamps, though, and the passage should say so.docs/async-trace.md:255-256: whentracing.otlpis set, spans are not delivered to streaming tail workers (workerd.capnp:181-182). Consider qualifying this sentence.
One earlier note is still open: a short guide to the tracing.otlp setting.
🔒 Security: ⚠️ 1 warning
The precise-clock blocker is fixed. One path remains: span exception timestamps can still reach tail workers at full precision.
Recommendation: 📝 Minor Comments
- [WARNING] When the embedder's OTLP invocation-span hook is installed and the worker has a streaming tail worker, a span exception's timestamp comes from the precise OTLP observer and is forwarded unchanged to the tail stream. The tail worker (user code) therefore sees a precise timestamp. (
src/workerd/io/tracer.c++:353)
View full review
🔄 Since last review: 1 finding resolved, 4 preserved, 1 new
Earlier BLOCKING is resolved. SpanSubmitter::getTime() now defaults to IoContext::current().now() (src/workerd/io/tracer.c++:779). UserSpanObserver::getTime() delegates to its submitter (tracer.c++:841-842). Only OtlpSpanSubmitter returns the precise system clock. Streaming-tail spans therefore get I/O-clock times at both ends again, and the mixed precise-start/coarse-end problem is gone.
Remaining leak: span exception timestamps. recordExceptionImpl (src/workerd/api/tracing.c++:301) takes its timestamp from the parent span's observer. When the invocation's root span comes from the embedder's OTLP hook, that observer is precise. WorkerTracer::addSpanException then hands the timestamp unchanged to reportExceptionToTailStream (tracer.c++:353). This happens whenever the hook is installed and the worker also has a streaming tail worker: the tail worker receives a nanosecond-precision timestamp for each span exception. That contradicts the stated goal that precise timestamps no longer reach user code. Fix: send getTime() (the I/O clock) to the tail stream and keep the precise value only for the OTLP submitter. Also add a test with an OTLP-backed root and a tail writer.
✅ All other checks passed
📖 AGENTS.md: ✅ LGTM
The new line in src/rust/AGENTS.md fits the existing guidance.
⚡ Performance: ✅ LGTM
The Rust SpanBuffer is bounded: at most 1024 open spans, with batches cut at 512 spans or 1 MiB, and it tracks the encoded size as spans are added. finish() sorts at most 1024 spans once per invocation.
Reviewed commit: 27fb47ae
There was a problem hiding this comment.
Superseded by the review of
78bb0518.
This review, of 5708482f
💬 Approved with comments — The earlier precise-timestamp leak to tail workers is fixed. Two WARNINGs remain in the opt-in workerd OTLP exporter (Durable Object id never set; exports have no concurrency cap and no timeout), plus INFO notes on the binding fetch filter and the docs.
Findings: 2 warnings · 4 info (2 are inline comments on the diff)
The decision is advisory: this review is a comment and never approves, blocks or requests changes on the pull request.
🔧 Fix with your agent
Paste the following into your coding agent to investigate and fix these issues.
Fix the following code review issues found in pull request #7686.
Before making changes, review each issue and present a brief summary table:
- For each finding, state whether you agree, disagree, or need clarification
- If you disagree (e.g., the fix would require disproportionate changes
for minimal ROI, or the finding is factually incorrect), explain why
- If you need to clarify something before deciding, ask those questions
- Then share your plan for which issues to tackle and in what order
## WARNING (2)
### [Code Quality] `OtlpServiceInfo.durableObjectId` is never set, so spans exported for Durable Objects never carry `cloudflare.durable_object.id`.
- **File:** src/workerd/server/factory/worker-factory.c++:1129
### [Code Quality] `exportSpans()` starts one POST task per batch with no cap on how many run at once and no timeout. A slow or hung collector makes memory grow without bound.
- **File:** src/workerd/server/factory/worker-factory-otlp.c++:27
## INFO (4)
### [Code Quality] The internal-binding fetch filter drops `fetch` spans for these hosts from every streaming tail worker and matches on host alone. Lowered to INFO: @wperron [non-author] says this is the current and expected behavior. Consider recording the compatibility decision in the commit message.
- **File:** src/workerd/api/http.c++:1922
### [Documentation] The Timing section should say that spans exported through native OTLP use precise timestamps.
- **File:** docs/async-trace.md:231
### [Documentation] The Outputs section should say that spans are not delivered to streaming tail workers when `tracing.otlp` is configured.
- **File:** docs/async-trace.md:255
### [Documentation] No Markdown guide explains how to configure the new `tracing.otlp` setting.
📋 Code Quality: ⚠️ 2 warnings · 1 info
Two earlier warnings in the workerd OTLP exporter are still present; the binding fetch filter is now an INFO.
Recommendation: 📝 Minor Comments
- [WARNING]
OtlpServiceInfo.durableObjectIdis never set, so spans exported for Durable Objects never carrycloudflare.durable_object.id. (src/workerd/server/factory/worker-factory.c++:1129) - [WARNING]
exportSpans()starts one POST task per batch with no cap on how many run at once and no timeout. A slow or hung collector makes memory grow without bound. (src/workerd/server/factory/worker-factory-otlp.c++:27) - [INFO] The internal-binding fetch filter drops
fetchspans for these hosts from every streaming tail worker and matches on host alone. Lowered to INFO: @wperron [non-author] says this is the current and expected behavior. Consider recording the compatibility decision in the commit message. (src/workerd/api/http.c++:1922)
View full review
Durable Object id is never exported (src/workerd/server/factory/worker-factory.c++:1129-1132). durableObjectId is computed at line 1086 and moved into the Trace at line 1100. The OtlpServiceInfo initializer sets only .serviceName and .entrypoint, so makeOtlpIdentityTags (otlp-span-submitter.c++:150) never emits cloudflare.durable_object.id.
Unbounded in-flight exports (src/workerd/server/factory/worker-factory-otlp.c++:25-30). Each batch adds a TaskSet task that POSTs a body of up to about 1 MiB. Nothing caps how many run at once, and postOtlpTraces sets no timeout. With a slow or hung collector, tasks and request bodies pile up until shutdown.
Internal-binding fetch filter (src/workerd/api/http.c++:1922). The filter runs for every worker and matches on host alone. @wperron [non-author] replied that this is the current and expected behavior, so it is noted here as INFO. The tail output change, which the D1 and Images instrumentation tests now reflect, is worth recording in the commit message as an accepted compatibility decision.
📝 Documentation: 💬 3 info
docs/async-trace.md should cover native OTLP timing, tail delivery, and the tracing.otlp setting.
Recommendation: 💬 Optional Nits
- [INFO] The Timing section should say that spans exported through native OTLP use precise timestamps. (
docs/async-trace.md:231) - [INFO] The Outputs section should say that spans are not delivered to streaming tail workers when
tracing.otlpis configured. (docs/async-trace.md:255) - [INFO] No Markdown guide explains how to configure the new
tracing.otlpsetting.
View full review
docs/async-trace.md:231-234: the passage is accurate for spans delivered to tail workers. Spans exported through native OTLP usesystemPreciseCalendarClock()(otlp-span-submitter.c++:204-205), and the passage should say so.docs/async-trace.md:255-256: whentracing.otlpis set, spans are not delivered to streaming tail workers. Qualify this sentence.- There is still no short guide to configuring
tracing.otlp(collector service, sampling, tail interaction).
✅ All other checks passed
🔒 Security: ✅ LGTM
recordExceptionImpl (src/workerd/api/tracing.c++:300-303) now passes the I/O-clock time from IoContext::current().now() to addSpanException, and that is what goes to the tail stream. The native OTLP root records its own precise timestamp separately. Tail workers therefore no longer receive precise exception timestamps.
📖 AGENTS.md: ✅ LGTM
The new line in src/rust/AGENTS.md fits the existing guidance.
⚡ Performance: ✅ LGTM
No performance issues found. The unbounded in-flight export concern is under Code Quality.
Reviewed commit: 5708482f
5708482 to
6f1f0e2
Compare
There was a problem hiding this comment.
Superseded by the review of
b33dc58b.
This review, of 78bb0518
💬 Approved with comments — Two WARNINGs remain in the opt-in workerd OTLP exporter: the Durable Object id is never exported, and exports have no concurrency cap and no timeout. There are also INFO notes on the binding fetch filter and the docs, and no BLOCKING issues.
Findings: 2 warnings · 1 info
The decision is advisory: this review is a comment and never approves, blocks or requests changes on the pull request.
🔧 Fix with your agent
Paste the following into your coding agent to investigate and fix these issues.
Fix the following code review issues found in pull request #7686.
Before making changes, review each issue and present a brief summary table:
- For each finding, state whether you agree, disagree, or need clarification
- If you disagree (e.g., the fix would require disproportionate changes
for minimal ROI, or the finding is factually incorrect), explain why
- If you need to clarify something before deciding, ask those questions
- Then share your plan for which issues to tackle and in what order
## WARNING (2)
### [Code Quality] `OtlpServiceInfo.durableObjectId` is never set, so exported Durable Object spans never carry `cloudflare.durable_object.id`.
- **File:** src/workerd/server/factory/worker-factory.c++:1129
### [Code Quality] `exportSpans()` starts one POST task per batch with no concurrency cap and no timeout, so a slow or hung collector makes memory grow without bound.
- **File:** src/workerd/server/factory/worker-factory-otlp.c++:27
## INFO (1)
### [Code Quality] The internal-binding fetch filter drops `fetch` spans for D1, Images, AI and Vectorize hosts from every streaming tail worker, matching on host alone. Kept as INFO because @wperron [non-author] says this is the current and expected behavior. Consider recording the compatibility decision in the commit message.
- **File:** src/workerd/api/http.c++:1922
📋 Code Quality: ⚠️ 2 warnings · 1 info
Both earlier WARNINGs are still present in the OTLP exporter, and the fetch-filter INFO stays open.
Recommendation: 📝 Minor Comments
- [WARNING]
OtlpServiceInfo.durableObjectIdis never set, so exported Durable Object spans never carrycloudflare.durable_object.id. (src/workerd/server/factory/worker-factory.c++:1129) - [WARNING]
exportSpans()starts one POST task per batch with no concurrency cap and no timeout, so a slow or hung collector makes memory grow without bound. (src/workerd/server/factory/worker-factory-otlp.c++:27) - [INFO] The internal-binding fetch filter drops
fetchspans for D1, Images, AI and Vectorize hosts from every streaming tail worker, matching on host alone. Kept as INFO because @wperron [non-author] says this is the current and expected behavior. Consider recording the compatibility decision in the commit message. (src/workerd/api/http.c++:1922)
View full review
Durable Object id is never exported (src/workerd/server/factory/worker-factory.c++:1129-1132)
durableObjectIdis computed earlier in the function and moved into theTrace.- The
OtlpServiceInfoinitializer sets only.serviceNameand.entrypoint, somakeOtlpIdentityTagsnever emitscloudflare.durable_object.id.
Unbounded in-flight exports (src/workerd/server/factory/worker-factory-otlp.c++:25-30)
- Each batch adds a
TaskSettask that POSTs a body of up to about 1 MiB. - Nothing limits how many tasks run at once, and the POST has no timeout.
- With a stalled collector, tasks and their request bodies pile up until shutdown.
- The new server test checks cancellation at shutdown. It does not check how much builds up while the server is running.
Internal-binding fetch filter (src/workerd/api/http.c++:1922)
- The filter matches on host only and applies to every streaming tail worker.
- It stays INFO because @wperron says this is the current and expected behavior.
- The commit message should record it as an accepted compatibility decision.
✅ All other checks passed
🔒 Security: ✅ LGTM
- Tail workers still get timestamps from the request's I/O clock (
src/workerd/api/tracing.c++:300-303). Only the native OTLP path uses the precise clock. - The collector is chosen by the operator's config and reached through a service channel at a fixed path, so user code controls neither the destination nor the path.
📖 AGENTS.md: ✅ LGTM
The new line in src/rust/AGENTS.md fits the existing guidance.
⚡ Performance: ✅ LGTM
No performance issues found. The unbounded in-flight export concern is covered under Code Quality.
📝 Documentation: ✅ LGTM
docs/async-trace.md:255-256: this passage says spans go to tail workers, but whentracing.otlpis configured, spans are not delivered to streaming tail workers. Qualify the statement.- No Markdown guide explains how to configure
tracing.otlp: the collector service, sampling, and how it interacts with tail workers.
Reviewed commit: 78bb0518
There was a problem hiding this comment.
Superseded by the review of
f301dcee.
This review, of b33dc58b
💬 Approved with comments — Both earlier WARNINGs are fixed: Durable Object IDs are now exported, and standalone exports are capped in concurrency and queue size and have a timeout. What remains is one INFO and one SUGGESTION on the exporter, and nothing blocks.
Findings: 1 info · 1 suggestion (2 are inline comments on the diff)
The decision is advisory: this review is a comment and never approves, blocks or requests changes on the pull request.
📋 Code Quality: 💬 1 suggestion
Both earlier warnings are fixed. One optional note remains: failed exports are logged at ERROR on every failure, with no throttling.
Recommendation: 💬 Optional Nits
- [SUGGESTION] Every failed OTLP batch is logged at ERROR with no throttling (
src/workerd/server/factory/worker-factory-otlp.c++:100)
View full review
🔄 Since last review: 2 findings resolved, 0 preserved, 1 new
Earlier findings, checked against this commit:
- Durable Object ID: fixed.
durableObjectIdis now passed asOtlpServiceInfo.durableObjectIdand emitted bymakeOtlpIdentityTags. The test checks both that the ID is present and that it is absent where it should be. - Unbounded exports: fixed.
worker-factory-otlp.c++now allows at most 16 POSTs at once, queues at most 4096 messages, batches up to about 1 MiB and times each request out after 10 s.unlink()cancels queued work at shutdown, andserver-test.c++covers each of these limits. - Internal-binding fetch filter (
http.c++): unchanged, and @wperron says this is the current and expected behaviour. One earlier note on it remains open.
New note:
taskFailedlogs every failed batch at ERROR with no throttling. Iftracing.otlppoints at a collector that isn't running, every invocation adds a log line. The queue-full path already usesLOG_WARNING_PERIODICALLY.
⚡ Performance: 💬 1 info
The export queue is limited by message count but not by total bytes.
Recommendation: 💬 Optional Nits
- [INFO] The export queue is limited by message count (4096) but not by total bytes (
src/workerd/server/factory/worker-factory-otlp.c++:47)
View full review
The earlier concern about unbounded in-flight exports is resolved.
Queue limit counts messages, not bytes
QUEUE_SIZElimits the number of queued messages, not their size.- If the collector is slow or down and messages are large, the queue can hold far more memory than the 4096 limit suggests.
- Typical messages are small, so this is unlikely to matter in practice.
- A cheap guard: track the bytes pending and drop new spans once they pass a limit.
✅ All other checks passed
🔒 Security: ✅ LGTM
- Collector destination: it comes from the operator's config and is reached through a service channel at the fixed path
/v1/traces. User code controls neither the destination nor the path. - Timestamps: only the native OTLP submitter uses the precise clock. Tail workers keep the request's coarsened I/O clock.
- Resource use: the exporter caps in-flight requests, queue length and request duration, and it drops overflow.
- Shutdown:
unlink()disables later exports before it cancels pending work.
📖 AGENTS.md: ✅ LGTM
The new line in src/rust/AGENTS.md is consistent with the guidance already there.
📝 Documentation: ✅ LGTM
No new documentation issues. The two documentation notes from earlier reviews (that tracing.otlp changes delivery to streaming tail workers, and that no guide covers the new setting) are still open, and they are the author's call.
Reviewed commit: b33dc58b
b33dc58 to
f301dce
Compare
There was a problem hiding this comment.
Superseded by the review of
71fec26e.
This review, of 7f6a7935
💬 Approved with comments — One WARNING is still open: with ExportSpans::NO and a streaming tail, a user span's updates and exceptions change the exported invocation span. It only affects embedders that use that mode.
Findings: 1 warning
Not opened by the correctness review: 5 changed files it was meant to read
src/opentelemetry/proto/collector/logs/v1/logs_service.protosrc/opentelemetry/proto/collector/trace/v1/trace_service.protosrc/opentelemetry/proto/common/v1/common.protosrc/opentelemetry/proto/resource/v1/resource.protosrc/opentelemetry/proto/trace/v1/trace.proto
The decision is advisory: this review is a comment and never approves, blocks or requests changes on the pull request.
🔧 Fix with your agent
Paste the following into your coding agent to investigate and fix these issues.
Fix the following code review issues found in pull request #7686.
Before making changes, review each issue and present a brief summary table:
- For each finding, state whether you agree, disagree, or need clarification
- If you disagree (e.g., the fix would require disproportionate changes
for minimal ROI, or the finding is factually incorrect), explain why
- If you need to clarify something before deciding, ask those questions
- Then share your plan for which issues to tackle and in what order
## WARNING (1)
### [Code Quality] With ExportSpans::NO and a streaming tail, user span renames, status changes and exceptions are recorded on the exported invocation span, because addSpanUpdate and addSpanException never check the span id.
- **File:** src/workerd/io/tracer.c++:282
📋 Code Quality: ⚠️ 1 warning
One warning is still open: user span updates and exceptions are recorded on the exported invocation span when ExportSpans::NO is used.
Recommendation: 📝 Minor Comments
- [WARNING] With ExportSpans::NO and a streaming tail, user span renames, status changes and exceptions are recorded on the exported invocation span, because addSpanUpdate and addSpanException never check the span id. (
src/workerd/io/tracer.c++:282)
View full review
Open: WorkerTracer::addSpanUpdate and addSpanException apply every span id to the invocation span
src/workerd/io/tracer.c++:282callsupdateInvocationSpan(update)without checkingspanId.src/workerd/io/tracer.c++:370callsrecordInvocationSpanException(...)without checking it either.- With
ExportSpans::NO, user spans go throughSequentialSpanSubmitter(worker-factory.c++). It forwards each user span's update and exception with that user span's own id. - So a user span's
updateName()renames the exported invocation span, itssetStatus()overwrites that span's status, and its exceptions are added to that span. recordInvocationLevelExceptionalready compares the id withSpanParent(invocationSpan).getSpanId(). Both of these paths need the same check.- The NO-mode tests in
tracer-test.c++would catch a regression here if they also sent a user-span update and a user-span exception.
✅ All other checks passed
🔒 Security: ✅ LGTM
- Timestamps: Exception timestamps in the streaming tail still come from the I/O clock. Only the native OTLP span records a precise time.
- Collector: The collector is set in the operator's configuration, and exports go only to the fixed
/v1/tracesand/v1/logspaths. - Headers: Exported request headers are limited to an allowlist.
📖 AGENTS.md: ✅ LGTM
The AGENTS.md additions agree with the repository's existing guidance.
⚡ Performance: ✅ LGTM
The export queue and the span and log buffers have fixed limits.
📝 Documentation: ✅ LGTM
The tracing.otlp schema docs and src/opentelemetry/AGENTS.md match the code.
Reviewed commit: 7f6a7935
There was a problem hiding this comment.
Superseded by the review of
bf27d046.
This review, of 71fec26e
💬 Approved with comments — One WARNING is still open: with ExportSpans::NO and a streaming tail, a user span's updates and exceptions change the exported invocation span. Only embedders that use that mode are affected.
Findings: 1 warning
Not opened by the correctness review: 5 changed files it was meant to read
src/opentelemetry/proto/collector/logs/v1/logs_service.protosrc/opentelemetry/proto/collector/trace/v1/trace_service.protosrc/opentelemetry/proto/common/v1/common.protosrc/opentelemetry/proto/resource/v1/resource.protosrc/opentelemetry/proto/trace/v1/trace.proto
The decision is advisory: this review is a comment and never approves, blocks or requests changes on the pull request.
🔧 Fix with your agent
Paste the following into your coding agent to investigate and fix these issues.
Fix the following code review issues found in pull request #7686.
Before making changes, review each issue and present a brief summary table:
- For each finding, state whether you agree, disagree, or need clarification
- If you disagree (e.g., the fix would require disproportionate changes
for minimal ROI, or the finding is factually incorrect), explain why
- If you need to clarify something before deciding, ask those questions
- Then share your plan for which issues to tackle and in what order
## WARNING (1)
### [Code Quality] With ExportSpans::NO and a streaming tail, user-span renames, status changes and exceptions are recorded on the exported invocation span, because addSpanUpdate and addSpanException never check the span id.
- **File:** src/workerd/io/tracer.c++:282
📋 Code Quality: ⚠️ 1 warning
Still open: user-span updates and exceptions change the exported invocation span in ExportSpans::NO mode.
Recommendation: 📝 Minor Comments
- [WARNING] With ExportSpans::NO and a streaming tail, user-span renames, status changes and exceptions are recorded on the exported invocation span, because addSpanUpdate and addSpanException never check the span id. (
src/workerd/io/tracer.c++:282)
View full review
WorkerTracer::addSpanUpdate and addSpanException ignore the span id (src/workerd/io/tracer.c++:282, :370)
addSpanUpdatecallsupdateInvocationSpan(update)and never comparesspanIdwith the invocation span's id.addSpanExceptioncallsrecordInvocationSpanException(...)with the same gap.- With
ExportSpans::NO, user spans go through theSequentialSpanSubmitterinworker-factory.c++. It forwards each user span's update and exception with that user span's own id. - So a user span's
updateName()renames the exported invocation span,setStatus()overwrites its status, and recorded exceptions are added to it.resolve_statusinbuffer.rsthen turns those exceptions into an error status. recordInvocationLevelException(tracer.c++:762) already checkscontext.getSpanId() != SpanParent(invocationSpan).getSpanId(). Both of these paths need the same check.- The NO-mode cases in
tracer-test.c++would catch this if they also sent a user-span update and a user-span exception.
The Rust test moves in buffer.rs/logs.rs and the changes to otlp-export-queue-test.c++ add no defects.
✅ All other checks passed
🔒 Security: ✅ LGTM
- Collector: Only the operator's configuration sets the collector. Exports go to the fixed
/v1/tracesand/v1/logspaths. - Timestamps: Exception timestamps in the streaming tail come from the I/O clock. Only the native OTLP span gets precise time.
- Input: The URL is parsed with
ada_url. When query redaction is on, the query and fragment are removed. - Limits: The span and log buffers are bounded.
📖 AGENTS.md: ✅ LGTM
The AGENTS.md additions agree with the repository's existing guidance.
⚡ Performance: ✅ LGTM
- Buffers: The span and log buffers cap open spans at 1024. Each batch is at most 512 items or 1 MiB, and batch sizes are tracked incrementally.
- Export queue: The export queue has a fixed size, a concurrency limit and a timeout.
📝 Documentation: ✅ LGTM
The tracing.otlp schema docs and src/opentelemetry/AGENTS.md match the code.
Reviewed commit: 71fec26e
71fec26 to
bf27d04
Compare
There was a problem hiding this comment.
Superseded by the review of
4989c65d.
This review, of bf27d046
💬 Approved with comments — One WARNING is still open: with ExportSpans::NO and a streaming tail, a user span's updates and exceptions change the exported invocation span. Only embedders that use that mode are affected.
Findings: 1 warning
Not opened by the correctness review: 5 changed files it was meant to read
src/opentelemetry/proto/collector/logs/v1/logs_service.protosrc/opentelemetry/proto/collector/trace/v1/trace_service.protosrc/opentelemetry/proto/common/v1/common.protosrc/opentelemetry/proto/resource/v1/resource.protosrc/opentelemetry/proto/trace/v1/trace.proto
The decision is advisory: this review is a comment and never approves, blocks or requests changes on the pull request.
🔧 Fix with your agent
Paste the following into your coding agent to investigate and fix these issues.
Fix the following code review issues found in pull request #7686.
Before making changes, review each issue and present a brief summary table:
- For each finding, state whether you agree, disagree, or need clarification
- If you disagree (e.g., the fix would require disproportionate changes
for minimal ROI, or the finding is factually incorrect), explain why
- If you need to clarify something before deciding, ask those questions
- Then share your plan for which issues to tackle and in what order
## WARNING (1)
### [Code Quality] With ExportSpans::NO and a streaming tail, user-span renames, status changes and exceptions are recorded on the exported invocation span, because addSpanUpdate and addSpanException never check the span id.
- **File:** src/workerd/io/tracer.c++:282
📋 Code Quality: ⚠️ 1 warning
Still open: user-span updates and exceptions change the exported invocation span in ExportSpans::NO mode.
Recommendation: 📝 Minor Comments
- [WARNING] With ExportSpans::NO and a streaming tail, user-span renames, status changes and exceptions are recorded on the exported invocation span, because addSpanUpdate and addSpanException never check the span id. (
src/workerd/io/tracer.c++:282)
View full review
WorkerTracer::addSpanUpdate and addSpanException ignore the span id (src/workerd/io/tracer.c++:282, :370)
addSpanUpdatecallsupdateInvocationSpan(update)and never comparesspanIdwith the invocation span's id.addSpanExceptioncallsrecordInvocationSpanException(...)with the same gap.- With
ExportSpans::NO, user spans go through theSequentialSpanSubmitterinworker-factory.c++. It forwards each user span's update and exception with that user span's own id. - So a user span's
updateName()renames the exported invocation span,setStatus()overwrites its status, and recorded exceptions are added to it.resolve_statusinbuffer.rsthen turns those exceptions into an error status. recordInvocationLevelException(tracer.c++:762) already checkscontext.getSpanId() != SpanParent(invocationSpan).getSpanId(). Both of these paths need the same check.- The NO-mode cases in
tracer-test.c++would catch this if they also sent a user-span update and a user-span exception.
The Rust test moves in buffer.rs/logs.rs and the changes to otlp-export-queue-test.c++ add no defects.
✅ All other checks passed
🔒 Security: ✅ LGTM
- Collector: Only the operator's configuration sets the collector. Exports go to the fixed
/v1/tracesand/v1/logspaths. - Timestamps: Exception timestamps in the streaming tail come from the I/O clock. Only the native OTLP span gets precise time.
- Input: The URL is parsed with
ada_url. When query redaction is on, the query and fragment are removed. - Limits: The span and log buffers are bounded.
📖 AGENTS.md: ✅ LGTM
The AGENTS.md additions agree with the repository's existing guidance.
⚡ Performance: ✅ LGTM
- Buffers: The span and log buffers cap open spans at 1024. Each batch is at most 512 items or 1 MiB, and batch sizes are tracked incrementally.
- Export queue: The export queue has a fixed size, a concurrency limit and a timeout.
📝 Documentation: ✅ LGTM
The tracing.otlp schema docs and src/opentelemetry/AGENTS.md match the code.
Reviewed commit: bf27d046 — same changes as 71fec26e, so this is that commit's review (no new review was run; line numbers are as of 71fec26e)
bf27d04 to
4989c65
Compare
There was a problem hiding this comment.
Superseded by the review of
e49b85d2.
This review, of 4989c65d
💬 Approved with comments — One WARNING is still open: with ExportSpans::NO and a streaming tail, a user span's updates and exceptions change the exported invocation span. Only embedders that use that mode are affected.
Findings: 1 warning
Not opened by the correctness review: 5 changed files it was meant to read
src/opentelemetry/proto/collector/logs/v1/logs_service.protosrc/opentelemetry/proto/collector/trace/v1/trace_service.protosrc/opentelemetry/proto/common/v1/common.protosrc/opentelemetry/proto/resource/v1/resource.protosrc/opentelemetry/proto/trace/v1/trace.proto
The decision is advisory: this review is a comment and never approves, blocks or requests changes on the pull request.
🔧 Fix with your agent
Paste the following into your coding agent to investigate and fix these issues.
Fix the following code review issues found in pull request #7686.
Before making changes, review each issue and present a brief summary table:
- For each finding, state whether you agree, disagree, or need clarification
- If you disagree (e.g., the fix would require disproportionate changes
for minimal ROI, or the finding is factually incorrect), explain why
- If you need to clarify something before deciding, ask those questions
- Then share your plan for which issues to tackle and in what order
## WARNING (1)
### [Code Quality] With ExportSpans::NO and a streaming tail, user-span renames, status changes and exceptions are recorded on the exported invocation span, because addSpanUpdate and addSpanException never check the span id.
- **File:** src/workerd/io/tracer.c++:282
📋 Code Quality: ⚠️ 1 warning
Still open: user-span updates and exceptions change the exported invocation span in ExportSpans::NO mode.
Recommendation: 📝 Minor Comments
- [WARNING] With ExportSpans::NO and a streaming tail, user-span renames, status changes and exceptions are recorded on the exported invocation span, because addSpanUpdate and addSpanException never check the span id. (
src/workerd/io/tracer.c++:282)
View full review
WorkerTracer::addSpanUpdate and addSpanException ignore the span id (src/workerd/io/tracer.c++:282, :370)
addSpanUpdatecallsupdateInvocationSpan(update)and never comparesspanIdwith the invocation span's id.addSpanExceptioncallsrecordInvocationSpanException(...)with the same gap.- With
ExportSpans::NO, user spans go through theSequentialSpanSubmitterinworker-factory.c++. It forwards each user span's update and exception with that user span's own id. - So a user span's
updateName()renames the exported invocation span,setStatus()overwrites its status, and recorded exceptions are added to it.resolve_statusinbuffer.rsthen turns those exceptions into an error status. recordInvocationLevelException(tracer.c++:762) already checkscontext.getSpanId() != SpanParent(invocationSpan).getSpanId(). Both of these paths need the same check.- The NO-mode cases in
tracer-test.c++would catch this if they also sent a user-span update and a user-span exception.
The Rust test moves in buffer.rs/logs.rs and the changes to otlp-export-queue-test.c++ add no defects.
✅ All other checks passed
🔒 Security: ✅ LGTM
- Collector: Only the operator's configuration sets the collector. Exports go to the fixed
/v1/tracesand/v1/logspaths. - Timestamps: Exception timestamps in the streaming tail come from the I/O clock. Only the native OTLP span gets precise time.
- Input: The URL is parsed with
ada_url. When query redaction is on, the query and fragment are removed. - Limits: The span and log buffers are bounded.
📖 AGENTS.md: ✅ LGTM
The AGENTS.md additions agree with the repository's existing guidance.
⚡ Performance: ✅ LGTM
- Buffers: The span and log buffers cap open spans at 1024. Each batch is at most 512 items or 1 MiB, and batch sizes are tracked incrementally.
- Export queue: The export queue has a fixed size, a concurrency limit and a timeout.
📝 Documentation: ✅ LGTM
The tracing.otlp schema docs and src/opentelemetry/AGENTS.md match the code.
Reviewed commit: 4989c65d — same changes as 71fec26e, so this is that commit's review (no new review was run; line numbers are as of 71fec26e)
4989c65 to
e49b85d
Compare
There was a problem hiding this comment.
Superseded by the review of
3fa1366a.
This review, of e49b85d2
💬 Approved with comments — One WARNING is still open: in ExportSpans::NO mode with a streaming tail, user-span updates and exceptions change the exported invocation span. This affects only embedders that use that mode. The protobuf pin has no issues.
Findings: 1 warning
Not opened by the correctness review: 7 changed files it was meant to read
build/deps/build_deps.jsoncbuild/deps/gen/build_deps.MODULE.bazelsrc/opentelemetry/proto/collector/logs/v1/logs_service.protosrc/opentelemetry/proto/collector/trace/v1/trace_service.protosrc/opentelemetry/proto/common/v1/common.protosrc/opentelemetry/proto/resource/v1/resource.protosrc/opentelemetry/proto/trace/v1/trace.proto
The decision is advisory: this review is a comment and never approves, blocks or requests changes on the pull request.
🔧 Fix with your agent
Paste the following into your coding agent to investigate and fix these issues.
Fix the following code review issues found in pull request #7686.
Before making changes, review each issue and present a brief summary table:
- For each finding, state whether you agree, disagree, or need clarification
- If you disagree (e.g., the fix would require disproportionate changes
for minimal ROI, or the finding is factually incorrect), explain why
- If you need to clarify something before deciding, ask those questions
- Then share your plan for which issues to tackle and in what order
## WARNING (1)
### [Code Quality] With ExportSpans::NO and a streaming tail, user-span renames, status changes and exceptions are recorded on the exported invocation span, because addSpanUpdate and addSpanException never check the span id.
- **File:** src/workerd/io/tracer.c++:282
📋 Code Quality: ⚠️ 1 warning
One open WARNING: user-span updates and exceptions change the exported invocation span in ExportSpans::NO mode.
Recommendation: 📝 Minor Comments
- [WARNING] With ExportSpans::NO and a streaming tail, user-span renames, status changes and exceptions are recorded on the exported invocation span, because addSpanUpdate and addSpanException never check the span id. (
src/workerd/io/tracer.c++:282)
View full review
addSpanUpdate and addSpanException ignore the span id (src/workerd/io/tracer.c++:282, :370)
addSpanUpdatecallsupdateInvocationSpan(update)for every span.updateInvocationSpan(tracer.c++:708-717) then sets the invocation span's name or status without comparing ids.addSpanExceptioncallsrecordInvocationSpanException(...)with the same gap.- With
ExportSpans::NO, user spans go throughSequentialSpanSubmitterinworker-factory.c++, which forwards each user span's update and exception with that span's own id. - So a user span's
updateName()renames the exported invocation span and itssetStatus()overwrites the invocation span's status. Its recorded exceptions are added to the invocation span, andresolve_statusinbuffer.rsturns them into an error status. recordInvocationLevelExceptionalready does the right check (context.getSpanId() != SpanParent(invocationSpan).getSpanId()). Both paths need the same guard.- The NO-mode cases in
tracer-test.c++would catch this if they also sent a user-span update and a user-span exception.
The protobuf pin is explained in build_deps.jsonc and agrees with the generated MODULE file and the lock.
✅ All other checks passed
🚀 Release: ✅ LGTM
This pull request is not a release. The protobuf 37.0-rc3 pin is a dependency change and its reason is written down in build_deps.jsonc.
🔒 Security: ✅ LGTM
- Collector: The collector comes only from the operator's
tracing.otlpsetting, and exports go to the fixed/v1/tracesand/v1/logspaths. - Limits: The span, log and export-queue buffers all have limits.
- Timestamps: The streaming tail still gets I/O-clock timestamps.
📖 AGENTS.md: ✅ LGTM
The AGENTS.md additions agree with the repository's existing guidance.
⚡ Performance: ✅ LGTM
The buffers cap open spans and batch sizes. The export queue has a fixed size, a concurrency limit and a timeout.
📝 Documentation: ✅ LGTM
The tracing.otlp schema docs and src/opentelemetry/AGENTS.md match the code and the vendored protos.
Reviewed commit: e49b85d2
e49b85d to
3fa1366
Compare
There was a problem hiding this comment.
Superseded by the review of
453d16f3.
This review, of 3fa1366a
💬 Approved with comments — One WARNING is still open: in ExportSpans::NO mode with a streaming tail, user-span updates, exceptions and attributes change the exported invocation span. This only affects embedders that use that mode.
Findings: 1 warning
Not opened by the correctness review: 5 changed files it was meant to read
src/opentelemetry/proto/collector/logs/v1/logs_service.protosrc/opentelemetry/proto/collector/trace/v1/trace_service.protosrc/opentelemetry/proto/common/v1/common.protosrc/opentelemetry/proto/resource/v1/resource.protosrc/opentelemetry/proto/trace/v1/trace.proto
The decision is advisory: this review is a comment and never approves, blocks or requests changes on the pull request.
🔧 Fix with your agent
Paste the following into your coding agent to investigate and fix these issues.
Fix the following code review issues found in pull request #7686.
Before making changes, review each issue and present a brief summary table:
- For each finding, state whether you agree, disagree, or need clarification
- If you disagree (e.g., the fix would require disproportionate changes
for minimal ROI, or the finding is factually incorrect), explain why
- If you need to clarify something before deciding, ask those questions
- Then share your plan for which issues to tackle and in what order
## WARNING (1)
### [Code Quality] With ExportSpans::NO and a streaming tail, user-span renames, status changes, exceptions and attributes are applied to the exported invocation span, because addSpanUpdate, addSpanException and addSpanAttribute never check the span id.
- **File:** src/workerd/io/tracer.c++:282
📋 Code Quality: ⚠️ 1 warning
One open WARNING: tracer methods apply user-span updates, exceptions and attributes to the invocation span in ExportSpans::NO mode.
Recommendation: 📝 Minor Comments
- [WARNING] With ExportSpans::NO and a streaming tail, user-span renames, status changes, exceptions and attributes are applied to the exported invocation span, because addSpanUpdate, addSpanException and addSpanAttribute never check the span id. (
src/workerd/io/tracer.c++:282)
View full review
User-span events change the exported invocation span (src/workerd/io/tracer.c++:282, :370, :658)
- With
ExportSpans::NO, user spans go throughSequentialSpanSubmitter(worker-factory.c++). It forwards each span's update and exception with the span's own id totracer.addSpanUpdate/tracer.addSpanException. Attributes reachaddSpanAttributefromapi/tracing.c++with the user span's context. addSpanUpdatecallsupdateInvocationSpan(update),addSpanExceptioncallsrecordInvocationSpanException(...)andaddSpanAttributecallssetInvocationSpanAttribute(...). None of them compares the id with the invocation span's id.- Result: a user span's
updateName()renames the exported invocation span, and itssetStatus()overwrites the invocation span's status. Its exceptions are recorded on the invocation span, whichresolve_statusthen turns into an error. Its attributes are tagged onto the invocation span. recordInvocationLevelExceptionalready guards withcontext.getSpanId() != SpanParent(invocationSpan).getSpanId(). The same guard is needed on these three paths.- The NO-mode cases in
tracer-test.c++do not send a user-span update, exception or attribute.
✅ All other checks passed
🚀 Release: ✅ LGTM
This pull request is not a release. The protobuf 37.0-rc3 pin is explained in build_deps.jsonc and agrees with the generated MODULE file and the lock.
🔒 Security: ✅ LGTM
- Collector: The collector comes only from the operator's
tracing.otlpsetting. - Limits: The buffers and the export queue have limits.
- Timestamps: The streaming tail still gets I/O-clock timestamps.
📖 AGENTS.md: ✅ LGTM
The AGENTS.md additions agree with the repository's existing guidance.
⚡ Performance: ✅ LGTM
No performance concerns.
📝 Documentation: ✅ LGTM
The tracing.otlp schema docs and src/opentelemetry/AGENTS.md match the code.
Reviewed commit: 3fa1366a
3fa1366 to
453d16f
Compare
There was a problem hiding this comment.
Superseded by the review of
fd5d49da.
This review, of 453d16f3
💬 Approved with comments — One WARNING is still open: in ExportSpans::NO mode with a streaming tail, user-span updates, exceptions and attributes change the exported invocation span. This only affects embedders that use that mode.
Findings: 1 warning
Not opened by the correctness review: 5 changed files it was meant to read
src/opentelemetry/proto/collector/logs/v1/logs_service.protosrc/opentelemetry/proto/collector/trace/v1/trace_service.protosrc/opentelemetry/proto/common/v1/common.protosrc/opentelemetry/proto/resource/v1/resource.protosrc/opentelemetry/proto/trace/v1/trace.proto
The decision is advisory: this review is a comment and never approves, blocks or requests changes on the pull request.
🔧 Fix with your agent
Paste the following into your coding agent to investigate and fix these issues.
Fix the following code review issues found in pull request #7686.
Before making changes, review each issue and present a brief summary table:
- For each finding, state whether you agree, disagree, or need clarification
- If you disagree (e.g., the fix would require disproportionate changes
for minimal ROI, or the finding is factually incorrect), explain why
- If you need to clarify something before deciding, ask those questions
- Then share your plan for which issues to tackle and in what order
## WARNING (1)
### [Code Quality] With ExportSpans::NO and a streaming tail, user-span renames, status changes, exceptions and attributes are applied to the exported invocation span, because addSpanUpdate, addSpanException and addSpanAttribute never check the span id.
- **File:** src/workerd/io/tracer.c++:282
📋 Code Quality: ⚠️ 1 warning
One open WARNING: tracer methods apply user-span updates, exceptions and attributes to the invocation span in ExportSpans::NO mode.
Recommendation: 📝 Minor Comments
- [WARNING] With ExportSpans::NO and a streaming tail, user-span renames, status changes, exceptions and attributes are applied to the exported invocation span, because addSpanUpdate, addSpanException and addSpanAttribute never check the span id. (
src/workerd/io/tracer.c++:282)
View full review
User-span events change the exported invocation span (src/workerd/io/tracer.c++:282, :370, :658)
- With
ExportSpans::NO, user spans go throughSequentialSpanSubmitter(worker-factory.c++). It forwards each span's update and exception with the span's own id totracer.addSpanUpdate/tracer.addSpanException. Attributes reachaddSpanAttributefromapi/tracing.c++with the user span's context. addSpanUpdatecallsupdateInvocationSpan(update),addSpanExceptioncallsrecordInvocationSpanException(...)andaddSpanAttributecallssetInvocationSpanAttribute(...). None of them compares the id with the invocation span's id.- Result: a user span's
updateName()renames the exported invocation span, and itssetStatus()overwrites the invocation span's status. Its exceptions are recorded on the invocation span, whichresolve_statusthen turns into an error. Its attributes are tagged onto the invocation span. recordInvocationLevelExceptionalready guards withcontext.getSpanId() != SpanParent(invocationSpan).getSpanId(). The same guard is needed on these three paths.- The NO-mode cases in
tracer-test.c++do not send a user-span update, exception or attribute.
✅ All other checks passed
🚀 Release: ✅ LGTM
This pull request is not a release. The protobuf 37.0-rc3 pin is explained in build_deps.jsonc and agrees with the generated MODULE file and the lock.
🔒 Security: ✅ LGTM
- Collector: The collector comes only from the operator's
tracing.otlpsetting. - Limits: The buffers and the export queue have limits.
- Timestamps: The streaming tail still gets I/O-clock timestamps.
📖 AGENTS.md: ✅ LGTM
The AGENTS.md additions agree with the repository's existing guidance.
⚡ Performance: ✅ LGTM
No performance concerns.
📝 Documentation: ✅ LGTM
The tracing.otlp schema docs and src/opentelemetry/AGENTS.md match the code.
Reviewed commit: 453d16f3 — same changes as 3fa1366a, so this is that commit's review (no new review was run; line numbers are as of 3fa1366a)
The runtime writes OTLP trace and log requests through its own prost structs in src/rust/otlp. Tests that decode those requests need the generated classes. Until now, edgeworker vendored the trace schemas for its tests, and nothing vendored the logs schemas. Add the trace, logs, common, resource and collector schemas from opentelemetry-proto v1.11.1, byte-identical below a provenance header, and expose them as //src/opentelemetry:otlp_cc_proto. protobuf becomes a direct dependency at 35.0, the version Perfetto already brings in, so the build resolves nothing new. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
src/rust/otlp turns one invocation's spans into OTLP/HTTP ExportTraceServiceRequests for a collector's /v1/traces, in the protobuf encoding. C++ reports each span's open, updates and close to a SpanBuffer through a cxx bridge, and gets back encoded requests to send. - proto: the part of the OTLP trace schema the encoder writes, as prost structs, so neither prost-build nor protoc is involved. - rules: what a Workers span looks like in OTLP. Its kind and status come from its tags, query strings are redacted from URL tags when asked, and fetches to bindings' internal hosts are left out. - buffer: batching. An invocation's spans go out bounded by count and bytes; the root closes the trace. - attributes: resource and default attributes, shared by every span. The crate has no C++ caller yet. Co-authored-by: William Perron <wperron@cloudflare.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Workers user spans reach a tracing backend today only through Streaming Tail Workers: the Workers Observability tail worker rebuilds them in JavaScript from tail events. Spans built that way are cut off when a Durable Object hands off work, carry coarse I/O-clock timestamps, and are sampled only after they already exist. This gives the runtime what it needs to export user spans itself: - OtlpSpanSubmitter, a span submitter that feeds the Rust encoder and hands its requests to an OtlpSpanExporter the embedder provides. - BaseTracer::setStartInvocationSpanFunc(), a hook that lets the embedder open each invocation's root span, so the invocation and all of its user spans form one exported trace. startExportedUserSpan() opens a span that is exported. - The root records the invocation's outcome and any exception reported for the invocation itself. - Exported spans get precise timestamps. Streaming tails, and so user code, still see the request's I/O clock. Nothing changes unless an embedder installs the hook. Co-authored-by: William Perron <wperron@cloudflare.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A new `tracing.otlp` setting names a service, typically an ExternalServer pointing at a collector's OTLP/HTTP port, such as a local Jaeger. workerd then exports every invocation's spans to it, which makes Workers traces visible in local development without a tail worker. A request arriving on a socket starts its trace with the Worker's invocation span, and invocations of the collector's own service are not traced. Spans carry the service name, entrypoint and, for Durable Objects, the object's id. The export queue matches edgeworker's limits: 16 concurrent POSTs, 4096 queued requests, batches of up to 1 MiB and a 10-second timeout. Requests are dropped when the queue is full or delivery fails, and queued exports are cancelled on shutdown. Co-authored-by: William Perron <wperron@cloudflare.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Some bindings (D1, AI, Images, Vectorize) reach their backend with an internal fetch to a placeholder host. Those fetch spans were dropped when they ended, but by then their id had already been handed to the backend as its parent, so any spans the backend exported were orphaned at the top of the trace. Decide when the fetch starts instead: a fetch to one of these hosts opens no user span, and the binding's own span (such as d1_run) becomes the subrequest's parent. Matching by host is still a stopgap until every binding talks to its backend over JSRPC. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
An invocation whose span is exported can now also export its console logs and the exceptions it reports, the events the streaming tail gets as `log` and `exception`. They become OTLP LogRecords in an ExportLogsServiceRequest for a collector's /v1/logs. This lets consumers such as Workers Observability get logs without a tail worker. The hook passed to BaseTracer::setStartInvocationSpanFunc() now returns a StartedInvocationSpan. It holds the span and, optionally, an InvocationLogSubmitter, which the tracer feeds until it is destroyed. startExportedInvocationSpan() opens an invocation's span. Its options say what the export carries: * ExportLogs::YES: the OtlpSpanSubmitter also takes the invocation's logs and hands them to OtlpSpanExporter::exportLogs(). * ExportSpans::NO: as the streaming tail worker samples logs apart from traces, the submitter records only the root span and exports no span, and the tracer parents no other span under it. The root still describes the invocation for its logs. * ExportInvocationLog::YES: when the root closes, the logs get a `cloudflare.invocation` record made from it: the span's name as the body, its final attributes, its end time and its ids. The Rust LogBuffer encodes and batches the records: * Each record carries the trace id, the id of the span that was active, the trace flags, and the spans' resource and default tags. * The body is the console arguments as an OTLP array, with object keys in their original order. A truncated message falls back to a string. * A console argument that was a native Error keeps its name, message and stack in `cloudflare.log.error_info`, as the streaming tail's `errorInfo` does: an array in argument order, with a kvlist for each Error and an empty value for any other argument. * An exception is a record with event_name "exception" and the exception.* attributes. * Batches are bounded like spans. They go out when the root span closes, and one record at a time after that. tracer-test decodes the requests with the vendored OTLP classes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
buffer.rs, logs.rs and rules.rs kept their tests in inline `mod tests` blocks; src/rust/AGENTS.md puts them in <module>-test.rs beside the module. lib.rs also drops the lint denies the wd_rust_* macros already apply. unreachable! is allowed everywhere, so it needs no deny of its own. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The lint job failed on three errors: - otlp-export-queue-test filled its payload with std::memset, which custom-ban-memset forbids. It now uses kj::ArrayPtr::fill(), and drops <cstring>. - buffer::Options has four bools, over struct_excessive_bools' limit. They are independent per-invocation switches, not states of one thing, so the lint is expected there rather than refactored away. - A logs.rs doc comment named serde_json without backticks. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
tracer-test aborts under ASan + UBSan before main runs, while protobuf registers the OTLP schemas' generated descriptors. EpsCopyInputStream::LimitToken poisons its token once it is used, and PopLimit() took the token by value. Clang 22 can give that parameter the stack slot of the moved-from token, so the move poisons the parameter and PopLimit() reads poisoned memory (protocolbuffers/protobuf#27298). protobuf 37.0 takes the token by rvalue reference (protocolbuffers/protobuf#28619). No stable release has the fix yet, so this freezes protobuf at 37.0-rc3 until 37.0 is out. Its dependencies are already at or below the versions workerd uses. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Before any case runs, the test builds about 25 startup snapshots, one per zygote Worker. That takes 6-10 seconds in a debug build on x86 and just over 15 seconds on edgeworker's arm64 debug build. Edgeworker gives medium tests 15 seconds, so the test's @all-autogates variant times out there in most runs; workerd's internal builds for #8405 and other PRs since have needed retries to get past it. Its cases take about 200ms, so it is slow to start rather than stuck. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
453d16f to
fd5d49d
Compare
There was a problem hiding this comment.
💬 Approved with comments — One WARNING is still open: with ExportSpans::NO and a streaming tail, a user span's renames, status changes, exceptions and attributes are applied to the exported invocation span. This only affects embedders that use that mode.
Findings: 1 warning
Not opened by the correctness review: 5 changed files it was meant to read
src/opentelemetry/proto/collector/logs/v1/logs_service.protosrc/opentelemetry/proto/collector/trace/v1/trace_service.protosrc/opentelemetry/proto/common/v1/common.protosrc/opentelemetry/proto/resource/v1/resource.protosrc/opentelemetry/proto/trace/v1/trace.proto
The decision is advisory: this review is a comment and never approves, blocks or requests changes on the pull request.
🔧 Fix with your agent
Paste the following into your coding agent to investigate and fix these issues.
Fix the following code review issues found in pull request #7686.
Before making changes, review each issue and present a brief summary table:
- For each finding, state whether you agree, disagree, or need clarification
- If you disagree (e.g., the fix would require disproportionate changes
for minimal ROI, or the finding is factually incorrect), explain why
- If you need to clarify something before deciding, ask those questions
- Then share your plan for which issues to tackle and in what order
## WARNING (1)
### [Code Quality] With ExportSpans::NO and a streaming tail, user-span renames, status changes, exceptions and attributes are applied to the exported invocation span: addSpanUpdate, addSpanException and addSpanAttribute never check the span id.
- **File:** src/workerd/io/tracer.c++:282
📋 Code Quality: ⚠️ 1 warning
One open WARNING: user-span events change the exported invocation span in ExportSpans::NO mode.
Recommendation: 📝 Minor Comments
- [WARNING] With ExportSpans::NO and a streaming tail, user-span renames, status changes, exceptions and attributes are applied to the exported invocation span: addSpanUpdate, addSpanException and addSpanAttribute never check the span id. (
src/workerd/io/tracer.c++:282)
View full review
User-span events change the exported invocation span (src/workerd/io/tracer.c++:282, :370, :658)
- With
ExportSpans::NO, user spans go throughSequentialSpanSubmitterinworker-factory.c++. It forwards each span's updates and exceptions totracer.addSpanUpdateandtracer.addSpanExceptionunder that span's own id.api/tracing.c++sends a user span's attributes toaddSpanAttributewith that span's context. addSpanUpdatecallsupdateInvocationSpan(update),addSpanExceptioncallsrecordInvocationSpanException(...)andaddSpanAttributecallssetInvocationSpanAttribute(...). None of the three checks that the id is the invocation span's id.- What happens:
- A user span's
updateName()renames the exported invocation span. - Its
setStatus()overwrites the invocation span's status. - Its exceptions are recorded on the invocation span, which then ends with an error status.
- Its attributes are added to the invocation span.
- A user span's
recordInvocationLevelExceptionalready has the needed check:context.getSpanId() != SpanParent(invocationSpan).getSpanId(). Add the same check to these three paths.tracer-test.c++has no ExportSpans::NO case that sends a user-span update, exception or attribute. Add one when fixing this.
✅ All other checks passed
🚀 Release: ✅ LGTM
This pull request is not a release. build_deps.jsonc explains the protobuf 37.0-rc3 pin, and the pin matches the generated MODULE file and the lockfile.
🔒 Security: ✅ LGTM
- Collector: Only the operator's
tracing.otlpsetting chooses the collector. - Limits: The span buffer, the log buffer and the export queue are all bounded.
- Exported data: Only selected request headers are exported, never authorization or cookie headers. Query strings can be redacted.
- Timestamps: The streaming tail still gets I/O-clock timestamps.
📖 AGENTS.md: ✅ LGTM
The AGENTS.md additions agree with the repository's existing guidance.
⚡ Performance: ✅ LGTM
No performance concerns.
📝 Documentation: ✅ LGTM
The tracing.otlp schema docs and src/opentelemetry/AGENTS.md match the code and the vendored protos.
Reviewed commit: fd5d49da
Spans are going away from streaming tail events: workerd now exports each invocation's spans itself, as OTLP (`tracing.otlp`, from cloudflare/workerd#7686). With `unsafeObservability`, Miniflare makes the observability collector workerd's OTLP collector. Its new `/v1/traces` endpoint decodes the protobuf export with a small reader for the fields workerd sets and stores the spans. The streaming tail now supplies only logs and exceptions, so `tail_worker_user_spans` is no longer enabled. `tracing.otlp` traces every service in the config, including Miniflare's own Workers. The collector keeps only the spans of the services Miniflare passes it (user workers and the Workflows services that run their workflows, with the worker names the tail props used to carry) and drops the rest. An invocation started by one of those dropped spans, such as the entry Worker's, would then point at a parent that is never stored, so the store keeps a span whose parent is missing as a root and parks the parent in `pending_parents`. If the parent arrives later, as a calling user worker's span does after its callee has been exported, the span is attached to it again. The config types add `tracing` from the merged workerd schema, so this needs a workerd release that includes cloudflare/workerd#7686. Co-Authored-By: Dan Lapid <dlapid@cloudflare.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Workers user spans reach a tracing backend today only through Streaming Tail Workers: the Workers Observability tail worker rebuilds them in JavaScript from tail events. Spans built that way are cut off when a Durable Object hands off work, carry coarse I/O-clock timestamps, and are sampled only after they already exist.
This gives the runtime what it needs to export user spans itself, as OTLP, the way other Cloudflare services already do:
workerd can also export on its own: a new
tracing.otlpsetting sends each invocation's spans to an OTLP/HTTP collector such as a local Jaeger. That makes Workers traces visible in local development without a tail worker.Nothing changes unless an embedder installs the hook or the setting is configured.