Skip to content

Export Workers user spans over OTLP - #7686

Merged
wperron merged 10 commits into
mainfrom
dlapid/user-tracing-otlp
Nov 19, 2026
Merged

wperron merged 10 commits into
mainfrom
dlapid/user-tracing-otlp

Conversation

@danlapid

Copy link
Copy Markdown
Collaborator

Workers user spans reach a tracing backend today only through Streaming Tail Workers: the Workers Observability tail worker rebuilds them in JavaScript from tail events. Spans built that way are cut off when a Durable Object hands off work, carry coarse I/O-clock timestamps, and are sampled only after they already exist.

This gives the runtime what it needs to export user spans itself, as OTLP, the way other Cloudflare services already do:

  • an OTLP encoder for spans, written in Rust, behind the existing span submitter interface;
  • a tracer hook that lets the embedder open each invocation's root span, so the invocation and all of its user spans form one exported trace;
  • precise timestamps for user spans, which no longer reach user code directly.

workerd can also export on its own: a new tracing.otlp setting sends each invocation's spans to an OTLP/HTTP collector such as a local Jaeger. That makes Workers traces visible in local development without a tail worker.

Nothing changes unless an embedder installs the hook or the setting is configured.

Comment thread src/workerd/io/otlp-span-submitter.h Resolved
Comment thread src/workerd/server/server.c++ Resolved
Comment thread src/workerd/server/server.h Resolved
Comment thread src/workerd/io/otlp-span-submitter.h Resolved
Comment thread src/rust/otlp/buffer.rs Resolved
@github-actions

github-actions Bot commented Oct 16, 2026 •

Copy link
Copy Markdown

All contributors have signed the CLA ✍️ ✅
Posted by the CLA Assistant Lite bot.

@wperron

wperron commented Oct 16, 2026

Copy link
Copy Markdown
Collaborator

I have read the CLA Document and I hereby sign the CLA

github-actions Bot added a commit that referenced this pull request Oct 16, 2026
@wperron
wperron force-pushed the dlapid/user-tracing-otlp branch from 82bb3ec to 9faabe2 Compare October 26, 2026 19:13
@github-actions

github-actions Bot commented Oct 27, 2026 •

Copy link
Copy Markdown

The generated output of @cloudflare/workers-types matches the snapshot in types/generated-snapshot 🎉

@wperron
wperron force-pushed the dlapid/user-tracing-otlp branch from 122e1d1 to f727d85 Compare October 28, 2026 15:21
@wperron
wperron requested a review from jasnell October 28, 2026 15:25
@wperron
wperron force-pushed the dlapid/user-tracing-otlp branch from f727d85 to 4f06a7e Compare October 30, 2026 14:13
@wperron
wperron marked this pull request as ready for review October 30, 2026 14:13
@wperron
wperron requested review from a team as code owners October 30, 2026 14:13

@clanky-the-clanker clanky-the-clanker Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Superseded by the review of 91d0101e.

This review, of 4f06a7ee

🛑 Significant concerns — 1 BLOCKING: UserSpanObserver::getTime() now uses the precise system clock for every user span, including the existing streaming-tail path. Tail workers, which are user code, now receive precise timestamps, and their spans mix a precise start time with a coarse I/O-clock end time. There are also 4 WARNINGs, among them an ungated change to the fetch spans tail workers receive and a Durable Object id tag that is never set.

Findings: 1 blocking · 4 warnings · 1 info · 1 suggestion (4 are inline comments on the diff)

The decision is advisory: this review is a comment and never approves, blocks or requests changes on the pull request.

🔧 Fix with your agent

Paste the following into your coding agent to investigate and fix these issues.

Fix the following code review issues found in pull request #7686.

Before making changes, review each issue and present a brief summary table:
- For each finding, state whether you agree, disagree, or need clarification
- If you disagree (e.g., the fix would require disproportionate changes
  for minimal ROI, or the finding is factually incorrect), explain why
- If you need to clarify something before deciding, ask those questions
- Then share your plan for which issues to tackle and in what order

## BLOCKING (1)

### [Security] `UserSpanObserver::getTime()` now uses `systemPreciseCalendarClock()` for every user span, including the existing streaming-tail path. Tail workers (user code) receive precise start and exception timestamps, which gets around the coarse I/O clock. Tail spans also combine a precise start with an I/O-clock end set by `adjustSpanTime()`, so durations can be zero or negative. The change is not gated.
- **File:** src/workerd/io/tracer.c++:838

## WARNING (4)

### [Code Quality] The internal-binding fetch filter drops `fetch` spans for D1, Images, AI and Vectorize hosts from every streaming tail worker, not only when OTLP export is enabled. Because it matches by host only, a user's own fetches to those hosts lose their spans as well.
- **File:** src/workerd/api/http.c++:1922

### [Code Quality] `OtlpServiceInfo.durableObjectId` is never set, so exported Durable Object spans never carry the `cloudflare.durable_object.id` tag that the header documents.
- **File:** src/workerd/server/factory/worker-factory.c++:1128

### [Code Quality] `exportSpans()` starts a new POST task for every batch, with no limit on how many run at once and no timeout. When the collector is slow or down, memory grows without limit and an ERROR is logged for every batch.
- **File:** src/workerd/server/factory/worker-factory-otlp.c++:27

### [Documentation] `docs/async-trace.md` still says user spans use the Spectre-safe clock, but `UserSpanObserver::getTime()` now uses the precise system clock.
- **File:** docs/async-trace.md:231

## INFO (1)

### [Documentation] The new `tracing.otlp` setting is described only in the `workerd.capnp` schema comments. A short guide, such as how to point a local workerd at Jaeger, would help users find it.

## SUGGESTION (1)

### [Code Quality] Rust unit tests are written as inline `mod tests` blocks in `buffer.rs` and `rules.rs`. The repository convention puts them in `buffer-test.rs` and `rules-test.rs` beside the module.
- **File:** src/rust/otlp/buffer.rs:309

📋 Code Quality: ⚠️ 3 warnings · 1 suggestion

An ungated change to tail-visible fetch spans, a Durable Object id tag that is never set, unbounded in-flight OTLP exports, and inline Rust tests.

Recommendation: 📝 Minor Comments

  • [WARNING] The internal-binding fetch filter drops fetch spans for D1, Images, AI and Vectorize hosts from every streaming tail worker, not only when OTLP export is enabled. Because it matches by host only, a user's own fetches to those hosts lose their spans as well. (src/workerd/api/http.c++:1922)
  • [WARNING] OtlpServiceInfo.durableObjectId is never set, so exported Durable Object spans never carry the cloudflare.durable_object.id tag that the header documents. (src/workerd/server/factory/worker-factory.c++:1128)
  • [WARNING] exportSpans() starts a new POST task for every batch, with no limit on how many run at once and no timeout. When the collector is slow or down, memory grows without limit and an ERROR is logged for every batch. (src/workerd/server/factory/worker-factory-otlp.c++:27)
  • [SUGGESTION] Rust unit tests are written as inline mod tests blocks in buffer.rs and rules.rs. The repository convention puts them in buffer-test.rs and rules-test.rs beside the module. (src/rust/otlp/buffer.rs:309)
View full review

Internal-binding fetch filter applies to everyone (src/workerd/api/http.c++:1616-1629, 1922)

fetchImplNoOutputLockAttempt now passes OmitUserSpan(isInternalBindingFetch(...)) for every worker, whether or not OTLP export is enabled. Every streaming tail worker stops receiving fetch spans for these hosts. The updated D1 and Images instrumentation tests confirm this. Matching is by host only, so a user's own fetch() to js.images.cloudflare.com or to http://d1/ also loses its span. Commit 837539b explains why the filter exists. Still, this is a visible change to tail output with no gate, and it contradicts the description's "nothing changes" claim. Consider gating it on OTLP export being active, or recording the compatibility decision explicitly.

Durable Object id is never exported (src/workerd/server/factory/worker-factory.c++:1128-1131)

The OtlpServiceInfo initializer sets only .serviceName and .entrypoint. The durableObjectId computed at lines 1086-1096 is moved into the Trace, so makeOtlpIdentityTags() never emits cloudflare.durable_object.id, although the doc comment in otlp-span-submitter.h promises it on every span.

Unbounded in-flight exports (src/workerd/server/factory/worker-factory-otlp.c++:25-49)

Each batch adds a TaskSet task that creates a new HTTP client and POSTs up to about 1 MiB. Nothing caps concurrency, and there is no timeout or drop policy. When the collector is slow or hung, tasks and their request bodies pile up for as long as the server runs. taskFailed also logs an ERROR for every failed batch. Consider capping the number of in-flight exports, dropping batches beyond the cap with spansDropped(), adding a timeout, and rate-limiting the log.

Inline Rust tests (src/rust/otlp/buffer.rs:309, src/rust/otlp/rules.rs:84)

The repository convention puts Rust unit tests in <module>-test.rs, never in an inline mod tests. wd_rust_crate already splits *-test.rs files into the test target.


📝 Documentation: ⚠️ 1 warning · 1 info

The timing description in docs/async-trace.md is now out of date, and tracing.otlp is not documented.

Recommendation: 📝 Minor Comments

  • [WARNING] docs/async-trace.md still says user spans use the Spectre-safe clock, but UserSpanObserver::getTime() now uses the precise system clock. (docs/async-trace.md:231)
  • [INFO] The new tracing.otlp setting is described only in the workerd.capnp schema comments. A short guide, such as how to point a local workerd at Jaeger, would help users find it.
View full review
  • docs/async-trace.md:231-234 says user-facing spans use the Spectre-safe clock. With this change, UserSpanObserver::getTime() uses the precise clock. Once the clock question in the security section is settled, update this text to match.
  • The new tracing.otlp setting has no guide in docs/. The schema comments in workerd.capnp are its only documentation.

🔒 Security: 🛑 1 blocking

The precise clock change also applies to the existing streaming-tail path, so tail workers receive precise timestamps.

Recommendation: 🔴 Serious Concerns

  • [BLOCKING] UserSpanObserver::getTime() now uses systemPreciseCalendarClock() for every user span, including the existing streaming-tail path. Tail workers (user code) receive precise start and exception timestamps, which gets around the coarse I/O clock. Tail spans also combine a precise start with an I/O-clock end set by adjustSpanTime(), so durations can be zero or negative. The change is not gated. (src/workerd/io/tracer.c++:838)
View full review

Precise timestamps reach tail workers, and tail spans mix two clocks (src/workerd/io/tracer.c++:837-840)

UserSpanObserver::getTime() used to return IoContext::current().now(). It now returns kj::systemPreciseCalendarClock().now(). This observer is not specific to OTLP. worker-factory.c++ installs it with a SequentialSpanSubmitter for every streaming tail worker, and the production embedder's tail path uses it too.

  • SpanBuilder takes its start time from observer->getTime() (trace.c++:1965). WorkerTracer::addSpanOpen then forwards that start time unchanged to the tail stream (tracer.c++:213-234). Exception timestamps (trace.c++:2089) are handled the same way.
  • A tail worker is user code. A worker can bracket any piece of its own code with enterSpan() and read nanosecond start times in its tail worker. That gets around the coarse I/O clock that exists to deny Workers a high-resolution timer. It also contradicts the description's statement that precise timestamps "no longer reach user code directly".
  • WorkerTracer::addSpanClose still calls adjustSpanTime(), which overwrites endTime with context.now() on the I/O clock (tracer.c++:534-560). Tail spans therefore have a precise start and a coarse end. The I/O clock does not advance while JavaScript runs, so the end time can be earlier than the start time. Existing tail consumers would then see zero or negative durations.
  • This change is not gated. It takes effect whether or not the OTLP hook or tracing.otlp is configured, which also contradicts "Nothing changes unless an embedder installs the hook or the setting is configured".

Fix: use the precise clock only for observers created by the OTLP export path, for example with a flag or a dedicated observer, and keep IoContext::now() on the tail-stream path.

Apart from this, the untrusted inputs into the Rust encoder are handled safely: UTF-8 is converted lossily, the open-span and batch buffers have size limits, and query strings are redacted by rebuilding url.full.


✅ All other checks passed

📖 AGENTS.md: ✅ LGTM

The addition to src/rust/AGENTS.md is consistent with the rest of that file.


⚡ Performance: ✅ LGTM

The issue of unbounded in-flight OTLP exports is covered in the code quality section. src/rust/otlp/buffer.rs limits open spans to MAX_OPEN_SPANS and limits batches by span count and by bytes.


Reviewed commit: 4f06a7ee

Comment thread src/workerd/io/tracer.c++ Resolved
Comment thread src/workerd/api/http.c++ Resolved
@wperron
wperron force-pushed the dlapid/user-tracing-otlp branch from 4f06a7e to 91d0101 Compare October 30, 2026 18:02

@clanky-the-clanker clanky-the-clanker Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Superseded by the review of 855b0965.

This review, of 91d0101e

🛑 Significant concerns — 1 BLOCKING: UserSpanObserver::getTime() now uses the precise system clock for every user span, including the existing streaming-tail path. Tail workers, which are user code, now receive precise timestamps, and their spans mix a precise start time with a coarse I/O-clock end time. There are also 4 WARNINGs, among them an ungated change to the fetch spans tail workers receive and a Durable Object id tag that is never set.

Findings: 1 blocking · 4 warnings · 1 info · 1 suggestion

The decision is advisory: this review is a comment and never approves, blocks or requests changes on the pull request.

🔧 Fix with your agent

Paste the following into your coding agent to investigate and fix these issues.

Fix the following code review issues found in pull request #7686.

Before making changes, review each issue and present a brief summary table:
- For each finding, state whether you agree, disagree, or need clarification
- If you disagree (e.g., the fix would require disproportionate changes
  for minimal ROI, or the finding is factually incorrect), explain why
- If you need to clarify something before deciding, ask those questions
- Then share your plan for which issues to tackle and in what order

## BLOCKING (1)

### [Security] `UserSpanObserver::getTime()` now uses `systemPreciseCalendarClock()` for every user span, including the existing streaming-tail path. Tail workers (user code) receive precise start and exception timestamps, which gets around the coarse I/O clock. Tail spans also combine a precise start with an I/O-clock end set by `adjustSpanTime()`, so durations can be zero or negative. The change is not gated.
- **File:** src/workerd/io/tracer.c++:838

## WARNING (4)

### [Code Quality] The internal-binding fetch filter drops `fetch` spans for D1, Images, AI and Vectorize hosts from every streaming tail worker, not only when OTLP export is enabled. Because it matches by host only, a user's own fetches to those hosts lose their spans as well.
- **File:** src/workerd/api/http.c++:1922

### [Code Quality] `OtlpServiceInfo.durableObjectId` is never set, so exported Durable Object spans never carry the `cloudflare.durable_object.id` tag that the header documents.
- **File:** src/workerd/server/factory/worker-factory.c++:1128

### [Code Quality] `exportSpans()` starts a new POST task for every batch, with no limit on how many run at once and no timeout. When the collector is slow or down, memory grows without limit and an ERROR is logged for every batch.
- **File:** src/workerd/server/factory/worker-factory-otlp.c++:27

### [Documentation] `docs/async-trace.md` still says user spans use the Spectre-safe clock, but `UserSpanObserver::getTime()` now uses the precise system clock.
- **File:** docs/async-trace.md:231

## INFO (1)

### [Documentation] The new `tracing.otlp` setting is described only in the `workerd.capnp` schema comments. A short guide, such as how to point a local workerd at Jaeger, would help users find it.

## SUGGESTION (1)

### [Code Quality] Rust unit tests are written as inline `mod tests` blocks in `buffer.rs` and `rules.rs`. The repository convention puts them in `buffer-test.rs` and `rules-test.rs` beside the module.
- **File:** src/rust/otlp/buffer.rs:309

📋 Code Quality: ⚠️ 3 warnings · 1 suggestion

An ungated change to tail-visible fetch spans, a Durable Object id tag that is never set, unbounded in-flight OTLP exports, and inline Rust tests.

Recommendation: 📝 Minor Comments

  • [WARNING] The internal-binding fetch filter drops fetch spans for D1, Images, AI and Vectorize hosts from every streaming tail worker, not only when OTLP export is enabled. Because it matches by host only, a user's own fetches to those hosts lose their spans as well. (src/workerd/api/http.c++:1922)
  • [WARNING] OtlpServiceInfo.durableObjectId is never set, so exported Durable Object spans never carry the cloudflare.durable_object.id tag that the header documents. (src/workerd/server/factory/worker-factory.c++:1128)
  • [WARNING] exportSpans() starts a new POST task for every batch, with no limit on how many run at once and no timeout. When the collector is slow or down, memory grows without limit and an ERROR is logged for every batch. (src/workerd/server/factory/worker-factory-otlp.c++:27)
  • [SUGGESTION] Rust unit tests are written as inline mod tests blocks in buffer.rs and rules.rs. The repository convention puts them in buffer-test.rs and rules-test.rs beside the module. (src/rust/otlp/buffer.rs:309)
View full review

Internal-binding fetch filter applies to everyone (src/workerd/api/http.c++:1616-1629, 1922)

fetchImplNoOutputLockAttempt now passes OmitUserSpan(isInternalBindingFetch(...)) for every worker, whether or not OTLP export is enabled. Every streaming tail worker stops receiving fetch spans for these hosts. The updated D1 and Images instrumentation tests confirm this. Matching is by host only, so a user's own fetch() to js.images.cloudflare.com or to http://d1/ also loses its span. Commit 837539b explains why the filter exists. Still, this is a visible change to tail output with no gate, and it contradicts the description's "nothing changes" claim. Consider gating it on OTLP export being active, or recording the compatibility decision explicitly.

Durable Object id is never exported (src/workerd/server/factory/worker-factory.c++:1128-1131)

The OtlpServiceInfo initializer sets only .serviceName and .entrypoint. The durableObjectId computed at lines 1086-1096 is moved into the Trace, so makeOtlpIdentityTags() never emits cloudflare.durable_object.id, although the doc comment in otlp-span-submitter.h promises it on every span.

Unbounded in-flight exports (src/workerd/server/factory/worker-factory-otlp.c++:25-49)

Each batch adds a TaskSet task that creates a new HTTP client and POSTs up to about 1 MiB. Nothing caps concurrency, and there is no timeout or drop policy. When the collector is slow or hung, tasks and their request bodies pile up for as long as the server runs. taskFailed also logs an ERROR for every failed batch. Consider capping the number of in-flight exports, dropping batches beyond the cap with spansDropped(), adding a timeout, and rate-limiting the log.

Inline Rust tests (src/rust/otlp/buffer.rs:309, src/rust/otlp/rules.rs:84)

The repository convention puts Rust unit tests in <module>-test.rs, never in an inline mod tests. wd_rust_crate already splits *-test.rs files into the test target.


📝 Documentation: ⚠️ 1 warning · 1 info

The timing description in docs/async-trace.md is now out of date, and tracing.otlp is not documented.

Recommendation: 📝 Minor Comments

  • [WARNING] docs/async-trace.md still says user spans use the Spectre-safe clock, but UserSpanObserver::getTime() now uses the precise system clock. (docs/async-trace.md:231)
  • [INFO] The new tracing.otlp setting is described only in the workerd.capnp schema comments. A short guide, such as how to point a local workerd at Jaeger, would help users find it.
View full review
  • docs/async-trace.md:231-234 says user-facing spans use the Spectre-safe clock. With this change, UserSpanObserver::getTime() uses the precise clock. Once the clock question in the security section is settled, update this text to match.
  • The new tracing.otlp setting has no guide in docs/. The schema comments in workerd.capnp are its only documentation.

🔒 Security: 🛑 1 blocking

The precise clock change also applies to the existing streaming-tail path, so tail workers receive precise timestamps.

Recommendation: 🔴 Serious Concerns

  • [BLOCKING] UserSpanObserver::getTime() now uses systemPreciseCalendarClock() for every user span, including the existing streaming-tail path. Tail workers (user code) receive precise start and exception timestamps, which gets around the coarse I/O clock. Tail spans also combine a precise start with an I/O-clock end set by adjustSpanTime(), so durations can be zero or negative. The change is not gated. (src/workerd/io/tracer.c++:838)
View full review

Precise timestamps reach tail workers, and tail spans mix two clocks (src/workerd/io/tracer.c++:837-840)

UserSpanObserver::getTime() used to return IoContext::current().now(). It now returns kj::systemPreciseCalendarClock().now(). This observer is not specific to OTLP. worker-factory.c++ installs it with a SequentialSpanSubmitter for every streaming tail worker, and the production embedder's tail path uses it too.

  • SpanBuilder takes its start time from observer->getTime() (trace.c++:1965). WorkerTracer::addSpanOpen then forwards that start time unchanged to the tail stream (tracer.c++:213-234). Exception timestamps (trace.c++:2089) are handled the same way.
  • A tail worker is user code. A worker can bracket any piece of its own code with enterSpan() and read nanosecond start times in its tail worker. That gets around the coarse I/O clock that exists to deny Workers a high-resolution timer. It also contradicts the description's statement that precise timestamps "no longer reach user code directly".
  • WorkerTracer::addSpanClose still calls adjustSpanTime(), which overwrites endTime with context.now() on the I/O clock (tracer.c++:534-560). Tail spans therefore have a precise start and a coarse end. The I/O clock does not advance while JavaScript runs, so the end time can be earlier than the start time. Existing tail consumers would then see zero or negative durations.
  • This change is not gated. It takes effect whether or not the OTLP hook or tracing.otlp is configured, which also contradicts "Nothing changes unless an embedder installs the hook or the setting is configured".

Fix: use the precise clock only for observers created by the OTLP export path, for example with a flag or a dedicated observer, and keep IoContext::now() on the tail-stream path.

Apart from this, the untrusted inputs into the Rust encoder are handled safely: UTF-8 is converted lossily, the open-span and batch buffers have size limits, and query strings are redacted by rebuilding url.full.


✅ All other checks passed

📖 AGENTS.md: ✅ LGTM

The addition to src/rust/AGENTS.md is consistent with the rest of that file.


⚡ Performance: ✅ LGTM

The issue of unbounded in-flight OTLP exports is covered in the code quality section. src/rust/otlp/buffer.rs limits open spans to MAX_OPEN_SPANS and limits batches by span count and by bytes.


Reviewed commit: 91d0101e — same changes as 4f06a7ee, so this is that commit's review (no new review was run; line numbers are as of 4f06a7ee)

@clanky-the-clanker clanky-the-clanker Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Superseded by the review of 27fb47ae.

This review, of 855b0965

🛑 Significant concerns — 1 BLOCKING, not yet fixed: UserSpanObserver::getTime() uses the precise system clock for every user span. Streaming tail workers, which are user code, therefore receive precise timestamps, and their spans mix a precise start with a coarse I/O-clock end. There are also 3 code-quality WARNINGs (an ungated tail-span filter, a Durable Object id that is never exported, unbounded OTLP exports) and a stale clock statement in the docs.

Findings: 1 blocking · 4 warnings · 2 info · 1 suggestion

Not opened by the correctness review: 1 changed file it was meant to read
  • src/workerd/server/server/config.rs

The decision is advisory: this review is a comment and never approves, blocks or requests changes on the pull request.

🔧 Fix with your agent

Paste the following into your coding agent to investigate and fix these issues.

Fix the following code review issues found in pull request #7686.

Before making changes, review each issue and present a brief summary table:
- For each finding, state whether you agree, disagree, or need clarification
- If you disagree (e.g., the fix would require disproportionate changes
  for minimal ROI, or the finding is factually incorrect), explain why
- If you need to clarify something before deciding, ask those questions
- Then share your plan for which issues to tackle and in what order

## BLOCKING (1)

### [Security] `UserSpanObserver::getTime()` uses `systemPreciseCalendarClock()` for every user span, including the streaming-tail path. Tail workers (user code) receive precise start and exception timestamps, which gets around the coarse I/O clock. Tail spans also combine a precise start with an I/O-clock end, so durations can be zero or negative. The change is not gated on OTLP export.
- **File:** src/workerd/io/tracer.c++:838

## WARNING (4)

### [Code Quality] The internal-binding fetch filter drops `fetch` spans for D1, Images, AI and Vectorize hosts from every streaming tail worker, not only when OTLP export is enabled. Because it matches by host only, a user's own fetches to those hosts lose their spans too.
- **File:** src/workerd/api/http.c++:1922

### [Code Quality] `OtlpServiceInfo.durableObjectId` is never set, so exported Durable Object spans never carry the `cloudflare.durable_object.id` tag that the header documents.
- **File:** src/workerd/server/factory/worker-factory.c++:1129

### [Code Quality] `exportSpans()` starts a new POST task for every batch, with no limit on how many run at once and no timeout. When the collector is slow or down, memory grows without bound and an ERROR is logged for every batch.
- **File:** src/workerd/server/factory/worker-factory-otlp.c++:27

### [Documentation] `docs/async-trace.md` still says user spans use the Spectre-safe clock, but `UserSpanObserver::getTime()` now uses the precise system clock.
- **File:** docs/async-trace.md:231

## INFO (2)

### [Documentation] `docs/async-trace.md:255` describes spans as always going to tail workers. When `tracing.otlp` is set, they are not delivered to streaming tail workers.
- **File:** docs/async-trace.md:255

### [Documentation] The new `tracing.otlp` setting is described only in the `workerd.capnp` schema comments. A short user guide, for example a local Jaeger setup, would help users find it.

## SUGGESTION (1)

### [Code Quality] The Rust unit tests in `buffer.rs` and `rules.rs` are inline `mod tests` blocks. The repository convention puts them in `buffer-test.rs` and `rules-test.rs` beside each module.
- **File:** src/rust/otlp/buffer.rs:309

📋 Code Quality: ⚠️ 3 warnings · 1 suggestion

All earlier findings are still present: an ungated internal-binding fetch filter, a Durable Object id that is never exported, unbounded OTLP exports, and inline Rust tests.

Recommendation: 📝 Minor Comments

  • [WARNING] The internal-binding fetch filter drops fetch spans for D1, Images, AI and Vectorize hosts from every streaming tail worker, not only when OTLP export is enabled. Because it matches by host only, a user's own fetches to those hosts lose their spans too. (src/workerd/api/http.c++:1922)
  • [WARNING] OtlpServiceInfo.durableObjectId is never set, so exported Durable Object spans never carry the cloudflare.durable_object.id tag that the header documents. (src/workerd/server/factory/worker-factory.c++:1129)
  • [WARNING] exportSpans() starts a new POST task for every batch, with no limit on how many run at once and no timeout. When the collector is slow or down, memory grows without bound and an ERROR is logged for every batch. (src/workerd/server/factory/worker-factory-otlp.c++:27)
  • [SUGGESTION] The Rust unit tests in buffer.rs and rules.rs are inline mod tests blocks. The repository convention puts them in buffer-test.rs and rules-test.rs beside each module. (src/rust/otlp/buffer.rs:309)
View full review

Internal-binding fetch filter applies to every worker (src/workerd/api/http.c++:1616-1629, 1922)

OmitUserSpan(isInternalBindingFetch(...)) applies to every worker, whether or not OTLP export is enabled. Every streaming tail worker stops getting fetch spans for these hosts; the D1 and Images instrumentation tests were updated to match. The function's own comment says it also drops the span of a user's own fetch to one of these hosts. Commit bcceb70 calls the filter a stopgap. Even so, this is a visible change to tail output, and the repository asks for that compatibility decision to be recorded or put behind a gate. Consider applying it only when OTLP export is active.

Durable Object id is never exported (src/workerd/server/factory/worker-factory.c++:1128-1131)

The OtlpServiceInfo initializer sets only .serviceName and .entrypoint. The durableObjectId computed earlier in that function goes only to the tracer. As a result, cloudflare.durable_object.id, which otlp-span-submitter.h documents, is never emitted.

Unbounded in-flight exports (src/workerd/server/factory/worker-factory-otlp.c++:25-32)

Each batch adds a TaskSet task that opens a new client and POSTs up to about 1 MiB. Nothing limits how many run at once, and there is no timeout. When the collector is slow or hung, tasks and their request bodies pile up without bound, and each failure logs an ERROR. Add a cap on in-flight requests that drops batches beyond it, plus a timeout.

Inline Rust tests (src/rust/otlp/buffer.rs:309, src/rust/otlp/rules.rs:84)

The repository convention (src/rust/AGENTS.md) is a <module>-test.rs file beside each module, not an inline mod tests block.


📝 Documentation: ⚠️ 1 warning · 2 info

The docs still describe user spans as using the Spectre-safe clock, the export-path description leaves out the OTLP mode, and tracing.otlp has no guide.

Recommendation: 📝 Minor Comments

  • [WARNING] docs/async-trace.md still says user spans use the Spectre-safe clock, but UserSpanObserver::getTime() now uses the precise system clock. (docs/async-trace.md:231)
  • [INFO] docs/async-trace.md:255 describes spans as always going to tail workers. When tracing.otlp is set, they are not delivered to streaming tail workers. (docs/async-trace.md:255)
  • [INFO] The new tracing.otlp setting is described only in the workerd.capnp schema comments. A short user guide, for example a local Jaeger setup, would help users find it.
View full review
  • docs/async-trace.md:231-234 says user-facing spans use the Spectre-safe clock. UserSpanObserver::getTime() now uses the precise clock. Update this text once the clock question in the security section is settled.
  • docs/async-trace.md:255-256 says spans go to tail workers and trace export. When tracing.otlp is set, spans are not delivered to streaming tail workers (workerd.capnp:181-182). Consider qualifying the sentence.
  • tracing.otlp is documented only in the workerd.capnp schema comments. A short guide in docs/, such as pointing a local workerd at Jaeger, would help users find it.

🔒 Security: 🛑 1 blocking

Not yet fixed: precise timestamps reach streaming tail workers, and tail spans mix two clocks.

Recommendation: 🔴 Serious Concerns

  • [BLOCKING] UserSpanObserver::getTime() uses systemPreciseCalendarClock() for every user span, including the streaming-tail path. Tail workers (user code) receive precise start and exception timestamps, which gets around the coarse I/O clock. Tail spans also combine a precise start with an I/O-clock end, so durations can be zero or negative. The change is not gated on OTLP export. (src/workerd/io/tracer.c++:838)
View full review

🔄 Since last review: 0 findings resolved, all preserved. The only file changed since then is src/workerd/server/server/config.rs, which moves the OTLP setup into OtlpTracing::from_config. That change raises no new concerns.

Precise timestamps reach tail workers, and tail spans mix two clocks (src/workerd/io/tracer.c++:837-840)

  • UserSpanObserver::getTime() returns kj::systemPreciseCalendarClock().now(). This observer is not specific to OTLP: streaming tail workers use it too.
  • WorkerTracer::addSpanOpen (tracer.c++:213-234) passes startTime to tailStreamWriter->report(...) unchanged. A worker can wrap any code in enterSpan() and read nanosecond start times in its tail worker. That gets around the coarse I/O clock, which exists to keep high-resolution timers away from Workers. It also contradicts the description's statement that precise timestamps "no longer reach user code directly".
  • addSpanClose still calls adjustSpanTime(). That function overwrites endTime with context.now() from the I/O clock (tracer.c++:545-547). A tail span therefore gets a precise start and a coarse end. The I/O clock does not advance while JavaScript runs, so the end can come before the start, giving zero or negative durations.
  • The change is not gated on OTLP export.

Fix: use the precise clock only for spans that go to the OTLP submitter (for example, let each submitter or observer choose its clock). Otherwise, coarsen timestamps before they reach the tail stream. Either way, start, end and exception times of one span should come from the same clock.


✅ All other checks passed

📖 AGENTS.md: ✅ LGTM

The new line in src/rust/AGENTS.md matches the rest of the guidance. The inline Rust tests that go against its test-file convention are reported in the code quality section.


⚡ Performance: ✅ LGTM

src/rust/otlp/buffer.rs caps open spans at 1024 and batches at 512 spans or 1 MiB, and tracks the encoded size as spans are added. The unbounded concurrency of OTLP POSTs is reported in the code quality section.


Reviewed commit: 855b0965

@clanky-the-clanker clanky-the-clanker Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Superseded by the review of 5708482f.

This review, of 27fb47ae

⚠️ Minor issues — The earlier blocker is fixed: streaming-tail spans use the I/O clock again, and only the OTLP submitter reads the precise clock. Four WARNINGs with runtime impact remain: span exception timestamps can still reach tail workers at full precision, the binding fetch filter changes tail output for every worker, the Durable Object id is never exported, and OTLP exports have no cap and no timeout.

Findings: 4 warnings · 2 info (1 is an inline comment on the diff)

Not opened by the correctness review: 3 changed files it was meant to read
  • src/workerd/io/BUILD.bazel
  • src/workerd/io/tracer.h
  • src/workerd/server/server/config.rs

The decision is advisory: this review is a comment and never approves, blocks or requests changes on the pull request.

🔧 Fix with your agent

Paste the following into your coding agent to investigate and fix these issues.

Fix the following code review issues found in pull request #7686.

Before making changes, review each issue and present a brief summary table:
- For each finding, state whether you agree, disagree, or need clarification
- If you disagree (e.g., the fix would require disproportionate changes
  for minimal ROI, or the finding is factually incorrect), explain why
- If you need to clarify something before deciding, ask those questions
- Then share your plan for which issues to tackle and in what order

## WARNING (4)

### [Security] When the embedder's OTLP invocation-span hook is installed and the worker has a streaming tail worker, a span exception's timestamp comes from the precise OTLP observer and is forwarded unchanged to the tail stream. The tail worker (user code) therefore sees a precise timestamp.
- **File:** src/workerd/io/tracer.c++:353

### [Code Quality] The internal-binding fetch filter drops `fetch` spans for the D1, AI, Images, Vectorize, Workflow and test hosts from every streaming tail worker, not only when OTLP export is on. Because it matches by host only, a user's own fetches to those hosts lose their spans too.
- **File:** src/workerd/api/http.c++:1922

### [Code Quality] `OtlpServiceInfo.durableObjectId` is never set, so exported Durable Object spans never carry `cloudflare.durable_object.id`.
- **File:** src/workerd/server/factory/worker-factory.c++:1129

### [Code Quality] `exportSpans()` starts a new POST task for every batch, with no limit on how many run at once and no timeout. A slow or hung collector makes memory grow without bound.
- **File:** src/workerd/server/factory/worker-factory-otlp.c++:27

## INFO (2)

### [Documentation] The Timing section should say that spans exported through native OTLP use precise timestamps, while streaming-tail spans keep the Spectre-safe I/O clock.
- **File:** docs/async-trace.md:231

### [Documentation] The Outputs section implies spans always go to tail workers. With `tracing.otlp` configured, they are not delivered to streaming tail workers.
- **File:** docs/async-trace.md:255

📋 Code Quality: ⚠️ 3 warnings

Three earlier warnings are still present: the binding fetch filter applies to every worker, the Durable Object id is never exported, and OTLP exports have no cap and no timeout.

Recommendation: 📝 Minor Comments

  • [WARNING] The internal-binding fetch filter drops fetch spans for the D1, AI, Images, Vectorize, Workflow and test hosts from every streaming tail worker, not only when OTLP export is on. Because it matches by host only, a user's own fetches to those hosts lose their spans too. (src/workerd/api/http.c++:1922)
  • [WARNING] OtlpServiceInfo.durableObjectId is never set, so exported Durable Object spans never carry cloudflare.durable_object.id. (src/workerd/server/factory/worker-factory.c++:1129)
  • [WARNING] exportSpans() starts a new POST task for every batch, with no limit on how many run at once and no timeout. A slow or hung collector makes memory grow without bound. (src/workerd/server/factory/worker-factory-otlp.c++:27)
View full review

Internal-binding fetch filter applies to every worker (src/workerd/api/http.c++:1922). OmitUserSpan(isInternalBindingFetch(...)) runs whether or not OTLP export is on. Every streaming tail worker loses the fetch spans for these hosts. Because the filter matches by host only, a user's own fetch() to one of them (for example the bare hosts d1 or vector-search) loses its span too. The D1 and Images instrumentation tests were edited to match, so this is a visible change to tail output. The repository's compatibility rules ask for that decision to be recorded or put behind a flag. Consider applying the filter only when OTLP export is active, or only to the binding's own internal fetcher.

Durable Object id is never exported (src/workerd/server/factory/worker-factory.c++:1129-1132). The OtlpServiceInfo initializer sets only .serviceName and .entrypoint. The durableObjectId computed at line 1086 is moved into the tracer at line 1100. As a result, cloudflare.durable_object.id, which otlp-span-submitter.h:67 declares, is never emitted.

Unbounded in-flight exports (src/workerd/server/factory/worker-factory-otlp.c++:27-32). Every batch adds a TaskSet task that POSTs a body of up to about 1 MiB. Nothing limits how many run at once, and there is no timeout. If the collector is slow or hung, tasks and their bodies pile up without bound, and each failure logs an ERROR. Cap the number of in-flight requests, drop batches beyond the cap (reporting them as dropped spans), and add a timeout.

One earlier note is still open and unchanged: the inline Rust tests in buffer.rs and rules.rs.


📝 Documentation: 💬 2 info

Two passages in docs/async-trace.md should mention native OTLP export.

Recommendation: 💬 Optional Nits

  • [INFO] The Timing section should say that spans exported through native OTLP use precise timestamps, while streaming-tail spans keep the Spectre-safe I/O clock. (docs/async-trace.md:231)
  • [INFO] The Outputs section implies spans always go to tail workers. With tracing.otlp configured, they are not delivered to streaming tail workers. (docs/async-trace.md:255)
View full review
  • docs/async-trace.md:231-234: streaming-tail spans use the Spectre-safe clock again, so the text is accurate for spans user code can see. Spans exported through native OTLP use precise timestamps, though, and the passage should say so.
  • docs/async-trace.md:255-256: when tracing.otlp is set, spans are not delivered to streaming tail workers (workerd.capnp:181-182). Consider qualifying this sentence.

One earlier note is still open: a short guide to the tracing.otlp setting.


🔒 Security: ⚠️ 1 warning

The precise-clock blocker is fixed. One path remains: span exception timestamps can still reach tail workers at full precision.

Recommendation: 📝 Minor Comments

  • [WARNING] When the embedder's OTLP invocation-span hook is installed and the worker has a streaming tail worker, a span exception's timestamp comes from the precise OTLP observer and is forwarded unchanged to the tail stream. The tail worker (user code) therefore sees a precise timestamp. (src/workerd/io/tracer.c++:353)
View full review

🔄 Since last review: 1 finding resolved, 4 preserved, 1 new

Earlier BLOCKING is resolved. SpanSubmitter::getTime() now defaults to IoContext::current().now() (src/workerd/io/tracer.c++:779). UserSpanObserver::getTime() delegates to its submitter (tracer.c++:841-842). Only OtlpSpanSubmitter returns the precise system clock. Streaming-tail spans therefore get I/O-clock times at both ends again, and the mixed precise-start/coarse-end problem is gone.

Remaining leak: span exception timestamps. recordExceptionImpl (src/workerd/api/tracing.c++:301) takes its timestamp from the parent span's observer. When the invocation's root span comes from the embedder's OTLP hook, that observer is precise. WorkerTracer::addSpanException then hands the timestamp unchanged to reportExceptionToTailStream (tracer.c++:353). This happens whenever the hook is installed and the worker also has a streaming tail worker: the tail worker receives a nanosecond-precision timestamp for each span exception. That contradicts the stated goal that precise timestamps no longer reach user code. Fix: send getTime() (the I/O clock) to the tail stream and keep the precise value only for the OTLP submitter. Also add a test with an OTLP-backed root and a tail writer.


✅ All other checks passed

📖 AGENTS.md: ✅ LGTM

The new line in src/rust/AGENTS.md fits the existing guidance.


⚡ Performance: ✅ LGTM

The Rust SpanBuffer is bounded: at most 1024 open spans, with batches cut at 512 spans or 1 MiB, and it tracks the encoded size as spans are added. finish() sorts at most 1024 spans once per invocation.


Reviewed commit: 27fb47ae

Comment thread src/workerd/io/tracer.c++ Resolved

@clanky-the-clanker clanky-the-clanker Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Superseded by the review of 78bb0518.

This review, of 5708482f

💬 Approved with comments — The earlier precise-timestamp leak to tail workers is fixed. Two WARNINGs remain in the opt-in workerd OTLP exporter (Durable Object id never set; exports have no concurrency cap and no timeout), plus INFO notes on the binding fetch filter and the docs.

Findings: 2 warnings · 4 info (2 are inline comments on the diff)

The decision is advisory: this review is a comment and never approves, blocks or requests changes on the pull request.

🔧 Fix with your agent

Paste the following into your coding agent to investigate and fix these issues.

Fix the following code review issues found in pull request #7686.

Before making changes, review each issue and present a brief summary table:
- For each finding, state whether you agree, disagree, or need clarification
- If you disagree (e.g., the fix would require disproportionate changes
  for minimal ROI, or the finding is factually incorrect), explain why
- If you need to clarify something before deciding, ask those questions
- Then share your plan for which issues to tackle and in what order

## WARNING (2)

### [Code Quality] `OtlpServiceInfo.durableObjectId` is never set, so spans exported for Durable Objects never carry `cloudflare.durable_object.id`.
- **File:** src/workerd/server/factory/worker-factory.c++:1129

### [Code Quality] `exportSpans()` starts one POST task per batch with no cap on how many run at once and no timeout. A slow or hung collector makes memory grow without bound.
- **File:** src/workerd/server/factory/worker-factory-otlp.c++:27

## INFO (4)

### [Code Quality] The internal-binding fetch filter drops `fetch` spans for these hosts from every streaming tail worker and matches on host alone. Lowered to INFO: @wperron [non-author] says this is the current and expected behavior. Consider recording the compatibility decision in the commit message.
- **File:** src/workerd/api/http.c++:1922

### [Documentation] The Timing section should say that spans exported through native OTLP use precise timestamps.
- **File:** docs/async-trace.md:231

### [Documentation] The Outputs section should say that spans are not delivered to streaming tail workers when `tracing.otlp` is configured.
- **File:** docs/async-trace.md:255

### [Documentation] No Markdown guide explains how to configure the new `tracing.otlp` setting.

📋 Code Quality: ⚠️ 2 warnings · 1 info

Two earlier warnings in the workerd OTLP exporter are still present; the binding fetch filter is now an INFO.

Recommendation: 📝 Minor Comments

  • [WARNING] OtlpServiceInfo.durableObjectId is never set, so spans exported for Durable Objects never carry cloudflare.durable_object.id. (src/workerd/server/factory/worker-factory.c++:1129)
  • [WARNING] exportSpans() starts one POST task per batch with no cap on how many run at once and no timeout. A slow or hung collector makes memory grow without bound. (src/workerd/server/factory/worker-factory-otlp.c++:27)
  • [INFO] The internal-binding fetch filter drops fetch spans for these hosts from every streaming tail worker and matches on host alone. Lowered to INFO: @​wperron [non-author] says this is the current and expected behavior. Consider recording the compatibility decision in the commit message. (src/workerd/api/http.c++:1922)
View full review

Durable Object id is never exported (src/workerd/server/factory/worker-factory.c++:1129-1132). durableObjectId is computed at line 1086 and moved into the Trace at line 1100. The OtlpServiceInfo initializer sets only .serviceName and .entrypoint, so makeOtlpIdentityTags (otlp-span-submitter.c++:150) never emits cloudflare.durable_object.id.

Unbounded in-flight exports (src/workerd/server/factory/worker-factory-otlp.c++:25-30). Each batch adds a TaskSet task that POSTs a body of up to about 1 MiB. Nothing caps how many run at once, and postOtlpTraces sets no timeout. With a slow or hung collector, tasks and request bodies pile up until shutdown.

Internal-binding fetch filter (src/workerd/api/http.c++:1922). The filter runs for every worker and matches on host alone. @​wperron [non-author] replied that this is the current and expected behavior, so it is noted here as INFO. The tail output change, which the D1 and Images instrumentation tests now reflect, is worth recording in the commit message as an accepted compatibility decision.


📝 Documentation: 💬 3 info

docs/async-trace.md should cover native OTLP timing, tail delivery, and the tracing.otlp setting.

Recommendation: 💬 Optional Nits

  • [INFO] The Timing section should say that spans exported through native OTLP use precise timestamps. (docs/async-trace.md:231)
  • [INFO] The Outputs section should say that spans are not delivered to streaming tail workers when tracing.otlp is configured. (docs/async-trace.md:255)
  • [INFO] No Markdown guide explains how to configure the new tracing.otlp setting.
View full review
  • docs/async-trace.md:231-234: the passage is accurate for spans delivered to tail workers. Spans exported through native OTLP use systemPreciseCalendarClock() (otlp-span-submitter.c++:204-205), and the passage should say so.
  • docs/async-trace.md:255-256: when tracing.otlp is set, spans are not delivered to streaming tail workers. Qualify this sentence.
  • There is still no short guide to configuring tracing.otlp (collector service, sampling, tail interaction).

✅ All other checks passed

🔒 Security: ✅ LGTM

recordExceptionImpl (src/workerd/api/tracing.c++:300-303) now passes the I/O-clock time from IoContext::current().now() to addSpanException, and that is what goes to the tail stream. The native OTLP root records its own precise timestamp separately. Tail workers therefore no longer receive precise exception timestamps.


📖 AGENTS.md: ✅ LGTM

The new line in src/rust/AGENTS.md fits the existing guidance.


⚡ Performance: ✅ LGTM

No performance issues found. The unbounded in-flight export concern is under Code Quality.


Reviewed commit: 5708482f

@wperron
wperron force-pushed the dlapid/user-tracing-otlp branch from 5708482 to 6f1f0e2 Compare November 2, 2026 14:58

@clanky-the-clanker clanky-the-clanker Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Superseded by the review of b33dc58b.

This review, of 78bb0518

💬 Approved with comments — Two WARNINGs remain in the opt-in workerd OTLP exporter: the Durable Object id is never exported, and exports have no concurrency cap and no timeout. There are also INFO notes on the binding fetch filter and the docs, and no BLOCKING issues.

Findings: 2 warnings · 1 info

The decision is advisory: this review is a comment and never approves, blocks or requests changes on the pull request.

🔧 Fix with your agent

Paste the following into your coding agent to investigate and fix these issues.

Fix the following code review issues found in pull request #7686.

Before making changes, review each issue and present a brief summary table:
- For each finding, state whether you agree, disagree, or need clarification
- If you disagree (e.g., the fix would require disproportionate changes
  for minimal ROI, or the finding is factually incorrect), explain why
- If you need to clarify something before deciding, ask those questions
- Then share your plan for which issues to tackle and in what order

## WARNING (2)

### [Code Quality] `OtlpServiceInfo.durableObjectId` is never set, so exported Durable Object spans never carry `cloudflare.durable_object.id`.
- **File:** src/workerd/server/factory/worker-factory.c++:1129

### [Code Quality] `exportSpans()` starts one POST task per batch with no concurrency cap and no timeout, so a slow or hung collector makes memory grow without bound.
- **File:** src/workerd/server/factory/worker-factory-otlp.c++:27

## INFO (1)

### [Code Quality] The internal-binding fetch filter drops `fetch` spans for D1, Images, AI and Vectorize hosts from every streaming tail worker, matching on host alone. Kept as INFO because @wperron [non-author] says this is the current and expected behavior. Consider recording the compatibility decision in the commit message.
- **File:** src/workerd/api/http.c++:1922

📋 Code Quality: ⚠️ 2 warnings · 1 info

Both earlier WARNINGs are still present in the OTLP exporter, and the fetch-filter INFO stays open.

Recommendation: 📝 Minor Comments

  • [WARNING] OtlpServiceInfo.durableObjectId is never set, so exported Durable Object spans never carry cloudflare.durable_object.id. (src/workerd/server/factory/worker-factory.c++:1129)
  • [WARNING] exportSpans() starts one POST task per batch with no concurrency cap and no timeout, so a slow or hung collector makes memory grow without bound. (src/workerd/server/factory/worker-factory-otlp.c++:27)
  • [INFO] The internal-binding fetch filter drops fetch spans for D1, Images, AI and Vectorize hosts from every streaming tail worker, matching on host alone. Kept as INFO because @​wperron [non-author] says this is the current and expected behavior. Consider recording the compatibility decision in the commit message. (src/workerd/api/http.c++:1922)
View full review

Durable Object id is never exported (src/workerd/server/factory/worker-factory.c++:1129-1132)

  • durableObjectId is computed earlier in the function and moved into the Trace.
  • The OtlpServiceInfo initializer sets only .serviceName and .entrypoint, so makeOtlpIdentityTags never emits cloudflare.durable_object.id.

Unbounded in-flight exports (src/workerd/server/factory/worker-factory-otlp.c++:25-30)

  • Each batch adds a TaskSet task that POSTs a body of up to about 1 MiB.
  • Nothing limits how many tasks run at once, and the POST has no timeout.
  • With a stalled collector, tasks and their request bodies pile up until shutdown.
  • The new server test checks cancellation at shutdown. It does not check how much builds up while the server is running.

Internal-binding fetch filter (src/workerd/api/http.c++:1922)

  • The filter matches on host only and applies to every streaming tail worker.
  • It stays INFO because @​wperron says this is the current and expected behavior.
  • The commit message should record it as an accepted compatibility decision.

✅ All other checks passed

🔒 Security: ✅ LGTM

  • Tail workers still get timestamps from the request's I/O clock (src/workerd/api/tracing.c++:300-303). Only the native OTLP path uses the precise clock.
  • The collector is chosen by the operator's config and reached through a service channel at a fixed path, so user code controls neither the destination nor the path.

📖 AGENTS.md: ✅ LGTM

The new line in src/rust/AGENTS.md fits the existing guidance.


⚡ Performance: ✅ LGTM

No performance issues found. The unbounded in-flight export concern is covered under Code Quality.


📝 Documentation: ✅ LGTM

  • docs/async-trace.md:255-256: this passage says spans go to tail workers, but when tracing.otlp is configured, spans are not delivered to streaming tail workers. Qualify the statement.
  • No Markdown guide explains how to configure tracing.otlp: the collector service, sampling, and how it interacts with tail workers.

Reviewed commit: 78bb0518

@clanky-the-clanker clanky-the-clanker Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Superseded by the review of f301dcee.

This review, of b33dc58b

💬 Approved with comments — Both earlier WARNINGs are fixed: Durable Object IDs are now exported, and standalone exports are capped in concurrency and queue size and have a timeout. What remains is one INFO and one SUGGESTION on the exporter, and nothing blocks.

Findings: 1 info · 1 suggestion (2 are inline comments on the diff)

The decision is advisory: this review is a comment and never approves, blocks or requests changes on the pull request.


📋 Code Quality: 💬 1 suggestion

Both earlier warnings are fixed. One optional note remains: failed exports are logged at ERROR on every failure, with no throttling.

Recommendation: 💬 Optional Nits

  • [SUGGESTION] Every failed OTLP batch is logged at ERROR with no throttling (src/workerd/server/factory/worker-factory-otlp.c++:100)
View full review

🔄 Since last review: 2 findings resolved, 0 preserved, 1 new

Earlier findings, checked against this commit:

  • Durable Object ID: fixed. durableObjectId is now passed as OtlpServiceInfo.durableObjectId and emitted by makeOtlpIdentityTags. The test checks both that the ID is present and that it is absent where it should be.
  • Unbounded exports: fixed. worker-factory-otlp.c++ now allows at most 16 POSTs at once, queues at most 4096 messages, batches up to about 1 MiB and times each request out after 10 s. unlink() cancels queued work at shutdown, and server-test.c++ covers each of these limits.
  • Internal-binding fetch filter (http.c++): unchanged, and @​wperron says this is the current and expected behaviour. One earlier note on it remains open.

New note:

  • taskFailed logs every failed batch at ERROR with no throttling. If tracing.otlp points at a collector that isn't running, every invocation adds a log line. The queue-full path already uses LOG_WARNING_PERIODICALLY.

⚡ Performance: 💬 1 info

The export queue is limited by message count but not by total bytes.

Recommendation: 💬 Optional Nits

  • [INFO] The export queue is limited by message count (4096) but not by total bytes (src/workerd/server/factory/worker-factory-otlp.c++:47)
View full review

The earlier concern about unbounded in-flight exports is resolved.

Queue limit counts messages, not bytes

  • QUEUE_SIZE limits the number of queued messages, not their size.
  • If the collector is slow or down and messages are large, the queue can hold far more memory than the 4096 limit suggests.
  • Typical messages are small, so this is unlikely to matter in practice.
  • A cheap guard: track the bytes pending and drop new spans once they pass a limit.

✅ All other checks passed

🔒 Security: ✅ LGTM

  • Collector destination: it comes from the operator's config and is reached through a service channel at the fixed path /v1/traces. User code controls neither the destination nor the path.
  • Timestamps: only the native OTLP submitter uses the precise clock. Tail workers keep the request's coarsened I/O clock.
  • Resource use: the exporter caps in-flight requests, queue length and request duration, and it drops overflow.
  • Shutdown: unlink() disables later exports before it cancels pending work.

📖 AGENTS.md: ✅ LGTM

The new line in src/rust/AGENTS.md is consistent with the guidance already there.


📝 Documentation: ✅ LGTM

No new documentation issues. The two documentation notes from earlier reviews (that tracing.otlp changes delivery to streaming tail workers, and that no guide covers the new setting) are still open, and they are the author's call.


Reviewed commit: b33dc58b

@wperron
wperron force-pushed the dlapid/user-tracing-otlp branch from b33dc58 to f301dce Compare November 2, 2026 19:25

@clanky-the-clanker clanky-the-clanker Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Superseded by the review of 71fec26e.

This review, of 7f6a7935

💬 Approved with comments — One WARNING is still open: with ExportSpans::NO and a streaming tail, a user span's updates and exceptions change the exported invocation span. It only affects embedders that use that mode.

Findings: 1 warning

Not opened by the correctness review: 5 changed files it was meant to read
  • src/opentelemetry/proto/collector/logs/v1/logs_service.proto
  • src/opentelemetry/proto/collector/trace/v1/trace_service.proto
  • src/opentelemetry/proto/common/v1/common.proto
  • src/opentelemetry/proto/resource/v1/resource.proto
  • src/opentelemetry/proto/trace/v1/trace.proto

The decision is advisory: this review is a comment and never approves, blocks or requests changes on the pull request.

🔧 Fix with your agent

Paste the following into your coding agent to investigate and fix these issues.

Fix the following code review issues found in pull request #7686.

Before making changes, review each issue and present a brief summary table:
- For each finding, state whether you agree, disagree, or need clarification
- If you disagree (e.g., the fix would require disproportionate changes
  for minimal ROI, or the finding is factually incorrect), explain why
- If you need to clarify something before deciding, ask those questions
- Then share your plan for which issues to tackle and in what order

## WARNING (1)

### [Code Quality] With ExportSpans::NO and a streaming tail, user span renames, status changes and exceptions are recorded on the exported invocation span, because addSpanUpdate and addSpanException never check the span id.
- **File:** src/workerd/io/tracer.c++:282

📋 Code Quality: ⚠️ 1 warning

One warning is still open: user span updates and exceptions are recorded on the exported invocation span when ExportSpans::NO is used.

Recommendation: 📝 Minor Comments

  • [WARNING] With ExportSpans::NO and a streaming tail, user span renames, status changes and exceptions are recorded on the exported invocation span, because addSpanUpdate and addSpanException never check the span id. (src/workerd/io/tracer.c++:282)
View full review

Open: WorkerTracer::addSpanUpdate and addSpanException apply every span id to the invocation span

  • src/workerd/io/tracer.c++:282 calls updateInvocationSpan(update) without checking spanId.
  • src/workerd/io/tracer.c++:370 calls recordInvocationSpanException(...) without checking it either.
  • With ExportSpans::NO, user spans go through SequentialSpanSubmitter (worker-factory.c++). It forwards each user span's update and exception with that user span's own id.
  • So a user span's updateName() renames the exported invocation span, its setStatus() overwrites that span's status, and its exceptions are added to that span.
  • recordInvocationLevelException already compares the id with SpanParent(invocationSpan).getSpanId(). Both of these paths need the same check.
  • The NO-mode tests in tracer-test.c++ would catch a regression here if they also sent a user-span update and a user-span exception.

✅ All other checks passed

🔒 Security: ✅ LGTM

  • Timestamps: Exception timestamps in the streaming tail still come from the I/O clock. Only the native OTLP span records a precise time.
  • Collector: The collector is set in the operator's configuration, and exports go only to the fixed /v1/traces and /v1/logs paths.
  • Headers: Exported request headers are limited to an allowlist.

📖 AGENTS.md: ✅ LGTM

The AGENTS.md additions agree with the repository's existing guidance.


⚡ Performance: ✅ LGTM

The export queue and the span and log buffers have fixed limits.


📝 Documentation: ✅ LGTM

The tracing.otlp schema docs and src/opentelemetry/AGENTS.md match the code.


Reviewed commit: 7f6a7935

@clanky-the-clanker clanky-the-clanker Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Superseded by the review of bf27d046.

This review, of 71fec26e

💬 Approved with comments — One WARNING is still open: with ExportSpans::NO and a streaming tail, a user span's updates and exceptions change the exported invocation span. Only embedders that use that mode are affected.

Findings: 1 warning

Not opened by the correctness review: 5 changed files it was meant to read
  • src/opentelemetry/proto/collector/logs/v1/logs_service.proto
  • src/opentelemetry/proto/collector/trace/v1/trace_service.proto
  • src/opentelemetry/proto/common/v1/common.proto
  • src/opentelemetry/proto/resource/v1/resource.proto
  • src/opentelemetry/proto/trace/v1/trace.proto

The decision is advisory: this review is a comment and never approves, blocks or requests changes on the pull request.

🔧 Fix with your agent

Paste the following into your coding agent to investigate and fix these issues.

Fix the following code review issues found in pull request #7686.

Before making changes, review each issue and present a brief summary table:
- For each finding, state whether you agree, disagree, or need clarification
- If you disagree (e.g., the fix would require disproportionate changes
  for minimal ROI, or the finding is factually incorrect), explain why
- If you need to clarify something before deciding, ask those questions
- Then share your plan for which issues to tackle and in what order

## WARNING (1)

### [Code Quality] With ExportSpans::NO and a streaming tail, user-span renames, status changes and exceptions are recorded on the exported invocation span, because addSpanUpdate and addSpanException never check the span id.
- **File:** src/workerd/io/tracer.c++:282

📋 Code Quality: ⚠️ 1 warning

Still open: user-span updates and exceptions change the exported invocation span in ExportSpans::NO mode.

Recommendation: 📝 Minor Comments

  • [WARNING] With ExportSpans::NO and a streaming tail, user-span renames, status changes and exceptions are recorded on the exported invocation span, because addSpanUpdate and addSpanException never check the span id. (src/workerd/io/tracer.c++:282)
View full review

WorkerTracer::addSpanUpdate and addSpanException ignore the span id (src/workerd/io/tracer.c++:282, :370)

  • addSpanUpdate calls updateInvocationSpan(update) and never compares spanId with the invocation span's id. addSpanException calls recordInvocationSpanException(...) with the same gap.
  • With ExportSpans::NO, user spans go through the SequentialSpanSubmitter in worker-factory.c++. It forwards each user span's update and exception with that user span's own id.
  • So a user span's updateName() renames the exported invocation span, setStatus() overwrites its status, and recorded exceptions are added to it. resolve_status in buffer.rs then turns those exceptions into an error status.
  • recordInvocationLevelException (tracer.c++:762) already checks context.getSpanId() != SpanParent(invocationSpan).getSpanId(). Both of these paths need the same check.
  • The NO-mode cases in tracer-test.c++ would catch this if they also sent a user-span update and a user-span exception.

The Rust test moves in buffer.rs/logs.rs and the changes to otlp-export-queue-test.c++ add no defects.


✅ All other checks passed

🔒 Security: ✅ LGTM

  • Collector: Only the operator's configuration sets the collector. Exports go to the fixed /v1/traces and /v1/logs paths.
  • Timestamps: Exception timestamps in the streaming tail come from the I/O clock. Only the native OTLP span gets precise time.
  • Input: The URL is parsed with ada_url. When query redaction is on, the query and fragment are removed.
  • Limits: The span and log buffers are bounded.

📖 AGENTS.md: ✅ LGTM

The AGENTS.md additions agree with the repository's existing guidance.


⚡ Performance: ✅ LGTM

  • Buffers: The span and log buffers cap open spans at 1024. Each batch is at most 512 items or 1 MiB, and batch sizes are tracked incrementally.
  • Export queue: The export queue has a fixed size, a concurrency limit and a timeout.

📝 Documentation: ✅ LGTM

The tracing.otlp schema docs and src/opentelemetry/AGENTS.md match the code.


Reviewed commit: 71fec26e

@wperron
wperron force-pushed the dlapid/user-tracing-otlp branch from 71fec26 to bf27d04 Compare November 18, 2026 18:27

@clanky-the-clanker clanky-the-clanker Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Superseded by the review of 4989c65d.

This review, of bf27d046

💬 Approved with comments — One WARNING is still open: with ExportSpans::NO and a streaming tail, a user span's updates and exceptions change the exported invocation span. Only embedders that use that mode are affected.

Findings: 1 warning

Not opened by the correctness review: 5 changed files it was meant to read
  • src/opentelemetry/proto/collector/logs/v1/logs_service.proto
  • src/opentelemetry/proto/collector/trace/v1/trace_service.proto
  • src/opentelemetry/proto/common/v1/common.proto
  • src/opentelemetry/proto/resource/v1/resource.proto
  • src/opentelemetry/proto/trace/v1/trace.proto

The decision is advisory: this review is a comment and never approves, blocks or requests changes on the pull request.

🔧 Fix with your agent

Paste the following into your coding agent to investigate and fix these issues.

Fix the following code review issues found in pull request #7686.

Before making changes, review each issue and present a brief summary table:
- For each finding, state whether you agree, disagree, or need clarification
- If you disagree (e.g., the fix would require disproportionate changes
  for minimal ROI, or the finding is factually incorrect), explain why
- If you need to clarify something before deciding, ask those questions
- Then share your plan for which issues to tackle and in what order

## WARNING (1)

### [Code Quality] With ExportSpans::NO and a streaming tail, user-span renames, status changes and exceptions are recorded on the exported invocation span, because addSpanUpdate and addSpanException never check the span id.
- **File:** src/workerd/io/tracer.c++:282

📋 Code Quality: ⚠️ 1 warning

Still open: user-span updates and exceptions change the exported invocation span in ExportSpans::NO mode.

Recommendation: 📝 Minor Comments

  • [WARNING] With ExportSpans::NO and a streaming tail, user-span renames, status changes and exceptions are recorded on the exported invocation span, because addSpanUpdate and addSpanException never check the span id. (src/workerd/io/tracer.c++:282)
View full review

WorkerTracer::addSpanUpdate and addSpanException ignore the span id (src/workerd/io/tracer.c++:282, :370)

  • addSpanUpdate calls updateInvocationSpan(update) and never compares spanId with the invocation span's id. addSpanException calls recordInvocationSpanException(...) with the same gap.
  • With ExportSpans::NO, user spans go through the SequentialSpanSubmitter in worker-factory.c++. It forwards each user span's update and exception with that user span's own id.
  • So a user span's updateName() renames the exported invocation span, setStatus() overwrites its status, and recorded exceptions are added to it. resolve_status in buffer.rs then turns those exceptions into an error status.
  • recordInvocationLevelException (tracer.c++:762) already checks context.getSpanId() != SpanParent(invocationSpan).getSpanId(). Both of these paths need the same check.
  • The NO-mode cases in tracer-test.c++ would catch this if they also sent a user-span update and a user-span exception.

The Rust test moves in buffer.rs/logs.rs and the changes to otlp-export-queue-test.c++ add no defects.


✅ All other checks passed

🔒 Security: ✅ LGTM

  • Collector: Only the operator's configuration sets the collector. Exports go to the fixed /v1/traces and /v1/logs paths.
  • Timestamps: Exception timestamps in the streaming tail come from the I/O clock. Only the native OTLP span gets precise time.
  • Input: The URL is parsed with ada_url. When query redaction is on, the query and fragment are removed.
  • Limits: The span and log buffers are bounded.

📖 AGENTS.md: ✅ LGTM

The AGENTS.md additions agree with the repository's existing guidance.


⚡ Performance: ✅ LGTM

  • Buffers: The span and log buffers cap open spans at 1024. Each batch is at most 512 items or 1 MiB, and batch sizes are tracked incrementally.
  • Export queue: The export queue has a fixed size, a concurrency limit and a timeout.

📝 Documentation: ✅ LGTM

The tracing.otlp schema docs and src/opentelemetry/AGENTS.md match the code.


Reviewed commit: bf27d046 — same changes as 71fec26e, so this is that commit's review (no new review was run; line numbers are as of 71fec26e)

@wperron
wperron force-pushed the dlapid/user-tracing-otlp branch from bf27d04 to 4989c65 Compare November 18, 2026 18:56

@clanky-the-clanker clanky-the-clanker Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Superseded by the review of e49b85d2.

This review, of 4989c65d

💬 Approved with comments — One WARNING is still open: with ExportSpans::NO and a streaming tail, a user span's updates and exceptions change the exported invocation span. Only embedders that use that mode are affected.

Findings: 1 warning

Not opened by the correctness review: 5 changed files it was meant to read
  • src/opentelemetry/proto/collector/logs/v1/logs_service.proto
  • src/opentelemetry/proto/collector/trace/v1/trace_service.proto
  • src/opentelemetry/proto/common/v1/common.proto
  • src/opentelemetry/proto/resource/v1/resource.proto
  • src/opentelemetry/proto/trace/v1/trace.proto

The decision is advisory: this review is a comment and never approves, blocks or requests changes on the pull request.

🔧 Fix with your agent

Paste the following into your coding agent to investigate and fix these issues.

Fix the following code review issues found in pull request #7686.

Before making changes, review each issue and present a brief summary table:
- For each finding, state whether you agree, disagree, or need clarification
- If you disagree (e.g., the fix would require disproportionate changes
  for minimal ROI, or the finding is factually incorrect), explain why
- If you need to clarify something before deciding, ask those questions
- Then share your plan for which issues to tackle and in what order

## WARNING (1)

### [Code Quality] With ExportSpans::NO and a streaming tail, user-span renames, status changes and exceptions are recorded on the exported invocation span, because addSpanUpdate and addSpanException never check the span id.
- **File:** src/workerd/io/tracer.c++:282

📋 Code Quality: ⚠️ 1 warning

Still open: user-span updates and exceptions change the exported invocation span in ExportSpans::NO mode.

Recommendation: 📝 Minor Comments

  • [WARNING] With ExportSpans::NO and a streaming tail, user-span renames, status changes and exceptions are recorded on the exported invocation span, because addSpanUpdate and addSpanException never check the span id. (src/workerd/io/tracer.c++:282)
View full review

WorkerTracer::addSpanUpdate and addSpanException ignore the span id (src/workerd/io/tracer.c++:282, :370)

  • addSpanUpdate calls updateInvocationSpan(update) and never compares spanId with the invocation span's id. addSpanException calls recordInvocationSpanException(...) with the same gap.
  • With ExportSpans::NO, user spans go through the SequentialSpanSubmitter in worker-factory.c++. It forwards each user span's update and exception with that user span's own id.
  • So a user span's updateName() renames the exported invocation span, setStatus() overwrites its status, and recorded exceptions are added to it. resolve_status in buffer.rs then turns those exceptions into an error status.
  • recordInvocationLevelException (tracer.c++:762) already checks context.getSpanId() != SpanParent(invocationSpan).getSpanId(). Both of these paths need the same check.
  • The NO-mode cases in tracer-test.c++ would catch this if they also sent a user-span update and a user-span exception.

The Rust test moves in buffer.rs/logs.rs and the changes to otlp-export-queue-test.c++ add no defects.


✅ All other checks passed

🔒 Security: ✅ LGTM

  • Collector: Only the operator's configuration sets the collector. Exports go to the fixed /v1/traces and /v1/logs paths.
  • Timestamps: Exception timestamps in the streaming tail come from the I/O clock. Only the native OTLP span gets precise time.
  • Input: The URL is parsed with ada_url. When query redaction is on, the query and fragment are removed.
  • Limits: The span and log buffers are bounded.

📖 AGENTS.md: ✅ LGTM

The AGENTS.md additions agree with the repository's existing guidance.


⚡ Performance: ✅ LGTM

  • Buffers: The span and log buffers cap open spans at 1024. Each batch is at most 512 items or 1 MiB, and batch sizes are tracked incrementally.
  • Export queue: The export queue has a fixed size, a concurrency limit and a timeout.

📝 Documentation: ✅ LGTM

The tracing.otlp schema docs and src/opentelemetry/AGENTS.md match the code.


Reviewed commit: 4989c65d — same changes as 71fec26e, so this is that commit's review (no new review was run; line numbers are as of 71fec26e)

@wperron
wperron force-pushed the dlapid/user-tracing-otlp branch from 4989c65 to e49b85d Compare November 18, 2026 20:06

@clanky-the-clanker clanky-the-clanker Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Superseded by the review of 3fa1366a.

This review, of e49b85d2

💬 Approved with comments — One WARNING is still open: in ExportSpans::NO mode with a streaming tail, user-span updates and exceptions change the exported invocation span. This affects only embedders that use that mode. The protobuf pin has no issues.

Findings: 1 warning

Not opened by the correctness review: 7 changed files it was meant to read
  • build/deps/build_deps.jsonc
  • build/deps/gen/build_deps.MODULE.bazel
  • src/opentelemetry/proto/collector/logs/v1/logs_service.proto
  • src/opentelemetry/proto/collector/trace/v1/trace_service.proto
  • src/opentelemetry/proto/common/v1/common.proto
  • src/opentelemetry/proto/resource/v1/resource.proto
  • src/opentelemetry/proto/trace/v1/trace.proto

The decision is advisory: this review is a comment and never approves, blocks or requests changes on the pull request.

🔧 Fix with your agent

Paste the following into your coding agent to investigate and fix these issues.

Fix the following code review issues found in pull request #7686.

Before making changes, review each issue and present a brief summary table:
- For each finding, state whether you agree, disagree, or need clarification
- If you disagree (e.g., the fix would require disproportionate changes
  for minimal ROI, or the finding is factually incorrect), explain why
- If you need to clarify something before deciding, ask those questions
- Then share your plan for which issues to tackle and in what order

## WARNING (1)

### [Code Quality] With ExportSpans::NO and a streaming tail, user-span renames, status changes and exceptions are recorded on the exported invocation span, because addSpanUpdate and addSpanException never check the span id.
- **File:** src/workerd/io/tracer.c++:282

📋 Code Quality: ⚠️ 1 warning

One open WARNING: user-span updates and exceptions change the exported invocation span in ExportSpans::NO mode.

Recommendation: 📝 Minor Comments

  • [WARNING] With ExportSpans::NO and a streaming tail, user-span renames, status changes and exceptions are recorded on the exported invocation span, because addSpanUpdate and addSpanException never check the span id. (src/workerd/io/tracer.c++:282)
View full review

addSpanUpdate and addSpanException ignore the span id (src/workerd/io/tracer.c++:282, :370)

  • addSpanUpdate calls updateInvocationSpan(update) for every span. updateInvocationSpan (tracer.c++:708-717) then sets the invocation span's name or status without comparing ids.
  • addSpanException calls recordInvocationSpanException(...) with the same gap.
  • With ExportSpans::NO, user spans go through SequentialSpanSubmitter in worker-factory.c++, which forwards each user span's update and exception with that span's own id.
  • So a user span's updateName() renames the exported invocation span and its setStatus() overwrites the invocation span's status. Its recorded exceptions are added to the invocation span, and resolve_status in buffer.rs turns them into an error status.
  • recordInvocationLevelException already does the right check (context.getSpanId() != SpanParent(invocationSpan).getSpanId()). Both paths need the same guard.
  • The NO-mode cases in tracer-test.c++ would catch this if they also sent a user-span update and a user-span exception.

The protobuf pin is explained in build_deps.jsonc and agrees with the generated MODULE file and the lock.


✅ All other checks passed

🚀 Release: ✅ LGTM

This pull request is not a release. The protobuf 37.0-rc3 pin is a dependency change and its reason is written down in build_deps.jsonc.


🔒 Security: ✅ LGTM

  • Collector: The collector comes only from the operator's tracing.otlp setting, and exports go to the fixed /v1/traces and /v1/logs paths.
  • Limits: The span, log and export-queue buffers all have limits.
  • Timestamps: The streaming tail still gets I/O-clock timestamps.

📖 AGENTS.md: ✅ LGTM

The AGENTS.md additions agree with the repository's existing guidance.


⚡ Performance: ✅ LGTM

The buffers cap open spans and batch sizes. The export queue has a fixed size, a concurrency limit and a timeout.


📝 Documentation: ✅ LGTM

The tracing.otlp schema docs and src/opentelemetry/AGENTS.md match the code and the vendored protos.


Reviewed commit: e49b85d2

@wperron
wperron force-pushed the dlapid/user-tracing-otlp branch from e49b85d to 3fa1366 Compare November 19, 2026 00:01

@clanky-the-clanker clanky-the-clanker Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Superseded by the review of 453d16f3.

This review, of 3fa1366a

💬 Approved with comments — One WARNING is still open: in ExportSpans::NO mode with a streaming tail, user-span updates, exceptions and attributes change the exported invocation span. This only affects embedders that use that mode.

Findings: 1 warning

Not opened by the correctness review: 5 changed files it was meant to read
  • src/opentelemetry/proto/collector/logs/v1/logs_service.proto
  • src/opentelemetry/proto/collector/trace/v1/trace_service.proto
  • src/opentelemetry/proto/common/v1/common.proto
  • src/opentelemetry/proto/resource/v1/resource.proto
  • src/opentelemetry/proto/trace/v1/trace.proto

The decision is advisory: this review is a comment and never approves, blocks or requests changes on the pull request.

🔧 Fix with your agent

Paste the following into your coding agent to investigate and fix these issues.

Fix the following code review issues found in pull request #7686.

Before making changes, review each issue and present a brief summary table:
- For each finding, state whether you agree, disagree, or need clarification
- If you disagree (e.g., the fix would require disproportionate changes
  for minimal ROI, or the finding is factually incorrect), explain why
- If you need to clarify something before deciding, ask those questions
- Then share your plan for which issues to tackle and in what order

## WARNING (1)

### [Code Quality] With ExportSpans::NO and a streaming tail, user-span renames, status changes, exceptions and attributes are applied to the exported invocation span, because addSpanUpdate, addSpanException and addSpanAttribute never check the span id.
- **File:** src/workerd/io/tracer.c++:282

📋 Code Quality: ⚠️ 1 warning

One open WARNING: tracer methods apply user-span updates, exceptions and attributes to the invocation span in ExportSpans::NO mode.

Recommendation: 📝 Minor Comments

  • [WARNING] With ExportSpans::NO and a streaming tail, user-span renames, status changes, exceptions and attributes are applied to the exported invocation span, because addSpanUpdate, addSpanException and addSpanAttribute never check the span id. (src/workerd/io/tracer.c++:282)
View full review

User-span events change the exported invocation span (src/workerd/io/tracer.c++:282, :370, :658)

  • With ExportSpans::NO, user spans go through SequentialSpanSubmitter (worker-factory.c++). It forwards each span's update and exception with the span's own id to tracer.addSpanUpdate / tracer.addSpanException. Attributes reach addSpanAttribute from api/tracing.c++ with the user span's context.
  • addSpanUpdate calls updateInvocationSpan(update), addSpanException calls recordInvocationSpanException(...) and addSpanAttribute calls setInvocationSpanAttribute(...). None of them compares the id with the invocation span's id.
  • Result: a user span's updateName() renames the exported invocation span, and its setStatus() overwrites the invocation span's status. Its exceptions are recorded on the invocation span, which resolve_status then turns into an error. Its attributes are tagged onto the invocation span.
  • recordInvocationLevelException already guards with context.getSpanId() != SpanParent(invocationSpan).getSpanId(). The same guard is needed on these three paths.
  • The NO-mode cases in tracer-test.c++ do not send a user-span update, exception or attribute.

✅ All other checks passed

🚀 Release: ✅ LGTM

This pull request is not a release. The protobuf 37.0-rc3 pin is explained in build_deps.jsonc and agrees with the generated MODULE file and the lock.


🔒 Security: ✅ LGTM

  • Collector: The collector comes only from the operator's tracing.otlp setting.
  • Limits: The buffers and the export queue have limits.
  • Timestamps: The streaming tail still gets I/O-clock timestamps.

📖 AGENTS.md: ✅ LGTM

The AGENTS.md additions agree with the repository's existing guidance.


⚡ Performance: ✅ LGTM

No performance concerns.


📝 Documentation: ✅ LGTM

The tracing.otlp schema docs and src/opentelemetry/AGENTS.md match the code.


Reviewed commit: 3fa1366a

@wperron
wperron force-pushed the dlapid/user-tracing-otlp branch from 3fa1366 to 453d16f Compare November 19, 2026 00:56

@clanky-the-clanker clanky-the-clanker Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Superseded by the review of fd5d49da.

This review, of 453d16f3

💬 Approved with comments — One WARNING is still open: in ExportSpans::NO mode with a streaming tail, user-span updates, exceptions and attributes change the exported invocation span. This only affects embedders that use that mode.

Findings: 1 warning

Not opened by the correctness review: 5 changed files it was meant to read
  • src/opentelemetry/proto/collector/logs/v1/logs_service.proto
  • src/opentelemetry/proto/collector/trace/v1/trace_service.proto
  • src/opentelemetry/proto/common/v1/common.proto
  • src/opentelemetry/proto/resource/v1/resource.proto
  • src/opentelemetry/proto/trace/v1/trace.proto

The decision is advisory: this review is a comment and never approves, blocks or requests changes on the pull request.

🔧 Fix with your agent

Paste the following into your coding agent to investigate and fix these issues.

Fix the following code review issues found in pull request #7686.

Before making changes, review each issue and present a brief summary table:
- For each finding, state whether you agree, disagree, or need clarification
- If you disagree (e.g., the fix would require disproportionate changes
  for minimal ROI, or the finding is factually incorrect), explain why
- If you need to clarify something before deciding, ask those questions
- Then share your plan for which issues to tackle and in what order

## WARNING (1)

### [Code Quality] With ExportSpans::NO and a streaming tail, user-span renames, status changes, exceptions and attributes are applied to the exported invocation span, because addSpanUpdate, addSpanException and addSpanAttribute never check the span id.
- **File:** src/workerd/io/tracer.c++:282

📋 Code Quality: ⚠️ 1 warning

One open WARNING: tracer methods apply user-span updates, exceptions and attributes to the invocation span in ExportSpans::NO mode.

Recommendation: 📝 Minor Comments

  • [WARNING] With ExportSpans::NO and a streaming tail, user-span renames, status changes, exceptions and attributes are applied to the exported invocation span, because addSpanUpdate, addSpanException and addSpanAttribute never check the span id. (src/workerd/io/tracer.c++:282)
View full review

User-span events change the exported invocation span (src/workerd/io/tracer.c++:282, :370, :658)

  • With ExportSpans::NO, user spans go through SequentialSpanSubmitter (worker-factory.c++). It forwards each span's update and exception with the span's own id to tracer.addSpanUpdate / tracer.addSpanException. Attributes reach addSpanAttribute from api/tracing.c++ with the user span's context.
  • addSpanUpdate calls updateInvocationSpan(update), addSpanException calls recordInvocationSpanException(...) and addSpanAttribute calls setInvocationSpanAttribute(...). None of them compares the id with the invocation span's id.
  • Result: a user span's updateName() renames the exported invocation span, and its setStatus() overwrites the invocation span's status. Its exceptions are recorded on the invocation span, which resolve_status then turns into an error. Its attributes are tagged onto the invocation span.
  • recordInvocationLevelException already guards with context.getSpanId() != SpanParent(invocationSpan).getSpanId(). The same guard is needed on these three paths.
  • The NO-mode cases in tracer-test.c++ do not send a user-span update, exception or attribute.

✅ All other checks passed

🚀 Release: ✅ LGTM

This pull request is not a release. The protobuf 37.0-rc3 pin is explained in build_deps.jsonc and agrees with the generated MODULE file and the lock.


🔒 Security: ✅ LGTM

  • Collector: The collector comes only from the operator's tracing.otlp setting.
  • Limits: The buffers and the export queue have limits.
  • Timestamps: The streaming tail still gets I/O-clock timestamps.

📖 AGENTS.md: ✅ LGTM

The AGENTS.md additions agree with the repository's existing guidance.


⚡ Performance: ✅ LGTM

No performance concerns.


📝 Documentation: ✅ LGTM

The tracing.otlp schema docs and src/opentelemetry/AGENTS.md match the code.


Reviewed commit: 453d16f3 — same changes as 3fa1366a, so this is that commit's review (no new review was run; line numbers are as of 3fa1366a)

danlapid and others added 10 commits November 19, 2026 08:42
The runtime writes OTLP trace and log requests through its own prost
structs in src/rust/otlp. Tests that decode those requests need the
generated classes. Until now, edgeworker vendored the trace schemas
for its tests, and nothing vendored the logs schemas.

Add the trace, logs, common, resource and collector schemas from
opentelemetry-proto v1.11.1, byte-identical below a provenance header,
and expose them as //src/opentelemetry:otlp_cc_proto. protobuf becomes
a direct dependency at 35.0, the version Perfetto already brings in,
so the build resolves nothing new.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
src/rust/otlp turns one invocation's spans into OTLP/HTTP
ExportTraceServiceRequests for a collector's /v1/traces, in the
protobuf encoding. C++ reports each span's open, updates and close to
a SpanBuffer through a cxx bridge, and gets back encoded requests to
send.

- proto: the part of the OTLP trace schema the encoder writes, as
  prost structs, so neither prost-build nor protoc is involved.
- rules: what a Workers span looks like in OTLP. Its kind and status
  come from its tags, query strings are redacted from URL tags when
  asked, and fetches to bindings' internal hosts are left out.
- buffer: batching. An invocation's spans go out bounded by count
  and bytes; the root closes the trace.
- attributes: resource and default attributes, shared by every span.

The crate has no C++ caller yet.

Co-authored-by: William Perron <wperron@cloudflare.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Workers user spans reach a tracing backend today only through
Streaming Tail Workers: the Workers Observability tail worker rebuilds
them in JavaScript from tail events. Spans built that way are cut off
when a Durable Object hands off work, carry coarse I/O-clock
timestamps, and are sampled only after they already exist.

This gives the runtime what it needs to export user spans itself:

- OtlpSpanSubmitter, a span submitter that feeds the Rust encoder and
  hands its requests to an OtlpSpanExporter the embedder provides.
- BaseTracer::setStartInvocationSpanFunc(), a hook that lets the
  embedder open each invocation's root span, so the invocation and
  all of its user spans form one exported trace.
  startExportedUserSpan() opens a span that is exported.
- The root records the invocation's outcome and any exception
  reported for the invocation itself.
- Exported spans get precise timestamps. Streaming tails, and so
  user code, still see the request's I/O clock.

Nothing changes unless an embedder installs the hook.

Co-authored-by: William Perron <wperron@cloudflare.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A new `tracing.otlp` setting names a service, typically an
ExternalServer pointing at a collector's OTLP/HTTP port, such as a
local Jaeger. workerd then exports every invocation's spans to it,
which makes Workers traces visible in local development without a
tail worker. A request arriving on a socket starts its trace with
the Worker's invocation span, and invocations of the collector's own
service are not traced. Spans carry the service name, entrypoint and,
for Durable Objects, the object's id.

The export queue matches edgeworker's limits: 16 concurrent POSTs,
4096 queued requests, batches of up to 1 MiB and a 10-second timeout.
Requests are dropped when the queue is full or delivery fails, and
queued exports are cancelled on shutdown.

Co-authored-by: William Perron <wperron@cloudflare.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Some bindings (D1, AI, Images, Vectorize) reach their backend with an
internal fetch to a placeholder host. Those fetch spans were dropped
when they ended, but by then their id had already been handed to the
backend as its parent, so any spans the backend exported were orphaned
at the top of the trace.

Decide when the fetch starts instead: a fetch to one of these hosts
opens no user span, and the binding's own span (such as d1_run) becomes
the subrequest's parent. Matching by host is still a stopgap until
every binding talks to its backend over JSRPC.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
An invocation whose span is exported can now also export its console
logs and the exceptions it reports, the events the streaming tail
gets as `log` and `exception`. They become OTLP LogRecords in an
ExportLogsServiceRequest for a collector's /v1/logs. This lets
consumers such as Workers Observability get logs without a tail
worker.

The hook passed to BaseTracer::setStartInvocationSpanFunc() now
returns a StartedInvocationSpan. It holds the span and, optionally,
an InvocationLogSubmitter, which the tracer feeds until it is
destroyed. startExportedInvocationSpan() opens an invocation's span.
Its options say what the export carries:

* ExportLogs::YES: the OtlpSpanSubmitter also takes the invocation's
  logs and hands them to OtlpSpanExporter::exportLogs().
* ExportSpans::NO: as the streaming tail worker samples logs apart
  from traces, the submitter records only the root span and exports
  no span, and the tracer parents no other span under it. The root
  still describes the invocation for its logs.
* ExportInvocationLog::YES: when the root closes, the logs get a
  `cloudflare.invocation` record made from it: the span's name as
  the body, its final attributes, its end time and its ids.

The Rust LogBuffer encodes and batches the records:

* Each record carries the trace id, the id of the span that was
  active, the trace flags, and the spans' resource and default tags.
* The body is the console arguments as an OTLP array, with object
  keys in their original order. A truncated message falls back to a
  string.
* A console argument that was a native Error keeps its name, message
  and stack in `cloudflare.log.error_info`, as the streaming tail's
  `errorInfo` does: an array in argument order, with a kvlist for
  each Error and an empty value for any other argument.
* An exception is a record with event_name "exception" and the
  exception.* attributes.
* Batches are bounded like spans. They go out when the root span
  closes, and one record at a time after that.

tracer-test decodes the requests with the vendored OTLP classes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
buffer.rs, logs.rs and rules.rs kept their tests in inline
`mod tests` blocks; src/rust/AGENTS.md puts them in
<module>-test.rs beside the module. lib.rs also drops the lint denies
the wd_rust_* macros already apply. unreachable! is allowed
everywhere, so it needs no deny of its own.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The lint job failed on three errors:

- otlp-export-queue-test filled its payload with std::memset, which
  custom-ban-memset forbids. It now uses kj::ArrayPtr::fill(), and
  drops <cstring>.
- buffer::Options has four bools, over struct_excessive_bools' limit.
  They are independent per-invocation switches, not states of one
  thing, so the lint is expected there rather than refactored away.
- A logs.rs doc comment named serde_json without backticks.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
tracer-test aborts under ASan + UBSan before main runs, while protobuf
registers the OTLP schemas' generated descriptors.
EpsCopyInputStream::LimitToken poisons its token once it is used, and
PopLimit() took the token by value. Clang 22 can give that parameter
the stack slot of the moved-from token, so the move poisons the
parameter and PopLimit() reads poisoned memory
(protocolbuffers/protobuf#27298).

protobuf 37.0 takes the token by rvalue reference
(protocolbuffers/protobuf#28619). No stable release has the fix yet,
so this freezes protobuf at 37.0-rc3 until 37.0 is out. Its
dependencies are already at or below the versions workerd uses.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Before any case runs, the test builds about 25 startup snapshots, one
per zygote Worker. That takes 6-10 seconds in a debug build on x86 and
just over 15 seconds on edgeworker's arm64 debug build. Edgeworker
gives medium tests 15 seconds, so the test's @all-autogates variant
times out there in most runs; workerd's internal builds for #8405 and
other PRs since have needed retries to get past it. Its cases take
about 200ms, so it is slow to start rather than stuck.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@wperron
wperron force-pushed the dlapid/user-tracing-otlp branch from 453d16f to fd5d49d Compare November 19, 2026 13:47

@clanky-the-clanker clanky-the-clanker Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💬 Approved with comments — One WARNING is still open: with ExportSpans::NO and a streaming tail, a user span's renames, status changes, exceptions and attributes are applied to the exported invocation span. This only affects embedders that use that mode.

Findings: 1 warning

Not opened by the correctness review: 5 changed files it was meant to read
  • src/opentelemetry/proto/collector/logs/v1/logs_service.proto
  • src/opentelemetry/proto/collector/trace/v1/trace_service.proto
  • src/opentelemetry/proto/common/v1/common.proto
  • src/opentelemetry/proto/resource/v1/resource.proto
  • src/opentelemetry/proto/trace/v1/trace.proto

The decision is advisory: this review is a comment and never approves, blocks or requests changes on the pull request.

🔧 Fix with your agent

Paste the following into your coding agent to investigate and fix these issues.

Fix the following code review issues found in pull request #7686.

Before making changes, review each issue and present a brief summary table:
- For each finding, state whether you agree, disagree, or need clarification
- If you disagree (e.g., the fix would require disproportionate changes
  for minimal ROI, or the finding is factually incorrect), explain why
- If you need to clarify something before deciding, ask those questions
- Then share your plan for which issues to tackle and in what order

## WARNING (1)

### [Code Quality] With ExportSpans::NO and a streaming tail, user-span renames, status changes, exceptions and attributes are applied to the exported invocation span: addSpanUpdate, addSpanException and addSpanAttribute never check the span id.
- **File:** src/workerd/io/tracer.c++:282

📋 Code Quality: ⚠️ 1 warning

One open WARNING: user-span events change the exported invocation span in ExportSpans::NO mode.

Recommendation: 📝 Minor Comments

  • [WARNING] With ExportSpans::NO and a streaming tail, user-span renames, status changes, exceptions and attributes are applied to the exported invocation span: addSpanUpdate, addSpanException and addSpanAttribute never check the span id. (src/workerd/io/tracer.c++:282)
View full review

User-span events change the exported invocation span (src/workerd/io/tracer.c++:282, :370, :658)

  • With ExportSpans::NO, user spans go through SequentialSpanSubmitter in worker-factory.c++. It forwards each span's updates and exceptions to tracer.addSpanUpdate and tracer.addSpanException under that span's own id. api/tracing.c++ sends a user span's attributes to addSpanAttribute with that span's context.
  • addSpanUpdate calls updateInvocationSpan(update), addSpanException calls recordInvocationSpanException(...) and addSpanAttribute calls setInvocationSpanAttribute(...). None of the three checks that the id is the invocation span's id.
  • What happens:
    • A user span's updateName() renames the exported invocation span.
    • Its setStatus() overwrites the invocation span's status.
    • Its exceptions are recorded on the invocation span, which then ends with an error status.
    • Its attributes are added to the invocation span.
  • recordInvocationLevelException already has the needed check: context.getSpanId() != SpanParent(invocationSpan).getSpanId(). Add the same check to these three paths.
  • tracer-test.c++ has no ExportSpans::NO case that sends a user-span update, exception or attribute. Add one when fixing this.

✅ All other checks passed

🚀 Release: ✅ LGTM

This pull request is not a release. build_deps.jsonc explains the protobuf 37.0-rc3 pin, and the pin matches the generated MODULE file and the lockfile.


🔒 Security: ✅ LGTM

  • Collector: Only the operator's tracing.otlp setting chooses the collector.
  • Limits: The span buffer, the log buffer and the export queue are all bounded.
  • Exported data: Only selected request headers are exported, never authorization or cookie headers. Query strings can be redacted.
  • Timestamps: The streaming tail still gets I/O-clock timestamps.

📖 AGENTS.md: ✅ LGTM

The AGENTS.md additions agree with the repository's existing guidance.


⚡ Performance: ✅ LGTM

No performance concerns.


📝 Documentation: ✅ LGTM

The tracing.otlp schema docs and src/opentelemetry/AGENTS.md match the code and the vendored protos.


Reviewed commit: fd5d49da

@wperron
wperron merged commit 0950838 into main Nov 19, 2026
24 checks passed
@wperron
wperron deleted the dlapid/user-tracing-otlp branch November 19, 2026 14:35
wperron added a commit to wperron/workers-sdk that referenced this pull request Nov 19, 2026
Spans are going away from streaming tail events: workerd now exports
each invocation's spans itself, as OTLP (`tracing.otlp`, from
cloudflare/workerd#7686). With `unsafeObservability`, Miniflare makes
the observability collector workerd's OTLP collector. Its new
`/v1/traces` endpoint decodes the protobuf export with a small reader
for the fields workerd sets and stores the spans. The streaming tail
now supplies only logs and exceptions, so `tail_worker_user_spans` is
no longer enabled.

`tracing.otlp` traces every service in the config, including
Miniflare's own Workers. The collector keeps only the spans of the
services Miniflare passes it (user workers and the Workflows services
that run their workflows, with the worker names the tail props used to
carry) and drops the rest. An invocation started by one of those
dropped spans, such as the entry Worker's, would then point at a parent
that is never stored, so the store keeps a span whose parent is missing
as a root and parks the parent in `pending_parents`. If the parent
arrives later, as a calling user worker's span does after its callee
has been exported, the span is attached to it again.

The config types add `tracing` from the merged workerd schema, so this
needs a workerd release that includes cloudflare/workerd#7686.

Co-Authored-By: Dan Lapid <dlapid@cloudflare.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants