Skip to content

feat(core): stamp a trace and span id on each test attempt in the artifacts manifest - #553

Open
Winify wants to merge 7 commits into
mainfrom
otel/1-manifest-trace-ids
Open

Winify wants to merge 7 commits into
mainfrom
otel/1-manifest-trace-ids

Conversation

@Winify

@Winify Winify commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

Stack

Part 1 of 4 of the OpenTelemetry export, replacing #485.

What

The artifacts manifest (devtools-artifacts-<sessionId>.json) can now link a run to its traces:

  • Each test attempt gets a trace and span id when it starts, recorded in a per-test attempts[] list. The test's last attempt is marked final, so a consumer reading the manifest can tell a retried test from its earlier attempts.
  • The manifest records the run's runId and, when the run was launched under a TRACEPARENT, that parent context.
  • WDIO entries carry start and end times, the framework-reported duration (Cucumber scenarios fall back to wall-clock time), and the final attempt's failure message.
  • A Cucumber scenario is listed once, and its steps are no longer listed as tests.
  • WDIO passes its retry count into the attempt record, so under specFileRetries the final flag and the attempt number agree.

Why

#485 created span ids when the manifest was uploaded, after the run ended. A test therefore had no id while it ran, so its span couldn't be linked live or carry a CI parent. Ids recorded when each attempt starts fix that. Pushing the same manifest again also gives the same spans.

Verification

pnpm build, pnpm test (3999 passed, 4 skipped), pnpm lint (0 errors) and pnpm docs:capabilities --check pass on this branch.

🤖 Generated with Claude Code

@vercel

vercel Bot commented Nov 10, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
devtools Ready Ready Preview Nov 13, 2026 11:17pm UTC

Request Review

@plum117

plum117 commented Nov 10, 2026

Copy link
Copy Markdown
Contributor

Reviewed the stack at 07b101a: ran the examples (WDIO mocha/Cucumber, Selenium, Nightwatch × telemetry/trace × retries × TRACEPARENT) and pushed each run to a fake receiver and to the #556 stack. Tests, lint, typecheck pass. Direction looks right. "read only" = found in the code, not run.

  1. Selenium retry: wrong attempt data. A flaky test is listed twice. final is on the failed attempt 0, so the push sends 3 spans, one of them "pass, final". Cause: the metadata uid is generateStableUid (changes on retry), but the ledger uid is deterministicUid.
  2. Absolute artifacts[].path. After test-results/ moves (CI artifact download), the push does not read the zips, gives no URL and exits 1. With relative paths, it works.
  3. specFileRetries: both attempts are attempt: 0. WDIO sends the real count as retry on runner:start; the service reporter can record it.
  4. Only the final attempt keeps error; the flaky attempt's failure is lost.
  5. Selenium/Nightwatch record no error and no test-level timing, so a failing span has no message. The same change in 3 adapters → core?
  6. (read only) suite-helpers.ts:192: in Selenium Cucumber, a repeated step text makes the scenario attempt: 1.
  7. (read only) attempt-tracker.ts:85: renumbering in recordOutcome and numbering by length in recordStart can give attempt numbers that repeat or are out of order.

@Winify
Winify force-pushed the otel/1-manifest-trace-ids branch from a528f4f to ae35f19 Compare November 11, 2026 11:42
@Winify
Winify marked this pull request as ready for review November 11, 2026 11:42
Winify and others added 3 commits November 12, 2026 10:16
…ifacts manifest

A manifest that lists only test states cannot link a run to the traces it
produced or to a CI parent span. Each attempt now mints its trace and span
id when it starts, and the manifest records the run id and any TRACEPARENT
parent, so an external consumer can address every attempt.

Co-Authored-By: Claude Haiku 5.5 <noreply@anthropic.com>
…m the real attempt

A retried Selenium test was listed twice with final on its failed attempt,
because the test uid carried a per-process counter the attempt ledger did not.
A specFileRetries rerun recorded attempt 0, the tracker could renumber an
attempt into a repeat, and only the final attempt kept its error.

Artifact paths are now written relative to the manifest directory so a moved
or downloaded results folder still resolves, and the manifest records cwd and
browser identity for the exporter to slice by.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
A retried Mocha test now reuses its first attempt's uid so the artifacts
manifest lists it once. The trace writer keyed a test's group on that uid,
so in a zip holding more than one test both attempts merged into a single
group and the examples smoke check counted one attempt where two ran.

Selenium stamps the attempt on each row of a retried attempt and the group
is keyed on the uid plus that attempt. A first attempt, and every adapter
that sets no attempt, exports exactly as before.

Also fixes the two test files that failed typecheck: a mock read through a
plainly typed hook, and a WebdriverIO-shaped result passed as a literal to
a parameter that deliberately omits the field the service ignores.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
@Winify
Winify force-pushed the otel/1-manifest-trace-ids branch from 0ea4b15 to 9d4fcee Compare November 12, 2026 09:16
@plum117
plum117 force-pushed the otel/1-manifest-trace-ids branch from 9d4fcee to 7104114 Compare November 12, 2026 11:03
Winify and others added 2 commits November 13, 2026 09:55
The manifest carries manifestVersion 2, since artifact paths, attempts[]
and specFile changed shape and a reader could not tell the formats apart.
Without a TRACEPARENT every attempt of a test now joins one trace derived
from the run id and the test's uid, so a retried test is not three
unrelated traces, while a large run is not one oversized trace. The
absolute cwd is gone: specFile is written relative to it instead. A final
attempt that matches no recorded attempt is reported, not silently
replaced by the last one.

Co-Authored-By: Claude Haiku 5.5 <noreply@anthropic.com>
…field

The shared wire schema gained CommandLog.attempt, and the Python CI job
fails when the generated types are out of date with it.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
@greptile-apps

greptile-apps Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

RetriggerView in GreptileConfidence Score: 4/5

[Medium impact] The changes reach test tracing and artifact reporting across several supported runners.

The queued Cucumber scenario path still needs fixing before merge.

Findings

  1. P1 Cucumber steps remain separate tests ▶
  2. P2 Test spans include service waits ▶

Summary

Adds trace and span IDs to test attempts in the artifacts manifest.

  • Each test attempt gets trace IDs that the manifest can link to its run.
  • Manifest test entries now include timing and failure details.
  • Retries stay separate in trace groups and per-test artifact names.
  • The version 2 manifest stores artifact paths relative to its own folder.

Reviews (2) · Last reviewed commit: "fix(selenium): list a Cucumber scenario,..." · Reviewed by Greptile

Comment on lines +135 to +139
const retriedUid = this.#retriedUid(name, file, signature, opts.attempt)
const uid = retriedUid ?? generateStableUid(file, signature)
const ledgerAttempt = this.#attemptTracker.recordStart(uid, file)
const test = createTestStats({
uid: generateStableUid(file, signature),
uid,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Retries overwrite earlier screenshots

Selenium retries now reuse the same uid, but writeScreenshotArtifact names screenshots using only testUid and sessionId. With screenshot: 'on', a passing retry overwrites the failed attempt's screenshot. Both manifest entries then point to the retry's image.

Pass the attempt number through the screenshot writer and include it in the filename and artifact key.

Comment on lines +38 to +41
const owner = stepOwnerUid(uid)
return (
(owner !== undefined && metadata.has(owner)) ||
meta.ancestry?.at(-1)?.kind === 'scenario'

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Cucumber steps remain separate tests

The new step filter does not recognize Selenium's Cucumber tree. ensureSuiteAndTestManager sets the root's file to process.cwd(), while startScenario sets only root.featureFile. The metadata walk therefore labels the scenario child as feature, not scenario.

Its steps fail both checks here, so they remain in tests[] and the scenario has no entry. Complete this change against Selenium's actual tree and record attempts under the scenario's uid.

Comment on lines +501 to +502
const metadata = this.#testMetadata.get(uid)
this.#attemptTracker.recordOutcome(uid, metadata?.state, metadata?.error)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Test spans include service waits

afterTest and afterScenario save the test's end time before waiting for command hooks, but recordOutcome stamps a fresh Date.now() afterward. Slow hooks therefore add the service's wait to attempts[].endTime, while the test's own endTime excludes it. The exporter in PR #555 uses the attempt record for span bounds.

Pass the captured end time into the tracker too, and extend the timing test to check the attempt record.

Winify and others added 2 commits November 13, 2026 21:11
A retried test's attempts share one uid, and the screenshot was named from the uid and the session alone, so a retry overwrote the earlier attempt's file and the manifest listed one path twice. The file and its artifact key now carry the attempt, the same way the per-test trace slice does.

Co-Authored-By: Claude Haiku 5.5 <noreply@anthropic.com>
…cts manifest

The Selenium root suite keeps the working directory as its file, so the scenario under it was read as the feature and no scenario was ever recorded: the manifest listed each step as a test. A suite whose feature file is a .feature now counts as the feature, and the scenario feeds the attempt ledger, so it gets its own attempts and trace ids.

Co-Authored-By: Claude Haiku 5.5 <noreply@anthropic.com>
@Winify

Winify commented Nov 14, 2026

Copy link
Copy Markdown
Contributor Author

@plum117 @vishnuv688 What do you think about this stack? Is this mergable?

The P1: Cucumber of Greptile is acknowledged, but I would not want to change it

The release would probably fail until we have a @wdio/opentelemetry npm package -- but it feels like now the whole wdio ecosystem has to have a release redesign

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants