Repository navigation
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
|
Reviewed the stack at 07b101a: ran the examples (WDIO mocha/Cucumber, Selenium, Nightwatch × telemetry/trace × retries ×
|
a528f4f to
ae35f19
Compare
…ifacts manifest A manifest that lists only test states cannot link a run to the traces it produced or to a CI parent span. Each attempt now mints its trace and span id when it starts, and the manifest records the run id and any TRACEPARENT parent, so an external consumer can address every attempt. Co-Authored-By: Claude Haiku 5.5 <noreply@anthropic.com>
…m the real attempt A retried Selenium test was listed twice with final on its failed attempt, because the test uid carried a per-process counter the attempt ledger did not. A specFileRetries rerun recorded attempt 0, the tracker could renumber an attempt into a repeat, and only the final attempt kept its error. Artifact paths are now written relative to the manifest directory so a moved or downloaded results folder still resolves, and the manifest records cwd and browser identity for the exporter to slice by. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
A retried Mocha test now reuses its first attempt's uid so the artifacts manifest lists it once. The trace writer keyed a test's group on that uid, so in a zip holding more than one test both attempts merged into a single group and the examples smoke check counted one attempt where two ran. Selenium stamps the attempt on each row of a retried attempt and the group is keyed on the uid plus that attempt. A first attempt, and every adapter that sets no attempt, exports exactly as before. Also fixes the two test files that failed typecheck: a mock read through a plainly typed hook, and a WebdriverIO-shaped result passed as a literal to a parameter that deliberately omits the field the service ignores. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
0ea4b15 to
9d4fcee
Compare
9d4fcee to
7104114
Compare
The manifest carries manifestVersion 2, since artifact paths, attempts[] and specFile changed shape and a reader could not tell the formats apart. Without a TRACEPARENT every attempt of a test now joins one trace derived from the run id and the test's uid, so a retried test is not three unrelated traces, while a large run is not one oversized trace. The absolute cwd is gone: specFile is written relative to it instead. A final attempt that matches no recorded attempt is reported, not silently replaced by the last one. Co-Authored-By: Claude Haiku 5.5 <noreply@anthropic.com>
…field The shared wire schema gained CommandLog.attempt, and the Python CI job fails when the generated types are out of date with it. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
[Medium impact] The changes reach test tracing and artifact reporting across several supported runners. The queued Cucumber scenario path still needs fixing before merge. FindingsSummaryAdds trace and span IDs to test attempts in the artifacts manifest.
Reviews (2) · Last reviewed commit: "fix(selenium): list a Cucumber scenario,..." · Reviewed by Greptile |
| const retriedUid = this.#retriedUid(name, file, signature, opts.attempt) | ||
| const uid = retriedUid ?? generateStableUid(file, signature) | ||
| const ledgerAttempt = this.#attemptTracker.recordStart(uid, file) | ||
| const test = createTestStats({ | ||
| uid: generateStableUid(file, signature), | ||
| uid, |
There was a problem hiding this comment.
Retries overwrite earlier screenshots
Selenium retries now reuse the same uid, but writeScreenshotArtifact names screenshots using only testUid and sessionId. With screenshot: 'on', a passing retry overwrites the failed attempt's screenshot. Both manifest entries then point to the retry's image.
Pass the attempt number through the screenshot writer and include it in the filename and artifact key.
| const owner = stepOwnerUid(uid) | ||
| return ( | ||
| (owner !== undefined && metadata.has(owner)) || | ||
| meta.ancestry?.at(-1)?.kind === 'scenario' |
There was a problem hiding this comment.
Cucumber steps remain separate tests
The new step filter does not recognize Selenium's Cucumber tree. ensureSuiteAndTestManager sets the root's file to process.cwd(), while startScenario sets only root.featureFile. The metadata walk therefore labels the scenario child as feature, not scenario.
Its steps fail both checks here, so they remain in tests[] and the scenario has no entry. Complete this change against Selenium's actual tree and record attempts under the scenario's uid.
| const metadata = this.#testMetadata.get(uid) | ||
| this.#attemptTracker.recordOutcome(uid, metadata?.state, metadata?.error) |
There was a problem hiding this comment.
Test spans include service waits
afterTest and afterScenario save the test's end time before waiting for command hooks, but recordOutcome stamps a fresh Date.now() afterward. Slow hooks therefore add the service's wait to attempts[].endTime, while the test's own endTime excludes it. The exporter in PR #555 uses the attempt record for span bounds.
Pass the captured end time into the tracker too, and extend the timing test to check the attempt record.
A retried test's attempts share one uid, and the screenshot was named from the uid and the session alone, so a retry overwrote the earlier attempt's file and the manifest listed one path twice. The file and its artifact key now carry the attempt, the same way the per-test trace slice does. Co-Authored-By: Claude Haiku 5.5 <noreply@anthropic.com>
…cts manifest The Selenium root suite keeps the working directory as its file, so the scenario under it was read as the feature and no scenario was ever recorded: the manifest listed each step as a test. A suite whose feature file is a .feature now counts as the feature, and the scenario feeds the attempt ledger, so it gets its own attempts and trace ids. Co-Authored-By: Claude Haiku 5.5 <noreply@anthropic.com>
|
@plum117 @vishnuv688 What do you think about this stack? Is this mergable? The P1: Cucumber of Greptile is acknowledged, but I would not want to change it The release would probably fail until we have a |
Stack
Part 1 of 4 of the OpenTelemetry export, replacing #485.
What
The artifacts manifest (
devtools-artifacts-<sessionId>.json) can now link a run to its traces:attempts[]list. The test's last attempt is markedfinal, so a consumer reading the manifest can tell a retried test from its earlier attempts.runIdand, when the run was launched under aTRACEPARENT, that parent context.specFileRetriesthefinalflag and the attempt number agree.Why
#485 created span ids when the manifest was uploaded, after the run ended. A test therefore had no id while it ran, so its span couldn't be linked live or carry a CI parent. Ids recorded when each attempt starts fix that. Pushing the same manifest again also gives the same spans.
Verification
pnpm build,pnpm test(3999 passed, 4 skipped),pnpm lint(0 errors) andpnpm docs:capabilities --checkpass on this branch.🤖 Generated with Claude Code