Repository navigation
feat(snapshots): add snapshots download - #1176
Conversation
Ports `sentry snapshots download`. Establishes the `snapshots` command group and downloads baseline snapshot images from the preprod system. Resolve a snapshot by `--snapshot-id`, or the latest baseline for an app via `--app-id` (optional `--branch` filter; `--project` numeric ID for org auth tokens). Ensures the downloadable archive is built — checking status, triggering a build, and polling (2s interval, 5-min timeout) — then downloads the ZIP and extracts the images to `--output` (default ./snapshots-base/), guarding against Zip-Slip path traversal. - API (preprod-artifacts.ts): getLatestBaseSnapshot, getSnapshotArchiveReady, triggerSnapshotArchiveBuild, waitForSnapshotArchive, downloadSnapshotArchive. - src/lib/snapshots/archive.ts: extractZipToDir (fflate, path-traversal safe). - Tests: extraction + Zip-Slip, API (incl. 404→null, trigger+poll), command (flag validation, app-id resolution, extraction).
|
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit f8bc5a5. Configure here.
Codecov Results 📊✅ Patch coverage is 89.22%. Project has 5188 uncovered lines. Files with missing lines (2)
Coverage diff@@ Coverage Diff @@
## main #PR +/-##
==========================================
+ Coverage 81.59% 81.62% +0.03%
==========================================
Files 405 408 +3
Lines 28124 28227 +103
Branches 18328 18384 +56
==========================================
+ Hits 22946 23039 +93
- Misses 5178 5188 +10
- Partials 1880 1887 +7Generated by Codecov Action |
- C1 (critical): latest-base response is snake_case (head_artifact_id/ image_count), matching the server + Rust struct — the camelCase schema would have failed on every real --app-id response. Parse snake_case and map to camelCase; add a schema-contract test (and drop the camelCase test mock that masked the bug). - M2: resolve the optional project via SENTRY_PROJECT (honoring org/project combo) then the configured default, instead of reading the raw env var. - L1: correct the download comment — the endpoint streams directly (no redirect); document the in-memory extraction ceiling. - L2: strip trailing slashes from the region URL before manual URL building. - L4: segment-aware Zip-Slip check (don't over-block "..name") + skip empty entry names; add absolute-path/empty-name/safe-name extraction tests. - L5: validate flag combinations before org resolution. - N1: docs say project ID or slug (server accepts either), not "numeric".
Self-review (subagent-assisted) + fixesA critical bug was found (and had been masked by a test asserting the wrong contract): C1 (critical): the Also fixed: M2 — optional project now resolves via Known limitation (documented): the archive is downloaded and extracted in memory — fine for typical screenshot baselines, but a very large archive (server caps ~1 GB) would need streaming extraction. Tracked as a follow-up alongside |
| while (!(await getSnapshotArchiveReady(org, snapshotId))) { | ||
| if (Date.now() >= deadline) { | ||
| throw new TimeoutError( | ||
| `Snapshot archive was not ready after ${ | ||
| SNAPSHOT_ARCHIVE_TIMEOUT_MS / 1000 | ||
| }s. The build may still be running; try again shortly.` | ||
| ); | ||
| } | ||
| await new Promise((r) => setTimeout(r, SNAPSHOT_ARCHIVE_POLL_MS)); | ||
| } |
There was a problem hiding this comment.
Bug: The timeout check in waitForSnapshotArchive is misplaced, allowing an extra API call to be made after the timeout deadline has passed.
Severity: MEDIUM
Suggested Fix
Move the timeout check to be the primary condition of the while loop, similar to the pattern in uploadBuild. The loop condition should be while (Date.now() < deadline). Inside the loop, fetch the snapshot status and break or return if it's ready. This ensures the deadline is respected before each API call.
Prompt for AI Agent
Review the code at the location below. A potential bug has been identified by an AI
agent. Verify if this is a real issue. If it is, propose a fix; if not, explain why it's
not valid.
Location: src/lib/api/preprod-artifacts.ts#L444-L453
Potential issue: The polling loop in `waitForSnapshotArchive` checks the timeout
condition after making an API call within the loop's condition. If the timeout deadline
passes during the sleep interval (`SNAPSHOT_ARCHIVE_POLL_MS`), the loop will execute one
more `getSnapshotArchiveReady()` call after the deadline has already been exceeded. This
violates the intended timeout contract, causing the function to overshoot its 5-minute
limit. The correct pattern, seen in `uploadBuild`, is to check the deadline in the
`while` condition before the API call is made.
Did we get this right? 👍 / 👎 to inform future reviews.
There was a problem hiding this comment.
Working as intended — this mirrors the legacy CLI's wait_for_archive exactly (check !ready in the while condition, timeout check in the body, then sleep). The "extra" status call in the loop condition is the final readiness check: if the archive becomes ready right at the deadline it succeeds rather than spuriously failing. Worst-case overshoot is one poll interval (~2s) against a 5-minute bound — a negligible, inherent property of soft-timeout poll loops. Changing to while (Date.now() < deadline) would drop that final check and diverge from the port source, so declining.
Fifth PR of the **build/Emerge** workstream. Completes the local `snapshots` tooling (after `snapshots download` #1176). ## `snapshots diff <base-dir> <head-dir>` Compares two local directories of snapshot images and writes a PNG diff mask per changed image + a JSON report. **No network requests.** - **Pure-JS engine:** `pixelmatch` implements the same YIQ-color-space, anti-aliasing-aware algorithm as the legacy CLI's native `odiff`; `pngjs`/`jpeg-js` decode PNG/JPEG. All pure-JS devDeps — bundle into the SEA binary, no per-platform binaries, no postinstall, no runtime deps. - **Behavior (matches legacy):** recursively collect images (png/jpg/jpeg via the shared `walkFiles`), categorize matched/added/removed (or `skipped` with `--selective`), **byte-identical fast-path**, then diff matched pairs — dimension mismatch → `layout_changed`, differing pixels → `changed` (mask written). `--threshold` (0–1 per-pixel sensitivity), `--no-antialiasing`, `--fail-on-diff` (non-zero exit on any change), `--output` (default `./diff-output/`). ### Why pure-JS over WASM/native (per your call) odiff has no WASM build (native Zig+SIMD, per-platform prebuilts + postinstall — incompatible with the bundled single-binary model), and its algorithm is equivalent to pixelmatch's. For a local baseline diff on screenshots, JS perf is more than adequate (and the byte-identical fast-path skips unchanged images). The diff core sits behind a module boundary, so a WASM engine can be swapped in later if perf ever matters. ## Files - `src/lib/snapshots/diff.ts` — `compareImages`, `collectImageFiles`, `categorizeImages`. - `src/commands/snapshots/diff.ts` + route registration. - devDeps: `pixelmatch`, `pngjs`, `jpeg-js`, `@types/pngjs` (all bundled; `check:deps` green). - Tests (13): compare (match/changed/layout), categorize (+selective), recursive collect; command (mask write, `--fail-on-diff` exit codes, missing-dir rejection). ## Follow-ups Streaming extraction for very large snapshot archives (from #1176); `snapshots upload` (objectstore session); iOS `build upload`. `typecheck`, `lint`, `check:deps`, `check:fragments` all pass.
Follow-up to `snapshots download` (#1176), closing the documented in-memory limitation. ## Problem `snapshots download` buffered the entire archive (`arrayBuffer()`) and then decompressed **every entry at once** (`unzipSync`). Peak memory ≈ compressed archive + all decompressed entries (~2× the archive) — an OOM risk on large baselines (the server caps archives at ~1 GB). ## Change Stream the archive straight to disk: - **`openSnapshotArchive`** (replaces `downloadSnapshotArchive`) returns the unconsumed HTTP `Response`. - **`extractZipStream`** feeds `response.body` through fflate's streaming `Unzip` with the **synchronous** `UnzipInflate` + `UnzipPassThrough` decoders (no web workers — SEA-safe), decompressing and writing each entry to disk as its chunks arrive. Yielding to the event loop between chunk pushes lets write streams flush, so peak memory is bounded by the in-flight entry, not the archive. Zip-Slip protection is preserved (shared `safeEntryDest` guard: skips traversal/absolute/empty/dir entries). The archive is server-produced (trusted), so no adversarial zip-bomb cap is added — the goal here is memory-bounding. ## Verification - All snapshot tests updated to stream (chunked async iterables) + a new large **deflated** multi-chunk entry case. - Verified end-to-end against a **real undici `ReadableStream`** (HTTP server → `fetch` → `extractZipStream`): a deflated entry split across the network stream extracts byte-for-byte. - `typecheck`, `lint`, `check:deps` pass; full suite 8289 passed. ## Notes Only stored (0) and DEFLATE (8) entries are supported — the formats Sentry's snapshot archives use.

Fourth PR of the build/Emerge workstream (after
build download#1170,build upload#1172, VCS metadata #1174). Establishes thesnapshotscommand group and its first subcommand.snapshots downloadDownloads baseline snapshot images from Sentry's preprod system and extracts them locally. Sentry SaaS only.
--snapshot-id <id>(direct) or--app-id <app>→ latest baseline (optional--branch;--projectnumeric ID for org auth tokens). The two ID flags are mutually exclusive;--branchrequires--app-id.TimeoutError).--output(default./snapshots-base/), with Zip-Slip protection (entries escaping the output dir are skipped).Files
src/lib/api/preprod-artifacts.ts—getLatestBaseSnapshot(404 →null),getSnapshotArchiveReady,triggerSnapshotArchiveBuild,waitForSnapshotArchive,downloadSnapshotArchive(auth dropped cross-origin on the storage redirect).src/lib/snapshots/archive.ts—extractZipToDir(fflate, path-traversal safe).src/commands/snapshots/{download,index}.ts+app.tsregistration.Follow-ups (not in this PR)
snapshots diff(needs an image-diff dependency decision — nativeodiffvs pure-JS);snapshots upload(blocked on the objectstore session).typecheck,lint,check:deps,check:fragmentsall pass.