Skip to content
This repository was archived by the owner on Oct 23, 2026. It is now read-only.

feat(snapshots): add snapshots download - #1176

Merged
BYK merged 3 commits into
mainfrom
byk/feat/snapshots-download
Jul 2, 2026
Merged

BYK merged 3 commits into
mainfrom
byk/feat/snapshots-download

Conversation

@BYK

@BYK BYK commented Jul 2, 2026

Copy link
Copy Markdown
Member

Fourth PR of the build/Emerge workstream (after build download #1170, build upload #1172, VCS metadata #1174). Establishes the snapshots command group and its first subcommand.

snapshots download

Downloads baseline snapshot images from Sentry's preprod system and extracts them locally. Sentry SaaS only.

  • Resolve the snapshot: --snapshot-id <id> (direct) or --app-id <app> → latest baseline (optional --branch; --project numeric ID for org auth tokens). The two ID flags are mutually exclusive; --branch requires --app-id.
  • Ensure the archive is built: checks status, and if not ready triggers a build and polls (2s interval, 5-min timeout via TimeoutError).
  • Download + extract: downloads the archive ZIP and extracts images to --output (default ./snapshots-base/), with Zip-Slip protection (entries escaping the output dir are skipped).

Files

  • src/lib/api/preprod-artifacts.ts — getLatestBaseSnapshot (404 → null), getSnapshotArchiveReady, triggerSnapshotArchiveBuild, waitForSnapshotArchive, downloadSnapshotArchive (auth dropped cross-origin on the storage redirect).
  • src/lib/snapshots/archive.ts — extractZipToDir (fflate, path-traversal safe).
  • src/commands/snapshots/{download,index}.ts + app.ts registration.
  • Tests (11 new): extraction incl. Zip-Slip; API incl. 404→null and trigger→poll→ready; command flag validation + app-id resolution + real extraction.

Follow-ups (not in this PR)

snapshots diff (needs an image-diff dependency decision — native odiff vs pure-JS); snapshots upload (blocked on the objectstore session).

typecheck, lint, check:deps, check:fragments all pass.

Ports `sentry snapshots download`. Establishes the `snapshots` command group
and downloads baseline snapshot images from the preprod system.

Resolve a snapshot by `--snapshot-id`, or the latest baseline for an app via
`--app-id` (optional `--branch` filter; `--project` numeric ID for org auth
tokens). Ensures the downloadable archive is built — checking status, triggering
a build, and polling (2s interval, 5-min timeout) — then downloads the ZIP and
extracts the images to `--output` (default ./snapshots-base/), guarding against
Zip-Slip path traversal.

- API (preprod-artifacts.ts): getLatestBaseSnapshot, getSnapshotArchiveReady,
  triggerSnapshotArchiveBuild, waitForSnapshotArchive, downloadSnapshotArchive.
- src/lib/snapshots/archive.ts: extractZipToDir (fflate, path-traversal safe).
- Tests: extraction + Zip-Slip, API (incl. 404→null, trigger+poll), command
  (flag validation, app-id resolution, extraction).
@github-actions

github-actions Bot commented Jul 2, 2026 •

Copy link
Copy Markdown
Contributor
PR Preview Action v1.8.1

QR code for preview link

🚀 View preview at
https://cli.sentry.dev/_preview/pr-1176/

Built to branch gh-pages at 2026-07-02 18:42 UTC.
Preview will be ready when the GitHub Pages deployment is complete.

@cursor cursor Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit f8bc5a5. Configure here.

Comment thread src/commands/snapshots/download.ts Resolved
@github-actions

github-actions Bot commented Jul 2, 2026 •

Copy link
Copy Markdown
Contributor

Codecov Results 📊

✅ Patch coverage is 89.22%. Project has 5188 uncovered lines.
✅ Project coverage is 81.62%. Comparing base (base) to head (head).

Files with missing lines (2)
File Patch % Lines
src/commands/snapshots/download.ts 86.36% ⚠️ 6 Missing and 4 partials
src/lib/api/preprod-artifacts.ts 87.50% ⚠️ 5 Missing and 3 partials
Coverage diff
@@            Coverage Diff             @@
##          main       #PR       +/-##
==========================================
+ Coverage    81.59%    81.62%    +0.03%
==========================================
  Files          405       408        +3
  Lines        28124     28227      +103
  Branches     18328     18384       +56
==========================================
+ Hits         22946     23039       +93
- Misses        5178      5188       +10
- Partials      1880      1887        +7

Generated by Codecov Action

Comment thread src/lib/api/preprod-artifacts.ts Resolved
- C1 (critical): latest-base response is snake_case (head_artifact_id/
  image_count), matching the server + Rust struct — the camelCase schema would
  have failed on every real --app-id response. Parse snake_case and map to
  camelCase; add a schema-contract test (and drop the camelCase test mock that
  masked the bug).
- M2: resolve the optional project via SENTRY_PROJECT (honoring org/project
  combo) then the configured default, instead of reading the raw env var.
- L1: correct the download comment — the endpoint streams directly (no redirect);
  document the in-memory extraction ceiling.
- L2: strip trailing slashes from the region URL before manual URL building.
- L4: segment-aware Zip-Slip check (don't over-block "..name") + skip empty
  entry names; add absolute-path/empty-name/safe-name extraction tests.
- L5: validate flag combinations before org resolution.
- N1: docs say project ID or slug (server accepts either), not "numeric".
@BYK

BYK commented Jul 2, 2026

Copy link
Copy Markdown
Member Author

Self-review (subagent-assisted) + fixes

A critical bug was found (and had been masked by a test asserting the wrong contract):

C1 (critical): the latest-base endpoint returns snake_case (head_artifact_id, image_count) — verified against both the Rust LatestBaseSnapshotResponse (no camelCase rename) and the live server serializer. My Zod schema expected camelCase, so every real --app-id resolution would have thrown "Unexpected response format" (only --snapshot-id worked). Now parses snake_case and maps to camelCase, with a schema-contract test that rejects camelCase (and the download flow test no longer mocks away the schema).

Also fixed: M2 — optional project now resolves via SENTRY_PROJECT (honoring org/project combo) then the configured default (was reading the raw env var); L2 — strip trailing slashes before manual URL building; L4 — segment-aware Zip-Slip check (no longer over-blocks legitimate ..name entries) + skip empty entry names (added absolute-path/empty-name tests); L5 — validate flag combos before org resolution; L1/N1 — corrected the (non-existent) redirect comment and the project ID/slug wording.

Known limitation (documented): the archive is downloaded and extracted in memory — fine for typical screenshot baselines, but a very large archive (server caps ~1 GB) would need streaming extraction. Tracked as a follow-up alongside snapshots diff/upload.

Comment thread src/lib/api/preprod-artifacts.ts Fixed
Comment on lines +444 to +453
while (!(await getSnapshotArchiveReady(org, snapshotId))) {
if (Date.now() >= deadline) {
throw new TimeoutError(
`Snapshot archive was not ready after ${
SNAPSHOT_ARCHIVE_TIMEOUT_MS / 1000
}s. The build may still be running; try again shortly.`
);
}
await new Promise((r) => setTimeout(r, SNAPSHOT_ARCHIVE_POLL_MS));
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bug: The timeout check in waitForSnapshotArchive is misplaced, allowing an extra API call to be made after the timeout deadline has passed.
Severity: MEDIUM

Suggested Fix

Move the timeout check to be the primary condition of the while loop, similar to the pattern in uploadBuild. The loop condition should be while (Date.now() < deadline). Inside the loop, fetch the snapshot status and break or return if it's ready. This ensures the deadline is respected before each API call.

Prompt for AI Agent
Review the code at the location below. A potential bug has been identified by an AI
agent. Verify if this is a real issue. If it is, propose a fix; if not, explain why it's
not valid.

Location: src/lib/api/preprod-artifacts.ts#L444-L453

Potential issue: The polling loop in `waitForSnapshotArchive` checks the timeout
condition after making an API call within the loop's condition. If the timeout deadline
passes during the sleep interval (`SNAPSHOT_ARCHIVE_POLL_MS`), the loop will execute one
more `getSnapshotArchiveReady()` call after the deadline has already been exceeded. This
violates the intended timeout contract, causing the function to overshoot its 5-minute
limit. The correct pattern, seen in `uploadBuild`, is to check the deadline in the
`while` condition before the API call is made.

Did we get this right? 👍 / 👎 to inform future reviews.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Working as intended — this mirrors the legacy CLI's wait_for_archive exactly (check !ready in the while condition, timeout check in the body, then sleep). The "extra" status call in the loop condition is the final readiness check: if the archive becomes ready right at the deadline it succeeds rather than spuriously failing. Worst-case overshoot is one poll interval (~2s) against a 5-minute bound — a negligible, inherent property of soft-timeout poll loops. Changing to while (Date.now() < deadline) would drop that final check and diverge from the port source, so declining.

@BYK
BYK merged commit e706e41 into main Jul 2, 2026
32 checks passed
@BYK
BYK deleted the byk/feat/snapshots-download branch July 2, 2026 18:56
BYK added a commit that referenced this pull request Jul 2, 2026
Fifth PR of the **build/Emerge** workstream. Completes the local
`snapshots` tooling (after `snapshots download` #1176).

## `snapshots diff <base-dir> <head-dir>`
Compares two local directories of snapshot images and writes a PNG diff
mask per changed image + a JSON report. **No network requests.**

- **Pure-JS engine:** `pixelmatch` implements the same YIQ-color-space,
anti-aliasing-aware algorithm as the legacy CLI's native `odiff`;
`pngjs`/`jpeg-js` decode PNG/JPEG. All pure-JS devDeps — bundle into the
SEA binary, no per-platform binaries, no postinstall, no runtime deps.
- **Behavior (matches legacy):** recursively collect images
(png/jpg/jpeg via the shared `walkFiles`), categorize
matched/added/removed (or `skipped` with `--selective`),
**byte-identical fast-path**, then diff matched pairs — dimension
mismatch → `layout_changed`, differing pixels → `changed` (mask
written). `--threshold` (0–1 per-pixel sensitivity),
`--no-antialiasing`, `--fail-on-diff` (non-zero exit on any change),
`--output` (default `./diff-output/`).

### Why pure-JS over WASM/native (per your call)
odiff has no WASM build (native Zig+SIMD, per-platform prebuilts +
postinstall — incompatible with the bundled single-binary model), and
its algorithm is equivalent to pixelmatch's. For a local baseline diff
on screenshots, JS perf is more than adequate (and the byte-identical
fast-path skips unchanged images). The diff core sits behind a module
boundary, so a WASM engine can be swapped in later if perf ever matters.

## Files
- `src/lib/snapshots/diff.ts` — `compareImages`, `collectImageFiles`,
`categorizeImages`.
- `src/commands/snapshots/diff.ts` + route registration.
- devDeps: `pixelmatch`, `pngjs`, `jpeg-js`, `@types/pngjs` (all
bundled; `check:deps` green).
- Tests (13): compare (match/changed/layout), categorize (+selective),
recursive collect; command (mask write, `--fail-on-diff` exit codes,
missing-dir rejection).

## Follow-ups
Streaming extraction for very large snapshot archives (from #1176);
`snapshots upload` (objectstore session); iOS `build upload`.

`typecheck`, `lint`, `check:deps`, `check:fragments` all pass.
BYK added a commit that referenced this pull request Jul 3, 2026
Follow-up to `snapshots download` (#1176), closing the documented
in-memory limitation.

## Problem
`snapshots download` buffered the entire archive (`arrayBuffer()`) and
then decompressed **every entry at once** (`unzipSync`). Peak memory ≈
compressed archive + all decompressed entries (~2× the archive) — an OOM
risk on large baselines (the server caps archives at ~1 GB).

## Change
Stream the archive straight to disk:
- **`openSnapshotArchive`** (replaces `downloadSnapshotArchive`) returns
the unconsumed HTTP `Response`.
- **`extractZipStream`** feeds `response.body` through fflate's
streaming `Unzip` with the **synchronous** `UnzipInflate` +
`UnzipPassThrough` decoders (no web workers — SEA-safe), decompressing
and writing each entry to disk as its chunks arrive. Yielding to the
event loop between chunk pushes lets write streams flush, so peak memory
is bounded by the in-flight entry, not the archive.

Zip-Slip protection is preserved (shared `safeEntryDest` guard: skips
traversal/absolute/empty/dir entries). The archive is server-produced
(trusted), so no adversarial zip-bomb cap is added — the goal here is
memory-bounding.

## Verification
- All snapshot tests updated to stream (chunked async iterables) + a new
large **deflated** multi-chunk entry case.
- Verified end-to-end against a **real undici `ReadableStream`** (HTTP
server → `fetch` → `extractZipStream`): a deflated entry split across
the network stream extracts byte-for-byte.
- `typecheck`, `lint`, `check:deps` pass; full suite 8289 passed.

## Notes
Only stored (0) and DEFLATE (8) entries are supported — the formats
Sentry's snapshot archives use.
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants