Skip to navigation
File SearchIndexingSupabase Storage

Index Supabase Manifest

Index an explicit list of objects from a Supabase Storage bucket, given as a JSON Lines manifest in the same bucket. One bulk job; nothing is listed or scanned.

One object per line, UTF-8, uncompressed. uri is required, the rest are optional:

{"uri": "s3://documents/inbox/2026-q4-msa.pdf", "source_identity": "contract-8841", "custom_metadata": {"department": "legal", "counterparty": "Northwind"}}
{"uri": "s3://documents/inbox/2026-q4-sow.pdf", "source_identity": "contract-8842", "size": 482113, "etag": "9b2cf535f27731c974343645a3985328"}
{"uri": "s3://documents/archive/2025-renewal.pdf", "source_identity": "contract-7310"}
{"uri": "s3://documents/inbox/handbook.pdf"}

Physical location vs. logical identity

uri is where the file currently lives: this bucket, this key. It must be inside bucket_name, and it is the only field Captain reads bytes from.

source_identity is what the document is, in your own terms. Captain keeps one document per identity, so the identity is what decides whether a later index is an update or a new document:

  • Give s3://documents/inbox/2026-q4-msa.pdf the identity contract-8841. Move the file to s3://documents/archive/2026-q4-msa.pdf and index it again with the same identity, and Captain updates that one document. Its chunks are replaced and the old ones are retired.
  • Omit source_identity and the identity is the uri. The same file at a new key is a different identity, so you get a second document and both versions answer queries.

A repeat of an identity already in the collection is skipped while skip_existing is true (the default), and replaced when you send overwrite_existing: true or skip_existing: false. While it is skipping, Captain compares content. A repeat whose content is unchanged is skipped. A repeat whose content differs is an identity conflict: it is neither skipped nor indexed, and the job reports it under identity_conflicts so that you can send it again with overwrite_existing: true. When either version has no checksum, the repeat is skipped. Use your own record id, not a path, whenever content can move or be delivered twice.

The other line fields

  • custom_metadata: merged over the job-level custom_metadata, key by key, for this file only. A line’s value wins on a shared key.
  • checksum and version_id: recorded on the document. A checksum that is the object’s ETag, written bare or as etag:..., is compared with the object before anything is read, and a line that does not match is reported as unreachable instead of being indexed. Write any other algorithm with its prefix, for example sha256:...; it is recorded but not verified. version_id pins the object version that is read.
  • size and etag: when both are present, as in a bucket listing export, Captain does not read the object’s metadata before indexing. That is what lets a large manifest start quickly.

Limits and error handling

A manifest holds up to 1,000,000 lines and 500 MB, with each line at most 10 KB. It must be plain UTF-8 .jsonl or .ndjson; a compressed body is refused. Per line, custom_metadata takes at most 50 keys, keys up to 64 characters and string values up to 1,024 characters. The job-level custom_metadata is capped at 64 KB serialized. It is attached to every file in the job, so metadata size times file count must stay under 2 GB: a few hundred bytes fits the full million lines, and the full 64 KB fits about 32,000 files. Larger jobs should carry less job-level metadata or be split.

A wrong manifest_uri is a 400 before the job is created, as is a manifest larger than the size cap. Request bodies are capped at 1 MB, which is why a large object list is a manifest in storage rather than an inline array.

Unusable lines (invalid JSON, not a JSON object, a uri outside the bucket, a duplicate uri, an unknown key) and unreachable objects (missing, access denied, unsupported file type) are counted on the job. GET /v2/jobs/{job_id} returns a manifest object with the totals and a sample of up to 20 of each, giving the line number or uri and a reason. Neither fails the job: a million-line manifest with a few thousand bad lines indexes the rest.

Headers:

  • Authorization: Bearer {api_key} - Captain API key for authentication
  • X-Organization-ID: Organization UUID

Returns: { job_id, status: “pending” }

Path parameters

collection_namestringRequired

Headers

authorizationstring or nullOptional

Request

This endpoint expects an object.
access_key_idstringRequired

Supabase S3 access key ID (Storage → S3 Connection → access key).

bucket_namestringRequired
endpoint_urlstringRequired

S3-compatible endpoint URL for your Supabase project’s Storage S3 connection, e.g. https://{project_ref}.storage.supabase.co/storage/v1/s3 (Storage → S3 Connection). Required.

manifest_uristringRequired

URI of the manifest, inside bucket_name (s3://bucket/path/manifest.jsonl or supabase://bucket/path/manifest.jsonl). UTF-8 JSON Lines, uncompressed, one object per line: {“uri”: …} plus optional source_identity, custom_metadata, checksum, version_id, size and etag. Up to 1,000,000 lines, 500 MB total and 10 KB per line.

processing_typeenumRequired

Document processing type. ‘advanced’ uses agentic OCR with AI-enhanced extraction for complex layouts, tables, figures, charts, and documents containing images. ‘basic’ provides reliable OCR optimized for general document indexing and high-volume processing. ‘auto’ inspects each PDF and uses advanced only when it has photos, drawings, tables or scanned pages (logos and signatures do not count), otherwise basic; each file bills at the tier it used.

Allowed values:
secret_access_keystringRequired

Supabase S3 secret access key (shown once when the S3 access key is created).

custom_metadatamap from strings to strings or integers or doubles or booleans or lists of strings or nullOptional

Custom metadata attached to every indexed chunk. A line’s own custom_metadata is merged over it for that file. Keys are strings; values: str, int, float, bool, or List[str].

mask_piibooleanOptionalDefaults to false

When true, detected PII (emails, phone numbers, SSNs, credit cards, names, and locations) is masked in the parsed content before it is embedded and stored — replaced with entity tags like <PERSON> and <EMAIL_ADDRESS>. For images (including images embedded in PDFs), PII text visible in the image is also pixel-redacted. Opt-in; defaults to false, which leaves content unchanged.

max_filesinteger or nullOptional>=1
Stop after this many manifest lines, in file order.
overwrite_existingbooleanOptionalDefaults to false

When true, files that already exist in the collection are re-indexed and replaced with zero downtime: the new version is built alongside the live one and atomically swapped in when complete, so the previous version keeps serving search results throughout the rebuild. The document keeps the same document_id across overwrites, and its status reads ‘updating’ in the document listing while the rebuild runs. Requires skip_existing=false. Setting both to true returns a 400 error.

parsing_scriptstring or nullOptional

Relative path to a JS parsing script for JSON files (e.g. ‘research/paper-parser’). When provided, .json files are processed through a sandboxed V8 isolate. Without this, .json files are indexed as raw text.

pii_engineenumOptionalDefaults to kev

Masking engine for this job. kev (default): the Kev classifier model on Captain’s infrastructure; supports pii_fields and pii_instructions; no fallback, so a file kev cannot mask fails to index and the rest of the job continues. jev: TypeSafe’s hosted Jev model, slightly more accurate, best-effort availability; supports custom fields and instructions and may list fallbacks in pii_fallback. presidio: the legacy pattern engine, built-in categories only (rejects pii_fields or pii_instructions with 422), never a fallback. Requires mask_pii: true. The earlier names captain-jev and captain-presidio keep working as before; new requests should use kev, jev or presidio.

Allowed values:
pii_fallbackobject or booleanOptional

Ordered fallback engines for jev, as {"engines": [...], "retry_budget_seconds": n}. Omitted or an empty engines list means no fallback. Only valid with pii_engine: "jev"; with kev or presidio a non-empty list answers 422. On the multipart file upload endpoint, send it as a JSON string form field. With the earlier engine name captain-jev it is true (default, fall back to captain-presidio) or false.

pii_fieldslist of objects or nullOptional

Additional PII categories to mask for this job, on top of the built-in set. Requires mask_pii to be true. Up to 20 fields. Each masked value is replaced with the field’s tag in angle brackets and appears in the masking report under the field’s name.

pii_instructionsstring or nullOptional<=1000 characters

Plain-language guidance that steers what counts as personal data for this job, for the built-in categories and any pii_fields. For example: ‘Names of hospital staff may stay; patient names, record numbers and bed assignments must be masked.’ Requires mask_pii to be true. Up to 1,000 characters.

regionstringOptionalDefaults to us-east-1

Region configured for your Supabase project’s S3 endpoint (Storage → S3 Connection). Required by the S3 protocol; defaults to ‘us-east-1’.

skip_existingbooleanOptionalDefaults to true

When true, files already indexed in the collection are skipped and will not be re-indexed with incoming changes. When false, all incoming files are indexed regardless of whether they already exist.

transcription_languagestring or nullOptional

AWS Transcribe language code for the spoken audio (e.g. ‘es-US’, ‘pt-BR’). Omit to auto-detect per file. Video and audio files only. Supported codes: https://docs.aws.amazon.com/transcribe/latest/dg/supported-languages.html

Response

Successful Response
job_idstring
statusstringOptionalDefaults to pending
custom_metadatamap from strings to any or nullOptional

The custom_metadata Captain accepted for this job, echoed back as validated. Null when none was supplied.

Errors

400
Bad Request Error
422
Unprocessable Entity Error