Content-hash dedup decides what work is skipped, replaced, or forced when objects are processed into collections — and lineage decides what re-runs when a source changes
Every object ingested into Mixpeek is fingerprinted with a SHA-256 content hash, and every document produced from it carries that hash in its system envelope alongside a full lineage record. Together they answer the two questions every pipeline eventually asks:
Have I already processed this? — the content hash.
What downstream work is stale now that this changed? — the lineage chain.
The hash is computed from the source content at ingestion and stored:
on the object (in the bucket), and
on every document derived from it (content_hash in the document’s system envelope).
Two objects with identical bytes produce the same hash — regardless of filename, upload time, or path. A changed file produces a new hash, which is what marks its derived documents as stale.“Latest version” of an asset is therefore a content-hash lookup, not a version counter you maintain yourself.
When you submit a batch, dedup_strategy controls how objects that were already processed in a prior batch are handled. Dedup is scoped to the (bucket, collection) pair — an object is a duplicate if the target collection already has documents produced from that same source object.
Strategy
Behavior
Cost profile
skip(default)
An object is reprocessed only if its content or the collection’s extractor configuration changed since it was last extracted; everything else is skipped
Free for unchanged content — no extraction, no inference
replace
Every object is reprocessed. Once its new documents are written, its earlier documents in the collection are deleted. If the batch fails, or fails on that object, its earlier documents are kept
Full extraction cost; use for a clean rebuild after the source content or the extractor configuration changed
force
Every object is reprocessed and its documents are overwritten in place. Documents a re-run no longer produces are not removed
Full extraction cost; prefer replace, which also removes those
skip reprocesses an object when its content or the extractor configuration changed since it was last extracted, and skips everything else. It does not delete documents a re-run no longer produces, for example the extra chunks left behind when a new chunking setting yields fewer of them. For a clean rebuild that removes those, use replace.
Derived assets stay in sync with their source through the combination of both mechanisms:
A source object’s content changes → its hash changes.
Re-submitting with replace (or re-syncing the source) reprocesses it — and every derived document downstream in its lineage (transcode → embed → tag) is rebuilt from the new source.
Sources whose hash didn’t change are skipped — unchanged content is never re-paid for.
This is what keeps N derived artifacts consistent with one upstream asset without a separate file system or job-tracking database: the document is the record, the hash is the version, and lineage is the dependency graph.
