Storage

Current format policy: Portable .tine files default to v5 (since 0.42). --for-legacy writes v4. The .tine/ repository format is v3. tine evidence export writes v4.

V3 repository and remote protocol

Implementing OpenTine in another language? This document is format policy and history. The normative, byte-level specification — envelope framing, oid derivation, both canonicalizations, the repository layout, the TINEPACK3 pack format, and all three signature schemes — is SPEC.md.

OpenTine 0.3.0 stores agent history under .tine/ using Git-shaped concepts, while retaining portable *.tine v2 compatibility files.

Advanced verbs appear here in their 0.16 group spelling (tine repo …, tine share …, tine trust …, tine agent …); the older top-level spellings such as tine fsck or tine promote still work unchanged. The full mapping is in CLI.md.

Object model

The immutable object types are:

A workspace snapshot (0.38+) is a tree blob: a blob at schema 2 whose canonical JSON body lists {mode, name, oid} entries, one tree per directory (SPEC §1.5.7). An event links the snapshot taken at its step in its tree field. fsck, gc, prune, pack, bundle and fetch/push walk a tree's entries like any other link; a 0.34 reader accepts trees as opaque blobs. opentine._snapshot.snapshot(repo, root, ignore=..., max_file_bytes=..., cache=...) stores one (deduplicated against earlier turns: unchanged files and directories keep their oids, and the returned cache skips rereading unchanged files); opentine._tree.link_tree(event_payload, tree_oid) links it from an event; tine rewind TREE --apply (or RUN@STEP --tree) restores it.

An object ID is TYPE:sha256:HEX, computed over the type, schema version, and canonical stored bytes. JSON uses RFC 8785/JCS encoding; blobs retain raw bytes. Credential redaction precedes canonicalization and hashing. Identical objects deduplicate naturally.

This is the key difference from portable v2 identity. A v3 repository id is a content hash, so two identical forks collapse to one run object. A v2 .tine fork id names the fork act — derived from lineage, the retained slice, the branch, the caller's declared intent, and a recorded nonce, stored in metadata.fork, and provable with verify_fork_id — so forking the same point twice produces two distinct runs rather than one shared id. Both are digests; neither lets an untrusted run id steer an output path. Importing a v2 run re-represents its steps as content-addressed event objects, but load_run reconstructs it with its original run id and metadata.fork, so a fork's verify_fork_id verdict survives the round-trip. A fork created inside a repository is a new content-addressed run object that carries no metadata.fork record, so verify_fork_id returns None (no verdict) for it.

Integer inputs outside JSON's interoperable ±(2**53−1) range are rejected; callers must encode 64-bit identifiers, nanosecond timestamps, or arbitrary- precision amounts as strings instead of accepting silent IEEE-754 rounding.

The dependency-free trusted kernel owns only canonical encoding, typed IDs, immutable envelopes, typed parent/causal-link checks, object verification, and a tiny repository protocol. CI caps the complete kernel and every production Python module at 250 physical lines.

Local layout and operations

.tine/ contains loose objects, refs, reflogs, deterministic compressed packs, configuration, shallow boundaries, and rebuildable indexes. The public API includes Repo.init/open, put/get, CAS update_ref, log, semantic diff, fork, verify/fsck, pack, fetch, and push.

from opentine import Repo

repo = Repo.init("workspace")
oid = repo.put("blob", b"artifact")
repo.update_ref("tags/example", oid, expected_old=None)
assert repo.fsck(deep=True).ok

update_ref is strict compare-and-swap: a contended ref lock is waited for (bounded, jittered backoff), but an expected_old that no longer matches raises RefCasMismatch (a ValueError) and is never retried, because the caller's new value was computed from a stale state. When the new value is a pure function of the current one (an append), opt in to re-read-and-retry instead:

repo.update_ref_rebase("tags/log", lambda current: make_entry_after(current))

Six ref namespaces are accepted — heads/, experiments/, promotions/, annotations/, tags/, and remotes/ — and any other name is refused. heads/*, experiments/*, and promotions/* move between run objects, and annotations/* between annotation objects; tags/* and remotes/* carry no type constraint and may label any immutable object. tine init creates refs/annotations, refs/heads, and refs/tags.

An annotations/* ref carries one further rule. Its target must be an annotation whose target_id is a run, and the ref name after annotations/ must equal that run's hex digest, so annotations/<digest> is the only annotations/* name that can point at an annotation of run:sha256:<digest>. A ref naming a different run, an annotation without a string target_id, or an annotation of a non-run object is refused rather than stored.

Attestations created by the v3 workflow target runs. These type rules are enforced on write, read, pack verification, remote ref updates, and by fsck.

fsck recomputes every ID, validates envelope canonicality, typed links, shallow boundaries, refs, and the event DAG. Ref updates are compare-and-swap; a stale expected value is rejected instead of silently overwriting another writer.

Semantic diff reports common/divergent events, cost, latency, tool path, content/blob and artifact changes, usage, billing, and evaluation scores. Exact common/only sets use object identity; the changed pairs are a sequence-position comparison of divergent events, not a causal merge alignment. Transcript line merging is not an operation. Agents select, compose, fork, resume, evaluate, attest, and promote run graphs instead.

Verb names: the repo- prefix rule

tine carries two command families. The legacy v2 verbs read and write portable .tine artifacts through the .tine_runs file index; the v3 verbs read and write objects and refs inside a .tine repository. They are different stores, so where both families need the same word, the v3 verb takes a repo- prefix.

The prefix is a collision marker, not a namespace. It appears only where a legacy verb already owns the plain name, which is why context, attest, evaluate, and promote are bare — v2 has no such verbs. Do not add a repo- prefix for symmetry.

repo-log is the one prefix without a collision: log was never a legacy verb, but repo-log shipped in 0.3.0 and renaming a released verb is a breaking change, so it is grandfathered. tests/test_repo_cli_parity.py asserts the rule and pins that single exception, so a second one cannot appear by accident.

The two families never share state. tine fork branches a run file and writes .tine_runs/<run-id>.tine; tine repo-fork branches a run object and moves a repository ref. A v2 artifact reaches the v3 side only through tine repo migrate --to-v3.

Read verbs and their --json contract

Five read verbs expose repository engines on the command line. Each renders for a human by default and emits exactly one JSON object with --json:

tine repo-show heads/main --repo . [--json]      # a whole run, as a step tree
tine context <event-oid> --repo . [--depth N]    # only that event's ancestors
tine repo-log [ref] --repo . [--limit N]         # the event ancestry, one per line
tine repo-diff <left> <right> --repo .           # the semantic diff of two runs
tine repo-search [query] --repo . [--limit N]    # completed runs, by content

repo-show accepts a ref name or a run:sha256:… oid; context requires an event:sha256:… oid and defaults to --depth 8, the same default the MCP context_slice tool uses, so an operator reproducing what an agent saw gets the same slice. Every one of the five is read-only and never writes a ref.

repo-diff takes a ref name or a run oid on either side, resolves each once, and compares the resolved objects, so the left_id/right_id it reports are exactly what was diffed. It is the semantic_diff MCP tool, so the human table and the JSON carry the same divergence sets and the same five summary dimensions. With --exit-code it follows git diff: 0 when the runs are identical, 1 when they differ. Identical means no event-level divergence — no only_left, no only_right, no changed — which the JSON also reports as identical. Without --exit-code a successful comparison always exits 0. A usage error is argparse's 2, which the verb itself never emits, so a script can distinguish "the runs differ" from "you typed the command wrong".

repo-search mirrors the MCP search_runs tool exactly: it defaults to successful_only (completed runs only, flipped by --include-unsuccessful) and to --limit 20, with --min-score and --model as the same optional filters. An empty query lists candidate runs; a query is matched case-insensitively against event input and output blob text, and the matching prefix comes back as matched_text.

A run's score prefers signed evaluations: when any exist, the best signed one decides it, and runs scored by signed evaluations rank above runs scored only by unsigned ones — any MCP client can write an unsigned evaluation, and through 0.9.0 one injected score decided the rank. Each result carries score_signed, and --signed-only (signed_only= in the library and in search_runs) ignores unsigned evaluations entirely. Signed means a signature block is present, not that it verified; check it with tine repo-verify and a trusted key.

Search is a bounded scan, not an index. It walks up to MAX_SEARCH_OBJECTS = 100,000 repository objects per invocation and reads blob text under further aggregate byte budgets, so its latency grows with repository size and it is deliberately not on any write path. --limit bounds the result set, not the scan — it is what keeps output readable, not what makes the command fast. Exceeding any of the scan budgets is a refusal, printed as a single tine repo-search: <message> line with exit 1, never a partial result presented as a complete one.

The derived index (0.44). tine repo-search --rebuild-index writes an SQLite FTS5 index to .tine/index/search.sqlite (format opentine-search/1) holding each run's stored prompts and completions, tool names and arguments, file paths named in those arguments, and run metadata (status, models, tags, annotation metadata). It is derived and never authoritative: it copies only what the store holds, after redaction; deleting it loses nothing. Once it exists, put_run and every pack import (push, fetch, bundle import) update it; if an update fails the file is removed and search scans again. While it is present repo-search (and tine search --repo, and MCP search_runs and recall) answer from it: every query word must occur (case- and diacritic-insensitive tokens, not a substring match), results rank by BM25, an empty query lists runs newest first, and the 10,000-candidate cap does not apply. Evaluation scores are still read from the attestations, for the returned rows only. --no-index forces the scan. A Python whose sqlite3 lacks FTS5 never builds the index and keeps scanning; --rebuild-index then exits with that reason. scripts/bench/search_index.py measures a cold 3-term query over 100,000 runs.

The JSON objects come from the same writer as tine show --json — literally the one function opentine/_cli_json.py:serialize, which every --json surface in the CLI calls: keys are sorted, every value passes json_safe, and each object carries command, which names the schema below. Within a major version fields are added, never renamed or removed. A failure that stops a verb producing a result — an unresolvable ref, a non-event id, a negative depth, a run beyond a shallow boundary — is a single tine <verb>: <message> line on stderr and exit 1, not JSON.

The five plumbing verbs have no --json flag

fsck, object, migrate-v3, fetch, and push print bare JSON unconditionally and deliberately take no --json flag. They are plumbing: a consistency report, an object dump, a migration receipt, and two transport receipts. None of them has a rich human rendering that --json would replace, so a flag to ask for JSON would only be a flag you could forget. Their output bytes are pinned by tests (tests/test_cli_surface_json.py, tests/test_repo_cli_routing.py, tests/test_cli_v3.py) and are part of the compatibility surface: scripts parse them as they stand. Everywhere else — including all four tine pricing subcommands — JSON is opt-in behind --json, and the human rendering is the default.

repo-diff emits every SemanticDiff field verbatim and adds only the envelope keys above, so a new engine field appears in the JSON without a schema change.

Write verbs

Three verbs mutate a repository. They are the operator half of the surface: MCP withholds promote_run unless the host opts in, because a model reading untrusted run content must not be able to move a release gate, but the person at the terminal already holds the repository.

tine trust attest <run-ref-or-oid> --signer NAME (--claim JSON | --claim-file PATH) \
    [--evidence OID]... [--key-env VAR | --key-file PATH | --ed25519-key-file PATH] \
    [--key-id ID] [--repo .] [--json]
tine trust evaluate <run-ref-or-oid> --evaluator NAME --score NAME=VALUE... \
    [--key-env VAR | --key-file PATH | --ed25519-key-file PATH] [--key-id ID] [--json]
tine trust promote <run-ref-or-oid> --name NAME [--expected-old OID] [--json]
tine repo-verify <attestation-oid | run-ref-or-oid> \
    [--key-env VAR | --key-file PATH | --pubkey PATH | --trust-embedded-key] \
    [--pin FINGERPRINT]... [--require-signature] [--signer NAME]... [--claim JSON] \
    [--repo .] [--json]

Each accepts a ref name or a run:sha256:… oid and resolves it first. That resolution is not cosmetic: attest hands target_id to the object store, whose link check requires an object that already exists, and promote hands its run into update_ref, which rejects a ref string outright. After resolving, each verb makes exactly one engine call — Repo.attest or Repo.promote — so a CLI-written object is byte-identical to the object the matching MCP tool writes. A target that resolves to no object, or to something that is not a run, is refused before anything is written.

attest stores the claim verbatim inside a content-addressed attestation. The claim must parse to a JSON object; a list, number, or string is refused, because every reader — the association scan behind repo-diff's summary.evaluations, and repo-search's score scan — reads a claim as a mapping. --evidence may be repeated and each value must be an existing object.

evaluate is attest with the claim fixed to {"kind": "evaluation", "scores": {…}} and signer taken from --evaluator — exactly what the MCP evaluate_run tool builds. There is deliberately only one evaluation claim shape in the format; a second would be scores no reader could see. Each --score must be a finite number, and a repeated name is refused rather than silently resolved.

Signing an attestation. Given a key — --key-env/--key-file for HMAC-SHA256, --ed25519-key-file for Ed25519, plus an optional --key-id recorded inside the block — attest and evaluate bind the signer label to that key at tine-attest/1. The algorithm is inferred from which flag names the key, so there is no --algorithm to disagree with it, and two key flags are refused rather than resolved by precedence. Given no key the verb writes the byte-identical unsigned object every release since 0.3.0 wrote, and the receipt still says unsigned: the label is then self-asserted, and saying otherwise would be the lie signing exists to remove. The signed view, the verdicts, and what a signature does and does not prove are in docs/SECURITY_MODEL.md.

tine repo-verify reads it back. It takes an attestation:sha256:… oid, or a run ref/oid — in which case it checks every attestation targeting that run, which is the release question ("is this approved, and by whom?") a promotion gate scripts. It reports tine verify's own verdicts (verified, verified-tofu, unsigned, no-key, mismatch, error; only the first two pass) and is fail-closed the same way: any key, --trust-embedded-key, or --require-signature arms the check and exits 1 unless every attestation verified, and --require-signature also refuses a run with no attestation at all. Unarmed, it is a report that exits 0. Scope a release gate with --signer NAME (repeatable) and --claim JSON — the attestation's claim must contain those keys with those values: the gate then passes when at least one selected attestation verifies, and unselected ones (an unsigned note, a signed evaluation that is not the approval) neither pass nor block it. Unscoped, a verified rejection passed like an approval and any writer's unsigned note blocked the release. Either flag arms the check. Each Ed25519 row reports its key_fingerprint, and --pin FINGERPRINT (repeatable, the full sha256: digest; alone it implies the embedded key) passes only attestations signed by a pinned key, as verified. MCP has no equivalent tool: checking a signature needs the operator's key material, and attest_run has no signing options for the same reason — run content must not be able to sign as an operator.

promote compare-and-swaps promotions/<name>. --expected-old omitted means expect no existing ref, so creating a promotion is the default and moving one always requires naming the value being replaced; the refusal on a conflict prints the current oid and the --expected-old invocation that would move it. There is no --force, and there will not be one: a release gate that can be overwritten without stating what it overwrote is not a gate.

Their --json object is a receipt, emitted only after the write has succeeded. A failure that stops the result — a CAS conflict, an unresolvable or non-run target, an invalid claim or score — is a single tine <verb>: <message> line on stderr with exit 1 and no JSON on stdout, so a consumer never has to parse an object to learn whether the repository changed. A usage error is argparse's 2, which these verbs never emit themselves.

repo-verify is a read verb, so unlike the three above its --json object is emitted whether or not the check passed — ok and the exit code carry that. Only a refusal (an unresolvable target, an unreadable key, two key flags) stays a single stderr line with no JSON.

Appending to a repository written by an older release is a release gate, tested the same way reading one is: every verb runs against a copy of every committed golden repository from 0.3.0 onward, and the copy must fsck clean afterwards.

Lineage verbs

Two more verbs branch a run graph. They are the CLI twins of the MCP fork_run_v3 and resume_run_v3 tools, and both make exactly one engine call, Repo.fork, with the same arguments the tool passes — so a CLI-written fork is the same content-addressed object the tool writes.

tine repo-fork <run-ref-or-oid> --from-event OID --ref REF \
    [--model M] [--prompt P] [--policy JSON] [--repo .] [--json]
tine repo-resume <run-ref-or-oid> --ref REF [--repo .] [--json]

repo-fork rebuilds a run from the causal closure of --from-event, so the forked run contains that event and its ancestors and nothing later. --model, --prompt, and --policy are the three overrides fork_run_v3 accepts; --policy must parse to a JSON object, and an omitted flag is identical to an absent override key, not to a null one.

repo-resume is the same fork taken at the run's last verified tip with overrides={"resume": True}, which is what makes the new run status: "running" instead of a plain branch. A run with no event tip cannot be resumed and is refused by name rather than crashing on an empty tip list.

Both accept a ref name or a run:sha256:… oid and resolve it first, then apply the run-type check resume_run_v3 performs; a target that is not a run is refused before anything is written.

--ref is required, has no default, and is not confined

--ref must be given on both verbs and there is deliberately no default. A fork's ref update is an unconditional overwrite — it compare-and-swaps against the value it just read — so a defaulted destination would mean a bare tine repo-fork RUN silently advancing whatever ref the default named. Every fork destination is an explicit operator choice.

The CLI is also deliberately not confined to experiments/*. That confinement is real, and it stays where it is: at the MCP boundary, in mcp_repository._writable_ref. Its threat model is model-controlled input — the ref an MCP client picks is chosen from run content it just read, so an unconfined fork tool would make "fork onto heads/main" a one-step prompt-injection payload. An operator at a terminal already holds the repository and can move any ref with tine trust promote or a direct write, so confining the CLI would buy no security and would only push people around it. The CLI still validates the ref name (canonicalizing it before the write, so a malformed ref is refused rather than reported after the object exists).

Their --json object is a receipt, emitted only after the write succeeded; a refusal is one tine repo-fork: <message> / tine repo-resume: <message> line on stderr with exit 1 and no JSON on stdout.

Forking a repository an older release wrote is a release gate exactly as appending to one is: both verbs run against a copy of every committed golden repository from 0.3.0 onward, and the copy must fsck clean afterwards.

short_id is deliberately absent from every v3 object. It is a twelve-character prefix, and a v3 oid's first twelve characters are the constant run:sha256: or event:sha25, so the field would be identical for every object in the repository. Human output shortens an oid as type:<12 hex digits> instead, dropping the hash-name segment rather than the digest.

entries carries no payload in either verb, exactly like the human line. Event payloads are unbounded recorded content; tine object <oid> --repo . is the verb that resolves one, with --resolve-blobs to follow its content blobs.

Everything these verbs print is recorded content — prompts, tool names, model strings, and ids an agent may have been fed by an attacker. Human rendering routes all of it through the CLI's terminal sanitizer, which strips control bytes and bidi overrides and escapes markup. --json does not sanitize: it is not a terminal, and a consumer must see the bytes as recorded.

V2 migration

tine init .
tine repo migrate --to-v3 legacy.tine --repo . --ref heads/imported

With a key flag (--key-env/--key-file/--pubkey/--trust-embedded-key/--pin) the source's signature must verify, or the import is refused unless --allow-unverified.

Migration:

1. stores the exact original v2 bytes as a legacy blob;
2. records the original checksum and signature-verification result;
3. redacts structured v3 content before canonicalization and hashing;
4. rebuilds object IDs and causal links;
5. stores a deterministic old-step-ID → event-ID map;
6. marks any v2 signature as legacy_blob_only.

Migration is fail-closed by default for a bad integrity digest and for any signature the caller requested to verify. --allow-unverified is an explicit escape hatch and the failed result remains recorded. The exact legacy blob is intentionally not redacted, so it can contain credentials present in the source artifact and should be treated as sensitive.

It never claims a v2 signature authenticates newly generated v3 objects. This also closes a v2 identity limitation: v2 compatibility saves can redact fields after a step ID was formed, whereas v3 always hashes the redacted stored bytes.

Agent-facing traces

Recorder starts a run with code/dirty patch, environment, policy, budget, and pricing manifests, then appends normalized events live. The code manifest marks capture failures explicitly when Git state is unavailable or a patch exceeds the 16 MiB capture ceiling. Untracked paths mark the worktree dirty and are listed, but their contents are not captured, so the manifest is explicitly incomplete until they are added or staged. Importers accept native OpenTine records, JSONL, and OpenTelemetry GenAI spans or complete OTLP/JSON exports. Malformed JSONL lines are skipped independently, large integers are string-preserved, and imported dependencies are ordered before recording. Span lookup is qualified by trace identity; duplicates and dependency cycles are rejected, while unresolved boundaries in partial traces are retained on the stored event as unresolved_span_refs. Framework importers best-effort normalize common serialized shapes from LangChain, LlamaIndex, AutoGen, CrewAI, and OpenAI Agents.

Import parsers can normalize larger input sources, but one stored Recorder run is limited to 3,000 events. The limit is checked before any event data is written; bulk imports preserve dependency order and commit one final run snapshot. This keeps even a worst-case graph with unique input and output blobs below the 10,000 object pack-negotiation ceiling.

Local search retrieves successful runs and evaluation scores. context_slice walks only parent and causal links needed for an event. Forking can replace model, prompt, or policy from the last good event. Evaluations and approvals are content-addressed, tamper-evident attestations; the signer is self-asserted unless the attestation is signed at tine-attest/1, which an operator does with tine trust attest --key-file … and checks with tine repo-verify. Promotion is a CAS ref update.

When an MCP server starts inside a repository, it adds search_runs, inspect_object, context_slice, semantic_diff, fork_run_v3, resume_run_v3, evaluate_run, and attest_run, plus verified object resources. promote_run is not among them by default. It is registered only when the host opts in with allow_promotion=True, because a promotion ref is a release gate and the run content an MCP client reads is untrusted, so text recorded inside a run can ask the model to promote a run of an attacker's choosing. fork_run_v3 and resume_run_v3 may write only experiments/* refs; the namespace is checked on the canonical ref name, so mainline, promotion, tag, and remote-tracking refs stay operator-only.

Promotion being an operator CLI verb (tine trust promote) does not change that default. The two surfaces are trusted differently on purpose: an operator at a terminal already holds the repository, while a model over MCP is acting on content it read out of that repository. allow_promotion remains False.

CLI ↔ MCP parity, and the four places they differ

Every MCP tool has a CLI verb and every v3 verb is accounted for. tests/test_repo_cli_parity.py holds the map and fails CI in both directions: a new tool without a verb is red, and so is a verb that is neither mapped to a tool nor listed as deliberately CLI-only.

The CLI-only verbs are init, fsck, pack, migrate-v3, fetch, push, and clone — repository administration and transport, each handing out a filesystem path, a network endpoint, or a credential that has no business being driven by run content — plus repo-show and repo-log, which render a run tree and an event ancestry for a terminal; a model reads the same objects through inspect_object and context_slice.

Four differences are deliberate. Each is asserted as a fact in the parity test, so making the surfaces agree turns CI red and forces the argument to be re-made.

1. promote is unconditional on the CLI, opt-in over MCP. No flag, no environment variable. The reasoning is the paragraph above.
2. repo-fork / repo-resume are not confined to experiments/*. The namespace confinement is an MCP-boundary control against model-chosen refs; see the lineage-verb section.
3. CLI --json receipts are a superset of the MCP return. fork_run_v3 returns {ref, run_id} — a model's working set. The CLI receipt adds what the operator typed and what it resolved to, because it is an audit artifact. Shared keys always carry the same values.
4. The CLI refuses non-finite evaluation scores; MCP passes them through. tine trust evaluate --score q=nan is refused at the argument door, naming the flag, before any engine call, because repo-search averages scores. evaluate_run takes an already-typed dict[str, float] and adds no check of its own, so the value travels into canonical_json and is refused there as a kernel error about JSON encoding. Nothing non-finite is stored either way — the kernel is the backstop — but the CLI is the stricter and more legible surface, and that asymmetry is recorded rather than levelled.

Search and inspection fail closed at explicit implementation ceilings: search indexes at most 100,000 objects, 10,000 candidate runs, and 100,000 aggregate event references while bounding structured and blob source bytes. Direct blob inspection returns a verified 512 KiB prefix; resolved inspection returns at most 1 MiB across 64 referenced blobs and marks truncated results explicitly.

Compression and packs

Since 0.11 Repo.init creates a version-2 repository (config.json carries "version": 2 and "storage": {"compression": …}) whose loose objects are stored compressed (zstd on Python 3.14 or with opentine[zstd], zlib otherwise). Object ids stay over the canonical bytes, so a compressed and a plain repository hold the same objects. OPENTINE_COMPRESS=0 or tine --no-compress creates the plain version-1 layout older releases read, and an existing version-1 repository is never written compressed until tine repo storage gc upgrades it.

The maintenance verbs below are also spelled under the repo group (0.16+): tine repo storage gc, tine repo storage repack, tine repo storage du, tine repo storage prune, tine repo storage compress, tine repo storage fsck and tine repo storage reindex are exactly tine gc, tine repack, tine du, tine prune, tine compress, tine fsck and tine reindex; both spellings take the same flags.

• tine repo storage gc moves loose objects into compressed local packs (.tine/packs/*.tpk + .tidx): a pack is written and read back object by object before anything loose is deleted (about 18x less disk on real sessions). Redundant copies of received transfer packs are removed unless --keep-transfer-packs.

• tine repo storage gc and tine repo storage repack hold .tine/gc.lock, so two packers never race. The lock (like every ref lock) records pid, host and start time; a lock left by a crashed process on the same host is taken over once that process is provably gone, on every platform and never by age alone (OPENTINE_GC_BREAK_LOCK=1 forces it). Any other lock is reported with its path. An interrupted gc loses nothing and can be re-run.

• tine repo storage repack merges every local pack into one.

• Behaviour change in 0.39: tine repo storage gc (and repack) prunes unreachable objects again by default (--prune=retention), but only those outside the retention window and never held ones. The window comes from .tine/retention.toml (default 14 days when the file is absent):

  keep_days = 30                  # default window
  legal_hold = ["blob:sha256:…"]  # never deleted
  [refs]                          # ref namespace (fnmatch) -> days
  "heads/release/*" = 365
  [status]                        # run status -> days
  failed = 7
  [tier]                          # assurance tier -> days
  high = 3650
  

  An object keeps the longest window that applies (rules only lengthen it); facts are read from a run object's ref, status and assurance_tier, other objects get keep_days. Prune's safety floor (MIN_GRACE, just over an hour) and quarantine still apply. --prune=never restores the 0.35-0.38 behaviour and --prune=now ignores the window (holds still kept). tine retention show prints the effective policy; tine retention check is a dry run listing what default gc would delete now. The same RetentionPolicy implements the remote RetentionHook (retain_until/before_delete).

• From 0.35 to 0.38 tine repo storage gc deleted no unreachable objects unless --prune=now. tine hold add|list|release <oid|ref> records holds in .tine/holds; prune, gc --prune=now, rewind --apply and remote deletes and purges keep held objects and everything they reach.

• tine repo storage gc --upgrade (0.13+, opt-in, never automatic) moves a repository to storage version 3: packs of large single-kind blocks, run snapshots stored as their parent plus the newly appended events, events without their empty default fields, and a binary index. Every read rebuilds the original bytes and checks them against the object id. About 2x smaller than version-2 packs on recorded sessions. opentine 0.12 and older refuse a version-3 repository (repository config is incompatible); tine repo storage gc --downgrade rewrites it for version 2 so they can open it again. Later gc, repack, auto-pack and prune keep writing v3 packs in a version-3 repository.

• tine repo storage gc --pack-refs (0.37+, opt-in) folds every loose ref into one .tine/packed-refs file (<oid> <ref> lines, sorted), so list_refs is one read instead of one open per ref; later gcs keep folding new loose refs. Reads look at the loose file first, then the packed value; writes still land loose (and shadow it); a delete removes both under packed-refs.lock. A loose ref is removed only when it still equals its packed value under its own lock, so a concurrent writer is never undone. Because opentine 0.36 and older never read packed-refs (they would see the refs as missing, and prune would treat their history as unreachable), the opt-in writes "refs": "packed" and "min_reader": "0.37.0" to config.json (object format 4 is required), and those releases refuse the repository; without the opt-in gc never touches a ref. tine repo storage gc --unpack-refs writes every ref loose again and lowers min_reader back to 0.26.0. Reflogs stay one file per ref.

• tine repo storage gc --blob-deltas (0.40+, opt-in) stores blobs in local v3 packs as blob deltas: the bytes that differ from a similar blob of the same pack (its base, chosen by the longest shared prefix plus suffix among the 16 blobs before it in recording order). A run's transcript, which put_transcript saves whole after every turn, then costs one turn per save. Every read rebuilds the canonical bytes and re-hashes each object of the chain against its id; a chain deeper than 64 or one that loops is refused with an error naming the object. The opt-in moves the repository to storage version 3 and object format 4 and writes "packs": "delta" plus "min_reader": "0.40.0", so opentine 0.39 and older refuse it instead of failing on a pack entry. Push, fetch, bundles and exports send canonical objects, so peers on older releases are unaffected. gc and auto-pack both write the form once opted in (auto-pack only because it met the 0.40 latency gate, docs/bench/0.40-cdc-bakeoff.md). tine repo storage gc --no-blob-deltas rewrites every pack without deltas, then lowers min_reader again (0.37.0 with packed refs, else 0.26.0).

• Pack indexes are read lazily (0.37+): the binary v3 .tidx is mapped into memory (read into memory on Windows, where a mapping would stop gc from replacing it) and searched through a fanout table, instead of every entry being parsed into one dict at a process's first object read. Opening an index checks its header, length, trailing SHA-256 and block table; each entry a lookup returns is checked on its own; the whole-file rules (sorted, unique, tiling) run for bulk walks (gc, du, verify_packs). has() asks the pack index before probing the loose path. Version-1 JSON indexes are still read.

• Auto-pack (0.37+) triggers on cold loose objects by count (256, or "auto_pack": {"objects": N} / OPENTINE_AUTO_PACK=N) or by the disk space they occupy (16 MiB, or "auto_pack": {"bytes": N} / OPENTINE_AUTO_PACK_BYTES=N), each file counted as at least one 4 KiB block.

• tine repo archive create FILE writes the whole repository (objects, refs, shallow boundaries) as one compressed file; tine repo archive restore FILE DEST verifies the pack digest and every object id and sets refs through the normal validated path, into a new repository only.

Push, fetch and clone are unchanged: transfer packs carry canonical bytes, so compressed and plain repositories exchange objects freely.

Packs and synchronization

TINEPACK3 packs are canonical manifests of verified envelopes, compressed with a SHA-256 checksum. Negotiation sends only objects reachable from wanted IDs that the receiver does not already have. Since 0.37 a server advertises "negotiation": ["haves", "tips"], and a fetching client then sends its ref tips (at most 4096 ref targets it holds) instead of every local oid as a have; the server's walk stops at a tip it holds, though runs met there still bring their associations (a redundant re-send at worst, never a missing object). A client with shallow boundaries, or facing a server that does not advertise tips (0.36 and older), sends haves exactly as before; a 0.36 client's haves are still answered by a 0.37 server. Packs support shallow boundaries and filtered fetch without pretending omitted history is present. Compressed transfers and decompressed manifests are capped at 256 MiB and trailing zlib streams/data are rejected. Repository-wide shallow state is validated and capped at 10,000 object IDs / 1 MiB, then cached for link verification; local object and negotiation listings stop at protocol limits before sorting. Client control responses are streamed under a 1 MiB raw-byte cap and non-identity HTTP content encodings are rejected; resumable offsets must advance, loops are bounded, transient short reads retain partial upload state, and abandoned upload state is reaped.

Peers pick the transfer format from each other's pack_formats:

• 0.37 and newer (with zstd) send TINEPACK5 (SPEC §2.8). Objects come in recording order, as one zstd long-distance frame (window 2^27, level 9) flushed into 4 MiB blocks that are read in order. Readers refuse a larger window.
• 0.34–0.36 peers get TINEPACK4.
• 0.11 and older peers get TINEPACK3.

Bodies compressed past 1000:1 (codec v2 frames) go only to a peer that advertises 0.35 or newer, and never into a .tine file.

scripts/bench/tinepack5.py measures the wire size and compress throughput of TINEPACK4 and TINEPACK5 on a synthetic corpus of similar runs with growing histories.

The pack layer is explicit about omitted history; the local read surface treats it in two different ways, and a reader of a shallow clone must know which. log, diff, and context_slice stop at the boundary silently: a cut object is skipped, and SemanticDiff carries no truncation marker, so its cost, latency, tool-path, and artifact aggregates are computed only over the events present locally and omit history beyond the boundary without saying so. load_run, fork, and resume instead refuse, with a typed error naming the object that lies beyond the repository's shallow fetch boundary and directing the caller to deepen the fetch. Treat aggregates read from a shallow clone as lower bounds until the clone is deepened.

The HTTP protocol exposes:

• unauthenticated capability discovery;
• authenticated ref discovery and missing-object negotiation;
• deterministic pack download and resumable, checksummed upload;
• depth/object-type filtered fetch;
• compare-and-swap ref updates.

Clients require HTTPS except when the base URL host is a literal loopback IP address (127.0.0.1, ::1), which is accepted over plain http:// with no opt-in, or when --allow-insecure development is requested explicitly. The hostname localhost is not a literal address and is refused without that flag. Authenticated clients ignore ambient proxy variables so local development credentials are not forwarded unexpectedly. Push uploads verified missing objects before attempting the CAS ref update.

Serverless sharing: bundles and plain git remotes

Since 0.15, runs can be shared with no opentine server.

• tine share bundle create FILE [REFS...] | --all writes one self-verifying file: TINEBUNDLE1 magic, a SHA-256 over the rest, a canonical JSON manifest (format, the pack's SHA-256 and length, ref -> object id) and an ordinary transfer pack. The pack carries each run's associations (annotations and attestations) exactly as a fetch would. --all bundles every local ref except remotes/*; a bare name means heads/NAME. The pack is TINEPACK3, which every release reads. With --pack-format TINEPACK5 (0.37+) it is TINEPACK5 with run deltas and slim events. That pack is much smaller, but only 0.37 and newer can read it.
• tine share bundle verify FILE checks it without a repository: size caps, both digests, canonical manifest, every ref name through normalize_ref (no .., absolute, upper-case, control characters or remotes/*), every ref target present in the pack, and every packed object against its id.
• tine share bundle unbundle FILE (alias import) verifies, installs the objects and moves refs only under remotes/<name>/… (heads/ dropped, like fetch); <name> is --as NAME or derived from the file name. A tracking ref that already exists moves only to a descendant (its forked_from chain) unless --force; refs move atomically (all or none). Re-importing is a no-op.
• tine share bundle push REMOTE [REFS...] and tine share bundle fetch REMOTE [--as NAME] carry a bundle over any git remote (GitHub, Gitea, a bare repository, Radicle's git, a local path). The remote gets one ref, refs/tine/bundle: a commit whose tree is a single bundle.tine blob, parented on the commit it replaces, so a push that raced another is refused by git's fast-forward check. git runs in a throwaway bare repository with an argument list, a scrubbed environment, the hardened -c set, only the file/https/ssh protocols (no ext::, no option-shaped remotes) and --timeout (default 120 s). The fetched blob is size-checked before it is read and fully verified before any local ref moves. Credentials come from your normal git/ssh setup.

Bundles are capped at the transfer-pack limit (256 MiB) plus a 4 MiB manifest. Unlike tine repo archive (whole repository, new destination only), a bundle carries chosen refs into an existing repository. Each bundle push uploads a full snapshot of the chosen refs, not a delta.

Reference self-hosted remote

Extension interfaces cover ObjectStore, IndexBackend, IdentityProvider, AuthorizationPolicy, KeyProvider, AuditSink, RetentionHook, and AdmissionPolicy. The reference deployment implements them with encrypted filesystem objects and SQLite. A StaticTokenIdentityProvider supports development identities. Retention hooks gate object deletion, and admission policies can reject pack bytes, object counts, or ref updates; resumable uploads invoke admission at declaration and again after pack inspection, so a policy can bound both bytes and object counts.

The security properties of that deployment — encryption at rest and key provision, identity and tenant-scoped authorization, the HMAC-chained audit log and its authenticated head outside SQLite, the listing and annotation ceilings, and the operator responsibilities that remain — are specified in SECURITY_MODEL.md and are not restated here.

tine repo serve --root DIR [--tenant T] ACTION runs operator maintenance on that storage instead of serving: refs, delete-ref NAME [--expect OID], associations OID, delete-objects OID… [--from FILE|-] [--dry-run] and purge [--grace-seconds N] [--dry-run], each printing JSON. It needs TINE_KMS_KEY (the server's key) and no token; see SECURITY_MODEL.md for what each keeps and refuses.

A fetch response may carry Opentine-Associations-Omitted: N — N runs whose annotations or attestations did not all fit in the pack. Clients that predate it ignore it; tine share fetch reports it as associations_omitted. Associations are budgeted by object count, encoded pack bytes, source bytes and the shallow-boundary list a depth-limited pack carries, so an association set that does not fit is left out rather than failing the fetch. A pack that still cannot be built with associations — the receiver's haves or a type filter can add shallow boundaries after the budget is checked — is rebuilt from the object graph alone and its runs reported as omitted; this applies to fetch, typed fetch and push. A graph that does not fit by itself still fails. The association phase also has a total read budget, twice the pack body budget, charged before every read and never returned when a set is rolled back; once it is spent, the remaining runs' associations are reported omitted rather than read, so one oversized evidence set shared by many runs is read a bounded number of times.

fetch adopts the advertised annotations/* refs of the runs it delivers — never one whose chain holds an annotation recorded as foreign at install, so a remote cannot plant a note through its ref for a run it only names as a shallow boundary. For a run the local repository already held, the remote's annotation head is stored as remotes/<name>/annotations/<digest> and left unadopted; tine fetch --adopt-annotations (fetch(..., adopt_annotations=True)) adopts those deliberately. The .tine/unadopted record is bounded: a foreign annotation that no longer fits is not installed (with a warning), and a damaged or over-limit record makes the unheaded fallback adopt nothing instead of failing. So do two or more independent unheaded chain tips for one run (with a warning); the next tag of that run writes a fresh head under its annotations/<digest> ref. Unheaded lookups use the non-authoritative target caches in .tine/indexes/. tine share clone refuses a target that already holds refs or objects (use fetch). A remote URL may not carry user:pass@, a query or a fragment — checked on the URL as given, before an embedded tenant (…/v1/tenants/acme) is split off; pass the token with --token-file or TINE_REMOTE_TOKEN — a remote tenant must match [a-z0-9][a-z0-9._-]*, and remote errors name only the HTTP status and path. tine repo serve purge --grace-seconds N with N below 60 needs --force (or --dry-run).

POST /search returns {"objects": [...], "truncated": bool} — at most 1,000 objects, newest first. GET /audit/verify returns {"ok", "status", "warnings"} (no chain head). A busy server answers 503; retry.

The 0.3.0 scope is an enterprise repository foundation. The bundled bounded WSGI server targets development and small self-hosted deployments, not turnkey high availability. S3-compatible object storage, PostgreSQL/search backends, managed identity configuration, and a hosted control plane remain production adapters or later 0.3.x work—not claims of this reference server.

Linking runs to git

A commit names the runs that produced it with Tine-Run: <run> trailers (added by the hook tine repo integrate git install-hook installs) or a note under refs/notes/tine (tine repo integrate git note). In a repository the run is named by its object id; in a runs directory by its run id. tine repo integrate git log --commit SHA and tine agent blame FILE:LINE --git follow those links back to runs and steps. Nothing about the link is stored in opentine's own objects: trailers and notes are plain git and travel with git push (push refs/notes/tine explicitly, as with any notes ref).