Skip to content

CLAUDE.md

Latest commit

 

History

History
376 lines (299 loc) · 16.8 KB

CLAUDE.md

File metadata and controls

376 lines (299 loc) · 16.8 KB
Raw

CLAUDE.md

General Workflow

When a task involves multiple steps (e.g., implement + commit + PR), complete ALL steps in sequence without stopping. If creating a branch, committing, and opening a PR, finish the entire chain.

Always commit after every turn. Don't wait for the user to ask — if you made changes, commit them before responding. Do not ask "shall I commit?" or "want me to commit?" — just commit. Committing is not a destructive or risky action; it is the expected default after every change.

PR descriptions should be concise and changelog-oriented: what changed, why, and how to use it. Do not include test plans, design decisions, or implementation details — those belong in specs and commit messages.

Never name private downstream projects or other private codebases in public code, tests, fixtures, documentation, commit messages, pull requests, or issues. Describe the reusable requirement generically and run the private-data scrub before publishing.

Project Overview

msgvault is an offline Gmail archive tool that exports and stores email data locally with full-text search capabilities. The goal is to archive 20+ years of Gmail data from multiple accounts, make it searchable, and eventually delete emails from Gmail once safely archived.

Architecture (Go)

Single-binary Go application:

msgvault/
├── cmd/msgvault/            # CLI entrypoint
│   └── cmd/                 # Cobra commands
├── internal/                # Core packages
│   ├── tui/                 # Bubble Tea TUI
│   ├── query/               # DuckDB query engine over Parquet
│   ├── store/               # SQLite database access
│   ├── deletion/            # Deletion staging and manifest
│   ├── gmail/               # Gmail API client
│   ├── sync/                # Sync orchestration
│   ├── oauth/               # OAuth2 flows (browser + device)
│   └── mime/                # MIME parsing
│
├── go.mod                   # Go module
└── Makefile                 # Build targets

Quick Commands

# Build
make build                    # Debug build
make build-release            # Release build (optimized)
make install                  # Install to ~/.local/bin or GOPATH
make test                     # Run tests
make lint                     # Run linter

# CLI usage
./msgvault init-db                                    # Initialize database
./msgvault add-account you@gmail.com                  # Browser OAuth
./msgvault add-account you@gmail.com --headless       # Device flow
./msgvault add-account you@acme.com --oauth-app acme  # Named OAuth app
./msgvault sync-full you@gmail.com --limit 100        # Sync with limit
./msgvault sync-full you@gmail.com --after 2024-01-01 # Sync date range
./msgvault sync-incremental you@gmail.com             # Incremental sync

# TUI and analytics
./msgvault tui                                        # Launch TUI (press 'a' inside to filter by account)
./msgvault tui --local                                # Force local (override remote config)
./msgvault build-cache                                # Build Parquet cache
./msgvault build-cache --full-rebuild                 # Full rebuild
./msgvault stats                                      # Show archive stats

# Apple Mail import
./msgvault import-messenger --me you@facebook.messenger ~/facebook-dyi   # Import Facebook Messenger DYI
./msgvault import-emlx                                # Auto-discover accounts
./msgvault import-emlx ~/Library/Mail                 # Explicit mail directory
./msgvault import-emlx --account me@gmail.com         # Specific account(s)
./msgvault import-emlx /path/to/dir --identifier me@gmail.com  # Manual fallback

# Microsoft Teams (delegated Graph)
./msgvault add-teams you@tenant.com          # Authorize Teams (browser OAuth)
./msgvault sync-teams you@tenant.com         # Sync Teams chats + channels
./msgvault sync-teams you@tenant.com --no-channels --limit 50

# Slack (user token from your own internal app)
./msgvault add-slack                         # Register a workspace (prompts for xoxp token)
./msgvault sync-slack                        # Sync channels, group DMs, DMs
./msgvault sync-slack --full                 # Repair: re-fetch + upsert in place

# Daemon mode (NAS/server deployment)
./msgvault serve                                      # Start HTTP API + scheduled syncs

# Maintenance
./msgvault repair-encoding                            # Fix UTF-8 encoding issues
./msgvault repair-dates                              # Preview missing/implausible date repairs
./msgvault repair-dates --apply                      # Apply repairs (and rebuild SQLite analytics cache)

# Agent skills
./msgvault skills install                             # Install agent skills (Claude Code, Codex)
./msgvault skills uninstall                           # Remove installed agent skills

Key Files

CLI (cmd/msgvault/cmd/)

  • root.go - Cobra root command, config loading

  • syncfull.go - Full sync command implementation

  • syncincremental.go - Incremental sync command

  • tui.go - TUI command, cache auto-build

  • build_cache.go - Parquet cache builder (DuckDB)

  • repair_encoding.go - UTF-8 encoding repair

  • import_emlx.go - Apple Mail .emlx import command

Core (internal/)

  • tui/model.go - Bubble Tea TUI model and update logic
  • tui/view.go - View rendering with lipgloss styling
  • query/engine.go - DuckDB query engine over Parquet files
  • query/sqlite.go - SQLite query engine (fallback)
  • store/store.go - SQLite database operations
  • store/schema.sql - Core SQLite schema
  • store/schema_sqlite.sql - FTS5 virtual table
  • deletion/manifest.go - Deletion staging and manifest generation
  • blobstore/blobstore.go - Attachment blob reads: packed CAS with loose-file fallback
  • gmail/client.go - Gmail API client with rate limiting
  • oauth/oauth.go - OAuth2 flows (browser + device)
  • sync/sync.go - Sync orchestration, MIME parsing
  • mime/parse.go - MIME message parsing

TUI Keybindings

  • j/k or ↑/↓ - Navigate rows
  • Enter - Drill down into selection
  • Esc or Backspace - Go back
  • Tab - Cycle views (Senders → Sender Names → Recipients → Recipient Names → Domains → Labels → Time)
  • s - Cycle sort field (Name → Count → Size)
  • r - Reverse sort direction
  • t - Jump to Time view (cycle granularity when already in Time)
  • a - Filter by account
  • f - Filter by attachments
  • Space - Toggle selection
  • A - Select all visible
  • x - Clear selection
  • d - Stage selected for deletion
  • D - Stage all messages matching current filter
  • / - Search
  • ? - Help
  • q - Quit

Database Schema

Core tables:

  • sources - Gmail accounts with history_id for incremental sync
  • conversations - Gmail thread abstraction
  • messages - Message metadata, foreign key to conversation
  • message_raw - Raw MIME blob (zlib compressed)
  • labels / message_labels - Gmail labels (many-to-many)
  • participants / message_recipients - From/To/Cc/Bcc addresses
  • attachments - Attachment metadata with content-hash deduplication
  • messages_fts - FTS5 virtual table
  • sync_runs / sync_checkpoints - Sync state for resumability

Schema files in internal/store/:

  • schema.sql - Core schema (SQLite; shared structure)
  • schema_sqlite.sql - SQLite FTS5 virtual table
  • schema_pg.sql - PostgreSQL-native schema with tsvector column + GIN index

Database backend: SQLite is the default. PostgreSQL is opt-in via [data].database_url and runs through the same store/query interfaces. See docs/internal/PG_STATUS.md for the current implementation state and follow-up work.

Test env: MSGVAULT_TEST_DB=postgres://... runs PostgreSQL-backed tests. pgvector tests require a PostgreSQL instance with the vector extension and the pgvector build tag.

There are two PostgreSQL configurations to cover: the pgvector build (make test-pg) and the shipped build, which has no pgvector tag (make test-pg-shipped). Run make test-pg-both rather than both of those — the tag changes the test binary of only nine packages, so the second full run would reprove the first. Fixtures build their schemas in concurrent batches (internal/testutil/pg_warm_pool.go) rather than one at a time: a fixture that finds the buffer empty builds several and the next few claim theirs for free. All of that happens inside the claiming fixture's own setup — the pool runs no background work, so it can never issue a statement while a test body is running. Each schema is still private to its test, still built by the same InitSchema() path, and still dropped on cleanup.

Parquet Analytics

The TUI uses denormalized Parquet files for fast aggregate queries (~3000x faster than SQLite JOINs).

~/.msgvault/
├── msgvault.db              # SQLite: System of record
└── analytics/               # Parquet: Aggregate analytics
    ├── messages/year=*/     # Partitioned by year
    └── _last_sync.json      # Incremental sync state

Workflow:

  1. Sync emails: ./msgvault sync-full you@gmail.com
  2. Launch TUI: ./msgvault tui (auto-builds cache if needed)

Parquet schema:

  • Denormalized: from_email, from_domain, to_emails[], labels[], etc.
  • Partitioned by year for efficient time-range queries
  • Compact: small fraction of SQLite size (excludes message bodies)

The TUI automatically builds/updates the Parquet cache on launch when new messages are detected.

Implementation Status

Completed

  • Gmail Sync: Full/incremental sync, OAuth (browser + headless), rate limiting, resumable checkpoints
  • MIME Parsing: Subject, body (text/HTML), attachments, charset detection
  • Parquet ETL: DuckDB-based SQLite → Parquet export with incremental updates
  • Query Engine: DuckDB over Parquet for fast aggregate analytics
  • TUI: Full-featured TUI with drill-down navigation, search, selection, deletion staging
  • Web UI: First-party daemon-served analytical interface for cross-modality search, grouping, timelines, files, people/domains, Saved Views, source status, deletions, settings, and optional task links
  • UTF-8 Repair: Comprehensive encoding repair for all string fields
  • Deletion Execution: Execute staged deletions via Gmail API (trash or permanent delete)

Not Yet Implemented

  • App-level encryption: Encrypt database and attachments at rest

Testing with Real Gmail Data

./msgvault init-db
./msgvault add-account you@gmail.com
./msgvault sync-full you@gmail.com --after 2024-12-01 --before 2024-12-15
./msgvault tui

Sync is read-only - no modifications to Gmail.

Test Data

Never use real people's names, email addresses, or identifiers in test fixtures. Use obviously synthetic names: alice, bob, Test User, user@example.com. Before committing test data, verify no real PII is present.

Exception: the provenance-documented Enron web UI fixture stored on the docs-fixtures orphan branch and its derived documentation screenshots may retain authentic names, addresses, and message text. This is a narrowly scoped public-research-data exception for the real-importer documentation smoke path; it does not permit copying those identities into ordinary tests or fixtures. The fixture manifest and README must record source provenance, attribution, the completed message-by-message sensitive-content review, and the reviewed artifact digest before publication.

Go Development

After making any Go code changes, always run go fmt ./... and go vet ./... before committing. Stage ALL resulting changes, including formatting-only files.

Testing

All Go tests use testify for assertions. Do NOT introduce new t.Errorf/t.Fatalf/t.Fatal/t.Error patterns — use assert.X or require.X instead.

Tests must exercise real behavior through the production path. Do not add tautological "fake TDD" tests that copy shell scripts into synthetic temporary trees, stub the primary commands, and only assert that the stub saw expected arguments or filenames. Those tests usually verify the fixture rather than the system. Prefer testing a real validator, parser, or public command output; for shell/docs workflows, use the actual repository script against the real docs tree or a built artifact. If a fake command is truly needed, the test must prove a stable external contract or regression that cannot be covered by the real path, and the commit message must explain why.

Do not add bash tests that grep shell scripts, workflows, config files, or docs for expected implementation text. Those checks are usually tautological. Use real execution, parser/tool-native validation, or a documented manual release check instead.

Mapping rule:

  • require.X — halts the test on failure (replaces what was t.Fatalf / t.Fatal). Use for setup operations or when subsequent assertions would be meaningless on failure.
  • assert.X — continues after failure (replaces what was t.Errorf / t.Error). Use for independent value checks where reporting multiple failures helps debugging.

Argument order: testify equality functions take (want, got), the opposite of Go stdlib convention. Always pass the expected value first: assert.Equal(t, want, got).

Imports:

import (
    "github.com/stretchr/testify/assert"
    "github.com/stretchr/testify/require"
)

Common conversions:

Stdlib Testify
if err != nil { t.Fatalf(...) } require.NoError(t, err)
if err != nil { t.Errorf(...) } assert.NoError(t, err)
if got != want { t.Errorf(...) } assert.Equal(t, want, got)
if !cond { t.Errorf(...) } assert.True(t, cond)
if len(s) != n { t.Errorf(...) } assert.Len(t, s, n)

Custom helpers: internal/testutil does NOT provide assertion wrappers — call testify directly. The non-assertion helpers (MakeSet, NewTestStore, builders, fixtures) are kept and may still be used.

Git Workflow

When committing changes, always stage ALL modified files (including formatting, generated files, and ancillary changes). Run git diff and git status before committing to ensure nothing is left unstaged.

Code Style & Linting

All code must pass formatting and linting checks before commit. A pre-commit hook is available via prek to enforce this automatically:

make install-hooks             # Install pre-commit hook via prek
make test                      # Run tests (SQLite default)
make test-pg-both              # Both PostgreSQL configurations, needs MSGVAULT_TEST_DB
make test-pg                   # PostgreSQL, pgvector build only
make test-pg-shipped           # PostgreSQL, shipped build only
make fmt                       # Format code (go fmt)
make lint                      # Run linter (auto-fix)
make lint-ci                   # Run linter (CI, no auto-fix)
go vet ./...                   # Check for issues

Standards:

  • Default gofmt configuration
  • Use error return values, wrap with context using fmt.Errorf
  • Table-driven tests

Code Conventions

  • Use Bubble Tea for TUI, lipgloss for styling
  • DuckDB for Parquet queries, go-duckdb driver
  • SQLite via marcboeker/go-duckdb for cache building, mattn/go-sqlite3 for store
  • Context-based cancellation for long operations
  • Route all DB operations through Store struct
  • Charset detection via gogs/chardet, encoding via golang.org/x/text/encoding

SQL Guidelines

  • Never use SELECT DISTINCT with JOINs - Use EXISTS subqueries instead (becomes semi-joins)

  • EXISTS is faster (stops at first match) and avoids duplicates at the source

  • Example - instead of:

    SELECT DISTINCT m.id FROM messages m
JOIN message_recipients mr ON mr.message_id = m.id
WHERE mr.recipient_type = 'from' AND ...

    Use:

    SELECT m.id FROM messages m
WHERE EXISTS (
    SELECT 1 FROM message_recipients mr
    WHERE mr.message_id = m.id AND mr.recipient_type = 'from' AND ...
)

  • Never JOIN or scan message_bodies in list/aggregate/search queries — this table is separated from messages specifically to keep the messages B-tree small for fast scans. Only access message_bodies via direct PK lookup (WHERE message_id = ?) when displaying a single message detail view. For text search, use FTS5 (messages_fts); if FTS is unavailable, search subject/snippet only.

Configuration

All data defaults to ~/.msgvault/:

  • ~/.msgvault/config.toml - Configuration file
  • ~/.msgvault/msgvault.db - SQLite database
  • ~/.msgvault/attachments/ - Content-addressed attachment storage
  • ~/.msgvault/tokens/ - OAuth tokens per account
  • ~/.msgvault/analytics/ - Parquet cache files

Override with MSGVAULT_HOME environment variable.

[data]
# data_dir = "~/custom/path"

[oauth]
client_secrets = "/path/to/client_secret.json"

# Named OAuth apps for Google Workspace orgs
# [oauth.apps.acme]
# client_secrets = "/path/to/acme_secret.json"

[sync]
rate_limit_qps = 5