Documentation
¶
Overview ¶
Package importer brings history over from another tool so day one here isn't a zero dashboard. It holds the format mappers (jsonl, csv, posthog, mixpanel, umami) and the batching senders shared by the two import surfaces: the `smolanalytics import` CLI (batches POSTed to /v1/events) and the MCP import_events tool (batches ingested straight into the running server's store). One parser, one batcher — the two paths cannot drift.
Index ¶
- Variables
- func MapAmplitude(r io.Reader, emit EmitFn, skip SkipFn) error
- func MapCSV(r io.Reader, emit EmitFn, skip SkipFn) error
- func MapJSONL(r io.Reader, emit EmitFn, skip SkipFn) error
- func MapMixpanel(r io.Reader, emit EmitFn, skip SkipFn) error
- func MapPostHog(r io.Reader, emit EmitFn, skip SkipFn) error
- func MapPostHogAPI(r io.Reader, emit EmitFn, skip SkipFn) error
- func MapUmami(r io.Reader, emit EmitFn, skip SkipFn) error
- func MapperFor(format string) (func(io.Reader, EmitFn, SkipFn) error, error)
- type EmitFn
- type HTTPSender
- type IngestSender
- type Remote
- type Sender
- type SkipFn
- type Summary
Constants ¶
This section is empty.
Variables ¶
var BatchSize = 5000
BatchSize is events per batch — half the server's 10k batch cap. A var so tests can shrink it to exercise multi-batch sends.
Functions ¶
func MapAmplitude ¶ added in v0.9.4
MapAmplitude reads Amplitude's Export API output: JSON, one event object per line, shaped
{"event_type":"Signed Up","user_id":"u1","event_time":"2024-01-01 12:00:00.000","$insert_id":"z","event_properties":{...},"user_properties":{...}}
Unlike Mixpanel, name/id/time are top-level: event_type→name, user_id (falling back to device_id then amplitude_id)→distinct_id, event_time (a space-separated stamp) →timestamp, $insert_id→event id so re-import is idempotent, and event+user properties merge into props. Amplitude's export is gzipped by default, so a raw .json.gz is auto-decompressed here.
func MapCSV ¶
MapCSV reads a generic CSV: header row, a name (or event) column, a distinct_id (or user_id / anonymous_id) column, an optional time (or timestamp) column, and every other column lands as a string property.
func MapJSONL ¶
MapJSONL reads our own export format: one /v1/events-shaped JSON object per line (GET /v1/export?format=jsonl). Ids are kept, so re-importing is idempotent.
func MapMixpanel ¶ added in v0.9.0
MapMixpanel reads Mixpanel's Raw Event Export (JSONL): one object per line shaped
{"event":"Signed up","properties":{"time":1704067200,"distinct_id":"u1","$insert_id":"z",...}}
Unlike our own JSONL, the name/id/time live INSIDE properties and time is a unix stamp, so feeding a Mixpanel export to --format=jsonl silently drops every row (no top-level name). $insert_id becomes the event id, so re-importing the same export is idempotent.
func MapPostHog ¶
MapPostHog reads PostHog's events CSV export (Activity → Export). Properties travel either as one embedded-JSON "properties" column or flattened into "properties.$browser"-style columns — both land as event properties here.
func MapPostHogAPI ¶ added in v0.32.0
MapPostHogAPI reads PostHog's REST shape, one JSON object per line:
{"id":"...","event":"$pageview","distinct_id":"u1","timestamp":"2026-01-01T00:00:00Z","properties":{…}}
Separate from MapPostHog because that one reads the CSV their UI exports, and the two genuinely differ: the API nests properties as real JSON rather than a quoted string column, and carries `id` where the CSV carries `uuid`. Feeding one to the other silently maps zero rows.
func MapUmami ¶
MapUmami reads Umami's website_event CSV export. Rows without an event_name are pageviews → "$pageview" with url_path as the "path" property (the exact shape our web view reads). session_id becomes distinct_id: Umami keeps no stable cross-session visitor id, so user-level reports treat each session as a user.
Types ¶
type EmitFn ¶
EmitFn ships one mapped event; its error aborts the import (it means a send failed, not a bad row). SkipFn counts a row that couldn't be mapped.
type HTTPSender ¶
type HTTPSender struct {
// contains filtered or unexported fields
}
HTTPSender accumulates events and POSTs them to /v1/events, flushing on count or on approximate body size (the server rejects requests over 4MB). This is the CLI's sender; progress lines go to out.
func NewHTTPSender ¶
func NewHTTPSender(host, key string, out io.Writer) *HTTPSender
func (*HTTPSender) Add ¶
func (s *HTTPSender) Add(e event.Event) (int, error)
Add queues one event, flushing when the batch is full. Returns how many events that flush sent (0 when it only queued).
func (*HTTPSender) Flush ¶
func (s *HTTPSender) Flush() (int, error)
Flush POSTs the queued batch. A rejected batch aborts the import; batches already sent stay stored, so a re-run only avoids duplicates for the jsonl format (ids are preserved there and the server dedupes on id).
type IngestSender ¶
type IngestSender struct {
// contains filtered or unexported fields
}
IngestSender batches events for a direct in-process ingest — the same flush thresholds as the HTTP sender (batch count + approximate encoded size), minus the HTTP hop. The MCP import_events tool uses it to write the server's own store, so both import paths ship identical batches.
func NewIngestSender ¶
func NewIngestSender(ingest func([]event.Event) error) *IngestSender
func (*IngestSender) Flush ¶
func (s *IngestSender) Flush() (int, error)
type Remote ¶ added in v0.32.0
type Remote struct {
Vendor string // "posthog" | "mixpanel"
Key string // API key/secret. Never persisted.
Project string // PostHog project id
Host string // vendor host override — PostHog is commonly self-hosted or on EU cloud
From time.Time
To time.Time
}
The vendor key is used for the length of one request and never written anywhere: not to the event store, not to settings, not to the audit log. A migration credential is the most dangerous secret a user will ever hand this tool — it reads their entire analytics history — and the only safe place for it is nowhere.
func (Remote) Fetch ¶ added in v0.32.0
Fetch opens the vendor's export stream. The caller closes it.
func (Remote) Format ¶ added in v0.32.0
Format is the mapper this vendor's stream should be parsed with.
func (Remote) Validate ¶ added in v0.32.0
Validate refuses an unbounded pull.
A date range is REQUIRED rather than defaulted. Defaulting to "everything" turns one mistyped call into a multi-year pull that burns the user's vendor rate limit and lands a pile of history they did not ask for — and on a metered plan here, bills them for it. Making the caller state the window means the size of the import is always something they chose.
type Sender ¶
Sender ships mapped events in batches. Add queues one event (flushing when the batch is full) and Flush ships whatever remains; both return how many events that call actually shipped.
type Summary ¶
type Summary struct {
Parsed int // rows mapped to events
Skipped map[string]int // rows dropped, counted per reason
Sent int // events actually shipped (0 on a dry run)
Preview []event.Event // the first 3 mapped events, for dry-run eyeballing
}
Summary is what one import run did — the exact counts, never estimates.
func Run ¶
Run parses src with the format's mapper and ships batches through send. On dryRun it parses and validates only — send is never called. A send error aborts the run (batches already shipped stay shipped); a bad row never does.
func (Summary) SkippedTotal ¶
SkippedTotal sums the per-reason skip counts.
Source Files