README.md

Deployment and observability

Telemetry is Stage 1, not Stage 10 — PLAN.md §1: every later stage is an optimisation whose success is undefined without measurement.

prometheus --config.file=deploy/prometheus.yml

Three scrape targets, because no one of them can diagnose a regression alone:

Source	Answers
engram-gateway :8000	What clients experienced — TTFT, ITL, rejections, per-tenant volume
vllm :8001	What the engine did — queue time, running vs waiting, prefix cache hit rate, KV utilisation
dcgm :9400	What the hardware did — memory bandwidth, power, temperature

Gateway TTFT climbing while vLLM's queue time stays flat means the delay is ours. Both climbing together means it is capacity.

---

Dashboard

No dashboard JSON is committed. Hand-authoring Grafana JSON that nobody has loaded produces a file that looks authoritative and is quietly broken; build the panels from the queries below, then export the dashboard from Grafana and commit that.

Latency — the headline row

# TTFT p95 by model
histogram_quantile(0.95, sum by (model, le) (rate(engram_ttft_seconds_bucket[5m])))

# Inter-token latency p95 — the streaming UX metric (target <40 ms)
histogram_quantile(0.95, sum by (model, le) (rate(engram_inter_token_latency_seconds_bucket[5m])))

# End-to-end p95
histogram_quantile(0.95, sum by (model, le) (rate(engram_request_duration_seconds_bucket[5m])))

Throughput — the business metric (PLAN.md §5.1)

# Output tokens/sec, the number that actually matters
sum by (model) (rate(engram_tokens_total{kind="completion"}[1m]))

# Requests/sec by outcome
sum by (status) (rate(engram_requests_total[1m]))

Saturation

sum by (model) (engram_requests_in_flight)
sum by (reason) (rate(engram_rejected_total[5m]))
sum by (tenant) (rate(engram_rate_limited_total[5m]))

Prefetch — the Stage 8 gate

Real in shadow mode too, where the wake counters stay at zero. The first query is the decision; the second is the bill it comes with.

# Prediction hit rate. Below ~0.5 there is nothing here worth waking for.
sum(rate(engram_prefetch_predictions_total{outcome="hit"}[30m]))
  / sum(rate(engram_prefetch_predictions_total[30m]))

# Why speculative wakes did not happen. A flat line on no_slot means the awake
# budget is saturated and prefetch_evict = false is doing nothing at all.
sum by (outcome) (rate(engram_prefetch_wakes_total[30m]))

# Churn. Nonzero only under prefetch_evict — models slept for a guess.
sum by (model) (rate(engram_prefetch_evictions_total[30m]))

Engine internals — from vLLM's own metrics, not ours

vllm:num_requests_running
vllm:num_requests_waiting
vllm:gpu_cache_usage_perc
vllm:gpu_prefix_cache_hit_rate          # the Stage 6 number
histogram_quantile(0.95, rate(vllm:time_in_queue_seconds_bucket[5m]))

GPU — read §5.1 before adding a utilisation panel

# Achieved memory bandwidth. This, not GPU_UTIL, is what decode efficiency means.
DCGM_FI_PROF_DRAM_ACTIVE
DCGM_FI_DEV_POWER_USAGE      # 575 W TGP per card; three of these is ~1.7 kW

---

Request records

[telemetry].access_log_path is a JSONL file, one row per request. It is not a debug log — PLAN.md §6 defines the Stage 5 cache policy as evaluated by replaying Stage 1 telemetry against the LRU baseline, and this is that input. Ship it somewhere durable now; a policy cannot be evaluated against traffic that was never recorded.

Prompt text is never written. prompt_fingerprint is a truncated hash of the prompt prefix, which is what makes offline prefix-cache hit-rate estimation possible without keeping a copy of tenant data.

---

Draining a worker

curl -X POST http://worker:8000/admin/drain    # blocks until in-flight finish

/readyz goes 503 immediately so the load balancer stops sending work; /healthz stays 200, because restarting a draining process truncates it rather than fixing it. SIGTERM does the same thing via the lifespan hook — give the orchestrator a terminationGracePeriodSeconds above [server].drain_grace_s, or it will kill the process mid-drain.