Engram

A multi-model LLM serving platform with a hierarchical memory manager — keeping many models warm across GPU VRAM, host RAM, and NVMe so that switching between them costs seconds instead of minutes.

Status: Stage 1 in progress. The gateway, telemetry, and load-test harness are written; the memory hierarchy below is not. Stage 0's measurements have not been taken, so every latency figure here is still a design target. The architecture and roadmap live in PLAN.md.

⚠️ Nothing in this repository has been executed. It was written on a machine with no Python and no GPU. Treat the first run of anything — spikes, gateway, tests — as part of the work. HANDOFF.md is the order to do it in.

A single GPU can hold one or two models. A useful assistant needs many — a chat model, a vision model, a coder, an embedder — and users switch between them unpredictably.

The naive answer is a process per model, which doesn't fit in VRAM. The other naive answer is loading on demand, but a cold start for a 27B model is 30–90 seconds of process spawn, weight load, and CUDA graph capture. Neither is acceptable on a request path.

Engram treats GPU memory as the top of a cache hierarchy rather than a fixed allocation. Models are promoted and demoted between tiers based on measured demand, so the model a user is about to ask for is already warm.

An engram is the physical trace a memory leaves behind — the substrate where something is stored so it can be recalled later. That is what this is.

Latencies are design targets, not measurements. Validating them is the first task in the roadmap.

The RAM tier is the important one, and it exists because of vLLM's sleep/wake API: sleep level=1 releases VRAM while keeping the process and CUDA context alive, so waking is a host-to-device copy rather than a cold start.

The memory manager is a state machine over model instances. It works at model granularity — it never touches individual tensors, because the inference engine owns its own allocator and KV cache.

stopped ──load──> loading ──> awake
                                │  ▲
                      sleep(1)  │  │  wake
                                ▼  │
                             sleep-1        (weights in host RAM, VRAM freed)
                                │  ▲
                      sleep(2)  │  │  wake
                                ▼  │
                             sleep-2        (weights on NVMe, process alive)
                                │
                             evicted ──> stopped

Which model sits in which state is decided by a cost-aware cache policy scoring frequency, recency, load cost, and priority — evaluated against a plain LRU baseline, and adopted only if it wins.

                        Client
                          │
                    API Gateway  ──── auth, rate limiting, quotas
                          │
                   Request Router  ──── capability + intent → smallest capable model
                          │
                      Scheduler  ──── admission control, backpressure, queueing
                          │
        ┌─────────────────┼─────────────────┐
        │                 │                 │
   Model Registry   Memory Manager     Telemetry
   metadata,        tier placement,    metrics, traces,
   versions         lifecycle          replay logs
        │                 │                 │
        └────────┬────────┴────────┬────────┘
                 │                 │
          Inference Engine    KV Cache Layer
             (vLLM)           (prefix + offload)
                 │
            GPU Worker Node

Engram builds: the lifecycle manager, cache policy, scheduler with admission control, request router, prefetch engine, and the control plane around them.

Engram does not build: the inference engine. Continuous batching, paged attention, prefix caching, and CUDA kernels come from vLLM. KV offload comes from LMCache. Metrics go to Prometheus and Grafana. Roughly a third of the originally proposed roadmap was work these already do — see PLAN.md §Appendix.

Stage 0 gates the rest: the caching design assumes a load cost that has not yet been measured on real hardware. Full analysis, corrected success criteria, and the risk register are in PLAN.md.

RTX 5090 worker nodes — 32 GB GDDR7, ~1.8 TB/s, compute capability sm_120.

Two constraints shape the design: there is no NVLink, so models are placed one per GPU rather than split across them; and consumer GeForce cards carry driver licensing restrictions for datacenter deployment, which needs resolving before any multi-node build-out.

The gateway needs no GPU. It ships a mock engine that fakes one with the latency targets from PLAN.md §5.2, which is enough to exercise the API, the metrics, the dashboards and the load-test harness end to end.

python -m venv .venv && source .venv/bin/activate
pip install -e ".[dev]"

cp engram.example.toml engram.toml
ENGRAM_AUTH_DISABLED=1 engram --check     # validate config, bind nothing
ENGRAM_AUTH_DISABLED=1 engram             # serve on :8000

curl localhost:8000/v1/chat/completions -H 'content-type: application/json' \
  -d '{"model":"mock-27b","messages":[{"role":"user","content":"hello"}],"stream":true}'

pytest
python loadtest/harness.py --model mock-27b --concurrency 8 --duration 20

Serving a real model means running a vLLM process per model and pointing a [[models]] entry at it — the gateway never loads weights itself. See the comments in engram.example.toml, and deploy/README.md for scraping and dashboards.

Requires CUDA 12.8+ and a PyTorch/vLLM build with sm_120 support on the worker nodes — older wheels will not run on Blackwell.

For a host that should just run the gateway, there is a Debian package: the gateway, its Python dependencies, a systemd unit, and a config with a generated API key. It installs nothing into the system site-packages and runs no pip at install time.

packaging/build-deb.sh                              # -> dist/engram_0.1.0-1_amd64.deb
sudo apt install ./dist/engram_0.1.0-1_amd64.deb
sudo systemctl start engram

curl -s localhost:8000/v1/chat/completions \
  -H "Authorization: Bearer $(sudo cat /etc/engram/api-key)" \
  -H 'content-type: application/json' \
  -d '{"model":"mock-27b","messages":[{"role":"user","content":"hello"}]}'

The mock engine is enabled by default, so a fresh install answers requests on a machine with no GPU. Config lands at /etc/engram/engram.toml, state under /var/lib/engram, logs under /var/log/engram.

The bundled dependencies include compiled extension modules, so the package is amd64 + Python 3.11 (Debian 12); build it on the release you are targeting. packaging/README.md has the layout, the portability trade-off, and the unauthenticated-/admin/ caveat.

PLAN.md              Architecture analysis, roadmap, risk register
DECISIONS.md         Resolved design decisions with reasoning
HANDOFF.md           First session on a GPU host — what to run, in what order
engram.example.toml  Annotated configuration template
spikes/              Stage 0 measurement harness — run these first
  s1_sleep_wake.py     vLLM sleep/wake latency (the load-bearing measurement)
  s2_toolchain.py      CUDA / PyTorch / vLLM validation on sm_120
  s3_kv_capacity.py    KV cache capacity and concurrency, FP16 vs FP8
  s4_bandwidth.py      Tier bandwidth — RAM, PCIe, VRAM, NVMe
src/engram/
  config.py            TOML config, validated at startup
  schemas.py           OpenAI-compatible wire types
  lifecycle.py         Readiness, admission slots, graceful drain
  api/                 Gateway — routes, auth, rate limiting, measurement
  engine/              Backends: vLLM over HTTP, and a GPU-free mock
  telemetry/           Prometheus series and the JSONL replay records
loadtest/harness.py  Closed- and open-loop load generator
deploy/              Prometheus scrape config, alert rules, dashboard queries
packaging/           Debian package — build script, systemd unit, maintainer scripts
tests/               Gateway behaviour, all against the mock engine

Stage 0 is still the gate. Start at spikes/README.md.

Recorded with reasoning in DECISIONS.md:

Two of these (topology, tenancy) rest on business facts and are marked for confirmation.

Apache License 2.0 — see LICENSE and NOTICE.

Copyright 2026 MegamTech. Development happens in the Megamtech Labs repository; this is the public mirror.