Local models

Atomic Agent can run language models on your own machine. With a local model there are no per-token fees, prompts are processed by a llama-server you control instead of a hosted API, and you can swap models whenever you like. Under the hood it talks to llama.cpp’s llama-server over a small HTTP API. In managed mode it downloads Atomic’s own nightly llama.cpp build (atomic-llama-cpp-turboquant-nightly) with TurboQuant KV-cache compression (turbo3); in external mode it talks to any llama-server you run.

Usage telemetry is on by default; turn it off with "analytics": { "enabled": false } in config.json.

This page explains the two ways to get a local model running — managed (Atomic Agent downloads and runs llama.cpp for you) and external (you point it at a llama-server you run yourself) — plus GPU acceleration, device selection, and the other surfaces that share the same model. If you’d rather call a hosted model, see cloud providers, and to combine a cloud model with local workers, see run modes.

Developer Preview

Atomic Agent is a Developer Preview. Model IDs, CLI flags, and config keys in this page are accurate to the current build but may shift between releases. Pin a release if you need stability.

Two modes at a glance

Managed

Atomic Agent downloads the llama.cpp binary and a GGUF model, picks a GPU, starts the daemon, and watches its health. One command and you’re running. Set localModels.mode: "managed".

External

You already run llama-server (or any OpenAI-compatible llama.cpp build) somewhere. Atomic Agent just connects to its URL. Set localModels.mode: "external".

The mode lives in your config file at <stateDir>/config.json:

{
  "localModels": {
    "mode": "managed",            // or "external"
    "url": "http://127.0.0.1:8080",
    "managed": {
      "modelId": "qwen-3.5-4b",
      "port": 19091
    }
  }
}

Note

localModels.url is the connection string Atomic Agent uses to reach the chat server. In managed mode it points at the daemon Atomic Agent spawns; in external mode it points at your own server. The defaults are 19091 for the managed chat daemon, 19092 for the embedding daemon, and http://127.0.0.1:8080 for localModels.url (llama.cpp’s own default, which is what an external llama-server listens on out of the box).

Managed mode (recommended)

In managed mode, Atomic Agent owns the full lifecycle of llama.cpp. The atomic-agent models commands drive everything.

Get a model running

# See which models are available and what's installed
atomic-agent models list

# Download a model's GGUF weights (idempotent — safe to re-run)
atomic-agent models pull qwen-3.5-4b

# Make it the active model
atomic-agent models use qwen-3.5-4b

# Start the daemon (installs or updates the llama.cpp backend; the model must already be pulled)
atomic-agent models start

# Check it's alive
atomic-agent models status

A fresh install starts in external mode with no model selected. The first-run picker in the TUI recommends a model sized to your RAM; atomic-agent models use <id> sets one and switches localModels.mode to managed. The catalog spans Qwen 3.5, 3.6 and 3.8, Gemma 4, NVIDIA Nemotron 3.5 and Meta Muse Glimmer, and you can add your own Hugging Face GGUFs from the TUI (they are stored in localModels.customModels). Run atomic-agent models list for the current set rather than trusting a count here, since the catalog moves between releases.

Most catalog chat models are vision-capable and pair with an mmproj projector (Nemotron 3.5 is text-only). The projector is a separate download: atomic-agent models pull <id> --mmproj, or the TUI’s Local Models panel. Without it the model runs text-only, and start prints vision disabled (mmproj missing …). The cost of vision is the extra memory the projector needs.

What models start actually does

When you start a managed daemon, Atomic Agent walks through a fixed sequence:

1. Verify prerequisites. Installs or updates the llama.cpp backend (when localModels.managed.autoUpdate is on, the default) and checks the GGUF model is on disk. It does not download the model: if it’s missing, start stops with model <id> not downloaded; run 'atomic-agent models pull <id>'.
2. Resolve a GPU device — enumerates GPUs and picks one (see GPU acceleration).
3. Spawn the daemon — launches llama-server detached, writes a PID file, and polls http://127.0.0.1:<port>/health until it returns ok (30s timeout).
4. Optionally start an embedding daemon — a second llama-server on a separate port for hybrid memory recall.

flowchart TD
    Start["atomic-agent models start"] --> Verify["Update backend<br/>check GGUF is pulled"]
    Verify --> GPU["Resolve GPU device<br/>(--list-devices / nvidia-smi)"]
    GPU --> Spawn["Spawn llama-server detached<br/>write PID file"]
    Spawn --> Health["Poll /health<br/>until ok or 30s timeout"]
    Health -->|ok| ChatUp["Chat daemon up<br/>(primary)"]
    Health -->|timeout| Fail["Fail + print log tail"]
    ChatUp --> Embed{"Embeddings<br/>enabled?"}
    Embed -->|yes| EmbedDaemon["Spawn embedding daemon<br/>on separate port"]
    Embed -->|no| Done["Ready"]
    EmbedDaemon -->|ok| Done
    EmbedDaemon -->|fail| Degrade["Chat continues<br/>FTS5-only memory fallback"]
    Degrade --> Done

Chat is primary, embeddings are optional

The chat daemon is the primary process — if it fails to start, that’s fatal. The embedding daemon is a graceful add-on: if it fails, your chat agent keeps working and memory recall falls back to keyword (FTS5) search. You won’t be blocked.

Manage the daemon and models

atomic-agent models stop          # SIGTERM, wait 3s, then SIGKILL; remove PID file
atomic-agent models update        # update the llama.cpp backend binary
atomic-agent models remove <id>   # delete a model's weights from disk
atomic-agent models devices       # list detected GPU devices
atomic-agent models use-device auto   # or: cpu | Vulkan0 | CUDA0 | MTL0 ...
atomic-agent models pull <id> --background   # keep downloading after the terminal closes
atomic-agent models pull <id> --mmproj       # also fetch the vision projector
atomic-agent models downloads     # list background downloads; `cancel <id>` or `clear`

atomic-agent models search <query> is the one subcommand that isn’t about the local runtime: it searches the models of your configured cloud providers by id, vendor and capability (for example claude vision or free tools), with --provider <id>, --limit <n>, --json and --refresh.

Embedding models have their own subcommands:

atomic-agent models list-embeddings
atomic-agent models pull-embedding nomic-embed-text-v1.5
atomic-agent models use-embedding nomic-embed-text-v1.5
atomic-agent models use-embedding --disable

The default embedding model is nomic-embed-text-v1.5. The catalog carries five: one Nomic, three BGE (bge-small-en-v1.5, bge-base-en-v1.5, and the multilingual bge-m3), and one Jina long-document model. Reach for bge-m3 if your notes aren’t all in English.

Caution

The daemon subcommands are for managed mode. In external mode, start refuses to run (external mode — nothing to start) and status just reports the external URL. atomic-agent models use <id> switches localModels.mode to managed for you. models search works in any mode.

External mode

If you already run llama-server (for example with a custom build, a shared GPU box, or a model Atomic Agent doesn’t bundle), use external mode and just point at it.

{
  "localModels": {
    "mode": "external",
    "url": "http://localhost:8080"
  }
}

Atomic Agent will probe that URL’s /health and /props endpoints at startup to detect the model profile and plan KV-cache slots. Everything else — tool grammar, hot-swap detection, prompting — works the same as managed mode.

Hot-swapping models behind llama-server

Atomic Agent watches the modelId field on each completion response. If you swap the model behind your llama-server mid-session (e.g. from Qwen to Gemma 4), it detects the mismatch, re-probes /props, rebuilds the tool-call grammar, and adapts — no restart needed. This relies on the modelId field, which older llama.cpp builds may not send.

GPU acceleration and device selection

Managed mode downloads a prebuilt llama.cpp build for your platform:

• macOS (Apple Silicon): Metal. Intel Macs are not supported in managed mode; use external mode.
• Linux x64: Vulkan, which covers NVIDIA, AMD and Intel GPUs.
• Linux arm64: a CUDA 13.3 build that also carries CPU backends, so it works without an NVIDIA driver.
• Windows x64: a CUDA build (12.4 or 13.3) when nvidia-smi reports a compatible NVIDIA driver, Vulkan otherwise. On Windows you can pin the build with localModels.managed.backendVariant (auto, vulkan, cuda-12.4, cuda-13.3 or cpu).

Atomic Agent enumerates devices by running llama-server --list-devices, with an nvidia-smi fallback for pre-download VRAM checks. Device ids look like Vulkan0, CUDA0 or MTL0.

atomic-agent models devices

How a device is chosen

• auto (default) — Atomic Agent picks the best GPU by VRAM, preferring a discrete GPU over an integrated one even if the integrated GPU reports more memory.
• A specific device — e.g. Vulkan0, CUDA0, MTL0.
• cpu — skip the GPU entirely.

atomic-agent models use-device auto
atomic-agent models use-device Vulkan0
atomic-agent models use-device cpu

The setting persists as localModels.managed.device in config. To spread one model across several GPUs, set localModels.managed.tensorSplit to ratios such as [3, 1].

VRAM budgeting

Atomic Agent computes a VRAM budget before loading:

• macOS uses 75% of system RAM, because Apple Silicon has unified memory. The conservative fraction avoids false “insufficient VRAM” warnings.
• Linux / Windows use the selected device’s reported VRAM.
• CPU mode has no GPU budget.

Vision needs headroom

A vision-capable model with its projector downloaded loads extra CLIP projector weights and runs with special batch flags (--image-min-tokens 560 and friends) so image embeddings aren’t near-noise. Budget for the mmproj projector on top of the weights themselves.

Context auto-sizing

localModels.managed.contextSize defaults to 0, which means auto: before launch, Atomic Agent reads the device’s free VRAM, subtracts the weights and projector, and sizes --ctx-size to fit (at least 32K, at most 256K or the model’s trained limit; 16K when no VRAM figure is available, for example on CPU). Because of the 32K floor, a very busy GPU can still fail to load. Set a positive integer to pin the context yourself when you’d rather have a predictable window.

The daemon’s lifetime

localModels.managed.stopOnExit defaults to true: when the last CLI session exits, the managed daemon shuts down with it rather than lingering in the background holding VRAM. Set it to false if you want the daemon to survive across invocations — worth it when you’re running many short commands and don’t want to pay the model load each time. Daemons you started with atomic-agent models start are never stopped this way; they keep running until atomic-agent models stop.

Reaching the same model from other surfaces

Whatever model you’re running — managed or external — is shared across every way you talk to Atomic Agent. They all connect to the same llama-server.

OpenAI-compatible HTTP

Run an OpenAI-compatible HTTP server and point any OpenAI SDK at it:

atomic-agent serve --host 127.0.0.1 --port 8787 --api-key <key>

Then call the chat completions endpoint:

curl http://127.0.0.1:8787/v1/chat/completions \
  -H "Authorization: Bearer <key>" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "atomic-agent",
    "messages": [{ "role": "user", "content": "List the files here." }],
    "stream": false
  }'

Notes:

• POST /v1/chat/completions supports both streaming (stream: true, SSE) and synchronous responses.
• GET /v1/models returns a single atomic-agent entry and needs no auth.
• GET /health is a no-auth liveness probe (also checks the underlying llama-server).
• Session IDs are derived deterministically from the prompt, or set explicitly with the X-Atomic-Session-Id header.
• The API key can also come from the ATOMIC_AGENT_API_KEY environment variable.

Tauri sidecar

For desktop apps, Atomic Agent ships a sidecar process that speaks newline-delimited JSON (NDJSON) over stdin/stdout. A Tauri (or any) host spawns it, pipes the streams, sends typed requests, and reads back streamed events.

atomic-agent-sidecar

• One active session per sidecar process. Start a new session and the previous one is disposed.
• Requests: start_session, send_message, cancel, approval_response, get_session, skill_*, ping, shutdown.
• Events stream back as you go: step_started, tool_call_started, assistant_delta, assistant_reply, approval_request, and more.
• The sidecar reads the same localModels.url / localModels.managed.port config as every other front end.

The sidecar uses the same managed/external model under the hood — it’s just another front end for the runtime.

Telegram

You can drive the same agent — and the same local model — from your phone via a single-user Telegram channel.

{
  "telegram": { "enabled": true }
}

Put your bot token in <stateDir>/.env as TELEGRAM_BOT_TOKEN (never in config.json — the token lives in the secrets file, mode 0600). Then pair as the owner and chat. Dangerous tool calls arrive as inline approve/deny buttons.

/help     show commands and setup
/status   active session, turn count, last error
/new      start a fresh session
/cancel   abort the running turn

Telegram is intentionally single-user: once paired, only the owner’s messages are accepted; everyone else is dropped.

Configuration reference

Change a single key with atomic-agent config set <key> <value> (for example atomic-agent config set localModels.managed.contextSize 32768). Avoid atomic-agent config set '<json>' for small edits: it replaces the whole file, and any key you leave out goes back to its default.

Key	Meaning
localModels.mode	managed (Atomic Agent runs llama.cpp) or external (you do). Default external
localModels.url	HTTP URL of the chat server (default http://127.0.0.1:8080)
localModels.completionMaxTokens	Max new tokens per completion (default 16384, clamped 64 to 131072)
localModels.useServerTemplate	auto (default), on or off: render prompts through the model’s own chat template instead of the built-in framing
localModels.thinking	auto (default), on or off: the chat template’s thinking switch
localModels.reasoningBudgetTokens	Thinking budget before a tool call for local reasoning models (default 1500, 0 = unbounded)
localModels.managed.modelId	Active model in managed mode (default none; e.g. qwen-3.5-4b)
localModels.managed.port	Chat daemon port (default 19091)
localModels.managed.device	auto, cpu, or a device id like Vulkan0
localModels.managed.backendVariant	Windows only: auto (default), vulkan, cuda-12.4, cuda-13.3 or cpu
localModels.managed.contextSize	--ctx-size; 0 (default) fits it to free VRAM
localModels.managed.tensorSplit	Multi-GPU split ratios, e.g. [3, 1] (default [], single device)
localModels.managed.parallel	llama-server request slots: auto (default) or 1 to 8. Caps how many Fusion workers run at once
localModels.managed.swaFull	auto (default), on or off: --swa-full for sliding-window models such as Gemma 4
localModels.managed.stopOnExit	Stop the daemon when the last CLI session exits (default true)
localModels.managed.autoUpdate	Auto-update the backend binary (default true)
localModels.managed.dataDirOverride	Store backend and model files somewhere else (default null)
localModels.embeddings.enabled	Run the secondary embedding daemon (default false)
localModels.embeddings.modelId	Embedding model (e.g. nomic-embed-text-v1.5)
localModels.embeddings.port	Embedding daemon port (default 19092)
localModels.embeddings.url	Embedding server URL (default http://127.0.0.1:19092)
localModels.download.connections	Parallel connections per model download (default 16)
localModels.download.hfEndpoint	Hugging Face endpoint or mirror (default https://huggingface.co)
localModels.customModels	Your own Hugging Face GGUFs, added from the TUI

Useful environment variables:

Variable	Effect
ATOMIC_AGENT_LLAMA_API_KEY	Bearer token for the llama-server (optional)
ATOMIC_AGENT_LLAMA_MAX_TOKENS	Max new tokens per completion — sets localModels.completionMaxTokens (default 16384, clamped 64 to 131072)
ATOMIC_AGENT_LLAMA_TEMPERATURE / _TOP_P / _TOP_K / _SEED	Sampling overrides (parsed at startup)
ATOMIC_AGENT_STATE_DIR	Root state directory holding config.json, .env, and model files

Cloud providers (opt-in)

Local llama.cpp is the default and the point of the product, but it isn’t the only option. Atomic Agent ships seven provider kinds, and you can mix them:

Kind	What it talks to
llama-server	Local llama.cpp — the default
openai-compatible	Any OpenAI-shaped endpoint (vLLM, LM Studio, a hosted API)
qwen-openai-compatible	The same, with Qwen’s tagged tool-call dialect
openrouter	OpenRouter’s model marketplace
aimlapi	AIMLAPI
gemini	Google Gemini
subscription-cli	Your signed-in Claude Code or Codex CLI subscription (no API key; set subscriptionCli.cli to claude or codex)

Ready-made presets

You rarely need to type a base URL. The TUI’s provider wizard ships twenty pre-verified presets: pick a vendor by name and it fills in the endpoint for you. Every one resolves to the ordinary openai-compatible kind, so a preset is a shortcut, not an eighth provider kind, and model lists still come from the server’s own /v1/models rather than a bundled list that goes stale.

Preset	Base URL	API-key env var
Anthropic (Claude)	https://api.anthropic.com	ANTHROPIC_API_KEY
Atomic Chat (local)	http://127.0.0.1:1337	ATOMIC_CHAT_API_KEY
Cerebras	https://api.cerebras.ai	CEREBRAS_API_KEY
DeepSeek	https://api.deepseek.com	DEEPSEEK_API_KEY
Fireworks AI	https://api.fireworks.ai/inference	FIREWORKS_API_KEY
Groq	https://api.groq.com/openai	GROQ_API_KEY
Hyperbolic	https://api.hyperbolic.xyz	HYPERBOLIC_API_KEY
LM Studio (local)	http://localhost:1234	LMSTUDIO_API_KEY
Mistral	https://api.mistral.ai	MISTRAL_API_KEY
Moonshot AI (Kimi)	https://api.moonshot.ai	MOONSHOT_API_KEY
Nous Research	https://inference-api.nousresearch.com	NOUS_API_KEY
Novita AI	https://api.novita.ai/openai	NOVITA_API_KEY
Ollama (local)	http://localhost:11434	OLLAMA_API_KEY
Ollama Cloud	https://ollama.com	OLLAMA_CLOUD_API_KEY
Perplexity	https://api.perplexity.ai	PERPLEXITY_API_KEY
Qwen (DashScope)	https://dashscope-intl.aliyuncs.com/compatible-mode	DASHSCOPE_API_KEY
SambaNova	https://api.sambanova.ai	SAMBANOVA_API_KEY
Sarvam	https://api.sarvam.ai	SARVAM_API_KEY
Together AI	https://api.together.xyz	TOGETHER_API_KEY
xAI (Grok)	https://api.x.ai	XAI_API_KEY

Each preset has its own env var, deliberately. An earlier shared OPENAI_COMPAT_API_KEY meant connecting a second vendor overwrote the first one’s key. Now Groq and Together can hold keys at the same time without touching each other, and adding a second key for the same vendor gets a numbered entry id (groq-2) that still resolves to the right service.

Three behaviours worth knowing:

• The three local presets (Atomic Chat, LM Studio and Ollama) point at a server on your own machine. There is no key to have, so the wizard saves them with an empty key and requests carry no Authorization header at all.
• Some presets list models before you paste a key. Nous Research (350+ ids), Novita, Ollama Cloud, SambaNova and Sarvam answer /v1/models unauthenticated, so you can browse the catalog first and add credentials later. The rest want a key before they’ll tell you anything.
• Anthropic is the one preset that doesn’t use Authorization: Bearer. It sends the key as x-api-key (the entry’s apiKeyHeader) and pins the anthropic-version header.

Cloud providers send your prompts off your machine

This is the setting that takes a chat off your machine, so it is strictly opt-in: nothing here is active until you add an llm.providers[] entry and point activeTextProvider at it (or pick Cloud or Fusion as the run mode). Once you do, every prompt routed to that provider, including whatever memory and file contents the agent injected, goes to a third party under their retention policy. Keep sensitive work on a local provider.

There is no atomic-agent llm command

Cloud providers are configured in exactly two places: the llm block of config.json, or the TUI’s LLM panel (which writes the same block). The CLI surface is run, skill, config, tui, serve, trace, task, memory, models, import, update and uninstall. models drives the local llama.cpp daemon (plus models search, which only searches your configured providers’ model lists). If you’re looking for a command to add a hosted key, it doesn’t exist.

Providers are declared in an llm.providers[] block, with activeTextProvider choosing which one handles a turn:

{
  "llm": {
    "activeTextProvider": "local",
    "providers": [
      { "id": "local", "kind": "llama-server" },
      {
        "id": "or",
        "kind": "openrouter",
        "apiKey": "sk-or-...",
        "defaultChatModel": "qwen/qwen3-max"
      }
    ]
  }
}

What goes in a provider entry

That example is the short form. A provider entry carries considerably more than id / kind / apiKey:

Field	Meaning
id	Name you refer to in activeTextProvider, activeEmbeddingProvider, and llm.fallback.chain
kind	One of the seven kinds above
baseUrl	API root for the OpenAI-shaped kinds. Stored without the /v1 suffix — call sites append it
url	llama-server base URL override (llama-server kind only)
apiKey	Credential. Empty string when the endpoint is local and unauthenticated
apiKeyEnvVar	Env var holding the key (set by the presets, e.g. GROQ_API_KEY); wins over apiKey when present
apiKeyHeader	Header that carries the key for services that don’t accept Authorization: Bearer (Anthropic uses x-api-key)
defaultChatModel	Model used when a turn doesn’t name one
defaultEmbeddingModel	Model used for embedding requests
headers	Extra HTTP headers sent verbatim on every request
supportsTools	Whether the endpoint handles parallel tool calls (defaults to true)
supportsVision	Whether the endpoint accepts images (defaults to true)
requestTimeoutMs	Per-request timeout
promptCache	auto, off, or explicit-markers
messageShape	native (default) or flat: how history is laid out for native tool calling
extraBody	Vendor-specific fields merged into the chat request body (model, messages, stream and tools can’t be overridden)
strictTools	Send OpenAI strict function tools (default off; services without strict mode reject the request)
providerPreferences	OpenRouter provider-routing object (openrouter kind only)
subscriptionCli	subscription-cli kind only: cli (claude or codex), plus optional binPath, extraArgs, streaming, maxBudgetUsd
userModels[]	Per-model capability and pricing declarations — see below

openai-compatible and qwen-openai-compatible refuse to boot without two fields

Both kinds require baseUrl and defaultChatModel. Leave either out and the provider factory throws at startup — openai-compatible provider "<id>" requires baseUrl and defaultChatModel — which takes the whole runtime down, not just that provider. A minimal working entry is four keys:

{
  "id": "vllm",
  "kind": "openai-compatible",
  "baseUrl": "https://vllm.internal.example",
  "apiKey": "sk-...",
  "defaultChatModel": "qwen/qwen3-32b"
}

The other five kinds are more forgiving: openrouter, aimlapi, and gemini each fall back to a built-in default chat model, llama-server reads its URL from localModels, and subscription-cli needs no key at all.

Declaring models yourself

Each provider entry can carry a userModels[] array describing individual models. Every entry needs an id and a kind (chat or embedding); the rest is optional:

{
  "id": "vllm",
  "kind": "openai-compatible",
  "baseUrl": "https://vllm.internal.example",
  "defaultChatModel": "qwen/qwen3-32b",
  "userModels": [
    {
      "id": "qwen/qwen3-32b",
      "kind": "chat",
      "contextWindow": 32768,
      "supportsVision": false,
      "supportsTools": "basic",
      "supportsPromptCache": false,
      "pricing": { "input": 0.15, "output": 0.6 }
    }
  ]
}

supportsTools is a four-way setting — none, basic, parallel, or strict — not a boolean, and embedding entries can also declare dim. Pricing takes input and output, plus optional cacheRead and cacheWrite.

Resolution runs in a fixed order: userModels beats the built-in catalog, which beats the defaults. That last step is the one to watch. A model nobody declared and the catalog doesn’t recognise silently resolves to a 128,000-token context window with parallel tool support and no vision — plausible-looking numbers that are simply wrong if your endpoint serves an 8K window or can’t do parallel calls. Nothing warns you; you find out when requests start getting truncated or tool calls come back malformed.

userModels[] is also the only way to give a model the catalog doesn’t know a price, which Atomic Agent uses where it shows a model’s pricing.

How tool calls travel

llm.toolTransport sits next to providers[] and decides how the model is asked for a tool call. It takes three values, and the default is easy to misread:

Value	Behaviour
auto (default)	Defers to whatever the active provider declares. Not a synonym for grammar
grammar	Force GBNF-constrained decoding, whatever the provider says
native_tools	Force the provider’s own function-calling API

Because auto follows the provider, the effective transport changes when you switch providers — and it changes again mid-chain when a fallback hands the turn to someone else. A local llama-server turn constrained by grammar and a hosted turn using native function calling are both auto. If you’re debugging tool-call behaviour that appears only on one provider, pin the transport explicitly before you go hunting anywhere else.

One more thing comes with the cloud path.

A fallback chain lets a request roll to the next provider when the active one errors — useful for riding out a hosted outage without dropping back to a cold local load, though it also means a failure can quietly move your prompt to a different vendor. Put only providers you’re equally comfortable with in one chain.

After the primary, the chain tries providers of the same class (cloud or local) first, then the others. Note that llm.fallback.appendLocal defaults to true: your local llama-server is silently appended to the chain, so a cloud outage can hand the turn to a local model whose answers differ sharply in quality. Set it to false if you would rather the request fail than switch. The chain also behaves as a circuit breaker — after failureThreshold consecutive failures (default 3) the provider is skipped for a cooldown that escalates 30s, 60s, 300s before the primary is probed again.

Run modes: Local, Cloud, Fusion

llm.runMode.mode names how a chat runs:

Mode	What runs the chat
local	The llama-server provider, on your machine
cloud	The active cloud provider
fusion	One model orchestrates and hands sub-tasks to several worker agents on a second provider, in parallel

In the TUI, switch with /runmode local, /runmode cloud or /runmode fusion (bare /runmode opens the switch, and ctrl+g then 1/2/3 does the same). /runmode status prints what will actually run, /runmode swap trades the orchestrator and worker legs, and /runmode workers N sets the default number of workers.

The default Fusion pairing is a cloud orchestrator with local workers: the cloud model plans and reviews, and local model workers do the bulk of the work. The orchestrator gets a fusion.delegate tool; one call runs several workers at once after a single fan-out approval (silenced from approval level 4). Local workers run one per llama-server request slot, so localModels.managed.parallel bounds how many run at once. Worker turns don’t read memory.

Fusion needs two providers. If no cloud provider is configured, or only one provider exists, Atomic Agent doesn’t switch to Fusion and tells you why. llm.activeTextProvider stays authoritative: if you switch the active provider by hand, you drop out of Fusion.

{
  "llm": {
    "runMode": {
      "mode": "fusion",
      "fusion": {
        "orchestratorProvider": "or",
        "workerProvider": "local",
        "workers": 2
      }
    }
  }
}

Key	Meaning
llm.runMode.mode	local, cloud or fusion
llm.runMode.fusion.orchestratorProvider	Provider id for the orchestrator. Default: the active cloud provider, else the first cloud provider
llm.runMode.fusion.workerProvider	Provider id for the workers. Default: the first llama-server provider. Must differ from the orchestrator
llm.runMode.fusion.workers	Default fan-out width, 1 to 8 (default 2). The orchestrator can ask for more in a single call; slots still bound it
llm.runMode.fusion.cloudWorkers	Ceiling on the fan-out when workers run on a cloud provider, 1 to 32 (default 4)
llm.runMode.fusion.workerReasoning	low, medium or high reasoning effort for worker completions (default: the provider’s own)

Pinned legs aren’t tied to a kind, so a local orchestrator with cloud workers is also allowed. Keep in mind that in Fusion the orchestrator’s prompts, and whatever it passes to cloud workers, go to that cloud provider.

Troubleshooting

• Daemon won’t come up. atomic-agent models start polls /health for 30 seconds; on timeout it prints the tail of the llama-server log. Re-run atomic-agent models status and check the log for load errors.
• Foreign daemon error. If a daemon was started by another user (e.g. with sudo) and you try to stop it as yourself, Atomic Agent raises a ForeignDaemonError rather than orphaning the process. Stop it as the original user.
• Embeddings failed but chat works. That’s by design — memory recall silently falls back to keyword (FTS5) search. Re-pull the embedding model or check its daemon port.
• Garbage memory recall. An embedding model’s pooling strategy must match how it was trained; the catalog encodes this. Use the catalog model IDs rather than hand-pointing at an arbitrary GGUF.
• GitHub rate limits during download. Backend downloads hit the GitHub API (unauthenticated ~60 req/h). Set GITHUB_TOKEN for heavy use or CI. GH_TOKEN works everywhere GITHUB_TOKEN does — model downloads, llama.cpp backend installs, and the app update check all read GITHUB_TOKEN first and fall back to GH_TOKEN, so the token the gh CLI already put in your environment is enough.

Related

CLI reference The full atomic-agent models, run, and serve command surface.

Configuration Every config.json block and ATOMIC_AGENT_* environment variable.

Memory How the embedding daemon powers hybrid recall over your notes.

HTTP API OpenAI-compatible endpoints, headers, and streaming details.

How to run an AI agent offline From the blog: a fully local setup, step by step, with no API key.

How to run Qwen 3.8 27B locally From the blog: memory math for 24 GB cards and two measured agent runs.