Run on a Mac

Build a Flint-compatible llama.cpp runtime on an M4/M5 Pro or Max with at least 48 GB of unified memory. You need Git, CMake, and the Xcode command-line tools. For automatic installation and optimized inference, use rec.

Set your artifact and runtime

Download the model through your runtime setup instructions. Set the exact values supplied with it:

export FLINT_RUNTIME_REPO="YOUR_RUNTIME_REPOSITORY_URL"
export FLINT_RUNTIME_REV="YOUR_RUNTIME_COMMIT"
export FLINT_GGUF="/absolute/path/to/your/model.gguf"
export FLINT_CONTEXT="YOUR_VALIDATED_CONTEXT_LENGTH"

FLINT_CONTEXT must be an integer token count. Do not substitute a model’s advertised maximum for the tested limit on your machine.

Build the supplied revision

git clone "$FLINT_RUNTIME_REPO" flint-runtime
cd flint-runtime
git checkout "$FLINT_RUNTIME_REV"
cmake -S . -B build -DCMAKE_BUILD_TYPE=Release -DGGML_METAL=ON
cmake --build build --config Release -j --target llama-server

If your runtime documentation specifies additional patches or build flags, apply those before building.

Start the server

./build/bin/llama-server \
  --model "$FLINT_GGUF" \
  --alias flint-local \
  --ctx-size "$FLINT_CONTEXT" \
  --n-gpu-layers 99 \
  --temp 1.0 --top-p 0.95 --top-k 20 \
  --host 127.0.0.1 --port 8080

This is the basic text-serving configuration. Tool use may require additional parser or grammar options from your runtime documentation.

Verify a response

In another terminal:

export FLINT_BASE_URL="http://127.0.0.1:8080/v1"
export FLINT_MODEL="flint-local"
export LOCAL_API_KEY="local"

This localhost server does not require authentication. LOCAL_API_KEY is the placeholder for SDKs that require a key; keep your Nebius credential in NEBIUS_API_KEY for hosted requests. Check this local server directly:

curl --fail-with-body --silent --show-error \
  "${FLINT_BASE_URL%/}/chat/completions" \
  -H "Content-Type: application/json" \
  --data '{"model":"flint-local","messages":[{"role":"user","content":"Reply with exactly FLINT_LOCAL_OK."}],"max_tokens":4096}'

Check the server log for successful model loading and Metal initialization. Keep the server running while you use the API or coding agent.

If your build reports an unknown architecture or fails while loading the model, check the artifact–runtime pairing before changing performance settings. See troubleshooting.

⌘I

Get started

Local inference

GPU inference

Harness

Resources