Skip to main content
Use a Flint-compatible llama.cpp revision and a matching GGUF. You need a working NVIDIA driver, a CUDA toolkit supported by that revision, Git, CMake, and a C++ compiler. Check nvidia-smi and nvcc --version before building.
CUDA compatibility, quantization support, and cache options depend on the supplied runtime revision. This guide does not establish support for every NVIDIA GPU or CUDA version.
1

Set your artifact and runtime

Download your model and use the values supplied in your runtime setup instructions:
FLINT_CONTEXT must be an integer token count that fits your tested configuration.
2

Build with CUDA

Apply any additional build flags or patches in your runtime documentation. Experimental cache or quantization options are not interchangeable across revisions.
3

Start the server

This requests GPU offload. Check the startup log to see which layers were actually placed on the GPU. If memory is insufficient, lower context first or use the supported offload configuration supplied for your GPU.
4

Make a request

In another terminal:
This localhost server does not require authentication. LOCAL_API_KEY is the placeholder for SDKs that require a key; keep your Nebius credential in NEBIUS_API_KEY for hosted requests. Check this local server directly:
For tool use, also apply and verify the parser or grammar configuration provided with your runtime.

​
Compare results consistently

Record GPU model, driver, runtime commit, model revision, quantization, context length, and number of concurrent requests. Measure time to first token separately from generation speed. A model that fits in memory may still perform differently with a long prompt or an agent’s tool history.