SGLang

Use Flint’s SGLang serving bundle to run an OpenAI-compatible server on NVIDIA GPUs. The bundle includes the model, tokenizer, Flint plugin, patched kernels, Dockerfile, and hardware-specific launch scripts.

Requirements

This recipe serves BF16 weights on one GPU per replica.

Requirement	Configuration
Validated GPUs	B300 (x86_64) or H200, with at least 141 GB VRAM per replica
Host	Linux with Docker and NVIDIA Container Toolkit configured for GPU access
Driver	CUDA 13-compatible driver, normally R580 or newer; tested B300 driver: 580.126.20
H200 compatibility	Tested with 570.133.20 through the image’s CUDA forward-compatibility libraries
Disk	About 80 GB for weights plus about 50 GB for the image; allow additional build/cache space
Build network access	Docker Hub, PyPI, files.pythonhosted.org, flashinfer.ai, and GitHub release assets

80 GB GPUs cannot run this single-GPU BF16 recipe. B200 and GB200/GB300 use the Blackwell configuration but have not been validated with this bundle. The Mac requirements apply to rec, not this deployment. For older datacenter drivers, confirm that the container runtime loads the forward-compatibility libraries at /usr/local/cuda/compat. A driver compatibility error should be resolved before loading weights.

Prepare the serving bundle

Download the BF16 model and serving kit from RecursiveSI/flint. The commands below pin the revision used for this guide so the weights, tokenizer, configurations, and launch scripts stay together.

If the repository requires access, sign in with a Hugging Face token that can read it. Use the model license published with the weights; the license in sglang/ applies to the serving code.

Run in Bash, using your Python environment:

python3 -m pip install -U huggingface_hub openai
hf auth login

Download into a new directory for this revision, verify its checksums, then build the image:

export FLINT_REV='1495c596e8cf84c1b024f9340ae57f7e7d4c9912'
export FLINT_IMAGE="flint-sglang:${FLINT_REV:0:12}"
hf download RecursiveSI/flint \
  --revision "$FLINT_REV" \
  --local-dir "flint-$FLINT_REV" && \
  cd "flint-$FLINT_REV" && \
  sha256sum -c --quiet SHA256SUMS && \
  docker build -t "$FLINT_IMAGE" sglang

The download includes roughly 80 GB of weights. Keep the complete directory, including SHA256SUMS and sglang/, for the launch commands below. When updating revisions, use a new directory: a Hugging Face download does not remove files deleted upstream. The bundled Dockerfile pins SGLang 0.5.16-cu130 by image digest and installs flint_sglang 0.1.2, FlashInfer 0.6.16.post1, and the Flint kernel patches. It includes PyTorch 2.11.0+cu130, Transformers 5.12.1, and sglang-router 0.3.2. A stock SGLang image alone does not contain this integration.

Start one replica

Run one of the following commands from the same bundle directory. Both mount the downloaded model at /model, use GPU 0, and publish the API only on localhost.

B300
H200

docker run -d --name flint --gpus '"device=0"' --ipc=host \
  -p 127.0.0.1:30000:30000 \
  -v "$PWD:/model" -e MODEL=/model "$FLINT_IMAGE" \
  bash /model/sglang/serve_b300.sh

docker run -d --name flint --gpus '"device=0"' --ipc=host \
  -p 127.0.0.1:30000:30000 \
  -v "$PWD:/model" -e MODEL=/model "$FLINT_IMAGE" \
  bash /model/sglang/serve_h200.sh

Check startup progress and readiness:

docker logs --tail 100 flint
curl --fail --silent --show-error http://127.0.0.1:30000/health_generate
curl --fail --silent --show-error http://127.0.0.1:30000/v1/models

First startup takes roughly four minutes on B300 with fast local storage; concurrent loads from shared storage can take up to 15 minutes. Wait for the readiness log and a successful health check before sending requests. The model ID is flint. The scripts listen on all interfaces inside the container and do not enable authentication by default. Keep the localhost mapping above for local clients. For a network deployment, restrict access to a private network or authenticated gateway; the scripts also accept --api-key for server authentication.

Exact server configuration

The launch scripts supply these flags automatically:

Flag	B300	H200
`--tp-size`	`1`	`1`
`--attention-backend`	`fa4`	`fa3`
`--moe-runner-backend`	`flashinfer_trtllm_routed`	`triton`
`--enable-fp32-lm-head`	Enabled	Enabled
`--context-length`	`262144`	`262144`
`--chunked-prefill-size`	`4096`	`4096`
`--swa-full-tokens-ratio`	`0.05`	`0.4`
`--mem-fraction-static`	`0.8`	`0.9`
`--disable-piecewise-cuda-graph`	Set	Not set
`--disable-flashinfer-autotune`	Set	Not set
`--tool-call-parser`	`flint`	`flint`
`--reasoning-parser`	`flint`	`flint`
`--grammar-backend`	`flint`	`flint`

Both scripts set SGLANG_EXTERNAL_MODEL_PACKAGE=flint_sglang and FLASHINFER_USE_CUDA_NORM=1. Keep these settings and the bundled tokenizer unchanged.

Environment variable	Purpose
`MODEL`	Local model directory; explicitly set to `/model` in the Docker examples
`PORT`	Replica port, default `30000`
`REVISION`	Immutable model revision when loading from a Hugging Face repository instead of a local path
`FLINT_TRITON_CACHE_BASE`	Kernel cache root; defaults to `$HOME/.triton/cache`, with a separate host/port subdirectory for each replica

Arguments after the script name override its defaults. For example, append --mem-fraction-static 0.85 to change the B300 memory allocation. Start with the tested defaults, then measure your workload before changing them. --mem-fraction-static allocates memory for weights and KV pools. --swa-full-tokens-ratio controls the sliding-window pool relative to the full-attention pool: increase it for many short concurrent requests, decrease it for fewer long requests. The 262,144-token context is specific to this BF16 configuration; input and generated tokens must fit within it. Tensor, expert, and pipeline parallelism, DP attention, quantized weights or KV cache, and speculative decoding have not been validated with this bundle. Do not replace the MoE runner with flashinfer_trtllm or triton_kernel, enable a MoE all-to-all backend, or switch attention to flashinfer: those paths do not preserve the required Flint routing or attention behavior.

Send requests and verify tools

Install the client dependency in your Python environment:

python3 -m pip install openai

A short request uses the standard OpenAI client:

from openai import OpenAI

client = OpenAI(base_url="http://127.0.0.1:30000/v1", api_key="unused")
response = client.chat.completions.create(
    model="flint",
    messages=[{"role": "user", "content": "Say hello in one sentence."}],
    temperature=1.0,
    top_p=0.95,
    max_tokens=4096,
    extra_body={"top_k": 20},
)
print(response.choices[0].message.content)

unused is a client placeholder for the localhost server above. If you enabled server authentication, use the configured key instead. Run the bundled examples from the model directory to check reasoning, tool calls, and follow-up answers:

python3 sglang/example_client.py --base-url http://127.0.0.1:30000/v1
python3 sglang/example_tools.py --base-url http://127.0.0.1:30000/v1

The examples exercise a full tool round trip, nested argument schemas, and parallel calls with canned tool results. Before connecting a coding harness, also verify streaming and the complete tool cycle in that client.

Client settings

Use temperature 1.0, top-p 0.95, and top-k 20. These are also the bundle’s generation defaults.
For difficult tasks, allow at least 32,768 output tokens, subject to the remaining context budget. Reasoning counts toward max_tokens; finish_reason: "length" can mean the answer or tool call was cut off.
Flint decides whether to reason on each turn. This integration has no supported reasoning toggle or budget; thinking and enable_thinking template settings have no effect. Preserve reasoning_content when replaying assistant messages.
Send message content as strings, and include tool definitions on every tool-enabled request. Return the received function.arguments JSON string unchanged in the assistant tool-call record, followed by one tool result per call in order.
Set strict: true on a function definition when its arguments must match its JSON schema. Keep all three Flint parser/grammar flags enabled.
For structured answers, use response_format with json_schema or json_object. The Flint grammar permits reasoning before the constrained answer. If both tools and a response format are supplied, only the response-format constraint applies.

For the general request interface, see SGLang’s request documentation.

Scale across GPUs and nodes

Scale this configuration with independent single-GPU replicas behind the bundled router. Each replica holds a complete copy of the weights; this does not split the model across GPUs. On each eight-GPU node, use the same bundle revision and image. Set FLINT_NODE_IP to that node’s private interface address. Run from its bundle directory:

export FLINT_NODE_IP='REPLACE_WITH_THIS_NODES_PRIVATE_IP'
docker run -d --name flint-8x --gpus all --ipc=host \
  -p "$FLINT_NODE_IP:30000-30007:30000-30007" \
  -v "$PWD:/model" -e MODEL=/model \
  -e SERVE_SCRIPT=serve_b300.sh "$FLINT_IMAGE" \
  bash /model/sglang/serve_node_8x.sh

For H200 nodes, set SERVE_SCRIPT=serve_h200.sh. Restrict worker ports to the router’s private network. Check each worker’s /health_generate endpoint before routing traffic. On a Linux host that can reach every worker, build or load the same image and set FLINT_IMAGE to its tag. Replace the example hostnames below with your private node names:

FLINT_NODES=(node0.internal node1.internal)
FLINT_WORKERS=()
for node in "${FLINT_NODES[@]}"; do
  for port in {30000..30007}; do
    FLINT_WORKERS+=("http://$node:$port")
  done
done

docker run -d --name flint-router --network host "$FLINT_IMAGE" \
  python3 -m sglang_router.launch_router \
  --host 127.0.0.1 --port 8000 --policy cache_aware \
  --worker-urls "${FLINT_WORKERS[@]}"

Clients on the router host use http://127.0.0.1:8000/v1, model flint. For remote clients, place an authenticated gateway in front of the router. Run the same client and tool checks against this endpoint. serve_node_8x.sh stops all eight replicas if one exits. Use one container per GPU when you need independent restarts, with a distinct GPU and published port for each container.

Use an existing container

For enroot/pyxis or another writable container environment, start from the exact base image pinned in the bundle’s sglang/Dockerfile. Inside that container:

export FLINT_BUNDLE=/path/to/flint-bundle
bash "$FLINT_BUNDLE/sglang/install.sh" "$FLINT_BUNDLE/sglang"
MODEL="$FLINT_BUNDLE" PORT=30000 \
  bash "$FLINT_BUNDLE/sglang/serve_h200.sh" --host 127.0.0.1

Use serve_b300.sh for B300. The installer changes the container’s Python packages and kernels in place; it needs network access and a writable Python installation. Save the resulting container before reuse. A fresh base image does not contain Flint support.

Troubleshooting

Symptom	Check
Unknown Flint architecture or missing parser	Build the bundled Dockerfile or run its installer; a stock image is insufficient.
GPU out of memory	Check GPU capacity, competing processes, context, and concurrency. Start with the hardware-specific defaults.
Literal reasoning or tool markers in `content`	Restore `--tool-call-parser flint --reasoning-parser flint --grammar-backend flint`.
Empty or unfinished answer	Inspect `reasoning_content` and `finish_reason`; increase the output budget within the context limit if it ended with `length`.
Tokenizer regex warning	Keep the shipped tokenizer; do not apply `fix_mistral_regex=True`.
Slow first startup	Check weight storage throughput and compilation logs, especially when starting many replicas together.

Contact support@recursive.com with the full model revision, image ID, GPU/driver details, launch flags, and relevant logs. For output problems, include a sanitized request and response. Remove keys and private prompt data before sharing.

Get started

Local inference

GPU inference

Harness

Resources

Requirements

Prepare the serving bundle

Start one replica

Exact server configuration

Send requests and verify tools

Client settings

Scale across GPUs and nodes

Use an existing container

Troubleshooting

​Requirements

​Prepare the serving bundle

​Start one replica

​Exact server configuration

​Send requests and verify tools

​Client settings

​Scale across GPUs and nodes

​Use an existing container

​Troubleshooting

Requirements

Prepare the serving bundle

Start one replica

Exact server configuration

Send requests and verify tools

Client settings

Scale across GPUs and nodes

Use an existing container

Troubleshooting