vLLM

Flint’s vLLM integration uses a dedicated model plugin with an OpenAI-compatible server. It requires a matching Flint export, tokenizer, and plugin installation.

The externally installable Flint vLLM package and release-model serving recipe are still being prepared. The configuration below documents the integration baseline for environments that already have the plugin. For a packaged GPU serving workflow, use SGLang.

Configuration baseline

Setting	Existing integration
vLLM version	`0.24.0`
Model implementation	`FlintForCausalLM`, registered by the `flint` plugin
Precision	Unquantized BF16 (`--dtype bfloat16`)
GPU evidence	One-B300 generation smoke test; eight B300s with data parallelism 8 and tensor parallelism 1 on an earlier checkpoint
Tensor parallelism	`1` for the tested single-GPU configuration
Basic launch context	`32768`; configure for the exact model export and available memory
Served model name	`Flint` in the example below; client IDs are case-sensitive

This is an integration baseline, not qualification of the final release checkpoint. The earlier tests do not establish H200 support, a minimum VRAM requirement, multi-node serving, or a 262,144-token vLLM context. The SGLang hardware and memory settings are specific to that runtime.

Launch with an installed Flint plugin

Before running the command, obtain a compatible environment with the Flint plugin installed and a complete BF16 model export. Keep the model configuration, weights, tokenizer, chat template, and generation configuration together. Installing stock vLLM alone does not install the Flint plugin. Replace the model path, then run:

VLLM_PLUGINS=flint vllm serve /path/to/flint-model \
  --host 127.0.0.1 --port 8000 \
  --served-model-name Flint \
  --dtype bfloat16 \
  --tensor-parallel-size 1 \
  --max-model-len 32768

VLLM_PLUGINS=flint selects an already installed plugin; it does not download one. If your deployment needs other vLLM plugins, preserve them in the comma-separated allowlist. A model configuration that loads successfully in Transformers does not establish that the vLLM model implementation is installed. This command is a local text-serving baseline. Flint-specific tool and reasoning parser settings for the external vLLM package remain to be published; do not copy SGLang’s parser flags into this command.

Verify the endpoint

After the server reports ready, check model discovery and a short chat:

curl --fail --silent --show-error http://127.0.0.1:8000/v1/models
curl --fail --silent --show-error http://127.0.0.1:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "Flint",
    "messages": [{"role": "user", "content": "Say hello in one sentence."}],
    "temperature": 1.0,
    "top_p": 0.95,
    "top_k": 20,
    "max_tokens": 4096
  }'

The request uses Flint’s recommended sampling settings. Keep input plus output within the server’s configured context. A length-limited response may need a larger output allowance and a compatible larger context; do not infer a release-wide maximum from this example. Before connecting a coding harness, verify streaming, parsed tool calls, and a complete tool round trip with the supplied parser configuration. A successful text response alone is insufficient. Use the SGLang guide for the bundle that includes Flint’s tool and reasoning parsers and runnable checks.

Troubleshooting

Symptom	Check
Unsupported `FlintForCausalLM` architecture	Confirm the plugin is installed in the same Python environment as vLLM and included in `VLLM_PLUGINS`.
Model loads but chat formatting fails	Use the tokenizer and chat template from the exact model export.
Client cannot find the model	Use the model ID returned by `/v1/models`; the example serves `Flint`.
Tools are missing or appear as raw text	Obtain the compatible tool/reasoning parser configuration before using an agent.
Quantization or LoRA errors	This integration baseline supplies BF16 inference; quantization and LoRA are not provided by it.

See vLLM’s plugin registration documentation for the extension mechanism. For a Flint package or deployment question, contact support@recursive.com.

Get started

Local inference

GPU inference

Harness

Resources

Configuration baseline

Launch with an installed Flint plugin

Verify the endpoint

Troubleshooting

​Configuration baseline

​Launch with an installed Flint plugin

​Verify the endpoint

​Troubleshooting

Configuration baseline

Launch with an installed Flint plugin

Verify the endpoint

Troubleshooting