Hugging Face Transformers

Flint is available on Hugging Face at recursivesi/flint. This guide shows how to load the public BF16 checkpoint in Python with a Flint-compatible Transformers installation.

The checkpoint is public, but its model card states that Transformers modeling code is not included yet. Direct Transformers loading still requires access to the integration fork below; the Flint PR remains open. For the public serving workflow, use SGLang.

Install the Flint fork

Use a separate Python environment with PyTorch supported by your hardware. The fork requires PyTorch 2.6 or newer; for CUDA, install the appropriate PyTorch build for your system. See Transformers installation. With Git authenticated to an account that can read the fork, run:

python3 -m venv .venv-flint
source .venv-flint/bin/activate
python -m pip install --upgrade pip

git clone --branch feat/flint-model --single-branch \
  https://github.com/huggingface/new-model-addition-rsi.git transformers-flint && \
  git -C transformers-flint checkout a7d66d92ce34cd44593b1cd1945f5290d725644e && \
  python -m pip install './transformers-flint[torch]'

Confirm the installed package includes Flint:

python -c "from transformers import FlintConfig, FlintForCausalLM; print('Flint support is installed')"

This checks the Python implementation without downloading model weights.

Load Flint in Python

Download the public Flint checkpoint through from_pretrained below. No private-model access is required. The example below adapts the fork’s AutoModel example to a pinned BF16 checkpoint. It has not been run end to end as part of this documentation update. The weights alone occupy about 79.6 GB; allow additional memory for activations, attention, and the KV cache. device_map="auto" places the model across available devices and may offload to CPU. This is a separate memory requirement from the quantized Mac model managed by rec.

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "recursivesi/flint"
revision = "2a247be0c0aec895ccb0ac7ca92229ec6df37288"

tokenizer = AutoTokenizer.from_pretrained(model_id, revision=revision)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    revision=revision,
    dtype="auto",
    device_map="auto",
    attn_implementation="eager",
    trust_remote_code=False,
)

messages = [{"role": "user", "content": "Say hello in one sentence."}]
inputs = tokenizer.apply_chat_template(
    messages,
    add_generation_prompt=True,
    return_tensors="pt",
    return_dict=True,
).to(model.device)

outputs = model.generate(
    **inputs,
    max_new_tokens=4096,
    do_sample=True,
    temperature=1.0,
    top_p=0.95,
    top_k=20,
)
print(tokenizer.decode(
    outputs[0, inputs["input_ids"].shape[-1]:],
    skip_special_tokens=True,
))

The installed fork supplies the model implementation; this example does not load remote model code. Keep the matching tokenizer and chat template unchanged. The public checkpoint is released under the Apache 2.0 license.

Attention and generation

Flint uses attention sinks. The fork defaults to eager attention, which computes attention scores in float32; SDPA does not support the required sinks. Start with a short prompt before attempting long contexts. For longer inputs, the fork documents sink-aware FlashAttention backends: kernels-community/vllm-flash-attn3 on Hopper and flash_attention_4 on Blackwell. Those paths require matching hardware and kernel dependencies; the basic example above uses eager attention. Generation returns model text, which can include reasoning or tool markup. Loading the model does not provide SGLang’s OpenAI-compatible API or its tool/reasoning parsers. Before connecting a coding harness through your own serving layer, verify the complete tool cycle. For a packaged server with these parsers, use SGLang.

Other runtimes

For Recursive’s optimized Mac engine and automatic harness setup, use rec. For GPU serving, Flint is supported in both vLLM and SGLang. Follow the setup guide for your preferred engine.

Get started

Local inference

GPU inference

Harness

Resources

Hugging Face Transformers

Install the Flint fork

Load Flint in Python

Attention and generation

Other runtimes

​Install the Flint fork

​Load Flint in Python

​Attention and generation

​Other runtimes

Install the Flint fork

Load Flint in Python

Attention and generation

Other runtimes