mojolearn
Reproduce the model. Reproduce the prediction.
Mojolearn makes machine-learning results reproducible down to the bits, across GPU vendors and CPU architectures. Identical mode goes beyond deterministic. Deterministic means the same result on every run on one machine; identical means the same bits on every supported GPU and on x86 and ARM CPUs. One Mojo codebase runs on Apple silicon, NVIDIA sm_89 (Ada: L40S, L40, L4, RTX 4090, RTX 6000 Ada) and AMD gfx942 (Instinct MI300X, MI325X) GPUs, as well as x86 and ARM CPUs, and covers over 100 algorithms and model components in 13 families, from tree ensembles to transformers and hybrid attention and state space models, supporting both training and inference.
Open source, Apache 2.0. Created by Andrew Hendel. See scope and requirements.
Install
mojolearn installs with pip on macOS (Apple silicon, M1 and newer) and Linux x86-64, with Python 3.10 to 3.14. The [verify] extra adds NumPy for the verification checks below.
python -m pip install "mojolearn[verify]"On a Mac, Metal support is built in. On Linux, pip also installs the NVIDIA and AMD GPU packages, and mojolearn selects a backend for the detected GPU. GPU execution requires a supported architecture and compatible driver. NVIDIA needs only the driver, with no CUDA toolkit; AMD needs the AMD GPU driver (amdgpu) and the ROCm HIP runtime. Each release ships kernels for specific GPU architectures, so check the install picker for yours. Then train a first model, or see troubleshooting if the install fails.
Applications
Consistent inference, straightforward verification
In
identicalmode, mojolearn returns the same bits on Apple, NVIDIA and AMD GPUs. The same weights, inputs, preprocessing, settings and mojolearn release give the same outputs on each. Compare output hashes to check a deployment change, test a fleet before rollout or investigate a discrepancy.Reproducible research
Rerun supported training workloads on another GPU vendor and compare model states exactly, making computational results easier to verify during replication and peer review. Seeds alone are not enough: preserve the input data, data order, preprocessing, initial model and optimizer state, configuration and library version too.
Repeatable fine-tuning
Start with pretrained weights and reproduce a supported fine-tuning run, or resume from a complete checkpoint on another supported vendor while preserving the training trajectory. Fine-tuning can need much less compute than pretraining, making the cost of exact replay easier to accommodate. The tradeoff depends on the workload: see the training timings and training identity evidence below.
Auditable model behavior
Preserve the model, inputs, preprocessing, configuration and release so another team can replay an output exactly. This could help investigate and document model behavior, including in regulated settings; numerical reproducibility alone does not establish regulatory compliance.
Apple FAST mode for classical ML and decision trees
Put the GPU in an Apple silicon MacBook to work on supported classical ML and tree workloads. In our M3 Ultra tests of supported classical ML and tree models, FAST mode was faster than CPU libraries in 213 of 231 measured training comparisons. These are mojolearn 0.8.34 results with one scored run per arm on that machine; MacBook performance depends on the model and workload.Compare Apple FAST timings and quality. FAST mode prioritizes speed and does not promise bitwise identity across vendors.
Verify on your hardware
Check against the recorded reference
Each release ships the output hashes we recorded for its checks. verify reruns representative inference and classical training checks on your GPU and compares each result with that reference, bit for bit. The report names the checks that ran and any failures or gaps.
python -m mojolearn verify --quickCompare two machines
You can also check two machines against each other directly, using supported Apple, NVIDIA or AMD GPUs, for example yours and a colleague's. Run the same check on both, with the same mojolearn release and settings and verification scope, and save each report. Copy both files to one machine with mojolearn installed. --compare checks that the inputs and protocols are compatible and compares the recorded output hashes. Matching hashes establish agreement for those recorded outputs. The comparison needs no GPU or network and does not load our reference table.
python -m mojolearn verify --quick --json-out mine.json # on one machine
python -m mojolearn verify --quick --json-out theirs.json # on another
python -m mojolearn verify --compare mine.json theirs.json # anywhereRead the final verdict and its coverage: agreement covers the recorded checks, not other workloads, and matching reports do not independently prove that the computations ran. In identical mode, mojolearn returns the same bits on Apple, NVIDIA and AMD GPUs.
Neural training
Quick verification excludes neural training. Run its full fixture set explicitly:
python -m mojolearn verify --neural-training --allThe verification guide explains the scopes, and the support matrix lists the GPU wheels in each release.
The numerical contract that makes bitwise identity possible
Mojo provides compilation across GPU backends. Getting their arithmetic to agree requires more work. Floating-point addition depends on order, and instruction selection and math libraries can change the final bits.
Fix the order of arithmetic
Adding the same floating-point values in a different order can change the answer. Reduction trees and accumulation rules keep that order stable across devices.
Control rounding and elementary functions
Fused operations, square root, exp and log can take different paths on different backends. The library uses explicit arithmetic rules and its own portable routines where needed.
Keep hardware scheduling separate
Thread layout, memory placement and kernel scheduling can vary while preserving the operations that determine the output bits. Mojo provides portability; mojolearn implements the numerical contract.
The numerical-pathway ledger explains the implementation choices. Neural training and default inference use full FP32 arithmetic. Some other algorithms, including tree histograms, use integer accumulation where appropriate.
The contract, and what it costs
Compare the same release, input bytes, model state, settings and supported execution configuration. In identical mode, mojolearn returns the same bits on Apple, NVIDIA and AMD GPUs. fast is available on Apple GPUs only and makes no cross-vendor identity promise. Results may change between releases. See the release notes and support matrix.
Identity has a cost: a fixed arithmetic order and portable numerical routines give up some throughput that vendor-specific implementations use. The measurements record timings, quality and scope for complete implementations, so a ratio does not isolate the cost of identity alone. Agreement is between mojolearn executions; it does not establish accuracy, fairness or fitness for a particular use, which need their own evaluation.
mojolearn is an alpha. Reproductions, bug reports and workloads where exact agreement would be useful are welcome, or get in touch.
From the paper
Bitwise-Identical Machine Learning Across Metal, CUDA, and HIP
Read the full paper (PDF)October 14, 2026 preprint · 22 pages
The abstract below describes the configurations verified in the paper, including its Apple and CPU results. The current release contract is described above.
The same machine-learning workload can produce different bits on different GPUs, changing predictions, learned models and subsequent training updates. mojolearn is a GPU machine-learning library whose default identical mode produces bitwise-identical results across verified Apple, NVIDIA and AMD GPUs and x86-64 and Arm CPUs. Results agree bit for bit, not merely within a numerical tolerance. It supports training and inference for over 100 machine-learning algorithms in 13 families, from decision trees to neural networks. Neural models trained in other frameworks can be served with identical outputs across vendors, and training workloads can be handed off between GPU vendors mid-run, or shared by GPUs of different vendors at once, while maintaining bitwise identity. Two training runs of a 162-million-parameter GPT-3 Small-shaped model follow different hardware routes collectively spanning Apple, NVIDIA and AMD GPUs and remain bitwise identical at all 5,000 optimizer steps, each processing 2.62 billion tokens. Neural training and inference run in full FP32 floating-point arithmetic, without reducing the computation to integers. To my knowledge, I am the first to enable these capabilities.
One Mojo codebase implements these algorithms across Metal, CUDA and HIP under a shared numerical contract. Mojo compiles the code for different GPUs, but the numerical contract, introduced in this work, fixes arithmetic order, rounding and elementary functions while allowing backend-specific scheduling.
A language model trained twice, with the same bits
A 162-million-parameter language model was trained twice from the same initialization, on the same 2.62 billion FineWeb-Edu tokens per run. The two routes used different hardware at every segment and together spanned NVIDIA, AMD and Apple. One segment trained jointly on an NVIDIA H100 and an AMD MI325X in separate machines for 400 steps.
The runs matched at all 5,000 optimizer steps in training state, summed gradients, all 64 per-shard losses and the learning rate. They wrote identical final checkpoint files, one on an Apple M3 Ultra and the other on an NVIDIA L40S.
4129921e99ed404db1ca4b7b94ebc43c0bee18e7711deb326b31ec89e0b402c84129921e99ed404db1ca4b7b94ebc43c0bee18e7711deb326b31ec89e0b402c8The training experiment
Two hardware routes. The same bits at every step.
162,147,840 parameters and 2.62 billion tokens per route. Each route resumes its checkpoint on the next segment’s hardware.
Segment 1
Steps 1 to 1,000
Route A2 × NVIDIA H100
1,000 steps
equal bitsRoute B1 × AMD MI325X
Segment 2
Steps 1,001 to 2,000
Route A2 × NVIDIA H100
1,000 steps
equal bitsRoute B2 × AMD MI300X
Segment 3
Steps 2,001 to 2,400
Route A1 × NVIDIA H100 + 1 × AMD MI325X
Joint training, separate machines400 steps
equal bitsRoute B1 × NVIDIA H100
Segment 4
Steps 2,401 to 3,900
Route A2 × AMD MI300X
1,500 steps
equal bitsRoute B2 × NVIDIA H100
Segment 5
Steps 3,901 to 4,900
Route A2 × NVIDIA H100
1,000 steps
equal bitsRoute B2 × AMD MI300X
Segment 6
Steps 4,901 to 5,000
Route A1 × Apple M3 Ultra
100 steps
equal bitsRoute B1 × NVIDIA L40S
Same initialization. 5,000 matching updates. Identical final checkpoint files.
Show the routes as a table
| Segment | Steps | Route A | Route B | Matching steps |
|---|---|---|---|---|
| 1 | 1 to 1,000 | 2 × NVIDIA H100 | 1 × AMD MI325X | 1,000 |
| 2 | 1,001 to 2,000 | 2 × NVIDIA H100 | 2 × AMD MI300X | 1,000 |
| 3 | 2,001 to 2,400 | 1 × NVIDIA H100 + 1 × AMD MI325X (joint training) | 1 × NVIDIA H100 | 400 |
| 4 | 2,401 to 3,900 | 2 × AMD MI300X | 2 × NVIDIA H100 | 1,500 |
| 5 | 3,901 to 4,900 | 2 × NVIDIA H100 | 2 × AMD MI300X | 1,000 |
| 6 | 4,901 to 5,000 | 1 × Apple M3 Ultra | 1 × NVIDIA L40S | 100 |
The model has GPT-3 Small’s depth and width with modern Llama-style blocks, and uses FP32 throughout. The model card has its architecture, quality evaluations and limitations, the weights and both routes’ evidence.