mirror of
https://github.com/ollama/ollama.git
synced 2026-09-04 10:17:56 +00:00
2b4a99376c
ModelOpt checkpoints apply a float32 global scale to every projection
output on top of the per-group quantization scales. Running the
multiply and the cast back to the activation dtype as separate eager
ops costs an extra kernel launch and a materialized intermediate per
projection.
Compile the multiply and cast into one kernel. On an M5 Max (medians
of order-swapped A/B runs against main; greedy outputs byte-identical):
qwen3.6:27b prefill 703 -> 769 t/s +7.9%
muse-glimmer:30b prefill 790 -> 843 t/s +6.7%
Speculative decode is unchanged within noise on both models. Only
checkpoints with a global scale are affected; single-scale nvfp4,
mxfp8, and affine checkpoints take the unchanged path.
|
||
|---|---|---|
| .. | ||
| create | ||
| imagegen/manifest | ||
| internal | ||
| mlxrunner | ||
| models | ||
| quant | ||
| safetensors | ||
| server | ||
| tokenizer | ||
| transfer | ||