Draft token embeddings were kept at source precision. A draft that
reuses its embedding as the output projection (the gemma4 assistant)
then reads the whole 537MB bf16 tensor on every draft step — about half
the step's cost. Draft quality only affects how many drafts are
accepted, so the output head now takes the requested type instead of the
8-bit type that protects a target's output quality.
gemma4:26b-mlx, M5 Max: MTP code decode 148 -> 157 tok/s (+26% -> +37%
over plain); prose goes from roughly zero to +2-5%; acceptance unchanged.
This is a rewrite of the create functionality for the MLX engine.
The core idea behind the create functionality is to break the import/convert into a pipeline of distinct phases:
* Read (scan the safetensors directory for the various bits of metadata)
* Classify (determine what the import type)
* Plan (determine any transforms that need to be done)
* Write (transform any data as necessary and write out the blobs)
* Create the manifest
Each architecture has a "policy" which determines how to convert the model correctly. A number of different formats for safetensors are supported including:
* nvfp4 (two formats: model optimized, torch)
* fp8 datatypes (convert to mxfp8)
* standard bf16 based weights
A number of cleanups/simplifications have been done including:
* using the baked in names for the tensors instead of munging them into something else
* unified 3d expert tensors (instead of separate per expert tensors)
* fewer unnecessary transforms to the various tensors in a model (keep a model as close to the source as possible)
* unified capability checking
* draft model handling (for MTP) is done on the same path
Image generation has been intentionally removed.
