ollama/fs
Daniel Hiltgen c053d1e985
MLX: make Qwen3.8 Flash Next memory efficient (#18078)
* MLX: reduce Qwen3.8 Flash Next memory use

For NVFP4 creation, retain source precision for sensitive QSA projections
and non-expert MTP weights. Keep routed experts and PLE at NVFP4, and
promote other eligible weights to MXFP8.

Keep NVFP4 PLE tables host-backed, transferring only selected packed rows
and their scales to MLX. Honor scalar and per-expert NVFP4 global scales
in MoE, and per-shard global scales in host-backed PLE.

Use compressed QSA block selections in fused indexed Metal attention for
long prefill batches, retaining the graph fallback. Co-locate component
loading and forward methods, with explicit quantization and cache ownership.

* mlx: skip materializing host-backed tensors

Let models exclude tensors already owned by host storage from MLX loading.
Avoid allocating and discarding the full Qwen PLE tables at startup while
preserving the resident fallback.

* address comments
2026-11-06 13:39:21 -08:00
..
gguf gguf: add the q2_0 tensor and file types (#18967) 2026-10-15 15:30:44 -07:00
safetensors MLX: make Qwen3.8 Flash Next memory efficient (#18078) 2026-11-06 13:39:21 -08:00
util/bufioutil