The previous mistral3 vision handler loaded but produced hallucinated
descriptions of real images (e.g. described a clear photo of a person
in a suit handing money as "abstract hexagonal pattern"). Solid colors
and large color blocks worked, suggesting the vision tower was producing
*some* signal but the spatial/feature relationships were scrambled.
Root cause: Ollama's mistral3 converter only applies its LLaMA-style
RoPE repack to TEXT-side `attn_q`/`attn_k` tensors — the
`if !HasPrefix(name, "v.")` guard in convert/convert_mistral.go::Tensors
skips vision tensors entirely. Vision Q/K therefore leave the converter
in raw HF/PyTorch order. Upstream's HF→GGUF flow does permute vision
Q/K with the vision head count, because pixtral's clip graph uses
`ggml_rope_ext` in mode 0 which expects the [n_head, head_dim/2, 2, ...]
LLaMA layout.
Fix: register a load-time op that applies LlamaModel.permute equivalently
on each F16 vision attn_q.weight / attn_k.weight (24 layers × 2 = 48
tensors for ministral-3 8B). Capture file offsets *before* renames
invalidate them, same pattern as promote_tensor_to_f32 and
register_concat_load.
Verified against the upstream-format migration mmproj (which gets the
same image right) — both 8B Ollama+compat and 14B migration now produce
matching, accurate descriptions ("hand giving money to person in suit,
cartoonish star/burst above their head, blue grid background").
Translates Ollama's monolithic mistral3 GGUF into the upstream pixtral
mmproj shape so `--mmproj <same-blob>` loads alongside the text model.
KV synthesis: rewrite arch to `clip`, copy `mistral3.vision.*` →
`clip.vision.*`, set `projector_type=pixtral`, `use_silu=true`,
inject pixtral image_mean/std defaults, copy
`mistral3.spatial_merge_size` → `clip.vision.spatial_merge_size`.
Tensor renames: `v.patch_conv`→`v.patch_embd`, `v.encoder_norm`→
`v.pre_ln`, `attn_output`→`attn_out`, `attn_norm`/`ffn_norm`→
`ln1`/`ln2`, `mm.linear_{1,2}`→`mm.{1,2}`, `mm.norm`→`mm.input_norm`,
`mm.patch_merger.merging_layer`→`mm.patch_merger`.
img_break: pixtral's loader strictly requires `v.token_embd.img_break`
(the embedding row for the [IMG_BREAK] separator token). Ollama
doesn't ship it as a separate tensor; the "right" value is row 12 of
token_embd.weight, but token_embd is Q4_K and per-row dequant is
heavyweight. Reclaim the orphan `output_norm.weight` slot (already
[n_embd] F32) and zero-fill via load op — pixtral.cpp adds img_break
to row separators, so a zero embedding makes [IMG_BREAK] insertion a
no-op without breaking the rest of the vision graph.
Adds `mistral3` to the Go-side `compatClipArches` allowlist so
ollama auto-passes `--mmproj <same-blob>` for vision requests.
Tested with ministral-3 (8B, just released): text + vision both work;
solid colored test images correctly identified (red/green/blue).
Ministral-3 (and other Ollama-published mistral3 models) load with this
handler. Differences vs upstream HF format:
- RoPE YaRN parameters: Ollama writes `rope.scaling.beta_fast` /
`beta_slow` while upstream wants the `yarn_*`-prefixed names.
- Attention temperature scale: Ollama writes `rope.scaling_beta`,
upstream reads `attention.temperature_scale` (same numeric value).
- The Ollama GGUF is monolithic — embedded `v.*` / `mm.*` tensors must
be hidden from the text loader, otherwise n_tensors mismatches.
Vision (clip) translation is not yet implemented; running with
--mmproj will still fail. Ministral-3 text generation works end-to-end.
gpt-oss: rename arch "gptoss" -> "gpt-oss" (incl. KV prefix), inject
the missing `expert_feed_forward_length` from the ffn_gate_exps shape,
and rename `attn_out`/`attn_sinks`/`ffn_norm` tensors to upstream's
`attn_output`/`attn_sinks.weight`/`post_attention_norm`. Also remove
the library/gpt-oss -> dhiltgen/gpt-oss redirect now that the compat
shim handles it directly.
lfm2: rename `output_norm.weight` -> `token_embd_norm.weight` and fix
a stale `lfm2.feed_forward_length` (some Ollama blobs claim 12288 on
a model whose ffn_gate is [2048, 8192]) by reading the real value off
the ffn_gate tensor shape.
Adds two helpers to compat-util: `copy_kv` (type-preserving generic
KV copy) and `rename_kv_prefix` (bulk-copy every KV with a given
prefix to a new prefix). Old keys are left in place — harmless because
the loader queries by exact name and only the new prefix matters.
Tested locally: gpt-oss:20b and lfm2.5-thinking now load + generate
coherently against an unmodified upstream llama-server build.
Added a "Maintenance" section to llama/compat/README.md and an inline
block to llama-ollama-compat-util.h enumerating the three places the
compat layer depends on something that isn't strictly guaranteed
upstream-public API:
1. Direct ggml_tensor::{type,ne,nb} field writes (public struct,
no sanctioned post-creation mutators).
2. const_cast on gguf_get_tensor_name's return to rename in place.
3. llama_model_loader forward-decl from src/ (used only as an
opaque pointer key).
Each entry ships with its escape hatch (small upstream patch or a
local replacement). All have been stable for years; they don't need
immediate action, but surfacing them makes the maintenance story
explicit for future readers.
Main translation unit (llama-ollama-compat.cpp) is now purely per-arch
dispatch: detect_* + handle_* for each arch, plus the 4 public entry
points. Dropped from 724 lines to 430.
Everything that doesn't depend on a specific arch moves to
llama-ollama-compat-util.{h,cpp}:
- gguf KV helpers (has_key, copy_{u32,f32}_kv, inject_{u32,f32,str,bool,
f32_arr}_if_missing, truncate_{str,data}_arr)
- ggml tensor helpers (any_tensor_with_prefix, rename_tensor,
rename_tensors_containing, set_tensor_{type,shape}, reclaim_slot_as,
tensor_file_offset)
- per-loader skip-prefix registry (add_skip_prefix, should_skip_tensor_prefix)
- LoadOp registry (register_load_op, take_load_op, read_at)
- common high-level transforms (promote_tensor_to_f32, register_concat_load)
New helpers introduced while splitting:
- inject_{u32,f32,str,bool,f32_arr}_if_missing — replaces the
has_key + gguf_set_val_* idiom we were using 20+ times.
- reclaim_slot_as — extracts the "rename an orphan tensor slot as a
synthesized one" pattern used by qwen35moe's patch_embed split. Clear
name + comment explains the workaround.
CMake: target_sources now globs llama/compat/*.cpp (CONFIGURE_DEPENDS),
so new .cpp files are picked up without CMake edits.
Nothing behaviorally changed. Verified gemma3 + qwen3.5 text + vision
still work end-to-end after a clean rebuild.
Extends the compat layer with the vision side for Ollama's monolithic
qwen3.5 blobs. All changes in llama/compat/ — no new upstream patch edits.
New generic infra (reused by gemma3's existing promotion):
- LoadOp registry (g_loadops). Any dest tensor whose name is registered
gets its bytes produced by a closure instead of being read straight
from disk. maybe_load_tensor consults it.
- promote_tensor_to_f32(meta, ctx, name) now captures the source offset
at registration time and becomes a LoadOp. Gemma3 already migrated.
- register_concat_load(meta, dest, {srcs...}) captures the file offsets
of N source tensors and registers a LoadOp that concatenates them.
Assumes sources concatenate along their slowest ggml axis — which in
C order means the dest bytes are src[0] || src[1] || ... .
- set_tensor_shape / set_tensor_type helpers for in-place edits.
qwen35moe clip handler (handle_qwen35moe_clip):
- Detection reuses detect_ollama_qwen35moe; additionally requires
embedded v.* tensors so we don't fire for text-only files.
- KV synth: clip.vision.* from qwen35moe.vision.* + sensible defaults
(feed_forward_length=4304, image_size=768, layer_norm_epsilon=1e-6,
is_deepstack_layers=false[27], image_mean/std=[0.5,0.5,0.5]).
- Arch rewrite: general.architecture=clip, projector_type=qwen3vl_merger.
- QKV merge per block (27x): captures q/k/v file offsets, registers a
concat LoadOp, renames attn_q -> attn_qkv and widens its shape from
[hidden, hidden] to [hidden, 3*hidden].
- patch_embed split: source [16,16,2,3456] F16 -> two dests
[16,16,3,1152] F32, permuting (c_out*3+c_in) packed_c back into
separate c_in/c_out dims. Matches upstream convert_hf's
Qwen3VLVisionModel.modify_tensors split.
- Tensor renames (substring-matched): pos_embed -> position_embd,
merger.norm -> post_ln, merger.linear_fc1/2 -> mm.0/mm.2,
mlp.linear_fc1/2 -> ffn_up/ffn_down, norm1/2 -> ln1/ln2.
- F16 -> F32 promote for v.position_embd.weight.
Ctx-pool trick for the sibling tensor:
clip.cpp sizes its ggml_context for exactly the gguf's tensor count
(+1). ggml_new_tensor to add v.patch_embd.weight.1 overflows. Since
v.blk.0.attn_k.weight is orphaned after the QKV merge (clip only
requests the merged attn_qkv), steal that slot: rename it to
v.patch_embd.weight.1 and reshape to [16,16,3,1152] F32. Its original
file offset is ignored; the LoadOp we register overrides the read.
Go side: adds qwen35moe to the auto-mmproj arch allowlist. ollama now
passes the monolithic blob as both --model and --mmproj for qwen3.5.
Verified end-to-end: ollama run qwen3.5:35b-a3b-q4_K_M with an image
correctly describes the image ("screenshot of a chat interface...
'open the browser, open never gonna give you up on youtube'..."). Text
inference still works on the same blob.
Text-only support for Ollama's qwen3.5 (qwen35moe) blobs. Vision is more
involved (QKV merge, patch_embed reshape/split) and follows in a later
commit.
Detection markers: qwen35moe.vision.*, qwen35moe.image_token_id,
qwen35moe.ssm.v_head_reordered, qwen35moe.feed_forward_length, mtp.*
tensors, embedded v.* tensors.
Fixes applied:
- attention.head_count_kv ARRAY -> UINT32 scalar (Ollama wrote per-layer
[0,0,0,2,...] over 40 layers; upstream wants the max non-zero scalar).
- rope.dimension_sections padded from [11,11,10] to [11,11,10,0]
(4-element M-RoPE convention).
- blk.N.ssm_dt -> blk.N.ssm_dt.bias rename for all 40 layers.
- skip prefixes v.*, mm.*, mtp.* from the text loader.
Tightens detect_ollama_gemma3 so it only matches files whose
general.architecture is actually "gemma3" — without this, qwen3.5 was
triggering gemma3 clip translation on the auto-mmproj pass and crashing
on tensor-shape mismatches.
Go side: gates the auto-mmproj behavior on a small arch allowlist
(currently just gemma3). When a clip handler exists for an arch, add it
to the map; until then the model runs text-only.
Verified end-to-end via `ollama run qwen3.5:35b-a3b-q4_K_M` on the
existing library/qwen3.5 blob — answers "2 + 2 equals **4**" with the
text loader's compat fixups firing and no --mmproj flag.
Round of post-first-arch cleanup. No behavioral change — text + vision
still pass end-to-end on Ollama gemma3:latest (4B), 1B, and 270m.
Removes:
- apply_tensor_transforms() and the TransformSpec registry. Registered
the +1 RMSNorm shift at one point but empirically Ollama's blobs
already have it baked in. YAGNI until an arch actually needs a
tensor-data transform; the hook can come back data-driven.
- Corresponding llama-model.cpp patch hunk (include + post-load call).
- copy_kv<> template plumbing. Direct copy_u32_kv / copy_f32_kv are
shorter and more readable.
- set_str wrapper (one-liner around gguf_set_val_str).
- find_tensor helper (only used once; inlined as a loop).
- convert_f16_to_f32 helper (one-line inner loop, inlined).
- set_f32_if_missing wrapper (inlined at 3 call sites).
Tightens:
- Clip tensor renames are now a table (kGemma3ClipRenames) iterated by
handle_gemma3_clip. Adding a rename is one row.
- translate_clip_metadata now reuses detect_ollama_gemma3 for its
"is this an Ollama blob?" check instead of a separate ad-hoc check.
Net:
llama-ollama-compat.cpp 488 -> 343 lines (-30%)
llama-ollama-compat.h 83 -> 63 lines (-24%)
upstream-edits.patch 20 -> 16 lines of real edits, 3 -> 2 files
Two reductions:
1. Drop the gguf_rename_tensor forwarder from gguf.h/gguf.cpp.
The rename-in-place trick it does (calling ggml_set_name on an embedded
ggml_tensor) can be done from outside gguf.cpp via:
char * p = const_cast<char *>(gguf_get_tensor_name(meta, id));
strncpy(p, new_name, GGML_MAX_NAME - 1);
That pointer aims into a mutable char[GGML_MAX_NAME] inside a std::vector
element; the const on the return type is API courtesy. This is defined
behavior and has no struct-layout dependency.
2. Drop the src/CMakeLists.txt hunk that added llama-ollama-compat.cpp to
the llama target. Replace with a target_sources() call in Ollama's
llama/server/CMakeLists.txt after FetchContent_MakeAvailable. Our
compat files now stay in llama/compat/ and are never copied into the
fetched _deps/ tree.
Net patch now touches 3 files, 20 lines, all pure call-site insertions:
src/llama-model-loader.cpp +8 (include + translate + 2x should_skip)
src/llama-model.cpp +4 (include + apply_tensor_transforms)
tools/mtmd/clip.cpp +8 (include + translate_clip + maybe_load)
Verified: fresh build from scratch (rm -rf build && cmake configure)
runs PATCH_COMMAND cleanly, compiles, and ollama run gemma3 still works
end-to-end for text + vision.
The clip.cpp tensor-read loop was the fattest hook in the patch — it
duplicated the host-vs-device buffer dispatch around a call into the
compat layer. Move that dispatch into our code (maybe_load_tensor),
so the upstream patch is a single conditional call.
Net: upstream patch drops from 48 lines across 6 files to 34 lines.
Every remaining edit is either a 1-line include, a 1-line function call,
or the gguf_rename_tensor shim (which accesses gguf_context internals
and has to live in gguf.cpp).
Verified end-to-end: text + vision both still correct after rebuild.
Previous handler only fired on vision-capable gemma3 (4B/12B/27B) because
its detection looked for `gemma3.mm.tokens_per_image` or embedded v.*/mm.*
tensors. The 1B blob has neither — but its old Ollama converter emitted:
- gemma3.rope.global.freq_base (upstream uses gemma3.rope.freq_base)
- gemma3.rope.local.freq_base (upstream uses gemma3.rope.freq_base_swa)
- tokenizer.ggml.add_{padding,unknown}_token
so llama.cpp would fall back to default rope_freq_base=10000 and produce
visibly-worse output.
Also inject rope.scaling.factor=8.0 / type=linear on 4B/12B/27B — those
variants ship with that scaling in their HF config to extend the native
~16k trained context to 131072. Without this KV, llama.cpp uses factor=1.0
and the positional embeddings are subtly off everywhere.
Detection now flips on any Ollama-specific marker. All three variants
verified end-to-end via `ollama run gemma3:{latest,1b,270m}`.
FetchContent's PATCH_COMMAND runs after each update step — including on
incremental rebuilds. `git apply` fails when the patch is already applied,
which bricks the build until the dev wipes build/ entirely.
Fix by routing the apply through a small apply-patch.cmake helper that
checks `git apply --reverse --check` first. If the patch cleanly reverses,
it's already applied and we skip. Otherwise apply forward. Both branches
surface real errors (drift against upstream, missing patch file, etc.).
Verified: fresh configure+build applies the patch once; re-running the
same commands is a no-op with no errors.
Two tiny Go-side changes that let the llama/compat shim take over gemma3:
1. llm/llama_server.go: when the GGUF has embedded v.* tensors and no
projector layer is declared, pass the model file itself as --mmproj.
The in-process compat layer translates the same file into both a
text-only view (for --model) and a clip-mmproj view (for --mmproj).
2. server/model_resolver.go: drop library/gemma3 from compatModelRedirects.
The compat layer handles it directly, so no dhiltgen/ republish is
needed. Other arches stay in the redirect list until they get their
own handler in llama/compat/llama-ollama-compat.cpp.
End-to-end verified: `ollama run gemma3` answers text and image prompts
against the existing library/gemma3 blob with no re-download.
Older Ollama builds ship GGUFs that diverge slightly from upstream llama.cpp
in arch names, KV keys, tensor names, and (for vision models) file layout
(text+vision in one monolithic file). This adds a self-contained compat
layer that translates those files in memory at load time, so
~/.ollama/models/blobs/* can be served by upstream llama-server with no
re-conversion and no re-download.
Structure:
llama/compat/
llama-ollama-compat.{h,cpp} — the shim (Ollama-owned, ~500 LOC)
upstream-edits.patch — ~48 lines of call-site hooks in 6 upstream files
compat.cmake — include()-able CMake fragment
README.md — what/why/how-to-regen
Integration: llama/server/CMakeLists.txt includes compat.cmake and passes
OLLAMA_LLAMA_CPP_COMPAT_PATCH_COMMAND to FetchContent_Declare via
PATCH_COMMAND. When OLLAMA_LLAMA_CPP_SOURCE is set (dev mode), the patch is
skipped so the developer's tree stays untouched.
Currently handles gemma3 (text + vision). Pattern is data-driven — adding
other archs is a new handle_<arch>() + one dispatch line. See README for
the per-arch checklist.
Verified end-to-end: `llama-server --model BLOB --mmproj BLOB` with an
Ollama gemma3:latest blob answers both text prompts ("Paris") and vision
prompts (correct image descriptions).
Remove the vendored GGML and llama.cpp backend, CGO runner, Go model
implementations, and sample. llama-server (built from upstream llama.cpp via
FetchContent) is now the sole inference engine for GGUF-based models.
(Safetensor based models continue to run on the new MLX engine.) This allows
us to more rapidly pick up new capabilities and fixes from llama.cpp as they
come out.
On windows this now requires recent AMD driver versions to support ROCm v7 as
llama.cpp currently does not support building against v6.
If you have a long running create, and start another ollama server with the
same model dir, the GC algorithm deletes the pending blobs and breaks the
create. This adds a 1h grace period to avoid deleting in-flight creation
operations.
Following up on #15560, this change now has e2b/e4b render differently
from 26b/31b.
For backwards compatibility, we take the existing renderer name `gemma4`
and make it do dynamic resolution based on the model name/size, but the
intended use is for the models to be republished with the renderer
variant specified explicitly: `gemma4-small` or `gemma4-large`.
After the rotating buffer has wrapped (c.offset > c.maxSize) a subsequent
L>1 Update() went through a slice-to-[0, c.idx) path that discarded all
slots in [c.idx, Dim), losing the older-but-still-in-window tokens the
first Q of the new batch needs for its sliding-window attention.
Linearize the circular buffer to logical order in that wrapped case so
the existing trim + concat preserves the last (maxSize - 1) old tokens.
When the buffer has not yet wrapped (c.offset <= c.maxSize), slots
[c.idx, Dim) are grow padding or stale post-rewind data, so keep
dropping them.
Converts SiLU/GELUApprox to compiled kernels and adds SwiGLU,
matching upstream mlx/mlx_lm's activations pattern. Routes llama,
qwen3, qwen3_5 (dense + MoE), and glm4_moe_lite MLP paths through
mlx.SwiGLU so each MLP invocation runs as one fused Metal/CUDA
kernel rather than a chain of per-op launches.
Wraps MLX's mlx_compile API so Go functions can be traced into fused
kernels. Contiguous elementwise chains collapse into a single
Metal/CUDA kernel instead of launching one per op.
Exposes Compile plus arity helpers (Compile1/2/3) that mirror Python's
@mx.compile decorator shape, lazily building the closure on first call
so package-level declarations work before the MLX dylib loads.
* gemma4: implement Gemma 4 model for MLX (text-only runtime)
* gemma4: two MoE + SWA prefill perf fixes
Two performance optimizations in the gemma4 forward pass
1. Memoize the sliding-window prefill mask across layers.
2. Softmax only over the selected experts in Router.Forward.
* review comments
Gemma 4 prompts differ when thinking is disabled for different sized
models: 26b/31b emit an empty thought block, while e2b/e4b do not.
Before #15490, our shared Gemma 4 renderer effectively matched the
e2b behavior. #15490 changed it to always emit the empty thought block,
which regressed e2b/e4b nothink behavior and led to #15536 (and possibly
This change restores the previous shared behavior by removing the empty
trailing thought block. It also renames the checked-in upstream chat
templates so the e2b and 31b fixtures are tracked separately.
A follow-up will split Gemma 4 rendering by model size.
Fixes: #15536
For some versions of Xcode, cmake builds are failing due to header problems in
cross-compiling during the generate phase. Since generate is producing arch
independent generated output, we can skip this during cross-compiling.
* mlx: add op wrappers for Conv2d, Pad, activations, trig, and masked SDPA
Add Conv2d, flexible Pad (with axes/mode), PadConstant, Maximum,
Minimum, Softplus, ReLU, GLU, Clamp, Sin, Cos, Clip,
ScaledDotProductAttentionMasked, and RoPEWithFreqs. Refactor
RoPEWithBase to delegate to RoPEWithFreqs.
* review comments
* mlx: fix ScaledDotProductAttentionMasked to consult the mask argument
Improve the MLX model creation pipeline with several model-agnostic changes:
- Rewrite supportsVision to use vision_config instead of architecture name
- Add supportsAudio for audio encoder detection
- Add alignment checking (isAligned) for quantization group sizes
- Support per-projection mixed quantization in MoE expert packing
- Record per-tensor quant metadata in safetensors blobs
- Parse per-tensor quant metadata at model load time
- Validate quantize output is non-empty before storing
- Fix pin/unpin cleanup in expert group quantization
- Promote v_proj/k_proj/down_proj to INT8 for INT4 base quant
- Add MetalIsAvailable() utility
- Skip audio encoder tensors from quantization
