model/base held the Model interface and the architecture registry while
model held weight loading and quant parameters, and every architecture
imported both. create/client was the CLI side of safetensors imports,
with the create command as its only caller, a duplicate of the command's
adapter error, and a name that read like a second API client. The
safetensors show helpers had their own package under x/ although it
already declared package server.
base merges into model, so base.Model and base.Register become
model.Model and model.Register; nothing in the two overlapped. The create
client's two files and their tests join package cmd, and the five names
the command called are no longer exported. The show helpers join the
server package, and the three entry points routes.go calls become
unexported like the helpers around them.
The MLX runner is the only Go inference runner left and is no longer
experimental, so its packages leave x/. The bindings become a top-level
mlx package beside the carried patches in mlx/compat, mirroring how
llama/ holds the llama.cpp integration, and the runner becomes mlxrunner
with the architectures nested under the package they implement.
Subpackages move with their parent unless listed.
x/mlxrunner/mlx mlx
x/internal/mlxthread mlx/mlxthread
x/internal/mlxthreadtest mlx/mlxthread/mlxthreadtest
x/internal/mlxtest mlx/mlxtest
x/quant mlx/quant
mlx/compat/*.patch mlx/compat/mlx-c (MLX patches go in mlx/compat/mlx)
x/mlxrunner mlxrunner
x/models/nn mlxrunner/nn
x/models/<arch> mlxrunner/model/<arch>
x/mlxrunner/imports.go mlxrunner/model/architectures (new package)
x/create create
x/safetensors fs/safetensors
x/tokenizer mlxrunner/tokenizer
Every package keeps its name, so the Go changes are the import path
rewrites the moves force, and the CMake, Dockerfile, CI cache keys, drift
check and Darwin payload script follow the new paths. Four edits are not
paths: the runner's blank architecture imports become the package
mlxrunner/model/architectures, so the list to extend for a new model sits
beside the architecture directories; a depguard rule keeps the two test
harnesses out of non-test code, as the x/internal placement used to; the
CI change filter's two entries for the long-deleted x/imagegen/mlx now
name the bindings' CMake project and the carried patches, so a change to
either builds the payload; and the tokenizer parity test reads its
fixtures from its own testdata instead of walking out of x/.
x/server and x/imagegen/manifest stay for the next two commits.
* create: add server-side MLX imports and drop GGUF conversion
Support safetensors imports through the MLX create pipeline both locally and on the server, including remote upload/staging, draft layer handling, cancellation propagation, transfer limits, and shared manifest/blob writing.
Limit GGUF create to wrapping existing GGUF inputs into Ollama manifests. Remove the in-tree safetensors-to-GGUF converter, server quantization path, and converter-only dependencies so GGUF conversion and quantization stay in llama.cpp tooling.
Keep the MLX path focused on supported safetensors model creation with validation before MLX work, and expose that flow without the --experimental CLI gate.
* address comments
* add client side gguf create fast path
* address comments
* rebase adjustments
Loading GGUF metadata is an expensive operation. Two caches had evolved to
mitigate this, and the two capability implementations produced inconsistent
results for some models.
This PR now extracts the metadata once per blob into a file at
<OLLAMA_MODELS>/metadata/sha256-<hex>.json. Only arrays over 4096 elements and
non-finite floats are left out.
Direct Capabilities() discovery now costs us instead of ms. /api/tags can
build directly from manifests and the extracted metadata.
progress: join the render loop in stop and do the final writes after the
goroutine exits, so Stop/StopAndClear cannot race an in-flight render on
the shared bufio.Writer.
sched: read the unload-mutable fields in runnerRef.LogValue only under a
successful refMu.TryLock and omit them when contended, since slog resolves
it on goroutines that may already hold refMu.
Safetensors gemma4 imports served by the MLX engine now answer image
and audio chats. Images run through both vision architectures: the
transformer tower (26B, 31B, e-series) and the 12B's encoder-free
unified embedder. Audio arrives through the same intake the ollama
API already accepts for gemma4 GGUFs — WAV bytes in the images field,
OpenAI input_audio parts, and /v1/audio/transcriptions uploads — with
the e2b/e4b checkpoints running clips through their conformer audio
encoder and the 12b unified checkpoint embedding the raw waveform
directly. Clips longer than 30 seconds are split evenly into chunks
of at most 30 seconds, cut at pauses, and encoded independently.
Each modality serves only checkpoints that carry it: 26B/31B have no
audio config and reject audio input, and checkpoints with an
unrecognized vision architecture still load as text-only models and
reject image requests.
The server previously hid the vision and audio capabilities for
gemma4 safetensors because the engine served neither. Both
suppressions are removed, and existing imports start advertising the
capabilities without re-importing since import already records them.
* Report cached prompt tokens
Add prompt_eval_cached_count to native responses and expose equivalent cached-token fields through the OpenAI- and Anthropic-compatible APIs. Keep prompt_eval_count as the logical input total while excluding cache hits from CLI and benchmark prefill rates. Surface processed and cached prompt counts in benchmark output.
Collect cache counts from llama-server and MLX, preserve coherent metrics across two-pass structured generation.
Fixes#8008
Related to #15758
* review comments
* Honor model generation defaults
Model-authored sampler defaults from GGUF metadata and HF generation_config.json were ignored, so built-in Ollama defaults could override model intent unless parameters were set in the Modelfile or request. The fix parses those defaults into model config and applies them before Modelfile/request options, preserving the expected precedence order.
* review comments
* address comments
When a builtin parser rejects model output, the completion callback wrote the
error to an unbuffered channel and returned. The callback cannot stop
generation -- it has no error return -- so the next chunk re-entered the
callback, hit the same parse error and blocked writing to a channel the
consumer had already stopped reading after emitting its 500. The completion
never returned, the goroutine leaked and the runner request was never
released, so retrying the same prompt hung with no log output until the client
gave up.
Record the parse error, cancel the completion, and report it once the
completion has returned. Parse failures landing on the final chunk were
already terminal, which is why non-thinking requests and the direct
qwen3-coder parser path failed cleanly and only thinking mode wedged.
ChatHandler and GenerateHandler share the defect: both run the same parser in
the same shape of callback behind a consumer that stops reading at the first
error. GenerateHandler had no cancel func at all, so one is added there.
Fixes#17825
As we look to bring Linux and Windows MLX support online, instead of blocking
downloads at the registry to avoid users wasting time downloading a model they
can't run, shift the logic to the local side which knows if MLX is present or not.
When a manifest contains a config and layer with the same digest, the
skipVerify map entry was overwritten by the config's cache-hit value
(true), replacing the layer's non-cache-hit value (false). This caused
verifyBlob to be skipped for the freshly downloaded blob.
A rogue OCI registry could exploit this by serving a manifest with
duplicate digests and redirecting blob downloads to internal endpoints.
The SSRF response would be written to disk, hash verification would be
skipped due to the map collision, and the blob would persist.
The fix uses logical AND when updating skipVerify: once any download of
a digest was not a cache hit, verification is always performed.
Fixes#15485
---------
Co-authored-by: Patrick Devine <patrick@ollama.com>
Add MLX support for Nemotron 3 Nano Omni, including the model implementation, Mamba2/recurrent pieces, MoE routing, and quantized NVFP4/MXFP8 expert paths.
Use a shared mapped MoE GatherQMM fast path under the generic moe_gather_qmm_mapped naming, with Metal-optimized NVFP4/MXFP8 block-mapped kernels and generic fallbacks for unsupported backends.
Serve the model's multi-token prediction head as a self-draft speculator, so speculative decoding needs no separate draft model.
Render the Nemotron prompt from the published chat template. The template the renderer was based on had drifted from the current reference; refreshing it surfaced five mismatches: stray leading newlines, the wrong turn separator and a trailing newline before the generation prompt; /think and /no_think toggles left in user turns; a trimmed system message the template leaves intact; a user block opened by a leading tool message; and Go scalar syntax for schema extras where the template applies Python str(), sending true/false/<nil> in place of True/False/None. Reference tests now render every case through the template itself.
Also harden the Nemotron parser path shared by both backends: while collecting thinking, preserve whitespace before partial </think>, <think>, and <tool_call> fakeouts, with streaming tests covering those cases.
* glimmer: implement the Muse Glimmer model
MLX model (language + vision encoder) with DFlash draft wiring, llama-server DFlash support and rope-interleave fix, renderer and parser, tokenizer fixes, and the import quantization policy.
* mlxrunner: report committed prefill chunks after the sweep and eval
The drafter's flush evaluates its report, and an eval that runs while the chunk's construction handles are still live cannot free any intermediate buffer. On media chunks that retention keeps the whole vision tower resident and grinds the Metal allocator at its limit until the request dies. Pin the report's inputs across the sweep, report after the chunk materializes, and release media items after the report so a drafter can still capture the rows its deferred flush embeds.
* ci: retry CUDA pre-release download
rootfs.diff_ids duplicated the manifest's layer digest list into the config blob and nothing ever read it. On per-tensor safetensors models the copy grows past 100KB and create excessively large config blobs with unused redundant data. Model identity is unaffected: it is the digest of the manifest itself, which already commits to every layer hash.
golangci-lint ran with only-new-issues, which filters findings down to the
lines a PR adds. That silently drops any issue a diff introduces at a
distance, where the report anchors to a line the diff never touched.
CI now is enabled to scan all files. This PR also fixes the last few
straggler lint glitches outside of integration, which I'll tackle
in a follow up PR.
Remove the x/imagegen tree (MLX image generation engine, Flux2/zimage
models, cache, C bindings) and all imagegen integration points:
- server: drop imagegen routes, scheduling, and generate handling
- api/cmd/docs: remove image generation API surface and docs
- middleware/openai: remove image endpoint support
- integration: remove imagegen test suites
- x/create: adopt the rewritten create pipeline from main; drop
imagegen create path (CreateImageGenModel, IsTensorModelDir,
model_index.json detection, Flux2KleinPipeline vision hack)
- retain x/imagegen/manifest (Ollama-store safetensors manifest
loader), still used by x/mlxrunner and x/create/client
- fix Windows MLX dl.dll install, MLX CMake version path, and the
show command after removing safetensors models
PsHandler iterated sched.loaded without holding loadedMu, racing with
scheduler goroutines that mutate the map. It also read runnerRef fields
(model, llama, expiresAt) that unload() and the expiration path mutate
under refMu, so a concurrently unloading runner could nil model out from
under the handler.
Instead of adding locking in routes.go, give the scheduler a small
snapshot API: loadedModels() copies the runner list under loadedMu, then
captures each runner's reporting fields under its refMu, respecting the
refMu-before-loadedMu lock ordering used by the expiration path. The
zero-expiresAt estimate for still-loading models moves into the
scheduler too, since it exists because of scheduler behavior.
Also remove the dead code Scheduler.GetRunner
In the PS output, expose the scheduler selected size (clamped by model context size) instead of always reporting the model max context. This will help provide a hint to clients to keep the context size below this value to avoid paging and poor performance on smaller VRAM systems.
* server: align generate with native chat templates
/api/generate rebuilt chat-like prompts through the Go template path even when the model selected its native GGUF Jinja chat template, so the same model rendered differently between generate and chat.
Route chat-like generate requests through the shared native chat preparation path, keep deprecated context and image handling working there, and keep explicit OLLAMA_GO_TEMPLATE overrides intact.
Fixes#16792
* review comments
Fall back to "{{ .Prompt }}" when lacking templates
This cleans up the capabilities logic so we can log more information about the various options we consider as well as the final template version we use.
* llama-server followups
Misc fixes for #16031
- Add back dropped ROCm build flag for multi-GPU support on windows
- Fix amdhip64_*.dll version detection for "latest" selection
- Fix embeddings API for consistent normalize behavior with prior versions
* ci: set up for automated llama.cpp update testing
* reduce batch for fa-disabled, and constrained vram
* mlx: fix v3 load bug on m5
Imagegen was incorrectly loading v3 first. This DRYs out the loading code so imagegen gets the same new v4/v3 selection logic.
* fix reload bug on embedding models
* bump version
* steer user how to enable iGPU when disabled
This change addresses some problems with GGUF conversion including:
* correctly naming the MoE tensors
* correctly quantizing the nextn.eh_proj.weight MTP tensor
* broad lint fixes to sidestep CI scope glitch
* runner: Remove CGO engines, use llama-server exclusively for GGML models
Remove the vendored GGML and llama.cpp backend, CGO runner, Go model
implementations, and sample. llama-server (built from upstream llama.cpp via
FetchContent) is now the sole inference engine for GGUF-based models.
(Safetensor based models continue to run on the new MLX engine.) This allows
us to more rapidly pick up new capabilities and fixes from llama.cpp as they
come out.
On windows this now requires recent AMD driver versions to support ROCm v7 as
llama.cpp currently does not support building against v6.
* llama/compat: load Ollama-format GGUFs in llama-server
Squashed from upstream/jmorganca/llama-compat on 2026-04-29.
Source tip: 0c33775d37.
Original source commits:
- 25223160d llama/compat: add in-memory shim so llama-server can load Ollama-format GGUFs
- 7449b539a llm,server: route Ollama-format gemma3 blobs through llama/compat
- 436f2e2b1 llama/compat: make patch-apply idempotent
- 8c2c9d4c8 llama/compat: extend gemma3 handler to cover 1B and 270M blobs
- 021389f7b llama/compat: shrink clip.cpp injection from 18 lines to 1
- 61b367ec2 llama/compat: shrink patch to pure call-site hooks (34 -> 20 lines)
- 36049361c llama/compat: simplify shim (gemma3-tested)
- 8fa664865 llama/compat: add qwen35moe text handler
- db0c74530 llama/compat: add qwen35moe vision (clip) support
- 2a388da77 llama/compat: split shared infra into a util TU
- 9a69a17dc llama/compat: document non-public API dependencies
- d0f38a915 llama/compat: add gpt-oss and lfm2 handlers
- 086071822 llama/compat: add mistral3 text handler (vision TODO)
- 63bde9ff7 llama/compat: add mistral3 vision (clip) support
- 3a57b89d5 llama/compat: apply LLaMA RoPE permute to mistral3 vision Q/K
- 99cb87439 llama/compat: add qwen35, gemma4, deepseek-ocr handlers
- 2c7850dba llama/compat: add nemotron_h_moe handler (latent FFN + MTP skip)
- 9e3b54225 llama/compat: add llama4 text + clip handlers
- 034fee349 llama/compat: add gemma4 clip handler (gemma4v projector)
- 9945c5a93 server: remove dhiltgen/* compat redirect table
- 5d4539101 llama/compat: rewrite gemma4 tokenizer model to BPE
- 7e0765327 llama/compat: add glm-ocr text handler + text-loader load-op hook
- f1bd1a25a llama/compat: add glm-ocr clip handler (glm4v projector)
- 4b5cf3420 llama/compat: collapse text-loader hook back to one new patch line
- eb4ecf4fc llama/compat: extend gemma4 clip handler to gemma4a (audio)
- a23a5e76f llama/compat: fix gemma4a per-block norm tensor mapping
- cd2dcaff4 llama/compat: add embeddinggemma handler
- 1ce8a6b26 llama/compat: add qwen3-vl + qwen2.5-vl handlers
- fd98ffa1e llama/compat: add gemma3n + glm4moelite handlers
- cc7bdf0bc llama/compat: handle null buft in maybe_load_tensor
- 0c33775d3 llama/compat: disable mmap when load_op transforms text-side tensors
* refine implementation
* ci: fix windows MLX build
* ci: fix windows llama-server build
* ci: fix windows rocm build
* ci: windows mlx tuning
Shorten long-tail on build, and get OllamaSetup.exe back under 2g limit
* ci: fix windows dependencies
* win: fix dependency gathering
* disable openmp
* win: arm64 cross-compile build
also DRY out CI steps
* scheduler improvements
* ci: improvements from #15982
* win: favor ninja for faster developer builds
* win: fix build
* win: fix arm64 cross-compile
* win: avoid spaces in compiler path
* misc discovery fixes, and bos handling
* lint fixes
* win: fix arm cross-compile build/CI bugs
* llama.cpp update
* win: handle multiple CRT dirs
* vulkan: add windows iGPU detection
* fix creation bugs for patched models, other refactoring work
* tune batch size for better performance
* ci and lint fixes
* fix repeat_last_n bug
* build: revamp build for better developer UX
* amd, sampler, qwen3next fixes
* version bump
* fix mlx build
* revamp GPU discovery
Scanning the output of llama-server is turning out to be too error prone across
llama.cpp updates, so this switches to a thin dynamic library load against the
bundled GGML libraries so more details can be gathered from the API.
* version bump
* missing file
* ci: fix cache miss on rocm build
* refine vulkan dep handling
* fix ps reporting bug on full GPU load
* improve cmake wiring for customized local builds
* version bump
* docker build arg cleanup
* improve windows exit error logs
* fix community gemma4 support and ci flakes
* fix mlx unit test
* tighten up ps logic to avoid double counting fit log lines
* version bump
* fix ps view for full gpu layer offload
* add MTP wiring for llama-server and create with GGUFs
* pick best template by capabilities
* version bump
* ci: harden apt repos
* remove unused cpu core discovery
* adjust batch default logic to reduce OOMs
* support larger tool calls
* fix audio support, template show
* qwen35 mtp patch support
* flesh out dtypes
* rocm deps
* version bump
* lint fix
* block broken gfx1150 on windows
* fix qwen3.5 moe mtp tensors in patch
* mmproj oom fallback and vulkan on by default
* qwen MTP compat fix
* version bump
* ci: fix WoA cross-compile
* ci: workaround ui tool in cross-compile
* version bump
* win: enable OpenMP for CPU builds
* build: improve developer UX
* ci: windows path workaround for CPU build
* win: fix WoA dependencies
* win: fix large offset reads for mmproj patched loads
* version bump
* fix vulkan dup detection
* add OLLAMA_IGPU_ENABLE and largely disable iGPUs by default
* opt-in MTP, win large offset, integraton fixes
* fix unit test scheduler interaction hang
* fix multi-gpu filtering
* version bump
* review comments
* fix thinking level
* fix linux rocm ordering and granite 3.3 template
* version bump
* ci fix - non-shallow MLX checkout
* bypass linux sysfs unit test on windows
---------
Co-authored-by: jmorganca <jmorganca@gmail.com>
This change updates the show API for MLX models to:
* display the correct quantization in mixed precision models
* not display the global_scale scalar value
* not duplicate the `tools` capability
* Reduce startup model hydration
Add a lightweight model list cache for tags and launch inventory, while keeping show cache population lazy. This avoids loading every local model at startup on large model stores.
* harden flaky scheduler unit test
* remove extra launch model metadata text
* review comments
* review comments
* app: harden update flows
This hardens the windows update flows and adds a new opt-in and CI triggered unit test to verify Mac/Windows updates with verification.
* test: harden unit tests for OLLAMA_MODELS being set
* app: harden updater
