Loading GGUF metadata is an expensive operation. Two caches had evolved to
mitigate this, and the two capability implementations produced inconsistent
results for some models.
This PR now extracts the metadata once per blob into a file at
<OLLAMA_MODELS>/metadata/sha256-<hex>.json. Only arrays over 4096 elements and
non-finite floats are left out.
Direct Capabilities() discovery now costs us instead of ms. /api/tags can
build directly from manifests and the extracted metadata.
progress: join the render loop in stop and do the final writes after the
goroutine exits, so Stop/StopAndClear cannot race an in-flight render on
the shared bufio.Writer.
sched: read the unload-mutable fields in runnerRef.LogValue only under a
successful refMu.TryLock and omit them when contended, since slog resolves
it on goroutines that may already hold refMu.
The runner compiled a format as a JSON Schema, the one grammar kind its
xgrammar binding exposed. A structural tag holds a schema as one node of
a larger tree and also expresses what a schema cannot: free text around
constrained spans, a thinking region that closes before constrained
content, tool calls pinned to their schemas.
The runner now compiles structural tags only; its client wraps the API's
formats into one, which compiles to the same grammar as before. The JSON
token and vocabulary caps go with it: neither bounds compile cost, which
follows the grammar's state count. The byte and nesting caps stay.
A structured-output request could not use a model's draft head: it
decoded one token at a time, at roughly half the speculative throughput
on a dense 27B MTP model.
The grammar is enforced during verification instead: each draft
position's logits are masked before rejection sampling, so an invalid
draft is never accepted and every emitted token obeys the grammar.
Drafts stay unconstrained; constraining the draft chain would stall its
pipelined forwards.
Speculative steps also now dispatch the drafts before the host builds
the verification graph, worth 4-8% end to end at a fixed draft depth on
MTP models, with or without a grammar.
* ci: rebuild MLX macOS test payloads the release can't supply
The MLX unit test dependency payload can go stale when a PR changes the
xgrammar wrapper (release dylib lacks the new symbols) or bumps MLX pins (no
matching release, so MLX tests silently skipped). Compare the matched tag's
build rules and wrapper sources against the checkout, rebuilding only
libollama_xgrammar.dylib on drift; with no matching release, build one Metal
variant at the platform default.
* address comments
Safetensors gemma4 imports served by the MLX engine now answer image
and audio chats. Images run through both vision architectures: the
transformer tower (26B, 31B, e-series) and the 12B's encoder-free
unified embedder. Audio arrives through the same intake the ollama
API already accepts for gemma4 GGUFs — WAV bytes in the images field,
OpenAI input_audio parts, and /v1/audio/transcriptions uploads — with
the e2b/e4b checkpoints running clips through their conformer audio
encoder and the 12b unified checkpoint embedding the raw waveform
directly. Clips longer than 30 seconds are split evenly into chunks
of at most 30 seconds, cut at pauses, and encoded independently.
Each modality serves only checkpoints that carry it: 26B/31B have no
audio config and reject audio input, and checkpoints with an
unrecognized vision architecture still load as text-only models and
reject image requests.
The server previously hid the vision and audio capabilities for
gemma4 safetensors because the engine served neither. Both
suppressions are removed, and existing imports start advertising the
capabilities without re-importing since import already records them.
Audio-capable models need mono PCM at their expected sample rate
before model-specific feature extraction. Like the image decoder set
in base, the supported audio containers are decided once here so
every model accepts the same formats: WAV, covering integer and float
PCM, extensible headers, and multi-channel downmix. Anything else is
rejected as unrecognized; supporting another container later means
one new decoder here, with no model or runner changes.
Input at other sample rates is resampled through a band-limiting
filter, so mismatched rates degrade gracefully instead of aliasing.
Decoded clips are capped at ten minutes: the declared rate comes from
an untrusted header, and the cap is what keeps a small file claiming
an absurdly low rate from resampling into an enormous allocation.
Models whose encoder takes clips only up to a fixed length split
longer clips with Split. Chunks are sized evenly, and each cut moves
to the quietest point within a few seconds of its even share, so a
boundary lands on a pause where the clip has one instead of severing
a word between two independently encoded chunks. The even sizing
bounds the search so no chunk can end up over the limit.
The bindings captured MLX error messages but checked almost no calls,
so a failure continued with a null output and surfaced later as zero
results, skipped evals, or an unrelated crash.
Wrap every call in mlxCheck. Paths that return an error or disable a
GPU kernel backend use mlxError instead, and the lookups where a
non-zero status means a miss read the buffer first and then treat the
status as data.
Also free the string handles behind Array.String and the log values
after the call that fills them; they were freed before it and leaked.
MLX runs on one goroutine locked to its OS thread, so the thread-local
error buffers and closure-based check helpers defended against a
calling pattern that is already invalid.
Replace them with a single buffer that the handler fills and Go reads
after every call. mlxError returns the captured message; mlxCheck
panics on it and passes the call's result through, so a checked call
is one expression. Only an int status carries a failure signal, which
lets a message next to a zero status be reported as an earlier
unchecked call.
Fix two tests that relied on errors being dropped: the laguna
mixed-precision fixture used an unsupported quantization group size,
and the compile callback test expected the callback's own panic.
* Report cached prompt tokens
Add prompt_eval_cached_count to native responses and expose equivalent cached-token fields through the OpenAI- and Anthropic-compatible APIs. Keep prompt_eval_count as the logical input total while excluding cache hits from CLI and benchmark prefill rates. Surface processed and cached prompt counts in benchmark output.
Collect cache counts from llama-server and MLX, preserve coherent metrics across two-pass structured generation.
Fixes#8008
Related to #15758
* review comments
* ci: wire up MLX unit tests for PR runs
Download the latest Darwin release payload matching the current MLX and MLX-C revisions so macOS PR tests can exercise MLX without rebuilding it. If no matching release exists after a pin bump, leave MLX tests skipped until the next release.
Add whole-tree race coverage and smoke-run committed benchmarks. Verify generated UI types, and stabilize tests exposed by the broader CI coverage.
* review comments
* mlx: run tests on one pinned worker
Keep MLX tests and benchmarks on a shared pinned thread while preserving Fatal, Skip, and Cleanup semantics. Also clean stale CI payloads and ensure updater workers shut down cleanly.
* addres comments
* llama.cpp: version bump b10729
Regenerate the compat hooks patch for b10729: upstream removed the
whole-tensor load_data_for read (last consumer was llama-quantize,
which now reads slabs via load_data_range). Keep the existing hook
surface (constructor, skip loops, load_all_data, mtmd/clip) unchanged
and add maybe_load_text_tensor_range, which materializes a text load
op's output once per tensor and serves the new (offset, size) slab
reads from that cache.
* address comments
* Honor model generation defaults
Model-authored sampler defaults from GGUF metadata and HF generation_config.json were ignored, so built-in Ollama defaults could override model intent unless parameters were set in the Modelfile or request. The fix parses those defaults into model config and applies them before Modelfile/request options, preserving the expected precedence order.
* review comments
* address comments
* fix(docs): correct typos found during code review
Non-functional changes only:
- Fixed minor spelling mistakes in comments
- Corrected typos in user-facing strings
- No variables, logic, or functional code was modified.
Signed-off-by: Marcel Petrick <mail@marcelpetrick.it>
* fix additional typos and shell-unsafe example in docs
---------
Co-authored-by: Patrick Devine <patrick@ollama.com>
The MLX gemma3 port implements only the text stack, while gemma3 as GGUF
runs on llama-server with vision. Once MLX takes priority for
architectures both engines support, a registered gemma3 would route the
model to the engine that cannot serve images. No gemma3 safetensors
manifests were ever published, so removing the architecture affects no
existing installs and keeps gemma3 on llama-server.
Pi's Edit() only set baseUrl when creating a new ollama provider entry.
On subsequent launches it preserved whatever baseUrl was already in
~/.pi/agent/models.json, so switching OLLAMA_HOST to a remote server had
no effect — Pi would still connect to localhost.
Edit() now ensures baseUrl reflects the current OLLAMA_HOST. Models()
returns nil when the stored baseUrl no longer matches, so the launcher
only calls Edit() when the host has actually drifted. User-customized api
and apiKey fields are still preserved.
