Claude Code adds a "tokens left" system message after every tool
result. Since ollama moves system messages to the front of the prompt,
this breaks the KV cache on every request.
A long prompt records restore points during prefill, but they only
reached the prefix trie when the prefill completed; a cancelled request
closed and released everything it had captured. Agent clients routinely
cancel long prefills — their timeouts are shorter than the minutes a
40k-token prompt takes — so every retry started the whole prompt over
and never got further than the timeout allowed, which presents as the
model hanging forever.
Closing a session now attaches every snapshot the prefill crossed, so a
retry resumes from the last one and makes progress across timeouts.
Scenario tests cover retries resuming exactly where a cancelled attempt
stopped and cancellations on divergent conversation variants.
Fixes#17839
A prefill that resumes partway into cached history — routine once
client timeouts interrupt long prompts — used to attach its captures
onto a node extended in place, so the stored snapshot spanned only the
tokens the prefill evaluated while the node's edge reached further
back. Restores walk node by node and trust each snapshot to cover its
node's edge; the short snapshot stranded the caches at mismatched
offsets and, on models with recurrent layers, ended up freeing all
cache state — a request matching 46k of a 47k-token prompt reprocessed
from zero.
Growth now never extends a node underneath its snapshots. New tokens
become a child node that carries exactly its own captures, and the
path stays compressed because non-user segments merge back into their
parent through the caches' snapshot Merge. Close already pages out
what it records, so every merge combines adjacent covered snapshots
and every stored snapshot spans exactly its node's edge.
When a session closes, every cache rests exactly at the end of the
segment the trie is about to record. That is the one moment the
segment's state can be captured for every layer, so close now pages
the new segment out itself instead of recording it without snapshots
and leaving the capture to a later path switch.
Path switching then has nothing left to capture and only rewinds and
pages in. The whole-state entry taken at close is released when the
next request grows past the segment; sliding-window layers pay the
same window copy a scheduled capture already costs.
Page-in restores a path node by node and trusts each stored snapshot to
cover its node's whole edge. A capture taken during prefill spans from
the previous capture or the prefill base, which need not line up with
the node it lands on: when a prefill resumes partway into cached
history, a capture can reach back before its node's start, and a
capture landing on a node that already has snapshots replaced them
with a shorter span that page-in then could not serve.
Clip each capture to its node's edge on attach, and keep the snapshots
the node already has instead of replacing them.
A prefill settles the drafter with the seed token after its last chunk,
leveling the draft caches with the targets; a cancelled prefill
returned before that, leaving the targets one token past the draft
caches and the recorded keys. The next request then had to move every
cache, and models with recurrent layers, which cannot rewind, fell back
to the last snapshot: a retry after a client timeout lost up to a full
snapshot interval of the prompt it had just evaluated.
Settle with the next prompt token on the cancelled path too. The caches
then rest level with the recorded keys, and a retry resumes exactly
where the prefill stopped.
When a builtin parser rejects model output, the completion callback wrote the
error to an unbuffered channel and returned. The callback cannot stop
generation -- it has no error return -- so the next chunk re-entered the
callback, hit the same parse error and blocked writing to a channel the
consumer had already stopped reading after emitting its 500. The completion
never returned, the goroutine leaked and the runner request was never
released, so retrying the same prompt hung with no log output until the client
gave up.
Record the parse error, cancel the completion, and report it once the
completion has returned. Parse failures landing on the final chunk were
already terminal, which is why non-thinking requests and the direct
qwen3-coder parser path failed cleanly and only thinking mode wedged.
ChatHandler and GenerateHandler share the defect: both run the same parser in
the same shape of callback behind a consumer that stops reading at the first
error. GenerateHandler had no cancel func at all, so one is added there.
Fixes#17825
Coding clients may insert runtime system messages after the initial user turn. The shared Qwen renderer rejected these transcripts before rendering, turning a potentially usable non-standard request into an HTTP 500.
Pass non-leading system turns through the existing raw ChatML path and warn when qwen3.8 encounters one. Extend the Anthropic tool-route integration scenario to cover this message pattern and remove the obsolete rejection test.
llama-server does not currently support WebP image payloads. Detect WebP media before forwarding, and transcode it to PNG. Pass all other media through unchanged.
Replace an existing vision integration image with a lossless WebP version so we now have coverage of JPG/PNG/WebP formats.
Fixes#17753
* qwen3.8: support developer instructions
Qwen3.8 does not define a developer role, while OpenAI-compatible coding agents commonly send developer instructions before user messages. Fold the leading system/developer instruction prefix into a single system turn before Qwen3.8 validation, preserving instruction precedence without changing Qwen3.5 or other renderer behavior.
Add streaming tool-call integration coverage for the native Ollama, OpenAI Chat Completions, OpenAI Responses, and Anthropic Messages request shapes. Each case exercises prior assistant tool calls, tool results, follow-up rendering, and parsed tool-call output. Add Qwen3.8 to the release tools sweep.
Removes an unnecessary unit test that should not have been included in the original 3.8 PR.
* review comments
Qwen3.8 keeps the Qwen3.5 model architecture and parser, but its chat template adds reasoning-effort and preserved-thinking semantics. Detect those template markers during safetensors import, select the qwen3.8 renderer, and cover thinking, tools, continuation, and malformed parser input.
Make indexed safetensors imports use the weight map's shard names instead of independently filtering files by the model-* convention. Reject unsafe shard paths, ignore unindexed tensors, and fail when an indexed weight is missing or stored in a different shard. Retain the conservative model-* scan when no index is present.
Treat Classification.Quantize as the effective tensor format and pass it to the manifest writer. This records file_type for automatic block-FP8-to-MXFP8 conversion and recognized prequantized inputs, preserves requested quantization and base-plus-draft behavior, and avoids claiming one type for mixed or unknown formats.
Normalize both supported convolution weight layouts with an explicit reshape. Add focused unit coverage for renderer selection, parser behavior, shard inventory, manifest metadata, and convolution layout; heavyweight reference-forward and release integration checks remain bring-up artifacts.
Updates Muse Glimmer Jinja reference template to the latest publisher version and mirror its explicit-system reasoning handling in the Go renderer.
Explicit system prompts now normalize "Reasoning effort" to "Reasoning strength" and skip adding a renderer-provided reasoning line when the prompt already contains one. This prevents duplicate or conflicting reasoning directives while preserving the default-system behavior.
Add reference tests for both normalization and deduplication, including Jinja-backed validation.
* launch: add Muse Code integration
Add `ollama launch muse` for Meta's Muse Code CLI.
Muse only takes a model catalog from settings.json (normally it fetches one from its provider and refuses to start otherwise), and that file's endpoint_transport is a global provider switch. So the integration writes a settings file under its own config root (~/.ollama/launch/muse-config via XDG_CONFIG_HOME), leaving a Meta-backed muse install untouched, and re-seeds it from muse's own persisted copy on later runs.
The launched model is preloaded so its catalog row carries the context length the server actually allocated, not the trained maximum; the loaded-context helpers move from cmd/agent_tui.go into cmd/launch for reuse.
Muse sends reasoning efforts outside Ollama's scale (minimal, xhigh, ultra), which were hard 400s; clamp them to the nearest tier in one helper shared by the chat and responses converters.
The registry entry stays Hidden (alias "muse-code"), like kimi and vscode.
* review comments
* skip muse test on windows (unsupported platform)
As we look to bring Linux and Windows MLX support online, instead of blocking
downloads at the registry to avoid users wasting time downloading a model they
can't run, shift the logic to the local side which knows if MLX is present or not.
ModelOpt checkpoints apply a float32 global scale to every projection
output on top of the per-group quantization scales. Running the
multiply and the cast back to the activation dtype as separate eager
ops costs an extra kernel launch and a materialized intermediate per
projection.
Compile the multiply and cast into one kernel. On an M5 Max (medians
of order-swapped A/B runs against main; greedy outputs byte-identical):
qwen3.6:27b prefill 703 -> 769 t/s +7.9%
muse-glimmer:30b prefill 790 -> 843 t/s +6.7%
Speculative decode is unchanged within noise on both models. Only
checkpoints with a global scale are affected; single-scale nvfp4,
mxfp8, and affine checkpoints take the unchanged path.
When a manifest contains a config and layer with the same digest, the
skipVerify map entry was overwritten by the config's cache-hit value
(true), replacing the layer's non-cache-hit value (false). This caused
verifyBlob to be skipped for the freshly downloaded blob.
A rogue OCI registry could exploit this by serving a manifest with
duplicate digests and redirecting blob downloads to internal endpoints.
The SSRF response would be written to disk, hash verification would be
skipped due to the map collision, and the blob would persist.
The fix uses logical AND when updating skipVerify: once any download of
a digest was not a cache hit, verification is always performed.
Fixes#15485
---------
Co-authored-by: Patrick Devine <patrick@ollama.com>
Request options are the model's published parameters and the request's
own options layered over the server defaults, so the default
repeat_penalty of 1.1 reaches every model whose parameters leave it
unset. No maker of the library's current models recommends 1.1: their
generation configs either omit the penalty, meaning 1.0, or pin 1.05.
llama.cpp dropped the same 1.1 default in 2024; vLLM, SGLang, and
transformers apply no penalty. An always-on penalty also distorts
output that legitimately repeats tokens, such as code, JSON, and long
reasoning traces.
The penalty is especially costly for speculative decoding, where
drafts are proposed without it: the penalized target rejects drafted
tokens and the depth controller backs off. On muse-glimmer 30B (DFlash
on M5 Max, HumanEval) the 1.1 default costs 13-16% of end-to-end
throughput at greedy and temperature 1 alike, and drops prose
acceptance at temperature 0.8 from 0.44 to 0.30. On qwen3.6-35B it
cuts the mean accepted draft length from 4.3 to 3.5 tokens and makes
the controller stop speculating on prose.
Defaulting to 1.0 disables the penalty unless a model's parameters or
the request set one. Across the library:
- qwen3, qwen3.6, and qwen3-coder pin their own values (1.0, 1.0, and
Qwen's recommended 1.05) and are unchanged.
- Everything else local now matches its maker's no-penalty
recommendation, including gemma2 through gemma4, muse-glimmer, both
laguna 2.1 models, qwen3.5 (previously 1.1 stacked on its
presence_penalty of 1.5), gpt-oss, deepseek-r1 and v3.1, the
nemotron family, granite4, the mistral and llama3/llama4 families,
phi4, glm4, llava, and devstral.
- qwen2.5 recommends 1.05 but ships no parameters, so it moves from
1.1 to 1.0 and still needs a parameters layer to conform.
- Cloud models (kimi-k3, deepseek-v4-flash) never receive these
defaults.
Small older models may repeat themselves more without the penalty
masking it; the remedy is a per-model parameter, not a penalty applied
to every model.
Select the 3.5 parser and renderer from its checkpoint template, preserve its prompt semantics, and map medium reasoning effort to the final-user annotation expected by the reference template.
Exercise parser and renderer registration, create-time metadata inference, and exact Jinja parity so created models cannot silently fall back to the Nemotron 3 renderer.
The model occasionally emits a <|message|> boundary token in the invoke
name region, echoing the header form `to=read<|message|>`. The existing
recovery handled the tag inside a terminated name (`name="read<|message|>">`)
but not the fleet-observed shape where the tag replaces the `">` terminator
itself (`name="read<|message|><atem:parameter ...`), which failed the call
with "malformed ATEM parameter".
Replace the strip-after-cut recovery with a single name scan shared by
parseGlimmerATEM and the content fallback: the name ends at the first `">`,
boundary tokens before it are dropped, and a parameter element immediately
after a dropped token means the token replaced the terminator. Well-formed
calls are unaffected — a boundary token is never legitimate before the
terminator, and parameter values (where the literal text is preserved) only
appear after it. Murkier garbles still fail loudly, the recipient
cross-check still applies, and the recovery WARN is retained.
Add MLX support for Nemotron 3 Nano Omni, including the model implementation, Mamba2/recurrent pieces, MoE routing, and quantized NVFP4/MXFP8 expert paths.
Use a shared mapped MoE GatherQMM fast path under the generic moe_gather_qmm_mapped naming, with Metal-optimized NVFP4/MXFP8 block-mapped kernels and generic fallbacks for unsupported backends.
Serve the model's multi-token prediction head as a self-draft speculator, so speculative decoding needs no separate draft model.
Render the Nemotron prompt from the published chat template. The template the renderer was based on had drifted from the current reference; refreshing it surfaced five mismatches: stray leading newlines, the wrong turn separator and a trailing newline before the generation prompt; /think and /no_think toggles left in user turns; a trimmed system message the template leaves intact; a user block opened by a leading tool message; and Go scalar syntax for schema extras where the template applies Python str(), sending true/false/<nil> in place of True/False/None. Reference tests now render every case through the template itself.
Also harden the Nemotron parser path shared by both backends: while collecting thinking, preserve whitespace before partial </think>, <think>, and <tool_call> fakeouts, with streaming tests covering those cases.
* glimmer: implement the Muse Glimmer model
MLX model (language + vision encoder) with DFlash draft wiring, llama-server DFlash support and rope-interleave fix, renderer and parser, tokenizer fixes, and the import quantization policy.
* mlxrunner: report committed prefill chunks after the sweep and eval
The drafter's flush evaluates its report, and an eval that runs while the chunk's construction handles are still live cannot free any intermediate buffer. On media chunks that retention keeps the whole vision tower resident and grinds the Metal allocator at its limit until the request dies. Pin the report's inputs across the sweep, report after the chunk materializes, and release media items after the report so a drafter can still capture the rows its deferred flush embeds.
* ci: retry CUDA pre-release download
Also bring back version tagging the MLX library with our git hash which was
accidentally dropped when imagegen was removed. Without this, the version
claimed to be the official tagged version, but we're typically using a git hash
with different content.
