Coding clients may insert runtime system messages after the initial user turn. The shared Qwen renderer rejected these transcripts before rendering, turning a potentially usable non-standard request into an HTTP 500.
Pass non-leading system turns through the existing raw ChatML path and warn when qwen3.8 encounters one. Extend the Anthropic tool-route integration scenario to cover this message pattern and remove the obsolete rejection test.
llama-server does not currently support WebP image payloads. Detect WebP media before forwarding, and transcode it to PNG. Pass all other media through unchanged.
Replace an existing vision integration image with a lossless WebP version so we now have coverage of JPG/PNG/WebP formats.
Fixes#17753
* qwen3.8: support developer instructions
Qwen3.8 does not define a developer role, while OpenAI-compatible coding agents commonly send developer instructions before user messages. Fold the leading system/developer instruction prefix into a single system turn before Qwen3.8 validation, preserving instruction precedence without changing Qwen3.5 or other renderer behavior.
Add streaming tool-call integration coverage for the native Ollama, OpenAI Chat Completions, OpenAI Responses, and Anthropic Messages request shapes. Each case exercises prior assistant tool calls, tool results, follow-up rendering, and parsed tool-call output. Add Qwen3.8 to the release tools sweep.
Removes an unnecessary unit test that should not have been included in the original 3.8 PR.
* review comments
Qwen3.8 keeps the Qwen3.5 model architecture and parser, but its chat template adds reasoning-effort and preserved-thinking semantics. Detect those template markers during safetensors import, select the qwen3.8 renderer, and cover thinking, tools, continuation, and malformed parser input.
Make indexed safetensors imports use the weight map's shard names instead of independently filtering files by the model-* convention. Reject unsafe shard paths, ignore unindexed tensors, and fail when an indexed weight is missing or stored in a different shard. Retain the conservative model-* scan when no index is present.
Treat Classification.Quantize as the effective tensor format and pass it to the manifest writer. This records file_type for automatic block-FP8-to-MXFP8 conversion and recognized prequantized inputs, preserves requested quantization and base-plus-draft behavior, and avoids claiming one type for mixed or unknown formats.
Normalize both supported convolution weight layouts with an explicit reshape. Add focused unit coverage for renderer selection, parser behavior, shard inventory, manifest metadata, and convolution layout; heavyweight reference-forward and release integration checks remain bring-up artifacts.
Updates Muse Glimmer Jinja reference template to the latest publisher version and mirror its explicit-system reasoning handling in the Go renderer.
Explicit system prompts now normalize "Reasoning effort" to "Reasoning strength" and skip adding a renderer-provided reasoning line when the prompt already contains one. This prevents duplicate or conflicting reasoning directives while preserving the default-system behavior.
Add reference tests for both normalization and deduplication, including Jinja-backed validation.
* launch: add Muse Code integration
Add `ollama launch muse` for Meta's Muse Code CLI.
Muse only takes a model catalog from settings.json (normally it fetches one from its provider and refuses to start otherwise), and that file's endpoint_transport is a global provider switch. So the integration writes a settings file under its own config root (~/.ollama/launch/muse-config via XDG_CONFIG_HOME), leaving a Meta-backed muse install untouched, and re-seeds it from muse's own persisted copy on later runs.
The launched model is preloaded so its catalog row carries the context length the server actually allocated, not the trained maximum; the loaded-context helpers move from cmd/agent_tui.go into cmd/launch for reuse.
Muse sends reasoning efforts outside Ollama's scale (minimal, xhigh, ultra), which were hard 400s; clamp them to the nearest tier in one helper shared by the chat and responses converters.
The registry entry stays Hidden (alias "muse-code"), like kimi and vscode.
* review comments
* skip muse test on windows (unsupported platform)
As we look to bring Linux and Windows MLX support online, instead of blocking
downloads at the registry to avoid users wasting time downloading a model they
can't run, shift the logic to the local side which knows if MLX is present or not.
ModelOpt checkpoints apply a float32 global scale to every projection
output on top of the per-group quantization scales. Running the
multiply and the cast back to the activation dtype as separate eager
ops costs an extra kernel launch and a materialized intermediate per
projection.
Compile the multiply and cast into one kernel. On an M5 Max (medians
of order-swapped A/B runs against main; greedy outputs byte-identical):
qwen3.6:27b prefill 703 -> 769 t/s +7.9%
muse-glimmer:30b prefill 790 -> 843 t/s +6.7%
Speculative decode is unchanged within noise on both models. Only
checkpoints with a global scale are affected; single-scale nvfp4,
mxfp8, and affine checkpoints take the unchanged path.
When a manifest contains a config and layer with the same digest, the
skipVerify map entry was overwritten by the config's cache-hit value
(true), replacing the layer's non-cache-hit value (false). This caused
verifyBlob to be skipped for the freshly downloaded blob.
A rogue OCI registry could exploit this by serving a manifest with
duplicate digests and redirecting blob downloads to internal endpoints.
The SSRF response would be written to disk, hash verification would be
skipped due to the map collision, and the blob would persist.
The fix uses logical AND when updating skipVerify: once any download of
a digest was not a cache hit, verification is always performed.
Fixes#15485
---------
Co-authored-by: Patrick Devine <patrick@ollama.com>
Request options are the model's published parameters and the request's
own options layered over the server defaults, so the default
repeat_penalty of 1.1 reaches every model whose parameters leave it
unset. No maker of the library's current models recommends 1.1: their
generation configs either omit the penalty, meaning 1.0, or pin 1.05.
llama.cpp dropped the same 1.1 default in 2024; vLLM, SGLang, and
transformers apply no penalty. An always-on penalty also distorts
output that legitimately repeats tokens, such as code, JSON, and long
reasoning traces.
The penalty is especially costly for speculative decoding, where
drafts are proposed without it: the penalized target rejects drafted
tokens and the depth controller backs off. On muse-glimmer 30B (DFlash
on M5 Max, HumanEval) the 1.1 default costs 13-16% of end-to-end
throughput at greedy and temperature 1 alike, and drops prose
acceptance at temperature 0.8 from 0.44 to 0.30. On qwen3.6-35B it
cuts the mean accepted draft length from 4.3 to 3.5 tokens and makes
the controller stop speculating on prose.
Defaulting to 1.0 disables the penalty unless a model's parameters or
the request set one. Across the library:
- qwen3, qwen3.6, and qwen3-coder pin their own values (1.0, 1.0, and
Qwen's recommended 1.05) and are unchanged.
- Everything else local now matches its maker's no-penalty
recommendation, including gemma2 through gemma4, muse-glimmer, both
laguna 2.1 models, qwen3.5 (previously 1.1 stacked on its
presence_penalty of 1.5), gpt-oss, deepseek-r1 and v3.1, the
nemotron family, granite4, the mistral and llama3/llama4 families,
phi4, glm4, llava, and devstral.
- qwen2.5 recommends 1.05 but ships no parameters, so it moves from
1.1 to 1.0 and still needs a parameters layer to conform.
- Cloud models (kimi-k3, deepseek-v4-flash) never receive these
defaults.
Small older models may repeat themselves more without the penalty
masking it; the remedy is a per-model parameter, not a penalty applied
to every model.
Select the 3.5 parser and renderer from its checkpoint template, preserve its prompt semantics, and map medium reasoning effort to the final-user annotation expected by the reference template.
Exercise parser and renderer registration, create-time metadata inference, and exact Jinja parity so created models cannot silently fall back to the Nemotron 3 renderer.
The model occasionally emits a <|message|> boundary token in the invoke
name region, echoing the header form `to=read<|message|>`. The existing
recovery handled the tag inside a terminated name (`name="read<|message|>">`)
but not the fleet-observed shape where the tag replaces the `">` terminator
itself (`name="read<|message|><atem:parameter ...`), which failed the call
with "malformed ATEM parameter".
Replace the strip-after-cut recovery with a single name scan shared by
parseGlimmerATEM and the content fallback: the name ends at the first `">`,
boundary tokens before it are dropped, and a parameter element immediately
after a dropped token means the token replaced the terminator. Well-formed
calls are unaffected — a boundary token is never legitimate before the
terminator, and parameter values (where the literal text is preserved) only
appear after it. Murkier garbles still fail loudly, the recipient
cross-check still applies, and the recovery WARN is retained.
Add MLX support for Nemotron 3 Nano Omni, including the model implementation, Mamba2/recurrent pieces, MoE routing, and quantized NVFP4/MXFP8 expert paths.
Use a shared mapped MoE GatherQMM fast path under the generic moe_gather_qmm_mapped naming, with Metal-optimized NVFP4/MXFP8 block-mapped kernels and generic fallbacks for unsupported backends.
Serve the model's multi-token prediction head as a self-draft speculator, so speculative decoding needs no separate draft model.
Render the Nemotron prompt from the published chat template. The template the renderer was based on had drifted from the current reference; refreshing it surfaced five mismatches: stray leading newlines, the wrong turn separator and a trailing newline before the generation prompt; /think and /no_think toggles left in user turns; a trimmed system message the template leaves intact; a user block opened by a leading tool message; and Go scalar syntax for schema extras where the template applies Python str(), sending true/false/<nil> in place of True/False/None. Reference tests now render every case through the template itself.
Also harden the Nemotron parser path shared by both backends: while collecting thinking, preserve whitespace before partial </think>, <think>, and <tool_call> fakeouts, with streaming tests covering those cases.
* glimmer: implement the Muse Glimmer model
MLX model (language + vision encoder) with DFlash draft wiring, llama-server DFlash support and rope-interleave fix, renderer and parser, tokenizer fixes, and the import quantization policy.
* mlxrunner: report committed prefill chunks after the sweep and eval
The drafter's flush evaluates its report, and an eval that runs while the chunk's construction handles are still live cannot free any intermediate buffer. On media chunks that retention keeps the whole vision tower resident and grinds the Metal allocator at its limit until the request dies. Pin the report's inputs across the sweep, report after the chunk materializes, and release media items after the report so a drafter can still capture the rows its deferred flush embeds.
* ci: retry CUDA pre-release download
Also bring back version tagging the MLX library with our git hash which was
accidentally dropped when imagegen was removed. Without this, the version
claimed to be the official tagged version, but we're typically using a git hash
with different content.
One vision path serves every qwen3.5/qwen3.6 registration, dense and
MoE. Rope positions are precomputed at prepare time as the request's
layout — the family uses interleaved M-RoPE — while text-only requests
keep the fused 1D rope path, which is numerically identical for
uniform channels. Image expansions are causal for this family, so
prefill chunks split them. The MTP head embeds prompt tokens, so it
scatters the delivered image features and applies the same position
tables, keeping speculative decoding working on image prompts. The
merger's exact erf GELU adds an Erf op to the MLX bindings.
A checkpoint whose config declares vision must ship its tower: missing
vision weights or a deepstack_visual_indexes request fail the load
rather than silently serving text-only or skipping the injections.
Text-only checkpoints, which carry no vision_config, load as before.
Verified tensor-by-tensor against HF transformers for all eight family
members and live on every published -mlx tag; published towers are
already bf16, so no re-import is needed.
Each media item's features are encoded lazily when a prefill chunk
first overlaps its expansion and stay pinned until the expansion is
fully evaluated. A chunk never ends strictly inside an atomic
expansion: a bidirectional run's early rows attend its later keys, so
its first evaluation must cover the whole run in one forward. Items
marked Causal are exempt and split at any boundary.
Draft models need the same request state — reference MTP drafters
embed prompt tokens with the image features merged in, and an M-RoPE
drafter cannot compute positions without the request's layout — so the
layout is stamped on every forward, target and draft alike, and the
MTP session holds feature rows across its deferred flush. The dflash
drafter ignores media: its context rows are target hiddens.
A prompt that references media arrives as text containing [img-N] tags
plus the media bytes. Prepare now splits on the tags, tokenizes the text
between them, and hands the model the resulting segments — text runs and
media in stream order — in a single PrepareMedia call. The model returns
the expanded stream with each media segment's placeholder expansion
spliced in place, described per item so the runner can key identity and
schedule encoding, along with any opaque request-scoped layout state it
derives while building the stream. Building the whole stream in one call
is what lets a model derive values that span items, and lets it choose
item granularity (one per image, or one per independently evaluable
tile).
The runner validates the model-authored items before trusting them —
ranges ordered, non-overlapping, in bounds, and covering every media
segment, since prefix-cache identity is keyed on them.
Unknown tag IDs fail the request, media the prompt never references is
ignored with a warning, and duplicate references are allowed, matching
the previous engine. A media request still produces no image output:
nothing feeds the features to the model yet, and no model implements
the media interface.
Media placeholders repeat one token ID, so two prompts with different
images would produce identical trie keys and falsely share cached state.
Substitute a per-item hash of the media bytes and preprocessing shape
across each item's expansion range at the key layer; the model still
sees real token IDs. Fold values carry a bit no token ID has, so a media
stream can never alias text, and the bigram packing for draft caches
composes unchanged, so draft restore points inherit the same identity.
Text-only prompts key exactly as before. Nothing records media items yet;
the change is inert until the prompt preparation wires them.
MLX checkpoints that include a vision tower are already tagged with the
vision capability at import, so the server accepts image chats and ships
the image bytes with the completion request. The MLX client dropped the
bytes, and the prompt's image tags were answered as literal text.
Carry the media through to the runner and fail the request with a clear
error when the loaded model has no media support. Nothing implements the
new media interface yet, so every media request now returns the error
rather than a silently wrong answer; later changes build the image path
on top of the same interface.
Vision towers are much more sensitive to weight quantization than
language layers: measured against the reference encoder on a real
image, 4-bit types and scale-only mxfp8 distort the projected image
features by 26-34% mean relative error (worst tokens near-orthogonal),
which shows up as degraded image recognition — down to complete
blindness for the small e-series towers under nvfp4. Affine 8-bit was
the only quantized format that matched the bf16 tower.
Keep vision tower tensors at source precision instead, matching the
audio tower's treatment and every vision component Ollama publishes in
GGUF form, including gemma4's own GGUF tags, which ship f16/f32 vision
beside 4-bit language weights. Towers are small and run once per image,
so neither size nor decode bandwidth argues for quantizing. Existing
MLX imports keep their quantized towers until re-imported.
rootfs.diff_ids duplicated the manifest's layer digest list into the config blob and nothing ever read it. On per-tensor safetensors models the copy grows past 100KB and create excessively large config blobs with unused redundant data. Model identity is unaffected: it is the digest of the manifest itself, which already commits to every layer hash.
Implements the DFlash draft checkpoint format: a few decoder layers
over fused target-layer outputs, which enter every layer as key/value
context while the block being drafted supplies the queries. The draft
has no embedding table or output head of its own; it borrows the
target's.
One model covers the known checkpoints. Attention weights normalize at
load to a q projection plus a fused k|v, stacking split checkpoints and
slicing fused ones, exact for quantized tensors; gate and up fuse the
same way. Optional tensors decide the output gate and per-tap norms,
config decides attention shape, and the architecture name decides only
laguna's context-norm convention.
A manifest can pair any draft with any target, so construction checks
the fit: tap ids inside the target's layers, matching hidden width, and
the target vocabulary covering the mask token. A bad pairing fails at
load.
A DFlash draft proposes a whole block per forward, which doesn't fit
the MTP session's one-token-per-call chain. Add a second drafting
session for block drafts: committed target features write straight
into the draft's context caches, and each round drafts a block in one
forward and samples it in one batched call, rolling the block's cache
entries back with the same mechanism speculative rounds use on the
target caches. The depth controller's search is capped at the deepest
draft the drafter can produce, since a depth it can never measure
would otherwise always look best.
Distribution aligns its rows with the end of the draft chain, so when
the caller passes no chain, every row sees the slot history unchanged.
That case already worked; only the row-count guard rejected it. The
guard now applies only when a chain is present, which is where more
rows than chain positions would silently drop history. A block drafter
needs the chainless case to sample its whole proposal batch in one
call.
The runner used to build caches by probing the model for an optional
NewCaches method, with one KV cache per layer as the fallback. A model
with a draft head appended the draft's cache slots to its own list, and
the speculative engine later recovered the two groups by comparing slot
identities, panicking when the lists didn't line up.
NewCaches is now a required method on both the model and the draft, and
each returns only the slots it writes. The runner concatenates the two
lists for the prefix cache and passes them to the speculative engine
separately, so snapshots and rollback apply to the target's slots and
the draft forward receives both groups as arguments. The identity
comparison, its panics, and the per-request rebinding are gone; the two
groups are fixed at load time.
A draft model conditions on state that the target produces during its
own forward pass. For an MTP head or an assistant model that state is
the final hidden state; for a block draft it is the concatenated
outputs of several layers. The choice belongs to the model, so Forward
now returns the conditioning state along with the hidden state to
unembed. Models without a special conditioning state return the final
hidden state for both, and the decode paths hand the value to the
drafter without looking at it.
The Responses API groups related tools by domain: a tool with type "namespace" carries the real function definitions in a nested tools array. The conversion dropped that array, leaving the model a single schema-less pseudo-function and making every namespaced call undeclarable.
Expand namespace declarations into their member functions with namespace-qualified names, since api.Tool carries only a flat function name.
Relates to #15921: full Responses API parity also wants the namespace preserved as a separate field on tool calls in the output, which needs new api surface and is not addressed here.
A lazy KV snapshot indexes into the cache's live buffer instead of owning
a copy, so it must be copied out before an append overwrites the slots it
names. appendKV checked for that only on the first append after a rewind,
and only against that append's own range: a still-lazy snapshot further
ahead in the buffer was overwritten without a copy when a later append
reached it. This happens when a request reuses a short prefix of a longer
cached conversation and prefills past one of the old conversation's
snapshots; restoring that snapshot later silently serves the new request's
KV in place of the old conversation's.
Scan every append instead. The overlap test already limits copies to
snapshots the current write clobbers, and appends outside a rewind refill
sit above every snapshot, so the steady-state scan walks a short list and
finds nothing. This restores the invariant Restore's lazy fast path relies
on: a snapshot still in its lazy state has never been overwritten.
* progress: fix data races on ticker, states, spinner, and bar state
NewProgress spawned start() which wrote p.ticker while stop() read and
cleared it with no synchronization; stop() and StopAndClear() also read
p.states and p.pos outside p.mu, Spinner's start() goroutine raced
Stop() and String() on s.value/s.stopped/s.ticker, and Bar.Set raced
Bar.String on currentValue/stopped/buckets (callback goroutine vs the
render goroutine). Detected by go test -race across cmd and cmd/launch
(~20 warnings; the Bar race is latent — never flagged because tests
don't interleave it, but real in production pull/push progress).
Create tickers before spawning the render goroutines and pass the
channel in, guard Progress internals with p.mu throughout stop() (via a
renderLocked core), and give Spinner and Bar their own mutexes.
* use a more idiomatic channel based done signal
Reworked our /v1/chat/completions streaming to match what api.openai.com actually sends,
chunk-for-chunk, based on captures I took of real OpenAI traffic.
What changed:
- finish_reason now goes on its own chunk with an empty delta {}, instead of riding on the last content
chunk. Precedence is length > tool_calls > the response's done reason > stop.
- role is only sent on the first chunk of a stream, not on every chunk.
- With stream_options.include_usage, usage goes out on its own chunk with choices: [] after the finish
chunk.
- A truncated response keeps finish_reason: "length" even when tool calls were streamed — it used to get
overwritten with "tool_calls". Fixed in both streaming and non-streaming paths.
- The metrics-only trailer response (empty message at end of stream) no longer produces a stray
delta:{"content":""} chunk before the finish chunk. A wholly empty completion still opens with a role
chunk.
- Every chunk in a stream shares one timestamp, from the response's CreatedAt.
DownloadNewRelease spawned a background checkForUpdate loop that read
package-level knobs (UpdateCheckInterval et al.) and returned without
waiting for it, so under -race the next test rewrote those globals while
the orphaned goroutine was still reading them. waitDownloadIdle (from
Cancel and WaitGroup-drain the loop before DownloadNewRelease returns,
and have TestCancelOngoingDownload join its download goroutine so the
drain is observable before the test exits.
Load the MTP head from the mtp.* tensors instead of freeing them and implement
Draft to propose one token per step, gated solely on the tensors being
present; a model whose head ships inline is its own draft via base.SelfDraft.
The runtime keeps sole ownership of the +1 RMSNorm shift (conversion passes
tensors through verbatim), and the head's norms shift under the same
original-format detection as the main stack, so nothing shifts twice.
