Compute the normalized residuals, query scores, and weighted sum in one
dispatch. Keep intermediate values in the threadgroup and preserve the
existing reduction order and rounding.
* mlxrunner: add Inkling 2 Mini
* mlx: fuse Inkling 2 Mini residual mixing on Metal
Fuse RMS normalization and query scoring in one kernel. Apply the
softmax weights in a second kernel to avoid storing per-query products.
* create: expose Inkling 2 Mini context length in model metadata
Copy model_max_length to max_position_embeddings during import so
ollama show reports the checkpoint's context length. Reject configs
that specify conflicting limits.
* mlxrunner: reduce Metal dispatches in Inkling 2 Mini decode
Reuse the prefill mask shader to compute relative position bias in one
dispatch for single-query attention. On M3 Ultra with Metal 3, compute
the loop controller's gain and forget projections in one matrix multiply.
* install: support preview channel
Allow install.sh and install.ps1 to select preview artifacts with
OLLAMA_CHANNEL=preview. Stable remains the default, OLLAMA_VERSION retains
precedence, and unsupported channels are rejected.
* address comments
When OLLAMA_NUM_PARALLEL is unset, select 1, 2, 4 or 8 slots
using available memory and estimated per-client storage at the
model's context limit.
Preserve explicit settings and retain the serial default under
memory pressure, for unknown cache layouts and on other backends.
* mlxrunner: adapt saved history to memory pressure
Retain Metal prefix history using observed memory headroom and execution
workspace rather than a fixed 8 GiB allowance. Release unused buffers and
cold history when system compression grows, preserving reusable paths and
active requests.
Cover pressure reclamation, snapshot ownership, memory observation, and
cacheless request isolation.
* address comments
* address comments
This removes dependence on socket buffering and large conversion work under the
race detector which should address a flaky CI test.
Block the response writer explicitly instead of generating 32 MiB of
audio to fill socket buffers. Verify generation finishes before reading
begins and all audio is delivered afterward.
Reuse completed prefix checkpoints from another slot, including one still
serving a request. Restore into recipient-owned state and copy full-attention
history directly without materializing an intermediate donor snapshot.
Cover donor lifetime, pending captures, cancellation, media and drafting
boundaries, merged history, and cached evaluation.
Abandoning a """ block left the REPL in multiline mode, so the next
line was swallowed or saved as the system message. Reset the multiline
state along with the buffer whenever readline returns an error.
Fixes#19340
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mlxrunner: batch speculative verification
Keep drafting active for concurrent generation by sharing target verification while preserving per-request sampling and cache rollback. Learn draft depth for each concurrent workload separately from isolated requests, and preserve probes that short requests never execute.
Materialize owned target features before a reader can pause, and cover acceptance, cancellation, handoffs, mixed decoding and retained-memory boundaries.
* address comments
Let packagers supply MLX-C and XGrammar binaries independently. Stage
MLX-C in the runner directory, build Ollama's XGrammar wrapper against
the supplied library, and allow additional Go linker flags.
On Windows, pinned host memory for an integrated GPU is charged to the
same WDDM shared-memory budget as its device buffers. llama-server turns
off mmap for iGPUs and places CPU-side weights in pinned host memory,
which --fit does not count, so a partially offloaded model overflows the
budget and the KV cache allocation fails. Pass --no-host for Windows
iGPUs so CPU-side weights stay in pageable memory.
* mlx: batch requests through a shared scheduler
Use one request queue and scheduler for generation, decisions, embeddings
and speech, including parallelism one. Share compatible model operations
while keeping request caches, sampling state and response streams independent.
Interleave prompt chunks with decoding and retain speculative decoding when
generation runs alone. Cover isolation, cancellation, slow readers, cache
reuse and mixed requests with unit and integration tests. Remove the stale
llama-server architecture restriction on parallel requests.
* mlx: prevent cache snapshots from retaining batch peers
Copy each request’s initial batched K/V row so a sliding-cache snapshot cannot retain the whole batch allocation while accounting for only one row.
Exclude grammar compilation waits from prompt timing and reuse row-selected attention masks across layers.
Return errors for requests rejected before admission and strengthen numerical and compiled-speech regression checks.
* address comments
Non-mmap loads put CPU-resident tensors in the Vulkan host buffer, which is
drawn from the same device budget as the weights. On a shared-memory iGPU that
roughly doubles the footprint and the load fails with "Failed to allocate
pinned memory" or, on Windows AMD, 0xe06d7363.
Direct I/O does not avoid the host buffer, so Vulkan uses mmap rather than
reusing the CUDA/ROCm dio policy.
Make --restore confirm before changing an active profile, leave a closed
app closed, and preserve existing settings and chat history. Let interactive
users finish ChatGPT's own quit prompt instead of force-quitting it.
Carry cancellation and timeouts through status checks, quit requests and
exit polling. Check cancellation before profile writes, and keep legacy
PID and singleton probes read-only until shutdown is confirmed, including
when a legacy client starts after preflight.
Keep context checks around process queries so a canceled probe cannot be
mistaken for a confirmed exit. Publish the legacy-quit test marker with an
atomic rename so cancellation cannot interrupt its write.
Validation: 591 launch tests, focused race checks, CLI build, and 100 repeated
legacy-quit cancellation checks with the race detector.
* Fix ChatGPT restore CI setup
Stub installed-app discovery in the late legacy-start consent test so it
runs without ChatGPT installed on the CI host.
Document the containedctx exception for the context implementation that
observes spinner cancellation synchronously. This preserves the existing
cancellation behavior while satisfying the repository lint configuration.
Add the Ripple model, renderer, parser and import registration.
Support per-layer NoPE in DFlash drafts.
Fuse MLP activation, value blending and attention gating.
Add reference fixtures and model/runner regression tests.
* MLX: reduce Qwen3.8 Flash Next memory use
For NVFP4 creation, retain source precision for sensitive QSA projections
and non-expert MTP weights. Keep routed experts and PLE at NVFP4, and
promote other eligible weights to MXFP8.
Keep NVFP4 PLE tables host-backed, transferring only selected packed rows
and their scales to MLX. Honor scalar and per-expert NVFP4 global scales
in MoE, and per-shard global scales in host-backed PLE.
Use compressed QSA block selections in fused indexed Metal attention for
long prefill batches, retaining the graph fallback. Co-locate component
loading and forward methods, with explicit quantization and cache ownership.
* mlx: skip materializing host-backed tensors
Let models exclude tensors already owned by host storage from MLX loading.
Avoid allocating and discarding the full Qwen PLE tables at startup while
preserving the resident fallback.
* address comments
These architectures were forced to num_parallel=1 because the older Go
engine's recurrent caches held one sequence. llama-server supports
multiple slots for all of them so we can allow parallelism now.
NVIDIA's libcusolver looks for its dependencies only in its own
directory, so libcublas and libnvJitLink resolved to the host's copies
and MLX failed to load when those didn't match the bundled ones. Preload
the bundled copies by full path before loading MLX.