Commit graph

6015 commits

Author SHA1 Message Date
Daniel Hiltgen
2bd4d6aabd
mlx: fuse Inkling residual mixing on Metal (#19808)
Compute the normalized residuals, query scores, and weighted sum in one
dispatch. Keep intermediate values in the threadgroup and preserve the
existing reduction order and rounding.
2026-11-20 20:05:50 -08:00
Daniel Hiltgen
23b94a2daf
mlx: inkling follow up (#19806) 2026-11-20 19:03:47 -08:00
Daniel Hiltgen
cecac8bf1b
mlxrunner: add Inkling 2 Mini (#19799)
* mlxrunner: add Inkling 2 Mini

* mlx: fuse Inkling 2 Mini residual mixing on Metal

Fuse RMS normalization and query scoring in one kernel. Apply the
softmax weights in a second kernel to avoid storing per-query products.

* create: expose Inkling 2 Mini context length in model metadata

Copy model_max_length to max_position_embeddings during import so
ollama show reports the checkpoint's context length. Reject configs
that specify conflicting limits.

* mlxrunner: reduce Metal dispatches in Inkling 2 Mini decode

Reuse the prefill mask shader to compute relative position bias in one
dispatch for single-query attention. On M3 Ultra with Metal 3, compute
the loop controller's gain and forget projections in one matrix multiply.
2026-11-20 18:38:44 -08:00
Daniel Hiltgen
afc7c06022
install: support preview channel (#19801)
* install: support preview channel

Allow install.sh and install.ps1 to select preview artifacts with
OLLAMA_CHANNEL=preview. Stable remains the default, OLLAMA_VERSION retains
precedence, and unsupported channels are rejected.

* address comments
2026-11-20 14:47:25 -08:00
Patrick Devine
0c666ce5ee
cmd: omit bias and gamma vectors from verbose safetensors show (#19803) 2026-11-20 14:44:24 -08:00
Eva H
5187f59581
test: stabilize Pi catalog timeout regressions (#19742) 2026-11-20 16:34:09 -05:00
Daniel Hiltgen
0b243a141a
mlxrunner: choose default parallelism from available memory (#19785)
When OLLAMA_NUM_PARALLEL is unset, select 1, 2, 4 or 8 slots
using available memory and estimated per-client storage at the
model's context limit.

Preserve explicit settings and retain the serial default under
memory pressure, for unknown cache layouts and on other backends.
2026-11-20 07:13:42 -08:00
Daniel Hiltgen
73ee71ddcc
mlxrunner: adapt saved history to memory pressure (#19678)
* mlxrunner: adapt saved history to memory pressure

Retain Metal prefix history using observed memory headroom and execution
workspace rather than a fixed 8 GiB allowance. Release unused buffers and
cold history when system compression grows, preserving reusable paths and
active requests.

Cover pressure reclamation, snapshot ownership, memory observation, and
cacheless request isolation.

* address comments

* address comments
2026-11-19 15:41:42 -08:00
Gaurav Jain
42ac2c0c1e
Merge pull request #19748 from ollama/claude/sched-shift-no-reload-18950
server, llm: share one runner across `shift` settings
2026-11-18 14:53:02 -08:00
Daniel Hiltgen
508d8be9c8
test: stabilize speech slow-reader coverage (#19746)
This removes dependence on socket buffering and large conversion work under the
race detector which should address a flaky CI test.

Block the response writer explicitly instead of generating 32 MiB of
audio to fill socket buffers. Verify generation finishes before reading
begins and all audio is delivered afterward.
2026-11-18 08:30:00 -08:00
Daniel Hiltgen
b8db8d5b5c
mlxrunner: reuse saved prefixes across request slots (#19677)
Reuse completed prefix checkpoints from another slot, including one still
serving a request. Restore into recipient-owned state and copy full-attention
history directly without materializing an intermediate donor snapshot.

Cover donor lifetime, pending captures, cancellation, media and drafting
boundaries, merged history, and cached evaluation.
2026-11-16 20:54:35 -08:00
Gaurav Jain
312c3c8d0c
Merge pull request #19731 from ollama/fix/repl-ctrl-c-multiline
cmd: leave multiline mode when Ctrl+C or Ctrl+G discards input
2026-11-16 19:33:27 -08:00
Gaurav Jain
169626b403 cmd: leave multiline mode when Ctrl+C or Ctrl+G discards input
Abandoning a """ block left the REPL in multiline mode, so the next
line was swallowed or saved as the system message. Reset the multiline
state along with the buffer whenever readline returns an error.

Fixes #19340

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 18:20:35 -08:00
Daniel Hiltgen
db5266a71d
mlxrunner: batch speculative verification (#19663)
* mlxrunner: batch speculative verification

Keep drafting active for concurrent generation by sharing target verification while preserving per-request sampling and cache rollback. Learn draft depth for each concurrent workload separately from isolated requests, and preserve probes that short requests never execute.

Materialize owned target features before a reader can pause, and cover acceptance, cancellation, handoffs, mixed decoding and retained-memory boundaries.

* address comments
2026-11-16 08:43:54 -08:00
Daniel Hiltgen
9a4b136efe
cmake: support compatible binary MLX-C and XGrammar (#19688)
Let packagers supply MLX-C and XGrammar binaries independently. Stage
MLX-C in the runner directory, build Ollama's XGrammar wrapper against
the supplied library, and allow additional Go linker flags.
2026-11-16 07:11:09 -08:00
saman-amd
15af2b261c
llm: keep weights out of pinned host memory on Windows iGPUs (#19636)
On Windows, pinned host memory for an integrated GPU is charged to the
same WDDM shared-memory budget as its device buffers. llama-server turns
off mmap for iGPUs and places CPU-side weights in pinned host memory,
which --fit does not count, so a partially offloaded model overflows the
budget and the KV cache allocation fails. Pass --no-host for Windows
iGPUs so CPU-side weights stay in pageable memory.
2026-11-15 10:46:15 -08:00
J Berg
a5864fcb33
api: preserve unmodeled tool schema keywords (#19686) 2026-11-13 16:00:07 -08:00
J Berg
5632619189
Map disabled thinking to the models lowest supported level (#19684) 2026-11-12 15:03:07 -08:00
Parth Sareen
9da7dfd5c3
cmd: show help for noninteractive bare invocations (#19557) 2026-11-12 14:35:02 -08:00
J Berg
361b3e17fd
Support deferred tool loading in claude code (#19578) 2026-11-12 13:24:56 -08:00
Anna Rahn
4a7b194120
Merge pull request #18972 from ollama/docs_restructuring
docs: restructuring to separate out cloud features
2026-11-12 10:28:01 -08:00
Eva H
8a89b76cc7
launch: refresh Pi metadata after sign-in and honor prompt args (#19575) 2026-11-10 17:43:58 -05:00
Daniel Hiltgen
6140670fd1
MLX, MLX-C: version bump (#19647) 2026-11-10 13:42:11 -08:00
Daniel Hiltgen
9b390157f2
llama.cpp: version bump b12257 (#19646) 2026-11-10 13:41:48 -08:00
Daniel Hiltgen
ab9295d03c
mlx: support concurrent generation (#19197)
* mlx: batch requests through a shared scheduler

Use one request queue and scheduler for generation, decisions, embeddings
and speech, including parallelism one. Share compatible model operations
while keeping request caches, sampling state and response streams independent.

Interleave prompt chunks with decoding and retain speculative decoding when
generation runs alone. Cover isolation, cancellation, slow readers, cache
reuse and mixed requests with unit and integration tests. Remove the stale
llama-server architecture restriction on parallel requests.

* mlx: prevent cache snapshots from retaining batch peers

Copy each request’s initial batched K/V row so a sliding-cache snapshot cannot retain the whole batch allocation while accounting for only one row.

Exclude grammar compilation waits from prompt timing and reuse row-selected attention masks across layers.

Return errors for requests rejected before admission and strengthen numerical and compiled-speech regression checks.

* address comments
2026-11-10 13:24:12 -08:00
Daniel Hiltgen
3f3fa58f7f
llm: use mmap for Vulkan integrated GPUs (#19640)
Non-mmap loads put CPU-resident tensors in the Vulkan host buffer, which is
drawn from the same device budget as the weights. On a shared-memory iGPU that
roughly doubles the footprint and the load fails with "Failed to allocate
pinned memory" or, on Windows AMD, 0xe06d7363.

Direct I/O does not avoid the host buffer, so Vulkan uses mmap rather than
reusing the CUDA/ROCm dio policy.
2026-11-10 10:42:12 -08:00
Eva H
f1ee5679fe
cmd/launch: use thinking metadata for OpenCode settings (#19409) 2026-11-09 17:29:01 -05:00
Eva H
1f23c7232e
Improve ChatGPT CLI restore and graceful restart (#19583)
Make --restore confirm before changing an active profile, leave a closed
app closed, and preserve existing settings and chat history. Let interactive
users finish ChatGPT's own quit prompt instead of force-quitting it.

Carry cancellation and timeouts through status checks, quit requests and
exit polling. Check cancellation before profile writes, and keep legacy
PID and singleton probes read-only until shutdown is confirmed, including
when a legacy client starts after preflight.

Keep context checks around process queries so a canceled probe cannot be
mistaken for a confirmed exit. Publish the legacy-quit test marker with an
atomic rename so cancellation cannot interrupt its write.

Validation: 591 launch tests, focused race checks, CLI build, and 100 repeated
legacy-quit cancellation checks with the race detector.

* Fix ChatGPT restore CI setup

Stub installed-app discovery in the late legacy-start consent test so it
runs without ChatGPT installed on the CI host.

Document the containedctx exception for the context implementation that
observes spinner cancellation synchronously. This preserves the existing
cancellation behavior while satisfying the repository lint configuration.
2026-11-09 12:29:51 -05:00
Patrick Devine
d63b702609
Add ripple preview support to the MLX runner (#19592)
Add the Ripple model, renderer, parser and import registration.
Support per-layer NoPE in DFlash drafts.
Fuse MLP activation, value blending and attention gating.
Add reference fixtures and model/runner regression tests.
2026-11-07 16:36:53 -08:00
Eva H
5e983d4f80
launch: fix Claude Code routing and retired model errors (#19587) 2026-11-06 21:28:13 -05:00
anra8571
04494de5b2 docs: merging to remove retired model recommendations 2026-11-06 15:21:50 -08:00
Daniel Hiltgen
c053d1e985
MLX: make Qwen3.8 Flash Next memory efficient (#18078)
* MLX: reduce Qwen3.8 Flash Next memory use

For NVFP4 creation, retain source precision for sensitive QSA projections
and non-expert MTP weights. Keep routed experts and PLE at NVFP4, and
promote other eligible weights to MXFP8.

Keep NVFP4 PLE tables host-backed, transferring only selected packed rows
and their scales to MLX. Honor scalar and per-expert NVFP4 global scales
in MoE, and per-shard global scales in host-backed PLE.

Use compressed QSA block selections in fused indexed Metal attention for
long prefill batches, retaining the graph fallback. Co-locate component
loading and forward methods, with explicit quantization and cache ownership.

* mlx: skip materializing host-backed tensors

Let models exclude tensors already owned by host storage from MLX loading.
Avoid allocating and discarding the full Qwen PLE tables at startup while
preserving the resident fallback.

* address comments
2026-11-06 13:39:21 -08:00
Daniel Hiltgen
eaf3f28586
windows: add CUDA 13.5 and SM 103 for ARM64 (#19580)
Install the separate ARM64 cross package in CI and retain SM 121
support. Other platforms are unchanged.
2026-11-06 13:33:36 -08:00
Matteo Celani
51e7030f94
README: add Quock to Mobile community integrations (#19361) 2026-11-06 12:41:45 -08:00
Daniel Hiltgen
7ed1eb9c01
llama.cpp: version bump b12177 (#19576) 2026-11-06 12:00:00 -08:00
liangliang205
6362e0a308
openai: return an empty array for /v1/models when no models are installed (#19539) 2026-11-05 16:57:40 -08:00
Marlon Ferreira
e968efb60a
openai: forward truncation option in responses requests (#19501)
Co-authored-by: Eva Ho <hoyyeva@gmail.com>
2026-11-05 16:56:45 -08:00
J Berg
02480ca2a4
cmd/launch: add Kimi Code CLI support (#19506) 2026-11-05 13:41:21 -08:00
Patrick Devine
8e264f8989
mlxrunner: close generation cache on shutdown (#19550) 2026-11-05 12:59:06 -08:00
Parth Sareen
f60dd2060f
cmd: restore the no-argument launcher (#19478)
Co-authored-by: Jeffrey Morgan <jmorganca@gmail.com>
2026-11-05 13:41:22 -05:00
Parth Sareen
32ce4379cc
cmd/launch: show gateway model names in Claude Desktop's model picker (#19533) 2026-11-05 10:40:54 -08:00
Daniel Hiltgen
84dc81db1f
llama-server: allow parallel requests for recurrent and vision models (#19545)
These architectures were forced to num_parallel=1 because the older Go
engine's recurrent caches held one sequence. llama-server supports
multiple slots for all of them so we can allow parallelism now.
2026-11-05 09:22:54 -08:00
Daniel Hiltgen
a6d3325162
llama.cpp: version bump b12111 (#19530)
Drop 002-metal-unary-cast-constexpr.patch: superseded by upstream fix
2026-11-04 17:31:38 -08:00
Patrick Devine
cdd4b55b43
mlx: support YaRN scaling in Qwen3 and Qwen3.5 (#19231) 2026-11-04 14:14:32 -08:00
Daniel Hiltgen
2e7b870fbc
mlx: reject unsupported CUDA devices (#19521)
* mlx: bump for CUDA architecture metadata

* mlx: reject unsupported CUDA devices

* address comments
2026-11-04 09:57:25 -08:00
Daniel Hiltgen
d666cdb246
Add AutoTrust decision model to MLX (#19399)
* Add AutoTrust decision scoring to MLX

* address comments

* Support AutoTrust 35B decision models in MLX

* Resolve AutoTrust choice limits during config loading

Apply publisher-specific defaults and validate requests against the resolved limit.
2026-11-04 09:47:37 -08:00
Eva H
e9fde36d75
launch: populate Pi models automatically (#19460) 2026-11-03 15:26:02 -08:00
Daniel Hiltgen
355eaee353
model: add MiMo V2.6 renderer for Qwen3.5 (#19400) 2026-11-03 07:09:06 -08:00
Daniel Hiltgen
ddf22489b6
mlx: load bundled CUDA libraries in-process on Linux (#19433)
NVIDIA's libcusolver looks for its dependencies only in its own
directory, so libcublas and libnvJitLink resolved to the host's copies
and MLX failed to load when those didn't match the bundled ones. Preload
the bundled copies by full path before loading MLX.
2026-11-03 06:45:30 -08:00
Parth Sareen
9a3e817776
server: decode cloud web search responses (#19476) 2026-11-02 16:59:50 -08:00