Commit graph

1137 commits

Author SHA1 Message Date
Daniel Hiltgen
23b94a2daf
mlx: inkling follow up (#19806) 2026-11-20 19:03:47 -08:00
Daniel Hiltgen
cecac8bf1b
mlxrunner: add Inkling 2 Mini (#19799)
* mlxrunner: add Inkling 2 Mini

* mlx: fuse Inkling 2 Mini residual mixing on Metal

Fuse RMS normalization and query scoring in one kernel. Apply the
softmax weights in a second kernel to avoid storing per-query products.

* create: expose Inkling 2 Mini context length in model metadata

Copy model_max_length to max_position_embeddings during import so
ollama show reports the checkpoint's context length. Reject configs
that specify conflicting limits.

* mlxrunner: reduce Metal dispatches in Inkling 2 Mini decode

Reuse the prefill mask shader to compute relative position bias in one
dispatch for single-query attention. On M3 Ultra with Metal 3, compute
the loop controller's gain and forget projections in one matrix multiply.
2026-11-20 18:38:44 -08:00
Gaurav Jain
42ac2c0c1e
Merge pull request #19748 from ollama/claude/sched-shift-no-reload-18950
server, llm: share one runner across `shift` settings
2026-11-18 14:53:02 -08:00
J Berg
5632619189
Map disabled thinking to the models lowest supported level (#19684) 2026-11-12 15:03:07 -08:00
Daniel Hiltgen
ab9295d03c
mlx: support concurrent generation (#19197)
* mlx: batch requests through a shared scheduler

Use one request queue and scheduler for generation, decisions, embeddings
and speech, including parallelism one. Share compatible model operations
while keeping request caches, sampling state and response streams independent.

Interleave prompt chunks with decoding and retain speculative decoding when
generation runs alone. Cover isolation, cancellation, slow readers, cache
reuse and mixed requests with unit and integration tests. Remove the stale
llama-server architecture restriction on parallel requests.

* mlx: prevent cache snapshots from retaining batch peers

Copy each request’s initial batched K/V row so a sliding-cache snapshot cannot retain the whole batch allocation while accounting for only one row.

Exclude grammar compilation waits from prompt timing and reuse row-selected attention masks across layers.

Return errors for requests rejected before admission and strengthen numerical and compiled-speech regression checks.

* address comments
2026-11-10 13:24:12 -08:00
Marlon Ferreira
e968efb60a
openai: forward truncation option in responses requests (#19501)
Co-authored-by: Eva Ho <hoyyeva@gmail.com>
2026-11-05 16:56:45 -08:00
Daniel Hiltgen
84dc81db1f
llama-server: allow parallel requests for recurrent and vision models (#19545)
These architectures were forced to num_parallel=1 because the older Go
engine's recurrent caches held one sequence. llama-server supports
multiple slots for all of them so we can allow parallelism now.
2026-11-05 09:22:54 -08:00
Daniel Hiltgen
d666cdb246
Add AutoTrust decision model to MLX (#19399)
* Add AutoTrust decision scoring to MLX

* address comments

* Support AutoTrust 35B decision models in MLX

* Resolve AutoTrust choice limits during config loading

Apply publisher-specific defaults and validate requests against the resolved limit.
2026-11-04 09:47:37 -08:00
Parth Sareen
9a3e817776
server: decode cloud web search responses (#19476) 2026-11-02 16:59:50 -08:00
Eva H
8861adaa77
openai: preserve readable Responses input content (#19192) 2026-10-28 23:46:40 -04:00
Daniel Hiltgen
3fbc609be4
llm: LFM3 GGUF fixes and llama-server speech (#19344)
Fixes for running LFM3 GGUFs on the llama-server runner, plus GGUF
text-to-speech that turns on once llama.cpp is bumped past b11901.

- BOS: honor an explicit tokenizer.ggml.add_bos_token. LFM3 GGUFs (and
  lfm2:24b) set it to false and expect the template's BOS, which the
  lfm2 default was stripping. The lfm2 default still applies when the
  key is absent. On lfm2:24b, WikiText PPL goes from 81.3 to 72.0.
- repeat_last_n -1 now maps to the context size; llama-server rejects
  negative values.
- Audio-only projectors (LFM3-Audio) no longer advertise vision, so
  clients don't send images the model can't take.
- /v1/audio/speech on GGUF models calls llama-server /tts chunk by
  chunk. Renderers opt in through a new SpeechPrompter (voices,
  voice parsing, chunking), so other speech models can add their own.
  The LFM3 voice and chunking helpers move from the MLX runner into
  model/renderers and are shared by both runners.
2026-10-27 18:56:22 -07:00
Daniel Hiltgen
f067033f91
mlx: add LFM3 and d1 models (#19333) 2026-10-27 08:47:18 -07:00
Jeffrey Morgan
675150d6ad
voice: add voice catalog for Phonon (#19242)
* mlxrunner: import and list Phonon catalog voices

* docs: document Phonon voices
2026-10-23 16:15:08 -07:00
SynapShift
f763dc70d2
decision: allow empty state for vllm-d3 (#19173)
* decision: allow empty state for vllm-d3

* server: reuse decoded state in System One tests

---------

Co-authored-by: Patrick Devine <patrick@ollama.com>
2026-10-23 14:13:14 -07:00
Eva H
1472203c47
launch: use allocated context for OpenCode (#19193) 2026-10-23 11:45:15 -07:00
Daniel Hiltgen
f52b9e742d
Update gemma renderers (#19241)
* renderers: add gemma4-nemotron renderer

Add a Gemma 4 renderer variant that renders assistant content before
its tool calls rather than after the tool responses. Models opt in with
RENDERER gemma4-nemotron; gemma4 auto-detection is unchanged.

Adds the corresponding chat template as a test fixture for Jinja parity
checks.

* mlx: support Gemma 4 targets with DFlash drafts

Load a draft-owned output head when its config requires one, while preserving the target-head path used by existing DFlash models.

Expose Gemma 4 hidden-state taps and raw logits for speculative decoding. Report safetensors target precision independently of quantized draft weights.
2026-10-23 07:12:11 -07:00
Eva H
a79f7d7455
anthropic: reuse resolved thinking for web search (#19098) 2026-10-22 15:23:42 -07:00
Jeffrey Morgan
6ef847b8fe
mlxrunner: add Phonon text-to-speech support (#19166)
* mlxrunner: add Phonon text-to-speech support

Add Phonon text-to-speech through the existing MLX runner, with an
OpenAI-compatible speech endpoint and CLI playback.

Include model loading, text preprocessing, streaming audio, and
regression tests.

* docs: add speech models

* fix min version

---------

Co-authored-by: Daniel Hiltgen <daniel@ollama.com>
2026-10-22 11:36:42 -07:00
Daniel Hiltgen
f20eb3b312
mlx: add d3 decision model support (#19196)
* mlx: add d3 decision model support

* bench: report loaded VRAM for decision runs
2026-10-22 08:50:14 -07:00
Eva H
efabf6bc45
anthropic: surface errors after streaming starts (#19188) 2026-10-21 19:00:18 -07:00
Daniel Hiltgen
52f168120e
server: read loaded runner models without refMu (#19070)
* server: read loaded runner models without refMu

Concurrent lookups of a warm model failed TryLock against each other and
fell back to a manifest reload, defeating the cache and making
TestLoadedModelLookupConcurrent flaky. Make runnerRef.model an
atomic.Pointer so lookups never take refMu.

* address comments
2026-10-20 12:19:18 -07:00
Eva H
e61ba8b470
openai: finish reasoning before client tool calls (#19146) 2026-10-19 18:51:48 -07:00
Eva H
4ed3ba13cf
server: cancel remote inference when clients disconnect (#19144) 2026-10-19 17:37:10 -04:00
Eva H
678722224b
openai: accept foreign compaction in Responses history (#19003) 2026-10-19 08:25:35 -04:00
Daniel Hiltgen
36a6e994d2
api: populate show renderer/parser fields and add template_source (#19095)
ShowResponse's renderer and parser fields existed but the server never
set them; they now carry the effective values (renderer resolves the
gemma4 legacy alias by model size). A new template_source names what
actually renders chat prompts: the configured renderer/parser (leaving
any packaged template inert), a Go template layer, the GGUF's embedded
jinja, harmony, or none.

The CLI prints renderer/parser/template source rows only when set, so
ollama show output is unchanged for models without rendering config.
2026-10-17 17:31:52 -07:00
Daniel Hiltgen
8be580ad09
mlx: add PPLX Decider v1.2 support (#19072)
Implement the Qwen3.5-backed decision scorer with noncausal attention,
the publisher readout, image handling, and System One prompt encoding.
2026-10-17 14:45:55 -07:00
Eva H
d1ed3fd13a
openai: allow Responses effort on non-thinking models (#19010) 2026-10-16 18:45:46 -07:00
Patrick Devine
4d2e250164
transfer: support proxies on registry redirect targets (#19065)
Behind a proxy, defer redirect-target DNS to the proxy; without one, keep the resolve-and-pin path. Trust default-registry/CDN hostnames with no local DNS. Fix a nil CancelFunc crash in blobDownload.release.
2026-10-16 15:00:17 -07:00
Eva H
8e4dad49f9
openai: clarify Responses authentication errors (#19004) 2026-10-14 22:35:06 -04:00
Devon Rifkin
957c8cd5ce
server: stop HTML-escaping cloud proxy request bodies (#19008) 2026-10-14 18:16:56 -07:00
J Berg
d10d1066ee
Merge pull request #18969 from ollama/jberg/systemone-cloud-proxy
Introduce clef and clef flash on cloud
2026-10-14 16:57:38 -07:00
Jonathan Berg
b852974584 use cloudPassthroughMiddleware 2026-10-14 16:42:45 -07:00
Jeffrey Morgan
7a5ae12ead
cli: add decision model support to ollama run (#18962) 2026-10-13 12:39:05 -07:00
Daniel Hiltgen
665f9e074c
Add Kev decision scoring to MLX (#18976)
* Add Kev decision scoring to MLX

Load Kev models with their FP32 decision head and release metadata.
Support System One scoring with shared-state caching and Kev-specific limits.

* address comments
2026-10-12 21:50:41 -07:00
Jeffrey Morgan
a527652553
server: fix multimodal embedding load with small num_ctx (#18973) 2026-10-12 17:43:50 -07:00
Jonathan Berg
d629978e44 Introduce clef and clef flash on cloud 2026-10-12 16:23:24 -07:00
Daniel Hiltgen
9d35332880
server: reuse verified manifests and loaded models in inference routes (#18947)
* server: reuse verified manifests and loaded models in inference routes

Resolve named manifests and runner selection in one verified cold lookup.
When a runner is loaded, reuse its model after checking that the v2 tag
still points to the same digest. Fully verify regular-file manifests,
including Windows copies.

Share the lookup across generate, chat, embed, and System One. Keep parser
selection request-local so reusing a model does not mutate its config.

Cover tag changes, runner selection, concurrent unloads, and existing
missing-model responses with regression tests and a lookup benchmark.

* address comments
2026-10-12 08:41:52 -07:00
Jeffrey Morgan
0c21bcf0ac llm: support multimodal embeddings through llama-server 2026-10-11 23:25:23 -07:00
Jeffrey Morgan
adc4aae9d6 llama: update to b11521 and use native decision support 2026-10-11 23:25:23 -07:00
Jeffrey Morgan
eab97e9f92
server: skip local model compatibility migration (#18908) 2026-10-09 10:36:11 -07:00
Daniel Hiltgen
531946e996
server: hide duplicate and downgrade guards from list (#18874)
When a legacy GGUF which requires our llama.cpp patch set is loaded, we do a
lazy conversion on disk to make it llama.cpp compatible.  To support
downgrades, we temporarily hold both the new and old GGUFs and create a shadow
v1 manifest tag to prevent the downgraded server from deleting the new blobs.
Showing these in the list output is confusing to users and API consumers.  The
legacy copies will be removed in a future release to recover the disk space.
2026-10-08 08:54:06 -07:00
Devon Rifkin
f864601538
server: proxy cloud usage and balance APIs (#18829)
Forward GET /api/usage and GET /api/balance through the local server
using the signed-in account.

Add API documentation and tests for proxying, signing, and
disabled cloud access.
2026-10-06 18:31:54 -07:00
Patrick Devine
0d0720e51f
routes: fix embed 413 vs 400 routing (#18822) 2026-10-06 10:46:39 -07:00
Patrick Devine
efe43c5561
model: add multimodal embeddings (#18820)
Implements the EmbeddingGemma2Model architecture on the MLX runner: 24-layer
bidirectional text encoder with PLE, shared gemma4 vision/audio towers,
mean-pool + L2 output. /api/embed accepts per-item media via input dicts.
2026-10-06 08:08:38 -07:00
Daniel Hiltgen
d9dec1509d
Reduce model lookup and MLX decision request overhead (#18806)
* Reduce model lookup and MLX decision request overhead

Avoid decoding unrelated manifests when resolving existing model names.
Reuse small Metal scratch buffers between decision requests, releasing
larger working sets and clearing scratch after scoring errors.

* address comments
2026-10-05 17:34:41 -07:00
Daniel Hiltgen
991421d4db
pull: allow RCs to pull matching min_version (#18790)
Strip off pre-release from the version string so RCs can pull
models for the same release.
2026-10-04 16:15:37 -07:00
Daniel Hiltgen
27a36513d2
mlx: Decision model improvements (#18776)
* reject instead of truncate on overflow

* tune route logic for faster warm runner lookup
2026-10-03 17:48:08 -07:00
Daniel Hiltgen
a00d083bb5
mlx: System one support (#18701)
* mlxrunner: restore System One scoring

Carry the MLX implementation and its tests from the original System One
commit. The API/compiler and llama-server implementation already exist on
the release branch, so retain those versions. Adapt the optional model test
to the renamed decision package and its explicit rendering/context setup.

(cherry picked from commit 26e546ddb3)

* Enable MLX System One and add regression coverage

* mlxrunner: add Laya support for System One

Import Laya checkpoints and implement bidirectional decision scoring.

* mlxrunner: release cached buffers after System One scoring

* decision: match Tev1 and Nimble publisher prompt formats

Select Tev1 prompts using renderer metadata and fix JSON formatting, with regression coverage for GGUF and MLX.

* mlx: support Clef and Clef Flash decision models

Add model import, quantization safeguards, and text/image scoring with the Clef joint decision head.

* support vision capabilities for decision models

* Make Clef scoring cancellable between prefill chunks

Preserve joint-head inputs and test cancellation, cleanup, and recovery.

* mlx: separate decision preparation and readout from forward

Let serial and batched scoring share model-owned preparation and readout
without duplicating architecture-specific logic in the runner.

* mlx: add Strands Decider support

* implement caching

---------

Co-authored-by: ParthSareen <parth.sareen@ollama.com>
2026-10-03 13:17:21 -07:00
Daniel Hiltgen
1c345795ab Merge remote-tracking branch 'upstream/main' into release_v0.40.0 2026-10-01 16:51:30 -07:00
Jeffrey Morgan
e4c0d18eb8
models: add clef support (#18741) 2026-10-01 15:52:04 -07:00