CERBERUS/ollama
1
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-08-04 14:56:15 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions
ollama/x/mlxrunner
History
Exact
Jesse Gross 123b1f2479 mlxrunner: raise the MTP pending-flush cap to 256 tokens 
Per-token cost of the batched head forward keeps falling until the
flush is large enough to reach the fastest kernels: NAX matmul tiles
for dense heads, and the segmented gather path for MoE heads, which
needs tokens*topK/experts >= 4. Measured across the qwen3.6 heads,
256 is the smallest cap past every threshold and within a few percent
of each head's per-token floor. The cost is bounded: up to 2.5 MiB of
pinned hiddens per request and a flush stall under one decode step.
2026-07-14 10:32:04 -07:00
..
batch mlxrunner: support draft heads that maintain draft caches 2026-06-22 15:25:45 -07:00
cache cache: stop recurrent conv state from pinning the forward buffer 2026-07-14 10:32:04 -07:00
mlx mlx: bump dependency (#16935) 2026-06-29 09:39:11 -07:00
model mlx: x/create rewrite (#16919) 2026-07-03 18:30:45 -07:00
sample mlxrunner: apply in-flight drafts to proposal penalty history 2026-06-22 15:25:45 -07:00
cache_trie.go mlxrunner: key the cache trie by token pairs for draft caches 2026-07-14 10:32:04 -07:00
cache_trie_test.go mlxrunner: key the cache trie by token pairs for draft caches 2026-07-14 10:32:04 -07:00
client.go mlx: configurable model load timeout (#14796) 2026-07-13 16:06:29 -07:00
imports.go models: add cohere2_moe (Command A / North) to the MLX engine (#16670) 2026-06-16 23:15:21 -07:00
mtp.go mlxrunner: raise the MTP pending-flush cap to 256 tokens 2026-07-14 10:32:04 -07:00
mtp_test.go mlxrunner: key the cache trie by token pairs for draft caches 2026-07-14 10:32:04 -07:00
pipeline.go mlxrunner: key the cache trie by token pairs for draft caches 2026-07-14 10:32:04 -07:00
prefix_cache.go mlxrunner: key the cache trie by token pairs for draft caches 2026-07-14 10:32:04 -07:00
prefix_cache_test.go mlxrunner: key the cache trie by token pairs for draft caches 2026-07-14 10:32:04 -07:00
runner.go mlxrunner: construct per-model state at load 2026-07-14 10:32:04 -07:00
server.go mlx: rework the MLX sampler (#16122) 2026-05-13 17:18:27 -07:00
speculate.go mlxrunner: key the cache trie by token pairs for draft caches 2026-07-14 10:32:04 -07:00
speculate_depth.go mlxrunner: choose the speculative draft length to maximize throughput 2026-06-22 15:25:45 -07:00
speculate_depth_test.go mlxrunner: choose the speculative draft length to maximize throughput 2026-06-22 15:25:45 -07:00
speculate_stats.go mlxrunner: choose the speculative draft length to maximize throughput 2026-06-22 15:25:45 -07:00
status_memory.go mlx: avoid status timeout during inference (#16086) 2026-05-11 16:03:38 -07:00
status_memory_test.go mlx: avoid status timeout during inference (#16086) 2026-05-11 16:03:38 -07:00
utf8_buffer.go consolidate the tokenizer (#14327) 2026-02-19 15:55:45 -08:00
utf8_buffer_test.go consolidate the tokenizer (#14327) 2026-02-19 15:55:45 -08:00