CERBERUS/ollama
1
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-10-23 20:11:28 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 5
ollama/mlxrunner
History
Exact
Daniel Hiltgen 195f4cdca1
mlx: match publisher tokenizer semantics (#18779) 
* mlx: match publisher tokenizer semantics

Honor pretokenizer stage order, split behavior, Unicode boundaries,
added-token normalization, and ranked BPE merges. Handle empty added
tokens and empty Metaspace input consistently.

Add shared Go/Python reference cases using published tokenizers, pulling
missing models directly and failing on errors, plus focused regressions
for configuration precedence, byte fallback, and parallel encoding.

* address comments
2026-10-04 07:45:15 -07:00
..
batch mlx: System one support (#18701) 2026-10-03 13:17:21 -07:00
cache mlx: System one support (#18701) 2026-10-03 13:17:21 -07:00
model mlx: Decision model improvements (#18776) 2026-10-03 17:48:08 -07:00
nn mlx: speed up Qwen 3.8 prompt processing (#18550) 2026-09-22 17:53:23 -07:00
sample
tokenizer mlx: match publisher tokenizer semantics (#18779) 2026-10-04 07:45:15 -07:00
xgrammar xgrammar: add standalone CMake project for native library (#18611) 2026-09-23 11:20:59 -07:00
cache_trie.go
cache_trie_test.go
client.go server: apply structured outputs in a single pass on thinking models 2026-09-22 11:24:11 -07:00
client_score.go mlx: System one support (#18701) 2026-10-03 13:17:21 -07:00
client_score_test.go mlx: System one support (#18701) 2026-10-03 13:17:21 -07:00
client_test.go server: apply structured outputs in a single pass on thinking models 2026-09-22 11:24:11 -07:00
dflash.go mlxrunner, cmd, server: fold three small packages into their users 2026-09-16 14:06:08 -07:00
dflash_test.go mlxrunner, cmd, server: fold three small packages into their users 2026-09-16 14:06:08 -07:00
grammar.go mlx: System one support (#18701) 2026-10-03 13:17:21 -07:00
grammar_mask_test.go
grammar_test.go mlx: System one support (#18701) 2026-10-03 13:17:21 -07:00
media.go mlxrunner, cmd, server: fold three small packages into their users 2026-09-16 14:06:08 -07:00
media_test.go mlxrunner, cmd, server: fold three small packages into their users 2026-09-16 14:06:08 -07:00
mtp.go
mtp_test.go server: apply structured outputs in a single pass on thinking models 2026-09-22 11:24:11 -07:00
pipeline.go mlx: System one support (#18701) 2026-10-03 13:17:21 -07:00
pipeline_test.go mlxrunner, cmd, server: fold three small packages into their users 2026-09-16 14:06:08 -07:00
prefix_cache.go mlx: System one support (#18701) 2026-10-03 13:17:21 -07:00
prefix_cache_scenario_test.go
prefix_cache_test.go mlx: System one support (#18701) 2026-10-03 13:17:21 -07:00
runner.go mlx: System one support (#18701) 2026-10-03 13:17:21 -07:00
score.go mlx: System one support (#18701) 2026-10-03 13:17:21 -07:00
score_hidden_test.go mlx: System one support (#18701) 2026-10-03 13:17:21 -07:00
score_test.go mlx: System one support (#18701) 2026-10-03 13:17:21 -07:00
server.go mlx: System one support (#18701) 2026-10-03 13:17:21 -07:00
speculate.go mlxrunner, cmd, server: fold three small packages into their users 2026-09-16 14:06:08 -07:00
speculate_depth.go
speculate_depth_test.go
speculate_stats.go
status_memory.go
status_memory_test.go
utf8_buffer.go
utf8_buffer_test.go
verify.go mlxrunner, cmd, server: fold three small packages into their users 2026-09-16 14:06:08 -07:00
verify_test.go