CERBERUS/ollama
1
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-10-21 11:01:33 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 2
ollama/mlxrunner/tokenizer
History
Exact
Daniel Hiltgen 195f4cdca1
mlx: match publisher tokenizer semantics (#18779) 
* mlx: match publisher tokenizer semantics

Honor pretokenizer stage order, split behavior, Unicode boundaries,
added-token normalization, and ranked BPE merges. Handle empty added
tokens and empty Metaspace input consistently.

Add shared Go/Python reference cases using published tokenizers, pulling
missing models directly and failing on errors, plus focused regressions
for configuration precedence, byte fallback, and parallel encoding.

* address comments
2026-10-04 07:45:15 -07:00
..
testdata mlx, mlxrunner: move the MLX engine out of x/ 2026-09-16 14:06:08 -07:00
tokenizer.go mlx: match publisher tokenizer semantics (#18779) 2026-10-04 07:45:15 -07:00
tokenizer_added.go mlx: match publisher tokenizer semantics (#18779) 2026-10-04 07:45:15 -07:00
tokenizer_benchmark_test.go mlx, mlxrunner: move the MLX engine out of x/ 2026-09-16 14:06:08 -07:00
tokenizer_bpe.go mlx: match publisher tokenizer semantics (#18779) 2026-10-04 07:45:15 -07:00
tokenizer_bpe_spans_test.go mlx: match publisher tokenizer semantics (#18779) 2026-10-04 07:45:15 -07:00
tokenizer_config_test.go mlx: match publisher tokenizer semantics (#18779) 2026-10-04 07:45:15 -07:00
tokenizer_correctness_test.go mlx: match publisher tokenizer semantics (#18779) 2026-10-04 07:45:15 -07:00
tokenizer_decode.go mlx, mlxrunner: move the MLX engine out of x/ 2026-09-16 14:06:08 -07:00
tokenizer_encode.go mlx: match publisher tokenizer semantics (#18779) 2026-10-04 07:45:15 -07:00
tokenizer_ggml_parity_test.go mlx, mlxrunner: move the MLX engine out of x/ 2026-09-16 14:06:08 -07:00
tokenizer_load.go mlx: match publisher tokenizer semantics (#18779) 2026-10-04 07:45:15 -07:00
tokenizer_load_test.go mlx: match publisher tokenizer semantics (#18779) 2026-10-04 07:45:15 -07:00
tokenizer_normalization_test.go mlx: match publisher tokenizer semantics (#18779) 2026-10-04 07:45:15 -07:00
tokenizer_pretokenizer.go mlx: match publisher tokenizer semantics (#18779) 2026-10-04 07:45:15 -07:00
tokenizer_pretokenizer_test.go mlx: match publisher tokenizer semantics (#18779) 2026-10-04 07:45:15 -07:00
tokenizer_reference_benchmark_test.go mlx: match publisher tokenizer semantics (#18779) 2026-10-04 07:45:15 -07:00
tokenizer_reference_model_test.go mlx: match publisher tokenizer semantics (#18779) 2026-10-04 07:45:15 -07:00
tokenizer_reference_test.go mlx: match publisher tokenizer semantics (#18779) 2026-10-04 07:45:15 -07:00