mirror of
https://github.com/ollama/ollama.git
synced 2026-07-11 02:04:06 +00:00
5c73c4e2ee
The KV cache previously used a tree structure which could store multiple divergent sequences, which is good for cache reuse. However, this is typically used in conjunction with paged attention so each node in the tree can store just a chunk of the KV cache and they can be stitched together later. We don't currently do this, so the cache was storing copies of the full cache for each past sequence. This redundancy plus the lack of resource limits, caused significant memory use as a conversation grew. Instead, this changes to store a single entry for the cache, which can be prefix matched. Although it is less ideal for multiple users, it largely matches Ollama's current behavior. It can be improved as additional pieces are fleshed out. |
||
|---|---|---|
| .. | ||
| cache | ||
| mlx | ||
| model | ||
| sample | ||
| cache.go | ||
| client.go | ||
| imports.go | ||
| pipeline.go | ||
| runner.go | ||
| server.go | ||
| server_stub.go | ||
| utf8_buffer.go | ||
| utf8_buffer_test.go | ||