Skip to content

LlamaCache.md

Latest commit

 

History

History
1535 lines (1027 loc) · 49.8 KB

LlamaCache.md

File metadata and controls

1535 lines (1027 loc) · 49.8 KB
Raw
title Llama Cache
module_name llama_cpp.llama_cache
source_file llama_cpp/llama_cache.py
last_updated 2026-09-17
version_target latest

Llama Cache

Overview

llama_cpp.llama_cache provides cache implementations for storing and restoring LlamaState objects or recurrent model state checkpoints.

The module is mainly used to speed up repeated inference workflows by reusing previously computed model state for matching token prefixes.

It defines several cache classes:

Class Purpose
BaseLlamaCache Abstract base class for llama.cpp state caches.
LlamaRAMCache In-memory LRU cache for LlamaState objects.
LlamaDiskCache Disk-backed cache using the diskcache library.
TrieNode Internal trie node used by LlamaTrieCache.
LlamaTrieCache Trie-based cache optimized for fast longest-prefix lookup.
HybridCheckpoint Dataclass representing one saved Hybrid/Recurrent checkpoint and its host-visible payload.
HybridCheckpointCache Checkpoint manager for Hybrid/Recurrent model state snapshots, with host and device-backed modes.

The public compatibility alias is:

LlamaCache = LlamaTrieCache

This means that code importing LlamaCache receives the trie-based cache implementation.

Defined in: llama_cpp/llama_cache.py

Related pages: Llama, Caching and state reuse.

Role in the API

The cache module provides reusable storage for model runtime state.

There are two main caching strategies:

  1. Token-prefix state caching

    Used by:

    • LlamaRAMCache
    • LlamaDiskCache
    • LlamaTrieCache
    • LlamaCache

    These caches map token sequences to llama_core.LlamaState objects. RAM and disk lookup select the cached key with the longest nonempty common prefix, even if that key later diverges from the query. Trie lookup requires the complete cached key to be a prefix of the query. All return the stored snapshot without truncating it.

  2. Hybrid / recurrent checkpoint caching

    Used by:

    • HybridCheckpoint
    • HybridCheckpointCache

    This is designed for Hybrid or recurrent models where rollback requires saving and restoring hidden state snapshots through low-level llama.cpp state APIs.

Public API Summary

API Type Public Description
BaseLlamaCache Abstract class Yes Base interface for cache implementations.
LlamaRAMCache Class Yes In-memory LRU cache with linear prefix lookup.
LlamaDiskCache Class Yes Disk-backed cache using diskcache.Cache.
LlamaTrieCache Class Yes Trie-based cache with efficient prefix lookup.
LlamaCache Alias Yes Backward-compatible alias for LlamaTrieCache.
HybridCheckpoint Dataclass Yes Represents one saved Hybrid/RNN checkpoint.
HybridCheckpointCache Class Yes Manages Hybrid/RNN state checkpoints.
TrieNode Class Internal Trie node used by LlamaTrieCache.

BaseLlamaCache

Overview

BaseLlamaCache is the abstract base class for llama.cpp cache implementations.

It defines a common dictionary-like interface for storing and retrieving llama_core.LlamaState objects by token sequence.

Subclasses are expected to implement:

  • cache_size
  • __getitem__
  • __contains__
  • __setitem__

Defined in: llama_cpp/llama_cache.py

Role in the API

BaseLlamaCache acts as the shared contract for cache implementations used by higher-level llama-cpp-python runtime code.

It is not intended to be used directly. Users should instantiate one of the concrete cache classes instead:

  • LlamaRAMCache
  • LlamaDiskCache
  • LlamaTrieCache
  • LlamaCache

Constructor: __init__

def __init__(self, capacity_bytes: int = (2 << 30)):
    ...
Parameter Type Default Required Description
capacity_bytes int 2 << 30 No Maximum cache capacity in bytes. The default is approximately 2 GiB.

Instance Variables

Name Type Description
capacity_bytes int Maximum allowed cache size in bytes. Concrete subclasses use this value to decide when eviction is required.

Properties

cache_size

@property
@abstractmethod
def cache_size(self) -> int:
    ...

Returns the current cache size in bytes.

Concrete implementations define how this value is calculated.

Core Methods

_find_longest_prefix_key

def _find_longest_prefix_key(
    self,
    key: Tuple[int, ...],
) -> Optional[Tuple[int, ...]]:
    ...

Finds the cached key with the longest token prefix matching the requested key.

In BaseLlamaCache, this method is only a placeholder and does not implement behavior.

Concrete subclasses may override it.

__getitem__

@abstractmethod
def __getitem__(self, key: Sequence[int]) -> "llama_core.LlamaState":
    ...

Retrieves a cached LlamaState.

The expected behavior is longest-prefix matching rather than strict exact-key lookup.

__contains__

@abstractmethod
def __contains__(self, key: Sequence[int]) -> bool:
    ...

Returns whether the cache contains a matching token prefix for the given key.

__setitem__

@abstractmethod
def __setitem__(
    self,
    key: Sequence[int],
    value: "llama_core.LlamaState"
) -> None:
    ...

Stores a LlamaState under a token sequence.

LlamaRAMCache

Overview

LlamaRAMCache is an in-memory cache for llama_core.LlamaState objects.

It stores token sequences in an OrderedDict and maintains an LRU eviction policy. Lookup selects the cached key with the longest nonempty common prefix with the query; the key need not be wholly contained in the query.

Defined in: llama_cpp/llama_cache.py

Role in the API

LlamaRAMCache is useful when users want fast in-process caching without writing state to disk.

It keeps all cached states in Python memory. This makes retrieval simple, but memory usage can grow quickly depending on the size of saved LlamaState objects.

Constructor: __init__

def __init__(self, capacity_bytes: int = (2 << 30), verbose: bool = False):
    ...
Parameter Type Default Required Description
capacity_bytes int 2 << 30 No Maximum total size of cached states in bytes.
verbose bool False No Whether to enable verbose behavior when computing token-prefix matches. This value is passed to Llama.longest_token_prefix.

Instance Variables

Name Type Description
capacity_bytes int Maximum cache capacity in bytes.
cache_state OrderedDict[Tuple[int, ...], llama_core.LlamaState] Stores cached token sequences and their corresponding LlamaState objects. The order is used for LRU eviction.
_current_size int Current total size of cached states in bytes.
verbose bool Passed to llama_core.Llama.longest_token_prefix during prefix comparison.

Properties

cache_size

@property
def cache_size(self):
    return self._current_size

Returns the current tracked memory usage of the cache in bytes.

Core Methods

_find_longest_prefix_key

def _find_longest_prefix_key(
    self,
    key: Tuple[int, ...],
) -> Optional[Tuple[int, ...]]:
    ...

Finds the cached token sequence with the longest prefix match against key.

This implementation scans every key in cache_state and calls:

llama_core.Llama.longest_token_prefix(k, key, self.verbose)

Complexity

Operation Complexity
Prefix lookup O(N * K)
LRU update O(1)
Size tracking O(1)

Where:

  • N is the number of cached entries.
  • K is the token sequence length.

__getitem__

def __getitem__(self, key: Sequence[int]) -> "llama_core.LlamaState":
    ...

Returns the cached LlamaState for the longest matching token prefix.

Behavior:

  1. Raises KeyError("Cache is empty") if the cache has no entries.
  2. Converts the input key to a tuple.
  3. Finds the cached key with the longest nonempty common prefix.
  4. Raises KeyError("Key not found") if no matching prefix exists.
  5. Moves the matched key to the end of cache_state to mark it as recently used.
  6. Returns the matched LlamaState.

__contains__

def __contains__(self, key: Sequence[int]) -> bool:
    ...

Returns True if any cached key is a prefix match for the requested token sequence.

Returns False if the cache is empty.

__setitem__

def __setitem__(self, key: Sequence[int], value: "llama_core.LlamaState"):
    ...

Stores a LlamaState in memory.

Behavior:

  1. Converts key to a tuple.
  2. If the key already exists, deletes the old entry.
  3. Inserts the new LlamaState.
  4. Adds value.nbytes to _current_size, falling back to value.llama_state_size for legacy state objects.
  5. Evicts least-recently-used entries while _current_size > capacity_bytes.
  6. Resets _current_size to 0 if the cache becomes empty.

Note: The current implementation increments _current_size by the new value size when replacing an existing key, but it does not subtract the old value size before deletion. This may cause size tracking to overcount replaced entries.

Example

from llama_cpp import Llama
from llama_cpp.llama_cache import LlamaRAMCache

llm = Llama(
    model_path="./models/model.gguf",
    cache=LlamaRAMCache(capacity_bytes=1 << 30),
)

response = llm("Q: What is llama.cpp?\nA:", max_tokens=64)

print(response["choices"][0]["text"])

Best Practices

  • Use LlamaRAMCache when cache speed is more important than persistence.
  • Keep capacity_bytes below available system memory.
  • Reuse the same cache instance across repeated prompts when prefix reuse is expected.
  • Prefer LlamaTrieCache or LlamaCache when many cached entries are expected and prefix lookup cost matters.

LlamaDiskCache

Overview

LlamaDiskCache is a disk-backed cache for llama_core.LlamaState objects.

It delegates storage, size limits, and eviction behavior to the external diskcache library without explicitly selecting an eviction policy.

Defined in: llama_cpp/llama_cache.py

Role in the API

LlamaDiskCache is useful when cached model states should persist beyond the current Python process or when RAM usage should be limited.

Compared with LlamaRAMCache, it may reduce memory pressure but can be slower due to disk I/O.

Constructor: __init__

def __init__(
    self,
    cache_dir: str = ".cache/llama_cache",
    capacity_bytes: int = (2 << 30),
    verbose: bool = False
):
    ...
Parameter Type Default Required Description
cache_dir str ".cache/llama_cache" No Directory used by diskcache.Cache to store cached state data.
capacity_bytes int 2 << 30 No Maximum disk cache size in bytes. Passed to diskcache.Cache(..., size_limit=capacity_bytes).
verbose bool False No Passed to Llama.longest_token_prefix when searching for the best prefix match.

Instance Variables

Name Type Description
cache_dir str Filesystem directory for the disk cache.
cache diskcache.Cache SQLite-backed disk cache object.
verbose bool Passed to token-prefix comparison logic.
capacity_bytes int Maximum configured cache capacity in bytes, inherited from BaseLlamaCache.

Properties

cache_size

@property
def cache_size(self):
    return self.cache.volume()

Returns the current disk cache volume in bytes using diskcache.Cache.volume().

Core Methods

_find_longest_prefix_key

def _find_longest_prefix_key(
    self,
    key: Tuple[int, ...],
) -> Optional[Tuple[int, ...]]:
    ...

Finds the cached key with the longest token-prefix match.

Behavior:

  1. Returns None immediately if the disk cache is empty.
  2. Iterates over self.cache.iterkeys().
  3. Uses llama_core.Llama.longest_token_prefix(k, key, self.verbose) to compare each cached key.
  4. Stops early if a perfect match is found.

Complexity

Operation Complexity
Prefix lookup O(N * K)
Disk iteration Depends on diskcache and filesystem
Exact-match early exit Supported

__getitem__

def __getitem__(self, key: Sequence[int]) -> "llama_core.LlamaState":
    ...

Retrieves the cached state associated with the longest matching prefix.

Behavior:

  1. Prints "LlamaDiskCache.__getitem__: called" to stderr.
  2. Raises KeyError("Cache is empty") if no entries exist.
  3. Converts key to a tuple.
  4. Finds the longest prefix key.
  5. Raises KeyError("Key not found") if no match exists.
  6. Reads and returns the cached LlamaState.

The read is non-destructive. Access tracking and eviction follow the effective diskcache settings; this wrapper does not explicitly enable an LRU policy.

__contains__

def __contains__(self, key: Sequence[int]) -> bool:
    ...

Returns whether the cache has any longest-prefix match for the given token sequence.

__setitem__

def __setitem__(self, key: Sequence[int], value: "llama_core.LlamaState"):
    ...

Stores a LlamaState in the disk cache.

Behavior:

  1. Prints "LlamaDiskCache.__setitem__: called" to stderr.
  2. Converts key to a tuple.
  3. Assigns the value to self.cache[tuple(key)].

diskcache handles capacity checks and eviction.

Example

from llama_cpp import Llama
from llama_cpp.llama_cache import LlamaDiskCache

cache = LlamaDiskCache(
    cache_dir=".cache/llama_cache",
    capacity_bytes=2 << 30,
)

llm = Llama(
    model_path="./models/model.gguf",
    cache=cache,
)

response = llm("Q: What is llama.cpp?\nA:", max_tokens=64)

print(response["choices"][0]["text"])

Best Practices

  • Use LlamaDiskCache when cache persistence is useful.
  • Place cache_dir on a fast local SSD when possible.
  • Avoid using slow network filesystems for high-throughput inference.
  • Consider LlamaTrieCache for workloads where many prefix lookups happen within a single process.

Common Pitfalls

  • Disk-backed caching can be slower than RAM caching.
  • The cache depends on the third-party diskcache package.
  • Prefix lookup still scans cached keys linearly, even though storage and eviction are handled by diskcache.
  • The implementation prints debug messages to stderr on get and set operations.

TrieNode

Overview

TrieNode is an internal helper class used by LlamaTrieCache.

Each node represents one position in a token-prefix tree.

Defined in: llama_cpp/llama_cache.py

Role in the API

TrieNode is not intended to be used directly by users.

It stores:

  • Child nodes keyed by token ID.
  • An optional LlamaState when the node marks the end of a cached token sequence.

Constructor: __init__

def __init__(self):
    ...

The constructor takes no parameters.

Instance Variables

Name Type Description
children Dict[int, TrieNode] Child trie nodes keyed by token ID.
state Optional[llama_core.LlamaState] Cached state stored at this node if the node represents a complete cached token sequence.

LlamaTrieCache

Overview

LlamaTrieCache is a trie-based cache implementation for llama_core.LlamaState objects.

It optimizes longest-prefix lookup by storing token sequences in a prefix tree rather than scanning all cached keys.

Defined in: llama_cpp/llama_cache.py

Role in the API

LlamaTrieCache is the preferred cache implementation for efficient prefix lookup.

It combines:

  • A trie for O(K) longest-prefix lookup.
  • An OrderedDict for O(1) LRU tracking.
  • Explicit byte-size tracking through _current_size.

The compatibility alias LlamaCache points to this class:

LlamaCache = LlamaTrieCache

Constructor: __init__

def __init__(self, capacity_bytes: int = (2 << 30)):
    ...
Parameter Type Default Required Description
capacity_bytes int 2 << 30 No Maximum cache size in bytes. Entries are evicted when tracked state size exceeds this value.

Instance Variables

Name Type Description
root TrieNode Root node of the token-prefix trie.
_current_size int Current total size of cached states in bytes.
lru_tracker OrderedDict[Tuple[int, ...], TrieNode] Tracks cached keys by recency. The value is the terminal TrieNode for that key.
capacity_bytes int Maximum cache capacity in bytes, inherited from BaseLlamaCache.

Properties

cache_size

@property
def cache_size(self) -> int:
    return self._current_size

Returns the current total size of cached states in bytes.

This is an O(1) operation.

Core Methods

_find_longest_prefix_node

def _find_longest_prefix_node(
    self,
    key: Tuple[int, ...]
) -> Tuple[Optional[TrieNode], Optional[Tuple[int, ...]]]:
    ...

Finds the trie node containing the longest cached prefix for the given token sequence.

Returns:

Tuple[Optional[TrieNode], Optional[Tuple[int, ...]]]

The first item is the matching trie node.

The second item is the matching cached key.

Behavior

  1. Starts at the root node.
  2. Checks whether the empty prefix has a cached state.
  3. Walks one token at a time through the trie.
  4. Updates the best match each time it reaches a node with a stored state.
  5. Stops when the token path no longer exists.

Complexity

Operation Complexity
Prefix lookup O(K)

Where K is the length of the requested token sequence.

__getitem__

def __getitem__(self, key: Sequence[int]) -> "llama_core.LlamaState":
    ...

Retrieves the LlamaState for the longest matching cached prefix.

Behavior:

  1. Converts key to a tuple.
  2. Finds the longest matching trie node.
  3. Raises KeyError if no prefix match exists.
  4. Moves the matched key to the end of lru_tracker.
  5. Returns the stored LlamaState.

__contains__

def __contains__(self, key: Sequence[int]) -> bool:
    ...

Returns True if any prefix of key is cached.

This lookup is O(K).

_prune

def _prune(self, key: Tuple[int, ...]):
    ...

Removes a cached key from the trie and prunes empty parent nodes.

This is an internal helper used during LRU eviction.

Behavior:

  1. Walks the trie path for the given key.
  2. Returns immediately if the key does not exist.
  3. Removes the stored state from the terminal node.
  4. Subtracts the state size from _current_size.
  5. Walks backward through the path and removes empty trie nodes.

__setitem__

def __setitem__(self, key: Sequence[int], value: "llama_core.LlamaState"):
    ...

Stores a LlamaState in the trie cache.

Behavior:

  1. Converts key to a tuple.
  2. Creates trie nodes for each token if needed.
  3. If the terminal node already has a state, subtracts the old state size.
  4. Stores the new state.
  5. Adds value.nbytes to _current_size, falling back to value.llama_state_size for legacy state objects.
  6. Updates lru_tracker.
  7. Evicts least-recently-used items while _current_size > capacity_bytes.

Example

from llama_cpp import Llama
from llama_cpp.llama_cache import LlamaCache

llm = Llama(
    model_path="./models/model.gguf",
    cache=LlamaCache(capacity_bytes=2 << 30),
)

response = llm("Q: What is llama.cpp?\nA:", max_tokens=64)

print(response["choices"][0]["text"])

Because LlamaCache is an alias for LlamaTrieCache, this example uses the trie-based cache.

Performance Characteristics

Cache Prefix Lookup LRU Tracking Storage
LlamaRAMCache O(N * K) O(1) RAM
LlamaDiskCache O(N * K) Delegated to diskcache Disk
LlamaTrieCache O(K) O(1) RAM

Where:

  • N is the number of cached entries.
  • K is the token sequence length.

Best Practices

  • Prefer LlamaCache for general use, because it currently aliases LlamaTrieCache.
  • Use LlamaTrieCache directly when you want explicit control over the cache implementation.
  • Use a realistic capacity_bytes value based on available RAM.
  • Use this cache when many prompts share prefixes.

Common Pitfalls

  • The cache still stores full LlamaState objects, which may be large.
  • RAM and trie capacity accounting uses LlamaState.nbytes: native state bytes plus owned token, score, and last-logit arrays. It excludes Python object and container overhead. Legacy objects fall back to llama_state_size. Disk usage instead comes from diskcache.Cache.volume().
  • TrieNode is internal and should not be manipulated directly.
  • Eviction removes entries from both lru_tracker and the trie.

LlamaCache

Overview

LlamaCache is a backward-compatible alias for LlamaTrieCache.

LlamaCache = LlamaTrieCache

This means users can import LlamaCache and receive the trie-based implementation.

Example

from llama_cpp import Llama
from llama_cpp.llama_cache import LlamaCache

cache = LlamaCache(capacity_bytes=2 << 30)

llm = Llama(
    model_path="./models/model.gguf",
    cache=cache,
)

Migration Notes

Older code may expect LlamaCache to refer to another cache implementation.

In the current source, LlamaCache resolves to LlamaTrieCache.

When documenting or debugging cache behavior, treat LlamaCache as equivalent to:

from llama_cpp.llama_cache import LlamaTrieCache as LlamaCache

HybridCheckpoint

Overview

HybridCheckpoint is a dataclass representing one saved snapshot of a Hybrid or Recurrent model state.

It is used by HybridCheckpointCache.

Defined in: llama_cpp/llama_cache.py

Role in the API

Hybrid or recurrent models may require sequence-state rollback rather than standard KV-cache truncation.

HybridCheckpoint stores the Python token position, native backend memory-position range, prefix verification hash, sequence id, and the serialized checkpoint payload visible to Python.

Its data field has different ownership semantics depending on the cache mode:

  • In host mode (on_device=False), data owns the host-serialized partial checkpoint; the attention prefix remains in the live context.
  • In device mode (on_device=True), data contains only the host-visible serialized portion. The large tensor payload is stored in llama_context-owned device buffers by llama.cpp, keyed by seq_id.

Dataclass Definition

@dataclass
class HybridCheckpoint:
    pos: int
    data: bytes
    hash_val: str
    size: int
    seq_id: int
    pos_min: int = -1
    pos_max: Optional[int] = None

Fields

Field Type Description
pos int Token position where this checkpoint was taken.
data bytes Serialized checkpoint payload visible to Python. In host mode this is the serialized partial state; in device mode this is only the host-visible portion.
hash_val str SHA-256 hash prefix used to verify exact token-prefix matching.
size int Number of bytes written by llama_state_seq_get_data_ext.
seq_id int Sequence id used by llama.cpp sequence-state APIs.
pos_min int Minimum native backend memory position covered by the checkpoint.
pos_max Optional[int] Maximum native backend memory position covered by the checkpoint. None is reserved for manually created legacy checkpoints.

Notes

HybridCheckpoint objects are normally created by HybridCheckpointCache.save_checkpoint.

Users usually do not need to instantiate this dataclass manually.

In device mode, old HybridCheckpoint Python objects may become stale if a newer checkpoint is saved for the same seq_id, because the device-side tensor payload is keyed by seq_id and may be overwritten.

HybridCheckpointCache

Overview

HybridCheckpointCache manages Hybrid/Recurrent model state checkpoints.

It is designed for models whose memory cannot always be safely truncated like a regular Transformer KV cache. Instead, rollback is implemented by saving and restoring sequence-state snapshots through llama.cpp state APIs.

The cache supports two operating modes:

  1. Host mode (on_device=False)

    • Partial checkpoint payloads are materialized as Python-owned bytes.
    • Multiple historical checkpoints per seq_id can coexist while their native prefixes remain valid.
    • This is the default mode and is useful for multi-turn rollback or deeper prefix reuse.

  2. Device mode (on_device=True)

    • LLAMA_STATE_SEQ_FLAGS_ON_DEVICE is forwarded to llama.cpp.
    • Tensor payloads are stored in llama_context-owned device buffers.
    • Python keeps only the host-visible serialized portion.
    • Only one active checkpoint per seq_id is safe because device payloads are keyed by seq_id.
    • This mode can reduce device-to-host copy overhead during checkpoint save/restore.

Defined in: llama_cpp/llama_cache.py

Role in the API

HybridCheckpointCache is a specialized cache manager for Hybrid/Recurrent model rollback.

It stores host-visible checkpoint data extracted from the llama.cpp backend through low-level C API functions:

  • llama_state_seq_get_size_ext
  • llama_state_seq_get_data_ext
  • llama_state_seq_set_data_ext

When on_device=True, tensor payloads are not treated as Python-owned bytes. They are stored by llama.cpp in llama_context-owned device buffers, while Python keeps the host-visible serialized portion and checkpoint metadata.

It is not a drop-in replacement for LlamaRAMCache, LlamaDiskCache, or LlamaTrieCache.

Constructor: __init__

def __init__(
    self,
    ctx: llama_cpp_lib.llama_context_p,
    max_checkpoints: int = 16,
    on_device: bool = False,
    verbose: bool = False
):
    ...
Parameter Type Default Required Description
ctx llama_cpp_lib.llama_context_p — Yes Borrowed low-level llama.cpp context pointer used for sequence-state save/restore. The cache does not own or free this context.
max_checkpoints int 16 No Maximum number of Python-side checkpoint entries to retain. If set to 0 or below, checkpointing is disabled.
on_device bool False No Whether to request llama.cpp to store checkpoint tensor payloads in llama_context-owned device buffers via LLAMA_STATE_SEQ_FLAGS_ON_DEVICE.
verbose bool False No Enables diagnostic messages printed to stderr.

Constructor Behavior

The constructor raises ValueError if ctx is None.

If max_checkpoints <= 0, checkpointing is disabled. In verbose mode, the cache reports that rollback capabilities are turned off. This mode is intended to avoid expensive state extraction for single-turn workflows.

When on_device=True, the cache forwards LLAMA_STATE_SEQ_FLAGS_ON_DEVICE to llama.cpp. In this mode, the cache keeps only one active checkpoint per seq_id by replacing old Python-side checkpoint metadata before saving a new checkpoint for the same seq_id.

Instance Variables

Name Type Description
_ctx llama_cpp_lib.llama_context_p Borrowed llama.cpp context pointer used for state extraction and restoration.
on_device bool Whether LLAMA_STATE_SEQ_FLAGS_ON_DEVICE is forwarded to llama.cpp state APIs.
verbose bool Enables debug output.
max_checkpoints int Maximum number of Python-side checkpoint entries retained. Values less than or equal to zero disable checkpointing.
checkpoints list[HybridCheckpoint] Python-side checkpoint registry. In host mode, entries own full checkpoint payloads. In device mode, entries own only host-visible metadata/payload portions.
_current_size int Python-tracked host-visible checkpoint size in bytes. In device mode, this does not include llama_context-owned device tensor storage.
_get_size_ext Callable Cached reference to llama_state_seq_get_size_ext.
_get_data_ext Callable Cached reference to llama_state_seq_get_data_ext.
_set_data_ext Callable Cached reference to llama_state_seq_set_data_ext.
_flags int Combined llama.cpp sequence-state flags, always including LLAMA_STATE_SEQ_FLAGS_PARTIAL_ONLY and optionally LLAMA_STATE_SEQ_FLAGS_ON_DEVICE.

Properties

cache_size

@property
def cache_size(self) -> int:
    return self._current_size

Returns the Python-tracked host-visible checkpoint size in bytes.

In host mode, this is close to the full serialized checkpoint payload size.

In device mode, this reports only the host-visible portion returned by llama.cpp. It does not include llama_context-owned device tensor storage.

Core Methods

clear

def clear(self):
    ...

Clears Python-side checkpoint metadata and resets _current_size to 0.

If the checkpoint list is already empty, it returns immediately.

In device mode, this does not explicitly release llama_context-owned device buffers. Those buffers are managed by llama.cpp and are associated with the context.

In verbose mode, it prints:

HybridCheckpointCache(clear): cleared

close

def close(self):
    ...

Releases Python-side checkpoint metadata and detaches cached references held by the cache.

Behavior:

  • Calls clear().
  • Sets _ctx to None.
  • Sets cached C API function references to None.

This method does not free the llama.cpp context itself, because the context is borrowed rather than owned by the cache.

__del__

def __del__(self) -> None:
    self.close()

Finalizer that calls close.

_hash_prefix

def _hash_prefix(self, tokens: List[int], length: int) -> str:
    ...

Computes a SHA-256 hash for the token prefix up to length.

Behavior:

  1. Returns "empty" if length <= 0.
  2. Clamps length to the actual token list length.
  3. Converts the selected token prefix into an array.array('i').
  4. Hashes the bytes with SHA-256.
  5. Returns the first 32 hex characters.

This hash is used to ensure checkpoints are restored only when the token prefix exactly matches.

Context mutation and checkpoint lifetime

Llama registers its hybrid cache with the owning LlamaContext through weak references. This registration neither keeps a discarded cache alive nor extends the native context's lifetime. Closing the context closes its registered caches before freeing native resources.

Wrapped memory clears and state loads invalidate affected checkpoints. Sequence removal preserves an earlier checkpoint only when its recorded native range is strictly before the removed suffix; shifts, copies, and sequence retention also invalidate affected entries. Device capture invalidates older device snapshots for the same sequence across registered caches before overwriting the slot.

These notifications cover registered caches on the same context. Raw C API calls, automatic SWA eviction, and dependencies on another context's shared KV memory are not automatically tracked. Advanced callers performing these operations must coordinate cache invalidation themselves. A cache constructed from a raw context pointer is not automatically registered.

_evict_checkpoints_if_needed

def _evict_checkpoints_if_needed(self) -> None:
    ...

Evicts old checkpoint entries using FIFO order until len(checkpoints) <= max_checkpoints.

In host mode, this evicts full Python-owned checkpoint payloads.

In device mode, this evicts Python-side checkpoint metadata only. Device tensor payloads are owned by llama_context.

Behavior:

  1. Checks whether the number of checkpoints exceeds max_checkpoints.
  2. Pops the oldest checkpoint entry from the front of the list.
  3. Subtracts its size from _current_size.
  4. Clamps _current_size to 0 if needed.
  5. Prints an eviction message in verbose mode.

find_best_checkpoint

def find_best_checkpoint(
    self,
    tokens: List[int],
    seq_id: int = 0
) -> Optional[HybridCheckpoint]:
    ...

Finds the longest valid checkpoint matching the given token prefix and sequence id.

The hash check prevents restoring a checkpoint that has the same length but belongs to a different prompt/history.

Returns None if:

  • Checkpointing is disabled.
  • There are no checkpoints.
  • No checkpoint matches the requested sequence id and token prefix.

Behavior:

  1. Returns immediately if max_checkpoints <= 0 or no checkpoints exist.
  2. Skips checkpoints whose seq_id differs from the requested seq_id.
  3. Skips checkpoints whose pos is greater than the current token length.
  4. Verifies token-prefix integrity using _hash_prefix.
  5. Returns the checkpoint with the largest matching pos.

save_checkpoint

def save_checkpoint(
    self,
    current_pos: int,
    tokens: List[int],
    seq_id: int = 0
) -> bool:
    ...

Extracts the current Hybrid/Recurrent model state from the C++ backend and stores it as a HybridCheckpoint.

Returns True if the checkpoint was saved successfully.

Returns False if:

  • Checkpointing is disabled.
  • The backend reports state size 0.
  • State extraction writes an unexpected number of bytes.

Behavior

  1. Returns immediately if max_checkpoints <= 0.
  2. In device mode, invalidates older device checkpoints for the same seq_id, including those in other registered caches on this context.
  3. Uses _flags to select partial-only state serialization, optionally with LLAMA_STATE_SEQ_FLAGS_ON_DEVICE.
  4. Calls _get_size_ext to query the required host-visible buffer size.
  5. Allocates a ctypes.c_uint8 buffer.
  6. Calls _get_data_ext to extract the host-visible checkpoint data.
  7. Copies the data into a Python bytes object.
  8. Computes a hash of the token prefix.
  9. Appends a new HybridCheckpoint.
  10. Increments _current_size.
  11. Evicts old checkpoint entries using FIFO order if the number of entries exceeds max_checkpoints.

Important Performance Note

The implementation intentionally bypasses checkpoint extraction when max_checkpoints <= 0.

This avoids potentially large synchronous checkpoint extraction costs for single-turn workflows.

When on_device=True, llama.cpp may keep large tensor payloads in context-owned device buffers instead of materializing them as Python-owned bytes. This can reduce device-to-host tensor copy overhead, but only one active checkpoint per seq_id is safe.

restore_checkpoint

def restore_checkpoint(
    self,
    cp: HybridCheckpoint,
    seq_id: int = 0
) -> bool:
    ...

Restores a previously saved checkpoint into the C++ backend.

Returns True if restoration succeeds.

Returns False if:

  • The checkpoint sequence id does not match the requested seq_id.
  • The cache is closed, or the exact checkpoint object is no longer tracked by it, in either host or device mode.
  • The current backend state size differs from the checkpoint size.
  • The backend does not report the expected number of restored bytes.
  • The backend cannot remove the memory suffix after the restored native pos_max.

Behavior

  1. Verifies cp.seq_id == seq_id.
  2. Rejects a closed cache or a checkpoint whose object identity is absent from its registry.
  3. Queries current expected host-visible state size from the backend.
  4. Verifies it matches cp.size.
  5. Copies checkpoint bytes into a ctypes buffer.
  6. Calls _set_data_ext to restore the state.
  7. Removes the remaining attention-memory suffix beginning at cp.pos_max + 1.
  8. On success, invalidates checkpoints at or beyond the restored suffix across registered caches on the same context.
  9. If native restoration or suffix removal fails, invalidates affected sequence checkpoints because native state may already have changed. Exceptions from these operations are propagated. Preliminary rejection in steps 1–4 does not invalidate current checkpoints.
  10. Returns whether both state restoration and suffix removal succeeded. High-level callers reset or rebuild after failure.

The native memory range is stored separately from pos because token counts do not always map one-to-one to backend positions, notably for multimodal inputs, custom position IDs, and SWA models.

Stale Checkpoint Guard

In device mode, Python does not own the full checkpoint tensor payload. The large tensor payload is stored inside llama_context device buffers keyed by seq_id.

If a newer checkpoint is saved for the same seq_id, an older HybridCheckpoint Python object may still exist outside the cache, but its device-side tensor payload may have been overwritten.

Host checkpoints also depend on attention memory that remains in the live context: they use PARTIAL_ONLY, not a complete LlamaState snapshot. restore_checkpoint therefore requires the exact object to remain registered in both modes. Retaining a Python reference does not preserve its validity after clear, eviction, invalidation, or closure. Matching serialized sizes alone does not establish that the required prefix is still available.

Disabled Dictionary Interface

HybridCheckpointCache inherits from BaseLlamaCache, but it intentionally disables the dictionary-style methods.

__getitem__

def __getitem__(self, key):
    raise NotImplementedError(
        "HybridCheckpointCache: pls use save_checkpoint or restore_checkpoint method"
    )

__setitem__

def __setitem__(self, key, value):
    raise NotImplementedError(
        "HybridCheckpointCache: pls use save_checkpoint or restore_checkpoint method"
    )

__contains__

def __contains__(self, key):
    raise NotImplementedError(
        "HybridCheckpointCache: pls use save_checkpoint or restore_checkpoint method"
    )

Users should use checkpoint-specific methods instead.

Example: Host-backed Checkpoints

from llama_cpp.llama_cache import HybridCheckpointCache

# `ctx` must be a valid llama.cpp context pointer.
checkpoint_cache = HybridCheckpointCache(
    ctx=ctx,
    max_checkpoints=16,
    on_device=False,
    verbose=True,
)

tokens = [1, 2, 3, 4]
current_pos = len(tokens)

saved = checkpoint_cache.save_checkpoint(
    current_pos=current_pos,
    tokens=tokens,
    seq_id=0,
)

if saved:
    checkpoint = checkpoint_cache.find_best_checkpoint(tokens, seq_id=0)

    if checkpoint is not None:
        restored = checkpoint_cache.restore_checkpoint(checkpoint, seq_id=0)
        print("Restored:", restored)

Host mode owns the serialized partial payload in Python bytes. Multiple historical checkpoints per seq_id can coexist while their required native prefixes remain valid.

Example: Device-backed Checkpoints

from llama_cpp.llama_cache import HybridCheckpointCache

# `ctx` must be a valid llama.cpp context pointer.
checkpoint_cache = HybridCheckpointCache(
    ctx=ctx,
    max_checkpoints=16,
    on_device=True,
    verbose=True,
)

tokens = [1, 2, 3, 4]
current_pos = len(tokens)

saved = checkpoint_cache.save_checkpoint(
    current_pos=current_pos,
    tokens=tokens,
    seq_id=0,
)

if saved:
    checkpoint = checkpoint_cache.find_best_checkpoint(tokens, seq_id=0)

    if checkpoint is not None:
        restored = checkpoint_cache.restore_checkpoint(checkpoint, seq_id=0)
        print("Restored:", restored)

In device mode, llama.cpp owns the large tensor payload in context-owned device buffers. Python keeps only the host-visible checkpoint data and metadata.

Only one active checkpoint per seq_id is safe.

Note: These examples assume ctx is already available from lower-level llama.cpp runtime code. Most high-level users do not manually create this cache. Instead, they configure it through the Llama constructor using ctx_checkpoints, checkpoint_interval, and checkpoint_on_device.

Best Practices

  • Use HybridCheckpointCache only for Hybrid or recurrent model workflows that require hidden-state rollback.
  • Keep on_device=False when you need multiple historical checkpoints for the same seq_id.
  • Use on_device=True when reducing device-to-host checkpoint copy overhead is more important than keeping many historical checkpoint payloads. Retain the checkpoint object so its sequence id and native memory-position range remain available for restoration.
  • Set max_checkpoints=0 for single-turn workflows where rollback is not needed.
  • Keep max_checkpoints small if checkpoint states are large.
  • Use find_best_checkpoint before calling restore_checkpoint.
  • Do not restore checkpoint objects removed from the cache, even if another Python reference still keeps them alive.
  • Do not use dictionary-style cache access with this class.

Common Pitfalls

  • Passing ctx=None raises ValueError.
  • max_checkpoints <= 0 disables checkpointing.
  • Restoring a checkpoint with the wrong seq_id fails.
  • Restore fails if the current backend state size no longer matches the checkpoint size.
  • Both host and device objects become stale after invalidation or eviction. Device capture additionally overwrites the previous slot for that sequence.
  • In device mode, cache_size does not include llama_context-owned device tensor storage.
  • clear() removes Python-side checkpoint metadata but does not explicitly free llama.cpp-owned device buffers.
  • close() detaches internal references; the object should not be reused afterward.
  • This class is not equivalent to LlamaCache.

Module Variables and Constants

LlamaCache

LlamaCache = LlamaTrieCache

Backward-compatible alias for LlamaTrieCache.

Users can import either:

from llama_cpp.llama_cache import LlamaCache

or:

from llama_cpp.llama_cache import LlamaTrieCache

Both refer to the trie-based cache implementation in the current source.

How the Cache Implementations Compare

Class Storage Prefix Lookup Eviction Persistence Best For
LlamaRAMCache RAM O(N * K) LRU No Small in-memory caches.
LlamaDiskCache Disk O(N * K) Delegated to diskcache Yes Persistent cache across runs.
LlamaTrieCache RAM O(K) LRU No Fast prefix lookup with many cached entries.
HybridCheckpointCache RAM Hash-verified checkpoint search FIFO by checkpoint count No Hybrid/Recurrent model rollback.

Recommended Entry Points

For most users:

from llama_cpp.llama_cache import LlamaCache

This currently gives the trie-based implementation.

For explicit cache selection:

from llama_cpp.llama_cache import LlamaRAMCache
from llama_cpp.llama_cache import LlamaDiskCache
from llama_cpp.llama_cache import LlamaTrieCache

For Hybrid/Recurrent models:

from llama_cpp.llama_cache import HybridCheckpointCache

Related Links