vllm.v1.kv_cache_interface
¶
Classes:
-
AttentionSpec– -
HiSparseHotPageSpec–Hot-cache pages, which kernels read as flat rows.
-
HiSparsePageSpec–Raw pages HiSparse lays out itself (hot and GPU-resident caches).
-
HiddenStateCacheSpec–Marker for hidden-state cache layers used by extract_hidden_states.
-
KVCacheConfig–The KV cache configuration of a model.
-
KVCacheGroup–Represents a group of model layers that share the same KV cache block table.
-
KVCacheLayerConfig–What one layer needs from the KV cache: the manager that allocates its
-
KVCacheSpec–A base class for specifying the KV cache format of one layer.
-
KVCacheTensor–A class for specifying how the workers should initialize the KV cache.
-
KVQuantMode–KV cache quantization mode.
-
MLAAttentionSpec– -
MambaSpec– -
ManagerConfig–Allocation policy of a layer: the
SingleTypeKVCacheManagerthat -
PrefixLookupGroup–KV cache groups looked up jointly for cache hits: they share a manager
Functions:
-
compute_layer_kv_cache_shape_bytes–Return the 4D logical shape
(B, H, N, C)where C is in bytes. -
compute_layout_strides–Byte strides in logical
[L, B, H, N, C]axis order. -
create_kv_cache_views–View a flat int8 buffer as one 4D
[B, H, N, C]view per layer. -
get_block_table_width–Return the width after optional alignment.
-
get_kv_quant_mode–Map a
kv_cache_dtypestring to aKVQuantMode. -
get_mamba_prefill_checkpoint_position–Return the reusable Mamba checkpoint boundary for a prefill.
-
group_kernel_blocks–View a kernel-block-granular layer cache with manager blocks as dim 0.
-
is_mamba_prefill_checkpoint_valid–Whether a backend can export the checkpoint in this query.
-
kv_cache_uses_per_token_head_scales–Return True if kv_cache_dtype needs per-token-head scales.
-
prefix_lookup_groups–Partition the prefix-cacheable groups into joint cache-hit lookups.
AttentionSpec
dataclass
¶
Bases: KVCacheSpec
Attributes:
-
kv_head_major_slots(bool) –Whether
num_head_slotsslots are grouped by KV head (H2ND: head -
max_tp_shards(int | None) –Distinct shards this cache splits into across TP; TP ranks beyond this
-
num_head_slots(int | None) –H of the logical
[B, H, N, C]page when packing diverges from one -
real_page_size_bytes(int) –Alias of
unpadded_page_size_bytes -
state_content_bytes(int | None) –C in bytes when packed; None means dense K/V content.
-
state_content_size_bytes(int) –Bytes per (head slot, stored state) cell of the page.
-
tokens_per_state(int | Fraction) –Tokens covered by one stored state. Ints > 1 compress multiple tokens
Source code in vllm/v1/kv_cache_interface.py
kv_head_major_slots = False
class-attribute
instance-attribute
¶
Whether num_head_slots slots are grouped by KV head (H2ND: head
h's K and V slots are adjacent), so every KV head is one contiguous run
of the page and the page re-shards across TP by byte offset.
max_tp_shards = None
class-attribute
instance-attribute
¶
Distinct shards this cache splits into across TP; TP ranks beyond this hold replicas. None: unknown, treated as not replicated.
num_head_slots = None
class-attribute
instance-attribute
¶
H of the logical [B, H, N, C] page when packing diverges from one
slot per KV head. None means one slot per KV head. Published by the backend.
real_page_size_bytes
property
¶
Alias of unpadded_page_size_bytes
TODO(lucas): follow up with TPU backend to see if we can remove this property.
state_content_bytes = None
class-attribute
instance-attribute
¶
C in bytes when packed; None means dense K/V content.
state_content_size_bytes
property
¶
Bytes per (head slot, stored state) cell of the page.
tokens_per_state = 1
class-attribute
instance-attribute
¶
Tokens covered by one stored state. Ints > 1 compress multiple tokens
into one state (DSv4 sparse MLA); fractions < 1 store multiple states per
token (Whisper block pooling: Fraction(1, block_pool_size)).
HiSparseHotPageSpec
dataclass
¶
Bases: HiSparsePageSpec
Hot-cache pages, which kernels read as flat rows.
Source code in vllm/v1/kv_cache_interface.py
HiSparsePageSpec
dataclass
¶
Bases: KVCacheSpec
Raw pages HiSparse lays out itself (hot and GPU-resident caches).
Source code in vllm/v1/kv_cache_interface.py
HiddenStateCacheSpec
dataclass
¶
Bases: MLAAttentionSpec
Marker for hidden-state cache layers used by extract_hidden_states.
Source code in vllm/v1/kv_cache_interface.py
KVCacheConfig
dataclass
¶
The KV cache configuration of a model.
Methods:
-
num_blocks_of–Number of blocks addressable by the pool backing
tensor. -
select_transfer_block_ids–Select block IDs for externally transferable cache groups.
Attributes:
-
cache_hit_alignment_tokens(int | None) –Token granularity of prefix-cache hits, resolved by the engine core.
-
has_mixed_precision_kv_cache(bool) –Whether device attention caches use more than one precision.
-
hash_block_size(int | None) –Tokens per prefix-cache block hash, resolved by the engine core.
-
hisparse_host_block_stride(int | None) –Physical bytes between consecutive HiSparse host blocks.
-
hisparse_host_num_blocks(int | None) –Capacity of the dedicated HiSparse host-block manager, when enabled.
-
hisparse_shared_host_pool(bool) –Whether local TP ranks share one physical HiSparse host pool.
-
kv_cache_groups(list[KVCacheGroup]) –The kv cache groups of the model.
-
kv_cache_layout(str | None) –The KV cache layout resolved by the engine core, adopted by all workers.
-
kv_cache_tensors(list[KVCacheTensor]) –How should model runner initialize the KV cache tensors for each layer
-
kv_tp_replicas(int) –Consecutive TP ranks holding identical KV for every layer (1: none).
-
needs_kv_cache_zeroing(bool) –Whether newly allocated attention blocks must be zeroed before use.
-
num_blocks(int) –The number of KV cache blocks
-
prefix_cache_retention_interval(int | None) –Resolved retention policy for local prefix-cache checkpoints.
-
prefix_cacheable_group_ids(tuple[int, ...]) –IDs of transferable groups eligible for hash-addressed stores.
-
prefix_cacheable_groups(tuple[KVCacheGroup, ...]) –Transferable groups eligible for hash-addressed stores.
-
transfer_group_ids(tuple[int, ...]) –IDs of cache groups that participate in external KV transfer.
-
transfer_group_index_by_layer(dict[str, int]) –Transfer-group tuple index for each participating layer.
-
transfer_groups(tuple[KVCacheGroup, ...]) –Cache groups that participate in external KV transfer.
Source code in vllm/v1/kv_cache_interface.py
842 843 844 845 846 847 848 849 850 851 852 853 854 855 856 857 858 859 860 861 862 863 864 865 866 867 868 869 870 871 872 873 874 875 876 877 878 879 880 881 882 883 884 885 886 887 888 889 890 891 892 893 894 895 896 897 898 899 900 901 902 903 904 905 906 907 908 909 910 911 912 913 914 915 916 917 918 919 920 921 922 923 924 925 926 927 928 929 930 931 932 933 934 935 936 937 938 939 940 941 942 943 944 945 946 947 948 949 950 951 952 953 954 955 956 957 958 959 960 961 962 963 964 965 966 967 968 969 970 971 972 973 974 975 976 977 | |
cache_hit_alignment_tokens = None
class-attribute
instance-attribute
¶
Token granularity of prefix-cache hits, resolved by the engine core.
has_mixed_precision_kv_cache
property
¶
Whether device attention caches use more than one precision.
hash_block_size = None
class-attribute
instance-attribute
¶
Tokens per prefix-cache block hash, resolved by the engine core.
hisparse_host_block_stride = None
class-attribute
instance-attribute
¶
Physical bytes between consecutive HiSparse host blocks.
hisparse_host_num_blocks = None
class-attribute
instance-attribute
¶
Capacity of the dedicated HiSparse host-block manager, when enabled.
hisparse_shared_host_pool = False
class-attribute
instance-attribute
¶
Whether local TP ranks share one physical HiSparse host pool.
kv_cache_groups
instance-attribute
¶
The kv cache groups of the model.
For models with only one type of attention, there is only one group that
contains all layers.
For models with multiple types of attention, there will be multiple groups,
see _get_kv_cache_config_uniform_page_size for more details.
kv_cache_layout = None
class-attribute
instance-attribute
¶
The KV cache layout resolved by the engine core, adopted by all workers.
kv_cache_tensors
instance-attribute
¶
How should model runner initialize the KV cache tensors for each layer
kv_tp_replicas = 1
class-attribute
instance-attribute
¶
Consecutive TP ranks holding identical KV for every layer (1: none).
needs_kv_cache_zeroing = field(init=False)
class-attribute
instance-attribute
¶
Whether newly allocated attention blocks must be zeroed before use.
Resolved across workers before scheduler projection and worker initialization.
num_blocks
instance-attribute
¶
The number of KV cache blocks
prefix_cache_retention_interval = None
class-attribute
instance-attribute
¶
Resolved retention policy for local prefix-cache checkpoints.
prefix_cacheable_group_ids
cached
property
¶
IDs of transferable groups eligible for hash-addressed stores.
prefix_cacheable_groups
cached
property
¶
Transferable groups eligible for hash-addressed stores.
transfer_group_ids
cached
property
¶
IDs of cache groups that participate in external KV transfer.
transfer_group_index_by_layer
cached
property
¶
Transfer-group tuple index for each participating layer.
transfer_groups
cached
property
¶
Cache groups that participate in external KV transfer.
num_blocks_of(tensor)
¶
Number of blocks addressable by the pool backing tensor.
Source code in vllm/v1/kv_cache_interface.py
select_transfer_block_ids(block_ids)
¶
Select block IDs for externally transferable cache groups.
Source code in vllm/v1/kv_cache_interface.py
KVCacheGroup
dataclass
¶
Represents a group of model layers that share the same KV cache block table. These layers are regarded as one layer in the KV cache manager.
Methods:
-
block_table_width–max_num_blocks_per_req(which accounts for DCP sharding: attention -
from_layers–A group of layers sharing a manager policy and block size.
-
max_allocated_blocks_per_request–Blocks one request at
max_model_lenholds in this group. -
max_num_blocks_per_req–The row length of the worker-side block table for this group, sized
Attributes:
-
distinct_specs(tuple[KVCacheSpec, ...]) –The distinct specs of the whole group's layers, for what kinds of
-
page_size_bytes(int) –Bytes one block of the group occupies across all its layers.
Source code in vllm/v1/kv_cache_interface.py
687 688 689 690 691 692 693 694 695 696 697 698 699 700 701 702 703 704 705 706 707 708 709 710 711 712 713 714 715 716 717 718 719 720 721 722 723 724 725 726 727 728 729 730 731 732 733 734 735 736 737 738 739 740 741 742 743 744 745 746 747 748 749 750 751 752 753 754 755 756 757 758 759 760 761 762 763 764 765 766 767 768 769 770 771 772 773 774 775 776 777 778 779 780 781 782 783 784 785 786 787 788 789 790 791 792 793 794 795 796 797 798 799 800 801 802 803 804 | |
distinct_specs
property
¶
The distinct specs of the whole group's layers, for what kinds of
cache it holds. The same on every pipeline stage, unlike
kv_cache_specs.
page_size_bytes
property
¶
Bytes one block of the group occupies across all its layers.
block_table_width(vllm_config, max_len)
¶
max_num_blocks_per_req (which accounts for DCP sharding: attention
KV is sharded, recurrent state replicated) padded to the manager's
block-table token alignment.
Source code in vllm/v1/kv_cache_interface.py
from_layers(layer_configs, **kwargs)
classmethod
¶
A group of layers sharing a manager policy and block size.
Source code in vllm/v1/kv_cache_interface.py
max_allocated_blocks_per_request(vllm_config)
¶
Blocks one request at max_model_len holds in this group.
Source code in vllm/v1/kv_cache_interface.py
max_num_blocks_per_req(vllm_config, max_len=None)
¶
The row length of the worker-side block table for this group, sized
for max_len tokens (default: max_model_len).
Source code in vllm/v1/kv_cache_interface.py
KVCacheLayerConfig
dataclass
¶
What one layer needs from the KV cache: the manager that allocates its blocks, that manager's policy, and the layer's tensor layout.
Attributes:
Source code in vllm/v1/kv_cache_interface.py
is_draft = field(default=False, kw_only=True, compare=False)
class-attribute
instance-attribute
¶
Whether a draft model owns this layer; excluded from equality so it never affects grouping.
KVCacheSpec
dataclass
¶
A base class for specifying the KV cache format of one layer.
Methods:
-
copy_with_new_block_size–Create a new KVCacheSpec from self but replacing the block size.
-
get_block_stride_alignment–Bytes the stride between physical blocks must be a multiple of.
Attributes:
-
block_stride_alignment(int | MultipleOf | None) –Required byte alignment between physical blocks, including packed layers.
-
dcp_sharded(bool) –Whether DCP shards this cache's token positions across ranks.
-
has_layer_views(bool) –Whether generic allocation creates per-layer cache views.
-
page_size_bytes(int) –The size of a page with
block_sizetokens in bytes. -
uses_slot_mapping(bool) –Whether the worker computes a per-token slot mapping for this cache.
Source code in vllm/v1/kv_cache_interface.py
188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 | |
block_stride_alignment = field(default=None, kw_only=True)
class-attribute
instance-attribute
¶
Required byte alignment between physical blocks, including packed layers.
MultipleOf(n): whole kernel blocks, when blocks are split into kernel
blocks of up to n tokens.
dcp_sharded = field(default=False, kw_only=True)
class-attribute
instance-attribute
¶
Whether DCP shards this cache's token positions across ranks.
has_layer_views
property
¶
Whether generic allocation creates per-layer cache views.
Specs without a per-layer shape keep the raw backing tensor and lay it out themselves. They also have no attention module of their own, so they take no part in attention-backend or kernel-block selection.
page_size_bytes
property
¶
uses_slot_mapping
property
¶
Whether the worker computes a per-token slot mapping for this cache. A cache written at slots that already address its pages takes none.
copy_with_new_block_size(block_size)
¶
get_block_stride_alignment()
¶
Bytes the stride between physical blocks must be a multiple of.
Source code in vllm/v1/kv_cache_interface.py
KVCacheTensor
dataclass
¶
A class for specifying how the workers should initialize the KV cache.
Placement of a set of same-shaped layers in the KV cache allocation.
Layer layers[l]'s page for block b starts at
offset + l * layer_stride + b * block_stride bytes into the backing
allocation of size bytes. Layer-outermost layouts give each layer a
contiguous region (layer_stride = page * num_blocks,
block_stride = page); block-outermost layouts make each block a
block of all layers' pages (layer_stride = page, block_stride =
the packed block). Tensors whose address ranges overlap
alias the same bytes: cache groups overlay each other, which is sound
because a block ID is owned by one group at a time.
Source code in vllm/v1/kv_cache_interface.py
KVQuantMode
¶
Bases: IntEnum
KV cache quantization mode.
Used by attention backends and kernels to dispatch quantization logic
without string matching on kv_cache_dtype.
Attributes:
-
is_nvfp4(bool) –True for NVFP4 packed quantization mode.
-
is_octave(bool) –True for any Octave quantization mode.
-
is_per_token_head(bool) –True for any per-token-head quantization mode.
-
is_turboquant(bool) –True for any turboquant quantization mode.
-
is_ultraquant(bool) –True for the UltraQuant 4-bit KV-cache format.
Source code in vllm/v1/kv_cache_interface.py
is_nvfp4
property
¶
True for NVFP4 packed quantization mode.
is_octave
property
¶
True for any Octave quantization mode.
is_per_token_head
property
¶
True for any per-token-head quantization mode.
is_turboquant
property
¶
True for any turboquant quantization mode.
is_ultraquant
property
¶
True for the UltraQuant 4-bit KV-cache format.
MLAAttentionSpec
dataclass
¶
Bases: AttentionSpec
Attributes:
-
topk_buffer_width(int | None) –Width of the model's sparse top-k indices buffer, including padding.
Source code in vllm/v1/kv_cache_interface.py
topk_buffer_width = None
class-attribute
instance-attribute
¶
Width of the model's sparse top-k indices buffer, including padding.
MambaSpec
dataclass
¶
Bases: KVCacheSpec
Attributes:
-
state_alignment(int) –Byte alignment of each state within a page.
Source code in vllm/v1/kv_cache_interface.py
state_alignment = 1
class-attribute
instance-attribute
¶
Byte alignment of each state within a page.
ManagerConfig
dataclass
¶
Allocation policy of a layer: the SingleTypeKVCacheManager that
enforces it and that manager's Params. Layers whose configs compare
equal may share a block table.
Source code in vllm/v1/kv_cache_interface.py
PrefixLookupGroup
¶
Bases: NamedTuple
KV cache groups looked up jointly for cache hits: they share a manager policy, block size and DCP sharding.
Attributes:
Source code in vllm/v1/kv_cache_interface.py
use_eagle = False
class-attribute
instance-attribute
¶
Whether the lookup takes the EAGLE last-block drop, which applies to every member since they are looked up jointly. Set by the coordinator.
compute_layer_kv_cache_shape_bytes(spec, num_blocks, kernel_block_size=None)
¶
Return the 4D logical shape (B, H, N, C) where C is in bytes.
Source code in vllm/v1/kv_cache_interface.py
compute_layout_strides(spec, num_blocks, num_layers, layout, kernel_block_size=None, fixed_strides=(None,) * 5)
¶
Byte strides in logical [L, B, H, N, C] axis order.
Source code in vllm/v1/kv_cache_interface.py
create_kv_cache_views(raw, spec, num_blocks, layout, kv_cache_tensor, kernel_block_size=None)
¶
View a flat int8 buffer as one 4D [B, H, N, C] view per layer.
Block b of layer l starts at the tensor offset plus its layer and
block stride contributions.
Source code in vllm/v1/kv_cache_interface.py
get_block_table_width(max_num_blocks, block_size, *, token_alignment=128)
¶
Return the width after optional alignment.
Source code in vllm/v1/kv_cache_interface.py
get_kv_quant_mode(kv_cache_dtype)
¶
Map a kv_cache_dtype string to a KVQuantMode.
Source code in vllm/v1/kv_cache_interface.py
get_mamba_prefill_checkpoint_position(num_tokens, hash_block_size, drop_eagle_block, num_hash_lookahead_tokens=0)
¶
Return the reusable Mamba checkpoint boundary for a prefill.
Source code in vllm/v1/kv_cache_interface.py
group_kernel_blocks(cache, num_blocks)
¶
View a kernel-block-granular layer cache with manager blocks as dim 0.
Kernel block splitting subdivides each manager block into uniformly strided
kernel blocks, so grouping is a pure view: (num_blocks * ratio, ...)
Source code in vllm/v1/kv_cache_interface.py
is_mamba_prefill_checkpoint_valid(query_start, query_end, checkpoint_position, hash_block_size, mamba_block_size, checkpoint_alignment)
¶
Whether a backend can export the checkpoint in this query.
Source code in vllm/v1/kv_cache_interface.py
kv_cache_uses_per_token_head_scales(kv_cache_dtype)
¶
Return True if kv_cache_dtype needs per-token-head scales.
prefix_lookup_groups(groups)
¶
Partition the prefix-cacheable groups into joint cache-hit lookups.