vllm.models.deepseek_v41.nvidia.specialized_decode.hopper.indexer
¶
The DeepSeek-V4.1 sparse-attention indexer on SM90, with length-bounded candidate selection.
DeepSeek-V4.1 selects a decode token's compressed keys in two levels. The
candidate source indexer (layer 20) keeps each row's best 2048 blocks of 8
compressed positions, and the indexers after it (layers 24, 28, 32 and 36,
the candidate consumers) keep their top 512 among those blocks only. vLLM's
indexer op sizes this work by the logits' width, max_model_len, which CUDA
graphs fix: the source ranks the blocks of the whole width with
torch.topk, whatever the row's length. A consumer runs like any other
indexer: DeepGEMM scores every key of the row, a mask kernel sets the keys
outside the candidate blocks to -inf, and the top-k reads the whole row
again. A row of 256K compressed positions so costs a consumer about 16 times
the candidates' work, three times over.
This layer keeps vLLM's op (sparse_attn_indexer) for every call except the
two-level calls on a step of only decode rows, and narrows the op to its rank's
prefill rows when a step splits them over the TP ranks
(indexer_metadata.py):
- the source (
_candidate_source) runs vLLM's decode path, DeepGEMM's logits and the configured top-k, but selects its candidate blocks with the specialized decode'sselect_candidate_blocks_cooperative, which ranks only the blocks below each row's length; - a consumer (
_compact_consumer) runs the SM80 package's CuTe DSL kernels on SM90's index cache as it is (pages of 128-byte E4M3 keys, then their FP32 scales).candidate_mqa_logitsscores only the row's candidate keys, into compact FP32 rows (columncis key8 * candidates[c // 8] + c % 8,-infpast the row's length), so a row costs at most2048 * 8keys however long its context;decode_topkkeeps the 512 best columns of each compact row and writes their key positions (all of a row's keys when it has at most 512); - on a split step, the other indexers (
_row_sharded) run the stock op over this rank's share of the prefill rows and gather the ranks' top-k rows back (../ampere/row_shard.py); the candidate rows of the two-level selection stay replicated.
The source's candidate set is torch.topk's with the newest block pinned,
except at the selection boundary: blocks tied there, and a boundary bin of
the cooperative kernel's FP16 radix more crowded than it keeps, are selected
approximately, so a row may hold fewer than 2048 candidates. Its rows are
unordered and -1 padded anywhere, but a row of at most 2048 blocks lists
block i in slot i, so every valid block lies within the first
min(2048, ceil(len / 8)) slots that a compact consumer scores; vLLM's
mask reads all slots. A consumer's key scores
scale * sum_h w_h * relu(q_h . k) as DeepGEMM computes it (up to the FP32
summation order), and its top-k is the same set as the masked dense row's, up
to ties at the threshold. Like vLLM's op, the call is a break of a breakable
CUDA graph capture, so its Python chooses the path at every step; a FULL graph
replays the path of its capture, a decode step.
Classes:
-
HopperIndexer–The DeepSeek-V4.1 indexer whose candidate source and consumers bound
-
HopperIndexerCache–The index K cache, with the SM90 metadata builder where it would
-
HopperSparseAttnIndexer–vLLM's indexer layer on
dsv41_hopper_sparse_attn_indexer.
Functions:
-
dsv41_hopper_sparse_attn_indexer–The stock
sparse_attn_indexerwith the same arguments, whose
HopperIndexer
¶
Bases: DeepseekV4Indexer
The DeepSeek-V4.1 indexer whose candidate source and consumers bound their candidate work by each row's length on decode steps.
Source code in vllm/models/deepseek_v41/nvidia/specialized_decode/hopper/indexer.py
HopperIndexerCache
¶
Bases: DeepseekV4IndexerCache
The index K cache, with the SM90 metadata builder where it would take the stock dense-logits one.
Source code in vllm/models/deepseek_v41/nvidia/specialized_decode/hopper/indexer.py
HopperSparseAttnIndexer
¶
Bases: SparseAttnIndexer
vLLM's indexer layer on dsv41_hopper_sparse_attn_indexer.
Methods:
-
forward_cuda–Top-k indices of the step's tokens, as vLLM's layer.
Source code in vllm/models/deepseek_v41/nvidia/specialized_decode/hopper/indexer.py
forward_cuda(hidden_states, q_quant, k, weights)
¶
Top-k indices of the step's tokens, as vLLM's layer.
Source code in vllm/models/deepseek_v41/nvidia/specialized_decode/hopper/indexer.py
_candidate_source(meta, num_tokens, kv_cache, q_fp8, weights, topk_tokens, head_dim, max_model_len, topk_indices_buffer, candidate_blocks, candidate_block_size, topk_backend)
¶
The source's top-k and candidate blocks of a step of only decode rows, as vLLM's op computes them but for the length-bounded block selection; False (nothing written) for any other step.
Source code in vllm/models/deepseek_v41/nvidia/specialized_decode/hopper/indexer.py
_compact_consumer(meta, num_tokens, kv_cache, q_fp8, weights, topk_tokens, topk_indices_buffer, candidate_blocks, candidate_block_size)
¶
A consumer's top-k of a step of only decode rows, from its candidate keys; False (nothing written) for any other step.
Source code in vllm/models/deepseek_v41/nvidia/specialized_decode/hopper/indexer.py
_dsv41_hopper_sparse_attn_indexer_fake(hidden_states, k_cache_prefix, kv_cache, q_quant, q_scale, k, weights, quant_block_size, scale_fmt, topk_tokens, head_dim, max_model_len, total_seq_lens, topk_indices_buffer, skip_k_cache_insert, use_pcp, pcp_shard_decode_requests, dense_mha_metadata_layer_name, use_fp4_cache=False, dcp_rank=0, dcp_world_size=1, cp_kv_cache_interleave_size=1, skip_topk_buffer_clear=False, candidate_blocks=None, candidate_block_size=0, candidate_write=False, topk_backend='auto')
¶
The op's output buffer, for tracing.
Source code in vllm/models/deepseek_v41/nvidia/specialized_decode/hopper/indexer.py
dsv41_hopper_sparse_attn_indexer(hidden_states, k_cache_prefix, kv_cache, q_quant, q_scale, k, weights, quant_block_size, scale_fmt, topk_tokens, head_dim, max_model_len, total_seq_lens, topk_indices_buffer, skip_k_cache_insert, use_pcp, pcp_shard_decode_requests, dense_mha_metadata_layer_name, use_fp4_cache=False, dcp_rank=0, dcp_world_size=1, cp_kv_cache_interleave_size=1, skip_topk_buffer_clear=False, candidate_blocks=None, candidate_block_size=0, candidate_write=False, topk_backend='auto')
¶
The stock sparse_attn_indexer with the same arguments, whose
two-level calls on steps of only decode rows bound their candidate work
by each row's length (_candidate_source, _compact_consumer), and
whose other calls score only this rank's prefill rows on a split step.
Source code in vllm/models/deepseek_v41/nvidia/specialized_decode/hopper/indexer.py
101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 | |