vllm.models.minimax_m3.nvidia.specialized_decode.kernels
¶
Kernel registry of the MiniMax-M3 specialized decode (NVIDIA, SM100 family).
Maps each decode stage of specialized_decode/stages.py to the
implementation that serves it and the envelope it is valid for: target arch,
TP size and rows per rank (rows = c * (K + 1), c concurrent requests, K draft
tokens). select returns None for any (stage, arch, tp, rows) outside
every envelope; the caller then runs the stock path for that stage.
Nothing is imported or compiled at import time: KernelImpl.load() imports
the implementing module lazily.
The kernels cover the sparse MoE layers 3-59; layers 0-2 run stock. The executor runs
steps of up to MAX_ROWS = 64 rows
(min(64, 8 * (K + 1))); each stage's envelope below is how far its kernel
goes, and a step with more rows runs that stage stock (select returns
None, fallback_stages names them for the build log):
==================== =======================================================
stage implementation
==================== =======================================================
front AR land + residual + RMSNorm + MXFP8 + QKV/index GEMV
(tcgen05) + q/k norm + RoPE + FP8 KV / index-K insert
index_score upstream minimax_m3_index_decode_score_cutedsl
page_topk top-16 pages per (row, index head)
sparse_decode 16-CTA cluster FP8 sparse decode with MXFP8 output
o_proj MXFP8 o_proj GEMV + router CTAs, Lamport push
allreduce_norm AR land + residual + norm + NVFP4/MXFP8 quant + top-4
moe fused NVFP4 routed + MXFP8 shared experts, push
dense_mlp stock (no entry)
==================== =======================================================
Rows per stage (max_rows) and what serves the rows above it:
==================== ==== ==================================================
stage rows above it
==================== ==== ==================================================
front 16 the "wide" variant to 64 (the kernel's activation
tile and its 20 boundary clusters are sized for 16
rows): stock AR + norm, MXFP8 quantize, the GEMV
QKV / index projection (rows = MMA N), the stock
q/k norm + RoPE + cache insert op
index_score 64 (grid = requests; only heads x qlen <= 32 binds)
page_topk 64 (grid = rows x heads)
sparse_decode 64 (grid = rows x heads clusters; beyond one wave of
clusters it runs in more than one wave)
o_proj 8 the "wide" variant (wide.py) to 64: the same
MXFP8 GEMV with the rows as the MMA's N (16 / 32 /
48 / 64), each weight tile read once; no router
CTAs, no push (BF16 partial)
allreduce_norm 8 stock (FlashInfer AR + Gemma norm, the stock
router GEMM)
moe 8 stock (at 32-64 rows x top-4 most experts are hit
and the stock grouped GEMMs stream each weight
once, at the HBM roofline; the fused engine's
slots stop at 8)
dense_mlp - stock at every row count
==================== ==== ==================================================
A stage's implementations are tried in order; the "wide" ones (variant)
serve only the rows above the fused pair's (min_rows).
Modules:
-
allreduce_norm–allreduce_norm stage: the post-attention boundary of a sparse layer (TP4).
-
front–Stage
front: the attention front of one sparse layer in one launch. -
front_kernel–CuTe DSL device code of stage
front(the host side isfront.py). -
geometry–Launch geometry of the specialized-decode kernels as a function of the SM
-
index_score–index_score stage: decode-time index page scores, served by upstream
-
moe–The
moestage of the MiniMax-M3 specialized decode: the fused NVFP4 routed + -
o_proj–o_proj stage (TP4): MXFP8 o_proj GEMV + router CTAs, one CuTe DSL launch.
-
page_topk–page_topk stage: per (decode row, index head) the 16 pages the sparse
-
page_topk_kernel–page_topk device code (CuTe DSL): exact indexer top-16 page selector for the sparse-attention decode.
-
post–The post family (attention -> MoE boundary): CuTe DSL kernels (
kernel.py) and -
sparse_decode–sparse_decode stage: FP8-KV sparse decode over the 16 pages
page_topk -
sparse_decode_kernel–CuTe DSL cluster sparse decode (FP8 e4m3 KV, 16 selected pages of 128 tokens, MXFP8 output).
-
wide–The kernels of steps wider than the fused kernels' rows (9..64 rows).
Functions:
-
fallback_stages–Stages with an implementation that does not cover
rowsat -
max_rows–The largest rows any
variantimplementation ofstageruns. -
select–The implementation serving
stagefor this envelope (of
fallback_stages(arch, tp, rows)
¶
Stages with an implementation that does not cover rows at
(arch, tp): they run stock for a step of that many rows.
Source code in vllm/models/minimax_m3/nvidia/specialized_decode/kernels/__init__.py
max_rows(stage, variant='')
¶
The largest rows any variant implementation of stage runs.
Source code in vllm/models/minimax_m3/nvidia/specialized_decode/kernels/__init__.py
select(stage, arch, tp, rows, variant=None)
¶
The implementation serving stage for this envelope (of
variant, when given), or None (run the stock path).