MiniMax-M3.1 specialized decode on NVIDIA GPUs: a decode executor over the
stock backbone, enabled by
additional_config={"decode_backend": "latency_specialized"}.
The executor runs FULL-graph decode steps of up to MAX_ROWS rows; every
other step runs the stock forward. It issues the same cross-rank collectives as
the stock layers, so the deployment check below is collective over the TP
group and every per-step decision reads host state that all TP ranks share.
The executor runs one precision setup: an fp8 (E4M3) KV cache and an fp8
(E4M3) index-K cache, MXFP8 attention and dense linears, NVFP4 routed experts.
NVFP4 (and BF16) KV and index caches are refused. The stock decode it is
checked against runs the same setup with the switch off. The resolved cache
dtypes are part of DecodeConfig, and each stage implementation declares
the ones it runs, so that more setups can be added stage by stage.
Modules:
-
draft_norm
–
The DSpark draft layers' Gemma RMSNorms fused with the MXFP8 quantize of
-
drafter
–
The DSpark drafter's sequential Markov sampling in K + 1 launches.
-
kernel_stages
–
The kernels/ registry entries behind the stages.py interfaces.
-
kernels
–
Kernel registry of the MiniMax-M3 specialized decode (NVIDIA, SM100 family).
-
peer_logits
–
LM head outputs in peer-mapped memory: the vocabulary-parallel logits
-
runtime
–
-
stages
–
The per-layer stages of the specialized decode.
-
stock
–
Every stage on the ops the stock layers run: the stock-op chain.
Classes:
-
CacheDtypes
–
The dtypes of the caches the attention stages read and write.
Functions:
-
cache_dtypes
–
The deployment's cache dtypes, or None if the executor refuses them
-
fused_finalize_max_tokens
–
The most tokens whose MoE top-k finalize the stock model leaves to the
-
max_rows
–
Rows of the largest executor step at query_len rows a request: whole
-
specialized_decode_config
–
The specialized decode settings, or None for the stock decode.
-
unsupported
–
Every requirement of the specialized decode this deployment misses.
CacheDtypes
Bases: NamedTuple
The dtypes of the caches the attention stages read and write.
Source code in vllm/models/minimax_m3/nvidia/specialized_decode/__init__.py
| class CacheDtypes(NamedTuple):
"""The dtypes of the caches the attention stages read and write."""
# The main K/V cache.
kv: CacheDtype
# The indexer's index-K cache.
index: CacheDtype # type: ignore[assignment]
|
cache_dtypes(vllm_config)
The deployment's cache dtypes, or None if the executor refuses them
(NVFP4, BF16, ...).
Source code in vllm/models/minimax_m3/nvidia/specialized_decode/__init__.py
| def cache_dtypes(vllm_config: VllmConfig) -> CacheDtypes | None:
"""The deployment's cache dtypes, or None if the executor refuses them
(NVFP4, BF16, ...)."""
kv = KV_CACHE_DTYPES.get(str(vllm_config.cache_config.cache_dtype))
index = INDEXER_KV_DTYPES.get(str(vllm_config.attention_config.indexer_kv_dtype))
if kv is None or index is None:
return None
return CacheDtypes(kv=kv, index=index)
|
fused_finalize_max_tokens(vllm_config, stock_cap)
The most tokens whose MoE top-k finalize the stock model leaves to the
next layer's fused finalize + all-reduce + norm + MXFP8 (stock_cap
without the switch).
With the specialized decode requested, every executor step defers, up to
MAX_ROWS: from 33 rows on that replaces the MoE's finalize, the
shared-expert add, FlashInfer's all-reduce + norm and the MXFP8 quantize of
the next QKV input (five launches) by the two
PDL-chained launches it already runs up to 32, and the wide front's GEMV
reads their MXFP8 rows. Host state only (every rank decides alike); the
deployment check itself runs in specialized_decode_config.
Source code in vllm/models/minimax_m3/nvidia/specialized_decode/__init__.py
| def fused_finalize_max_tokens(vllm_config: VllmConfig, stock_cap: int) -> int:
"""The most tokens whose MoE top-k finalize the stock model leaves to the
next layer's fused finalize + all-reduce + norm + MXFP8 (`stock_cap`
without the switch).
With the specialized decode requested, every executor step defers, up to
`MAX_ROWS`: from 33 rows on that replaces the MoE's finalize, the
shared-expert add, FlashInfer's all-reduce + norm and the MXFP8 quantize of
the next QKV input (five launches) by the two
PDL-chained launches it already runs up to 32, and the wide front's GEMV
reads their MXFP8 rows. Host state only (every rank decides alike); the
deployment check itself runs in `specialized_decode_config`.
"""
extra = vllm_config.additional_config
backend = extra.get("decode_backend", "auto") if isinstance(extra, dict) else "auto"
if backend != "latency_specialized":
return stock_cap
return max(stock_cap, MAX_ROWS)
|
max_rows(query_len)
Rows of the largest executor step at query_len rows a request: whole
requests of up to MAX_REQUESTS, at most MAX_ROWS.
Source code in vllm/models/minimax_m3/nvidia/specialized_decode/__init__.py
| def max_rows(query_len: int) -> int:
"""Rows of the largest executor step at `query_len` rows a request: whole
requests of up to `MAX_REQUESTS`, at most `MAX_ROWS`."""
if query_len < 1:
raise ValueError(f"query_len must be positive, got {query_len}")
return min(MAX_ROWS, MAX_REQUESTS * query_len)
|
specialized_decode_config(vllm_config)
The specialized decode settings, or None for the stock decode.
Collective over the TP group when requested: every rank raises if any rank
cannot run the specialized decode.
Source code in vllm/models/minimax_m3/nvidia/specialized_decode/__init__.py
| def specialized_decode_config(vllm_config: VllmConfig) -> DecodeConfig | None:
"""The specialized decode settings, or None for the stock decode.
Collective over the TP group when requested: every rank raises if any rank
cannot run the specialized decode.
"""
extra = vllm_config.additional_config
backend = extra.get("decode_backend", "auto") if isinstance(extra, dict) else "auto"
if backend == "auto":
return None
if backend != "latency_specialized":
raise ValueError("Supported decode_backend values are auto/latency_specialized")
missing = unsupported(vllm_config)
supported = torch.tensor([0 if missing else 1], dtype=torch.int32)
# The GPU check is rank-local, and ranks that disagree would run different
# collectives and hang: every rank raises if any rank misses a requirement.
dist.all_reduce(supported, op=dist.ReduceOp.MIN, group=get_tp_group().cpu_group)
if not supported.item():
reason = "; ".join(missing) or "another tensor parallel rank to support it"
raise ValueError(f"decode_backend=latency_specialized needs {reason}.")
query_len = 1 + _num_speculative_tokens(vllm_config)
dtypes = cache_dtypes(vllm_config)
assert dtypes is not None
cap = max_rows(query_len)
step_rows = tuple(
sorted(
{
size
for size in vllm_config.compilation_config.cudagraph_capture_sizes or []
if 0 < size <= cap
}
)
)
logger.info_once(
"MiniMax-M3.1 specialized decode on, for FULL-graph decode steps of up "
"to %d rows (%d rows a request; %s KV cache, %s index cache).",
cap,
query_len,
dtypes.kv,
dtypes.index,
)
kernel = vllm_config.kernel_config
return DecodeConfig(
cache_dtypes=dtypes,
query_len=query_len,
autotune=kernel.enable_flashinfer_autotune is not False,
jit_warmup=bool(kernel.enable_jit_warmup),
step_rows=step_rows,
)
|
unsupported(vllm_config)
Every requirement of the specialized decode this deployment misses.
Source code in vllm/models/minimax_m3/nvidia/specialized_decode/__init__.py
| def unsupported(vllm_config: VllmConfig) -> list[str]:
"""Every requirement of the specialized decode this deployment misses."""
parallel = vllm_config.parallel_config
attention = vllm_config.attention_config
cache = vllm_config.cache_config
spec = vllm_config.speculative_config
compilation = vllm_config.compilation_config
cudagraph_mode = compilation.cudagraph_mode
quant = vllm_config.quant_config
capture_sizes = compilation.cudagraph_capture_sizes or []
query_len = 1 + _num_speculative_tokens(vllm_config)
requirements = {
"a B200/GB200 (SM100) or B300/GB300 (SM103) GPU": (
current_platform.is_cuda()
and (
current_platform.is_device_capability(100)
or current_platform.is_device_capability(103)
)
),
"async scheduling": bool(vllm_config.scheduler_config.async_scheduling),
"tensor parallel size "
+ " or ".join(str(tp) for tp in SUPPORTED_TP_SIZES)
+ " over the whole world, without pipeline, data (DP attention), expert "
"or context parallelism or micro-batching": (
parallel.tensor_parallel_size in SUPPORTED_TP_SIZES
and parallel.world_size == parallel.tensor_parallel_size
and parallel.pipeline_parallel_size == 1
and parallel.data_parallel_size == 1
and not parallel.enable_expert_parallel
and parallel.decode_context_parallel_size == 1
and parallel.prefill_context_parallel_size == 1
and not parallel.use_ubatching
),
f"no speculative decoding, or DSpark with at most "
f"{MAX_SPECULATIVE_TOKENS} draft tokens and without adaptive "
"verification": (
spec is None
or (
spec.method == "dspark"
and _num_speculative_tokens(vllm_config) <= MAX_SPECULATIVE_TOKENS
and not spec.enable_adaptive_verification
)
),
# The executor reads and writes fp8 KV only: NVFP4 KV is refused.
"an fp8 (E4M3) KV cache (--kv-cache-dtype fp8; not nvfp4 or bf16)": (
cache.cache_dtype in KV_CACHE_DTYPES
),
# Likewise for the index-K cache: NVFP4 and MXFP4 are refused.
"an fp8 (E4M3) indexer cache (--attention-config "
'\'{"indexer_kv_dtype": "fp8"}\'; not nvfp4 or bf16), without '
"indexer context parallelism": (
attention.indexer_kv_dtype in INDEXER_KV_DTYPES
and not attention.indexer_decode_cp
and not attention.minimax_m3_indexer_cp
),
"the MiniMax-M3.1 checkpoint format: MXFP8 attention and dense linears "
"with NVFP4 routed experts (modelopt_mxfp8, moe_quant_algo NVFP4), on "
"the default (TRTLLM-gen) MoE backend": (
quant is not None
and quant.get_name() == "modelopt_mxfp8"
and getattr(quant, "moe_quant_algo", None) == "NVFP4"
and vllm_config.kernel_config.moe_backend != "deep_gemm_mega_moe"
),
"FULL CUDA graphs for decode batches only (FULL_AND_PIECEWISE or "
"FULL_DECODE_ONLY)": (
cudagraph_mode is not None
and cudagraph_mode.decode_mode() == CUDAGraphMode.FULL
and cudagraph_mode.mixed_mode() != CUDAGraphMode.FULL
),
f"a CUDA graph capture size of at most {max_rows(query_len)} rows "
f"(up to {MAX_REQUESTS} requests of {query_len} rows)": (
any(size <= max_rows(query_len) for size in capture_sizes)
),
# The decode loop runs in forward; a compiled forward would trace it.
"no torch.compile of the model": (
compilation.mode != CompilationMode.STOCK_TORCH_COMPILE
),
# Kernel stages may bind weight addresses at build.
"no sleep mode": not vllm_config.model_config.enable_sleep_mode,
"no LoRA or batch invariance, and no KV transfer other than "
+ " or ".join(STEP_LEVEL_KV_CONNECTORS)
+ " (CPU KV offload)": (
vllm_config.lora_config is None
and (
vllm_config.kv_transfer_config is None
or vllm_config.kv_transfer_config.kv_connector
in STEP_LEVEL_KV_CONNECTORS
)
and not envs.VLLM_BATCH_INVARIANT
),
f"a KV block size of {PAGE_SIZE} (--block-size {PAGE_SIZE})": (
cache.block_size == PAGE_SIZE
),
f"a max model length of at most {MAX_PAGES * PAGE_SIZE} tokens "
f"({MAX_PAGES} pages of {PAGE_SIZE})": (
-(-int(vllm_config.model_config.max_model_len) // PAGE_SIZE) <= MAX_PAGES
),
}
return [need for need, ok in requirements.items() if not ok]
|