vllm.models.deepseek_v41.nvidia.specialized_decode.moe.step
¶
The MoE of a specialized decode step.
Every MoE layer whose experts can stop after GEMM2 leaves its routed top-k reduction, the shared-expert add and the all-reduce to the next boundary's fused kernel, including the layers before an Engram layer and the last one, whose boundaries take the kernel's residual-only variant:
- up to
routed_expert_gemm.MAX_ROWSrows, on the low-batch routed-expert GEMMs (routed_expert_gemm); - above that, on the runner's own
_forward_implwith the deferral raised to the step's rows for the call (deferred_finalize).
The DSpark draft's layers finalize their MoE in the layer instead (the fused
kernel's finalize is built for the target's top-k): up to
LOW_BATCH_MAX_ROWS rows on the routed-expert GEMMs and their finalize
kernel, above that on _forward_impl. Either way their output stays
TP-partial for the next boundary, though their stock layers reduce it.
Classes:
-
MoEStep–One MoE layer of the decode step; with
finalize, one that finalizes
Functions:
-
deferred_finalize–Within the block, let the experts leave the top-k reduction of calls of
-
defers_finalize–Whether
ffn's experts can leave the top-k reduction to the fused
MoEStep
¶
One MoE layer of the decode step; with finalize, one that finalizes
its own MoE (the DSpark draft's).
Methods:
-
run–The layer's TP-partial output, unfinalized where it can defer.
Source code in vllm/models/deepseek_v41/nvidia/specialized_decode/moe/step.py
79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 | |
_forward_low_batch(gemm, hidden_states, input_ids)
¶
The runner's _forward_impl with the routed experts on the
low-batch GEMMs; the gate and the shared experts run as the runner runs
them. Returns the unfinalized routed output and the shared output.
Source code in vllm/models/deepseek_v41/nvidia/specialized_decode/moe/step.py
_run_finalized(x, input_ids, step)
¶
The layer's TP-partial output with its top-k reduction done here: the shared output plus the finalized routed output, as the stock forward adds them before its all-reduce.
Source code in vllm/models/deepseek_v41/nvidia/specialized_decode/moe/step.py
run(x, input_ids, step)
¶
The layer's TP-partial output, unfinalized where it can defer.
Source code in vllm/models/deepseek_v41/nvidia/specialized_decode/moe/step.py
deferred_finalize(config, max_num_tokens)
¶
Within the block, let the experts leave the top-k reduction of calls of
up to max_num_tokens tokens to the caller; then restore the layer's own
deferral, which the stock forward reads.
Source code in vllm/models/deepseek_v41/nvidia/specialized_decode/moe/step.py
defers_finalize(ffn, max_num_tokens)
¶
Whether ffn's experts can leave the top-k reduction to the fused
all-reduce + mHC: the modular TRT-LLM MXFP4 experts without routed scaling,
on a TP-only deployment.