vllm.models.deepseek_v41.nvidia.specialized_decode.ampere.all_gather
¶
The TP all-gathers of the SM80 decode step, in CuTe DSL.
Three kinds of tensor are all-gathered across the TP group in a decode step:
- the row-sharded sparse-attention indexer's top-k: in a step of at least
two requests, each rank scores and top-k's only its share of the
indexer's rows (
indexer.py: whole requests, equal shares, a short or empty one padded with -1), then gathers the selected int32 indices (index_topk= 512 a row) so every rank has all rows; - the vocabulary-parallel logits of the target and the DSpark draft model
(
columns.py): every rank has every row's logits over its slice of the vocabulary, 16,160 BF16 values at TP8, and needs the whole vocabulary; - the Engram layers' embedded hash heads and
wkvoutput, split by heads and by output columns across the ranks.
vLLM gathers the first through its custom all-gather, which loads every peer's buffer between two barriers (about 15 us a call on 8x A100, median), and the others through NCCL's ring, 30 to 170 us a call, each followed by a copy from rank-major blocks into the layout the caller wants.
AllGatherRows pushes instead, through a Lamport mailbox as the SM80
all-reduce does (all_reduce.py): every slot holds a sentinel, the int32
INT32_MIN (FP32 negative zero), until a peer's store replaces it. A call
gathers rows rows of fragments 16-byte fragments from every rank into
out, in one of two layouts:
- rank-major (the indexer): rank
s's rows are rowss * rowsonward ofout, and this rank's own rows are already in place (the indexer scores its share into its slice ofout); - side by side (logits, Engram): row
jofoutis every rank's rowjin rank order, and this rank's own row is copied in fromlocal.
A unit of work is a pair of a row j and a hop h in 1 .. world - 1;
for it, each thread of a CTA stores its fragments of local row j into the
mailbox of rank rank + h, then polls the fragments of row j that rank
rank - h stores into its own mailbox, writes them to out and resets
the slots. Side by side, the units of hop 1 also write this rank's own row
from the fragments they push. No phase waits for anything but its own data,
and nothing is copied afterwards.
A unit on one rank waits only for the same unit on one peer. Units go to
CTAs in increasing order, ctas apart, with at most one CTA per SM: the
lowest unfinished unit is then in progress on every rank, so every call
completes (the argument of all_reduce.py, which also needs every kernel
beside a call to finish without waiting on it).
Each unit keeps the generation (of three) its next call uses, advanced by every call that has the unit. All ranks make the same calls with the same row counts, so a unit's generation agrees across ranks. A peer writes the same slot again three calls later, which needs this rank's pushes of two calls later, issued after this call completed. The mailbox starts as sentinels and the generations at zero: no per-launch initialization, CUDA-graph safe.
A sender stores a 32-bit word equal to the sentinel as zero, as vLLM's
Lamport all-gather turns FP32 negative zero into positive zero; the
indexer's rows (indices of at least -1) never hold it, and in BF16 rows it
is a positive zero followed by a negative one, which becomes a positive zero
(the sign of a zero changes no logit's probability and no product). Side by
side, this rank's own row is written the same way, so every rank's out
is bitwise the same. Every other value arrives unchanged.
Classes:
-
AllGatherRows–The push-based Lamport all-gather of rows of 16-byte fragments; see
-
AmpereAllGather–The mailbox and per-unit generations of
AllGatherRowsfor one TP
Functions:
-
poll–Wait until the slot at
addressholds no sentinel, read it into -
sanitize_sentinel–word, or zero if it equals the sentinel.
AllGatherRows
¶
The push-based Lamport all-gather of rows of 16-byte fragments; see the module docstring.
world ranks, rows of fragments fragments, at most capacity rows
a rank, gathered rank-major or side_by_side.
Methods:
-
mailbox_bytes–Bytes of a rank's mailbox: a slot per generation, source rank and
-
out_offset–Byte offset of rank
source's fragment of rowrowin -
pull–Wait for the source's fragment of row
row, write it toout -
push–Store a fragment of local row
rowinto the destination's -
slot–Byte offset of a slot's fragment in a mailbox.
Source code in vllm/models/deepseek_v41/nvidia/specialized_decode/ampere/all_gather.py
120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 | |
mailbox_bytes()
¶
Bytes of a rank's mailbox: a slot per generation, source rank and row.
Source code in vllm/models/deepseek_v41/nvidia/specialized_decode/ampere/all_gather.py
out_offset(source, row, rows, fragment)
¶
Byte offset of rank source's fragment of row row in
out.
Source code in vllm/models/deepseek_v41/nvidia/specialized_decode/ampere/all_gather.py
pull(out, mailbox, generation, source, row, rows, fragment)
¶
Wait for the source's fragment of row row, write it to out
and reset its slot.
Source code in vllm/models/deepseek_v41/nvidia/specialized_decode/ampere/all_gather.py
push(local, out, destination, generation, rank, row, rows, hop, fragment)
¶
Store a fragment of local row row into the destination's
mailbox (and, side by side at hop 1, into this rank's out).
Source code in vllm/models/deepseek_v41/nvidia/specialized_decode/ampere/all_gather.py
slot(generation, source, row, fragment)
¶
Byte offset of a slot's fragment in a mailbox.
Source code in vllm/models/deepseek_v41/nvidia/specialized_decode/ampere/all_gather.py
AmpereAllGather
¶
The mailbox and per-unit generations of AllGatherRows for one TP
group and row size; construction is collective over the group.
Parameters:
-
(group¶ProcessGroup) –The TP group's device process group.
-
(row_bytes¶int) –The bytes of a row a rank contributes, a multiple of 16.
-
(capacity¶int) –The most rows a rank contributes to a call.
-
(device¶device) –This rank's device.
-
(side_by_side¶bool, default:False) –Lay the ranks' rows out side by side (row
jof the output is every rank's rowj), else rank-major.
Methods:
-
__call__–Gather every rank's rows of
localintoout; rank-major, -
gather_columns–local's all-gather along its last dimension, `[rows, world * -
supports–Whether a call takes
localintoout: contiguous 16-byte
Source code in vllm/models/deepseek_v41/nvidia/specialized_decode/ampere/all_gather.py
305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 | |
__call__(local, out)
¶
Gather every rank's rows of local into out; rank-major,
local is this rank's slice of out (left as it is).
Source code in vllm/models/deepseek_v41/nvidia/specialized_decode/ampere/all_gather.py
gather_columns(local)
¶
local's all-gather along its last dimension, [rows, world *
columns] (side by side), or None when a call does not take it.
Source code in vllm/models/deepseek_v41/nvidia/specialized_decode/ampere/all_gather.py
supports(local, out)
¶
Whether a call takes local into out: contiguous 16-byte
aligned rows of row_bytes, at most capacity local rows and
out holding every rank's; rank-major, local must be this
rank's slice of out.
Source code in vllm/models/deepseek_v41/nvidia/specialized_decode/ampere/all_gather.py
_kernel(device_index, fragments, world, capacity, side_by_side)
cached
¶
AllGatherRows compiled for a device and shape.
Source code in vllm/models/deepseek_v41/nvidia/specialized_decode/ampere/all_gather.py
poll(address, words)
¶
Wait until the slot at address holds no sentinel, read it into
words and reset it to the sentinel.
Source code in vllm/models/deepseek_v41/nvidia/specialized_decode/ampere/all_gather.py
sanitize_sentinel(word)
¶
word, or zero if it equals the sentinel.