vllm.models.minimax_m3.nvidia.specialized_decode.kernels.page_topk_kernel
¶
page_topk device code (CuTe DSL): exact indexer top-16 page selector for the sparse-attention decode.
256 threads, 4 witness lanes per warp; the selection is exact, so the table is the same for any launch shape, bit for bit.
Per (decode row, index head) one CTA selects, from the row's cap = min(n_valid, num_cols, 8192) pages, the
forced pages (the first init and the last local) plus the need best other pages by (fp32 key desc, page
asc): every thread holds KPT order-preserving keys in registers; per warp the R-th largest lane maximum is a witness,
and the K-th largest witness (K = ceil(need / R)) bounds a pool that provably holds the top need. A pool of at
most 64 entries is ranked by warp 0 (certificate); otherwise an exact block-wide MSD radix select runs on the
register keys. Winners are written in ascending page order as physical page ids (-1 tail when fewer than 16
pages are valid).
split_mode (launches whose rows may exceed single_max pages): grid (rows x heads, splits). A row of at most
single_max pages runs the one-CTA select in split 0 (the other splits exit before the grid wait); a longer row
is cut into chunk-page CTAs that write their exact local top-need as key << 32 | ~page to part; the
last chunk CTA to arrive (per-row counter in cnt, threadfence pattern, re-armed for graph replay) ranks the
union and writes the row. No CTA waits for another.
PDL: n_valid and the page rows are read before griddepcontrol.wait (they are written before the predecessor),
the scores after it; dependents are released right after the wait.
Classes:
-
Smem–Shared memory of one CTA (the CUDA
struct Smem; the V2 per-warp counts are not used by V4).
Functions:
-
ctz–Index of the lowest set bit (
__ffs(v) - 1for v != 0). -
ldcg_u32–pred ? ld.global.cg.b32 [addr] : 0(a predicated load: no access when pred is 0). -
ldcg_u64–pred ? ld.global.cg.b64 [addr] : 0. -
okey–Order-preserving uint32 key of the fp32 bits.
-
pack64–hi << 32 | loof two 32-bit words. -
shr_u32–Logical right shift; shift amounts >= 32 give 0 (PTX clamps).
-
smem_addr–shared::cta address of int32 element
idxofarr.
Smem
¶
Shared memory of one CTA (the CUDA struct Smem; the V2 per-warp counts are not used by V4).
Source code in vllm/models/minimax_m3/nvidia/specialized_decode/kernels/page_topk_kernel.py
_ld(arr, idx)
¶
arr[idx] (a free function: the DSL carries the receivers of method calls through dynamic regions).
_radix_select(k, incm, need, s, tid, kpt)
¶
Exact top-need (key desc, page asc) among the included keys (bit j of incm): block-wide MSD radix select,
8-bit passes until <= 32 candidates, mass-tie path in page order. Winners -> s.wsel[0..need) (warp 0 synced).
Source code in vllm/models/minimax_m3/nvidia/specialized_decode/kernels/page_topk_kernel.py
343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 | |
_select_body(s, srw, orw, cap, init, fend, flo, need, tid, diag, has_diag, kpt, part_mode, part, c0)
¶
Witness-select body for one row (or one chunk when part_mode: srw and pages chunk-local, winners -> part).
Source code in vllm/models/minimax_m3/nvidia/specialized_decode/kernels/page_topk_kernel.py
473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 567 568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596 597 598 599 600 601 602 603 | |
_single_tiers(s, srw, orw, cap, init, fend, flo, need, tid, diag, has_diag)
¶
The one-CTA select: 6, 16 or 32 keys per thread.
Source code in vllm/models/minimax_m3/nvidia/specialized_decode/kernels/page_topk_kernel.py
_stage_rows(s, prw, cap, tid)
¶
s.spid[p] = page_rows[p] for p < cap: SPID_BATCH predicated loads first, then the stores.
Source code in vllm/models/minimax_m3/nvidia/specialized_decode/kernels/page_topk_kernel.py
_store_page_order(s, need, base, orw, lane, from_spid, prw)
¶
Winners' pages in s.wsel[0..need) -> page-order store at orw[base + pos] (warp 0; V4: smem reads).
Source code in vllm/models/minimax_m3/nvidia/specialized_decode/kernels/page_topk_kernel.py
_store_partial(s, need, srw, c0, part, lane)
¶
Chunk CTA: winners in s.wsel[0..need) (chunk-local pages) -> 16 entries key << 32 | ~(c0 + page), zero tail.
Source code in vllm/models/minimax_m3/nvidia/specialized_decode/kernels/page_topk_kernel.py
ctz(v, *, loc=None, ip=None)
¶
Index of the lowest set bit (__ffs(v) - 1 for v != 0).
Source code in vllm/models/minimax_m3/nvidia/specialized_decode/kernels/page_topk_kernel.py
ldcg_u32(addr, pred, *, loc=None, ip=None)
¶
pred ? ld.global.cg.b32 [addr] : 0 (a predicated load: no access when pred is 0).
Source code in vllm/models/minimax_m3/nvidia/specialized_decode/kernels/page_topk_kernel.py
ldcg_u64(addr, pred, *, loc=None, ip=None)
¶
pred ? ld.global.cg.b64 [addr] : 0.
Source code in vllm/models/minimax_m3/nvidia/specialized_decode/kernels/page_topk_kernel.py
okey(raw)
¶
Order-preserving uint32 key of the fp32 bits.
pack64(hi, lo, *, loc=None, ip=None)
¶
hi << 32 | lo of two 32-bit words.
Source code in vllm/models/minimax_m3/nvidia/specialized_decode/kernels/page_topk_kernel.py
shr_u32(v, s, *, loc=None, ip=None)
¶
Logical right shift; shift amounts >= 32 give 0 (PTX clamps).
Source code in vllm/models/minimax_m3/nvidia/specialized_decode/kernels/page_topk_kernel.py
smem_addr(arr, idx, *, loc=None, ip=None)
¶
shared::cta address of int32 element idx of arr.