vllm.models.minimax_m3.nvidia.specialized_decode.kernels.front_kernel
¶
CuTe DSL device code of stage front (the host side is front.py).
A operand in shared memory at bucket 8, early x polling, Lamport sentinel hand-off, one weight phase. Every float
operation is spelled with an explicit rounding mode (add.rn, fma.rn, *.ftz, and FMA only where intended),
so the outputs do not depend on compiler contraction.
Grid: 4 * (heads + meta) CTAs, cluster 4, 512 threads, one CTA per SM (~226 KB shared memory). Per head cluster:
- warp 15: TMEM allocation, weight tile TMA (12 tiles of 128 x 128 E4M3, 128-byte swizzle) and scale atoms, their scale copy into TMEM (A operand in TMEM at bucket 16);
- clusters c < rows, warps 0..14: boundary row c: Lamport push of the partial into every rank's exchange (unless the producer pushed), poll every rank's partial, residual add, Gemma RMSNorm over the cluster (DSMEM), MXFP8 of the normed row into this rank's private hand-off buffer (two parities, sentinel-armed);
- every CTA: re-arm the other parity, poll its K slice of every row's MXFP8 activation into shared memory, 48 tcgen05 block-scaled MMAs (M 128 weight rows, N 8/16 rows, K 32), K-split reduce over the cluster (st.async);
- warps 0..3: per-head Gemma RMSNorm + NeoX RoPE, FP8 Q / index-Q, FP8 K/V and index-K cache inserts;
- optionally one more cluster writes the step metadata of the layer.
Compile-time parameters: the bucket (MAXT 8 / 16), rows (0 = runtime), the TP world, the per-rank head geometry (nq, nkv, niq: TP4 16/1/1, TP8 8/1/1), the rotary dim, the page size and the exchange mode.
Functions:
-
asl–TMEM layout: bucket 8 reads A (weights) straight from shared memory; bucket 16 stages A into TMEM.
-
bf16_round–float(bf16(x)) (RN).
-
bf16x2–bf16(lo) | bf16(hi) << 16 (RN).
-
e4m3x2–E4M3 (RN, satfinite) of lo (byte 0) and hi (byte 1), zero-extended.
-
f1–One-operand op: "rsqrt.approx.ftz", "rsqrt.approx", "rcp.approx.ftz", "abs".
-
fop–op.f32 a, b, e.g. op = "add.rn", "mul.rn.ftz", "div.full.ftz", "max", "min". -
ldg_v4–ld.global.v4.b32, zeros when pred is 0.
-
ldv_v4_pred–ld.volatile.global.v4 when pred, else four EMPTY words is not wanted: zeros (the caller masks the poll).
-
mbar_wait–Spin on mbarrier.try_wait.parity (sem: "" or ".acquire.cluster").
-
poll_empty–Re-poll a 16-byte exchange item (first load w0..w3) while pred and any word is EMPTY; traps after SPIN_CAP.
-
poll_x–The Lamport hand-off poll of one thread: up to four 16-byte activation items (bits 0..3 of pend) and one scale
-
ptx–A side-effecting PTX statement without operands.
-
ptx_p–A PTX statement executed when pred != 0.
-
ptx_rp–A PTX statement with one register operand ($0), executed when pred != 0.
-
sdesc–tcgen05 shared memory descriptor of a 16-byte aligned shared address.
-
stg_v4–st.global.v4.b32 (explicit .global: a generic store to an NVLS multicast address faults).
-
tc_ld16–Two tcgen05.ld 32x32b.x8 (columns c, c + 8) and one wait.
-
tc_ld8–tcgen05.ld 32x32b.x8 and its wait in one statement (the values exist only after the wait).
-
tc_mma_ss–Block-scaled MMA, A and B from shared memory descriptors.
-
tc_mma_ts–Block-scaled MMA, A from TMEM, B from a shared memory descriptor.
-
ue8m0–UE8M0 of x rounded up, satfinite (the low byte of cvt.rp.satfinite.ue8m0x2.f32 with a 0 partner).
-
ue8m0_value–The float value of a UE8M0 code (through bf16, as __nv_fp8_e8m0 -> float).
-
zero_byte_mask–0x80 in every byte of t that is zero, else 0 (exact per byte).
_boundary(MAXT, WORLD, PREPUSHED, t, crank, tid, lane, warp, bx, warp_sq, cluster_sq, xbar, x, x2, fc2, perm, scales, topk, scale_fp32, residual, norm_w, eps, out_residual, ws_uc, flags, xq, xsf, ctr, rank, npeers, single_round, peers)
¶
Boundary row t (FlashInfer one-shot all-reduce + norm rounding), on warps 0..14; 192 threads do the math.
Source code in vllm/models/minimax_m3/nvidia/specialized_decode/kernels/front_kernel.py
1465 1466 1467 1468 1469 1470 1471 1472 1473 1474 1475 1476 1477 1478 1479 1480 1481 1482 1483 1484 1485 1486 1487 1488 1489 1490 1491 1492 1493 1494 1495 1496 1497 1498 1499 1500 1501 1502 1503 1504 1505 1506 1507 1508 1509 1510 1511 1512 1513 1514 1515 1516 1517 1518 1519 1520 1521 1522 1523 1524 1525 1526 1527 1528 1529 1530 1531 1532 1533 1534 1535 1536 1537 1538 1539 1540 1541 1542 1543 1544 1545 1546 1547 1548 1549 1550 1551 1552 1553 1554 1555 1556 1557 1558 1559 1560 1561 1562 1563 1564 1565 1566 1567 1568 1569 1570 1571 1572 1573 1574 1575 1576 1577 1578 1579 1580 1581 1582 1583 1584 1585 1586 1587 1588 1589 1590 1591 1592 1593 1594 1595 1596 1597 1598 1599 1600 1601 1602 1603 1604 1605 1606 1607 1608 1609 1610 1611 1612 1613 1614 1615 1616 1617 1618 1619 1620 1621 1622 1623 1624 1625 1626 1627 1628 1629 1630 1631 1632 1633 1634 1635 1636 1637 1638 1639 1640 1641 1642 1643 1644 1645 1646 1647 1648 1649 1650 1651 1652 1653 1654 1655 1656 1657 1658 1659 1660 1661 1662 1663 1664 1665 1666 1667 1668 1669 1670 1671 1672 1673 1674 1675 1676 1677 1678 1679 1680 1681 1682 1683 1684 1685 1686 1687 1688 1689 1690 1691 1692 1693 1694 1695 1696 1697 1698 1699 1700 1701 1702 1703 1704 | |
asl(maxt)
¶
TMEM layout: bucket 8 reads A (weights) straight from shared memory; bucket 16 stages A into TMEM.
Source code in vllm/models/minimax_m3/nvidia/specialized_decode/kernels/front_kernel.py
bf16_round(x, *, loc=None, ip=None)
¶
float(bf16(x)) (RN).
Source code in vllm/models/minimax_m3/nvidia/specialized_decode/kernels/front_kernel.py
bf16x2(lo, hi, *, loc=None, ip=None)
¶
bf16(lo) | bf16(hi) << 16 (RN).
Source code in vllm/models/minimax_m3/nvidia/specialized_decode/kernels/front_kernel.py
e4m3x2(lo, hi, *, loc=None, ip=None)
¶
E4M3 (RN, satfinite) of lo (byte 0) and hi (byte 1), zero-extended.
Source code in vllm/models/minimax_m3/nvidia/specialized_decode/kernels/front_kernel.py
f1(op, a, *, loc=None, ip=None)
¶
One-operand op: "rsqrt.approx.ftz", "rsqrt.approx", "rcp.approx.ftz", "abs".
Source code in vllm/models/minimax_m3/nvidia/specialized_decode/kernels/front_kernel.py
fop(op, a, b, *, loc=None, ip=None)
¶
op.f32 a, b, e.g. op = "add.rn", "mul.rn.ftz", "div.full.ftz", "max", "min".
Source code in vllm/models/minimax_m3/nvidia/specialized_decode/kernels/front_kernel.py
ldg_v4(addr, pred, *, loc=None, ip=None)
¶
ld.global.v4.b32, zeros when pred is 0.
Source code in vllm/models/minimax_m3/nvidia/specialized_decode/kernels/front_kernel.py
ldv_v4_pred(addr, pred, *, loc=None, ip=None)
¶
ld.volatile.global.v4 when pred, else four EMPTY words is not wanted: zeros (the caller masks the poll).
Source code in vllm/models/minimax_m3/nvidia/specialized_decode/kernels/front_kernel.py
mbar_wait(bar, parity, sem='', *, loc=None, ip=None)
¶
Spin on mbarrier.try_wait.parity (sem: "" or ".acquire.cluster").
Source code in vllm/models/minimax_m3/nvidia/specialized_decode/kernels/front_kernel.py
poll_empty(addr, w0, w1, w2, w3, pred, *, loc=None, ip=None)
¶
Re-poll a 16-byte exchange item (first load w0..w3) while pred and any word is EMPTY; traps after SPIN_CAP.
Source code in vllm/models/minimax_m3/nvidia/specialized_decode/kernels/front_kernel.py
poll_x(pend, g0, g1, g2, g3, gw, s0, s1, s2, s3, sw, *, loc=None, ip=None)
¶
The Lamport hand-off poll of one thread: up to four 16-byte activation items (bits 0..3 of pend) and one scale word (bit 4), all loaded in one round per spin; an item is accepted once none of its bytes is the sentinel (0x80 for activations, 0xFF for scales) and is stored to shared memory.
Source code in vllm/models/minimax_m3/nvidia/specialized_decode/kernels/front_kernel.py
ptx(text, *, loc=None, ip=None)
¶
ptx_p(text, pred, *, loc=None, ip=None)
¶
A PTX statement executed when pred != 0.
Source code in vllm/models/minimax_m3/nvidia/specialized_decode/kernels/front_kernel.py
ptx_rp(text, a, pred, *, loc=None, ip=None)
¶
A PTX statement with one register operand ($0), executed when pred != 0.
Source code in vllm/models/minimax_m3/nvidia/specialized_decode/kernels/front_kernel.py
sdesc(saddr, lbo, sbo, layout)
¶
tcgen05 shared memory descriptor of a 16-byte aligned shared address.
Source code in vllm/models/minimax_m3/nvidia/specialized_decode/kernels/front_kernel.py
stg_v4(addr, w0, w1, w2, w3, pred, *, loc=None, ip=None)
¶
st.global.v4.b32 (explicit .global: a generic store to an NVLS multicast address faults).
Source code in vllm/models/minimax_m3/nvidia/specialized_decode/kernels/front_kernel.py
tc_ld16(taddr, *, loc=None, ip=None)
¶
Two tcgen05.ld 32x32b.x8 (columns c, c + 8) and one wait.
Source code in vllm/models/minimax_m3/nvidia/specialized_decode/kernels/front_kernel.py
tc_ld8(taddr, *, loc=None, ip=None)
¶
tcgen05.ld 32x32b.x8 and its wait in one statement (the values exist only after the wait).
Source code in vllm/models/minimax_m3/nvidia/specialized_decode/kernels/front_kernel.py
tc_mma_ss(d, da, db, idesc, acc, sfa, sfb, *, loc=None, ip=None)
¶
Block-scaled MMA, A and B from shared memory descriptors.
Source code in vllm/models/minimax_m3/nvidia/specialized_decode/kernels/front_kernel.py
tc_mma_ts(d, ta, db, idesc, acc, sfa, sfb, *, loc=None, ip=None)
¶
Block-scaled MMA, A from TMEM, B from a shared memory descriptor.
Source code in vllm/models/minimax_m3/nvidia/specialized_decode/kernels/front_kernel.py
ue8m0(x, *, loc=None, ip=None)
¶
UE8M0 of x rounded up, satfinite (the low byte of cvt.rp.satfinite.ue8m0x2.f32 with a 0 partner).
Source code in vllm/models/minimax_m3/nvidia/specialized_decode/kernels/front_kernel.py
ue8m0_value(code, *, loc=None, ip=None)
¶
The float value of a UE8M0 code (through bf16, as __nv_fp8_e8m0 -> float).
Source code in vllm/models/minimax_m3/nvidia/specialized_decode/kernels/front_kernel.py
zero_byte_mask(t, *, loc=None, ip=None)
¶
0x80 in every byte of t that is zero, else 0 (exact per byte).