Skip to content

vllm.models.minimax_m3.nvidia.specialized_decode.kernels.page_topk_kernel

page_topk device code (CuTe DSL): exact indexer top-16 page selector for the sparse-attention decode.

256 threads, 4 witness lanes per warp; the selection is exact, so the table is the same for any launch shape, bit for bit.

Per (decode row, index head) one CTA selects, from the row's cap = min(n_valid, num_cols, 8192) pages, the forced pages (the first init and the last local) plus the need best other pages by (fp32 key desc, page asc): every thread holds KPT order-preserving keys in registers; per warp the R-th largest lane maximum is a witness, and the K-th largest witness (K = ceil(need / R)) bounds a pool that provably holds the top need. A pool of at most 64 entries is ranked by warp 0 (certificate); otherwise an exact block-wide MSD radix select runs on the register keys. Winners are written in ascending page order as physical page ids (-1 tail when fewer than 16 pages are valid).

split_mode (launches whose rows may exceed single_max pages): grid (rows x heads, splits). A row of at most single_max pages runs the one-CTA select in split 0 (the other splits exit before the grid wait); a longer row is cut into chunk-page CTAs that write their exact local top-need as key << 32 | ~page to part; the last chunk CTA to arrive (per-row counter in cnt, threadfence pattern, re-armed for graph replay) ranks the union and writes the row. No CTA waits for another.

PDL: n_valid and the page rows are read before griddepcontrol.wait (they are written before the predecessor), the scores after it; dependents are released right after the wait.

Classes:

  • Smem –

    Shared memory of one CTA (the CUDA struct Smem; the V2 per-warp counts are not used by V4).

Functions:

  • ctz –

    Index of the lowest set bit (__ffs(v) - 1 for v != 0).

  • ldcg_u32 –

    pred ? ld.global.cg.b32 [addr] : 0 (a predicated load: no access when pred is 0).

  • ldcg_u64 –

    pred ? ld.global.cg.b64 [addr] : 0.

  • okey –

    Order-preserving uint32 key of the fp32 bits.

  • pack64 –

    hi << 32 | lo of two 32-bit words.

  • shr_u32 –

    Logical right shift; shift amounts >= 32 give 0 (PTX clamps).

  • smem_addr –

    shared::cta address of int32 element idx of arr.

Smem

Shared memory of one CTA (the CUDA struct Smem; the V2 per-warp counts are not used by V4).

Source code in vllm/models/minimax_m3/nvidia/specialized_decode/kernels/page_topk_kernel.py
class Smem:
    """Shared memory of one CTA (the CUDA `struct Smem`; the V2 per-warp counts are not used by V4)."""

    def __init__(self):
        sp = cutlass.AddressSpace.smem
        self.spid = cutlass.Array(
            cutlass.Int32, MAX_CAP, space=sp, alignment=16
        )  # main-KV page rows (single rows)
        self.pk = cutlass.Array(
            cutlass.Int64, MAX_PART, space=sp, alignment=16
        )  # pool / merge: key << 32 | ~page
        self.hist = cutlass.Array(
            cutlass.Int32, 256 + 4, space=sp, alignment=16
        )  # radix; [256..259] ctl
        self.ckey = cutlass.Array(cutlass.Int32, 32, space=sp, alignment=16)
        self.cpg = cutlass.Array(cutlass.Int32, 32, space=sp, alignment=16)
        self.ws = cutlass.Array(
            cutlass.Int32, 32, space=sp, alignment=16
        )  # per-warp witness
        self.wsel = cutlass.Array(
            cutlass.Int32, TOPK, space=sp, alignment=16
        )  # winners' pages (any order)
        self.cnt = cutlass.Array(cutlass.Int32, 4, space=sp, alignment=16)

_ld(arr, idx)

arr[idx] (a free function: the DSL carries the receivers of method calls through dynamic regions).

Source code in vllm/models/minimax_m3/nvidia/specialized_decode/kernels/page_topk_kernel.py
def _ld(arr, idx):
    """arr[idx] (a free function: the DSL carries the receivers of method calls through dynamic regions)."""
    return arr.load(idx=idx)

_radix_select(k, incm, need, s, tid, kpt)

Exact top-need (key desc, page asc) among the included keys (bit j of incm): block-wide MSD radix select, 8-bit passes until <= 32 candidates, mass-tie path in page order. Winners -> s.wsel[0..need) (warp 0 synced).

Source code in vllm/models/minimax_m3/nvidia/specialized_decode/kernels/page_topk_kernel.py
@cute.jit
def _radix_select(k, incm, need, s, tid, kpt: cutlass.Constexpr):
    """Exact top-`need` (key desc, page asc) among the included keys (bit j of incm): block-wide MSD radix select,
    8-bit passes until <= 32 candidates, mass-tie path in page order. Winners -> s.wsel[0..need) (warp 0 synced)."""
    warp = tid >> 5
    lane = tid & 31
    prefix = cutlass.Int32(0)
    krem = need
    sh = cutlass.Int32(24)
    npass = cutlass.Int32(0)
    go = cutlass.Int32(1)
    cute.arch.sync_threads()
    while go != cutlass.Int32(0):
        _st(s.hist, cutlass.Int32(0), idx=tid)  # NT == 256 bins
        if tid == 0:
            _st(s.hist, cutlass.Int32(0), idx=256 + 3)
        cute.arch.sync_threads()
        first = npass == cutlass.Int32(0)
        for j in cutlass.range_constexpr(kpt):
            if (_bit(incm, j) != cutlass.Int32(0)) & (
                first | (shr_u32(k[j], sh + cutlass.Int32(8)) == prefix)
            ):
                atom_add_shared(
                    smem_addr(s.hist, shr_u32(k[j], sh) & cutlass.Int32(255)),
                    cutlass.Int32(1),
                )
        cute.arch.sync_threads()
        if warp == 0:
            c = [
                _ld(s.hist, cutlass.Int32(255 - j) - cutlass.Int32(8) * lane)
                for j in range(8)
            ]
            tot = cutlass.Int32(0)
            for j in cutlass.range_constexpr(8):
                tot = tot + c[j]
            incl = tot
            for o in cutlass.range_constexpr(5):
                t = shfl_up(incl, cutlass.Int32(1 << o))
                incl = _sel(lane >= cutlass.Int32(1 << o), incl + t, incl)
            a = incl - tot
            if (a < krem) & (incl >= krem):
                bin_ = cutlass.Int32(-1)
                above = cutlass.Int32(0)
                inb = cutlass.Int32(0)
                for j in cutlass.range_constexpr(8):
                    hit = (bin_ < cutlass.Int32(0)) & (a + c[j] >= krem)
                    bin_ = _sel(
                        hit, cutlass.Int32(255 - j) - cutlass.Int32(8) * lane, bin_
                    )
                    above = _sel(hit, a, above)
                    inb = _sel(hit, c[j], inb)
                    a = a + c[j]
                kr = krem - above
                _st(s.hist, (prefix << cutlass.Int32(8)) | bin_, idx=256)
                _st(s.hist, kr, idx=257)
                _st(
                    s.hist,
                    _sel(
                        need - kr + inb <= cutlass.Int32(32),
                        cutlass.Int32(1),
                        cutlass.Int32(0),
                    ),
                    idx=258,
                )
        cute.arch.sync_threads()
        prefix = _ld(s.hist, 256)
        krem = _ld(s.hist, 257)
        stop = _ld(s.hist, 258)
        if stop != cutlass.Int32(0):
            go = cutlass.Int32(0)
        else:
            sh = sh - cutlass.Int32(8)
            npass = npass + cutlass.Int32(1)
            go = _sel(npass < cutlass.Int32(4), cutlass.Int32(1), cutlass.Int32(0))
    if sh >= cutlass.Int32(0):
        thr = prefix << sh
        for j in cutlass.range_constexpr(kpt):
            if (_bit(incm, j) != cutlass.Int32(0)) & _ge_u32(k[j], thr):
                at = atom_add_shared(smem_addr(s.hist, 256 + 3), cutlass.Int32(1))
                _st(s.ckey, k[j], idx=at)
                _st(s.cpg, tid + cutlass.Int32(NT * j), idx=at)
    else:
        tv = prefix
        nt = krem
        for j in cutlass.range_constexpr(kpt):
            if (_bit(incm, j) != cutlass.Int32(0)) & _gt_u32(k[j], tv):
                at = atom_add_shared(smem_addr(s.hist, 256 + 3), cutlass.Int32(1))
                _st(s.ckey, k[j], idx=at)
                _st(s.cpg, tid + cutlass.Int32(NT * j), idx=at)
        run = cutlass.Int32(0)
        for j in cutlass.range_constexpr(kpt):
            if run < nt:  # block-uniform
                tie = (_bit(incm, j) != cutlass.Int32(0)) & (k[j] == tv)
                bm = ballot(_sel(tie, cutlass.Int32(1), cutlass.Int32(0)))
                cute.arch.sync_threads()
                if lane == 0:
                    _st(s.hist, popc(bm), idx=warp)
                cute.arch.sync_threads()
                woff = cutlass.Int32(0)
                tot = cutlass.Int32(0)
                for w in cutlass.range_constexpr(W):
                    cw = _ld(s.hist, w)
                    woff = woff + _sel(cutlass.Int32(w) < warp, cw, cutlass.Int32(0))
                    tot = tot + cw
                if tie & (run + woff + popc(bm & lanemask_lt()) < nt):
                    at = atom_add_shared(smem_addr(s.hist, 256 + 3), cutlass.Int32(1))
                    _st(s.ckey, k[j], idx=at)
                    _st(s.cpg, tid + cutlass.Int32(NT * j), idx=at)
                run = run + tot
    cute.arch.sync_threads()
    if warp == 0:
        nc = _ld(s.hist, 256 + 3)
        ki = _sel(lane < nc, _ld(s.ckey, lane), cutlass.Int32(0))
        pi = _sel(lane < nc, _ld(s.cpg, lane), cutlass.Int32(INT_MAX))
        r = cutlass.Int32(0)
        for j in cutlass.range_constexpr(32):
            kj = shfl_idx(ki, cutlass.Int32(j))
            pj = shfl_idx(pi, cutlass.Int32(j))
            r = r + _sel(
                _gt_u32(kj, ki) | ((kj == ki) & (pj < pi)),
                cutlass.Int32(1),
                cutlass.Int32(0),
            )
        win = (lane < nc) & (r < need)
        bw = ballot(_sel(win, cutlass.Int32(1), cutlass.Int32(0)))
        if win:
            _st(s.wsel, pi, idx=popc(bw & lanemask_lt()))
        cute.arch.sync_warp()

_select_body(s, srw, orw, cap, init, fend, flo, need, tid, diag, has_diag, kpt, part_mode, part, c0)

Witness-select body for one row (or one chunk when part_mode: srw and pages chunk-local, winners -> part).

Source code in vllm/models/minimax_m3/nvidia/specialized_decode/kernels/page_topk_kernel.py
@cute.jit
def _select_body(
    s,
    srw: cutlass.Int64,
    orw: cutlass.Int64,
    cap,
    init,
    fend,
    flo,
    need,
    tid,
    diag: cutlass.Int64,
    has_diag: cutlass.Constexpr,
    kpt: cutlass.Constexpr,
    part_mode: cutlass.Constexpr,
    part: cutlass.Int64,
    c0,
):
    """Witness-select body for one row (or one chunk when `part_mode`: srw and pages chunk-local, winners -> `part`)."""
    warp = tid >> 5
    lane = tid & 31
    # one batch of independent loads (the predicate depends only on pre-wait data)
    k = []
    for j in cutlass.range_constexpr(kpt):
        p = tid + cutlass.Int32(NT * j)
        k.append(
            ldcg_u32(
                srw + cutlass.Int64(p) * cutlass.Int64(4),
                _sel(p < cap, cutlass.Int32(1), cutlass.Int32(0)),
            )
        )
    incm = cutlass.Int32(0)
    lm = cutlass.Int32(0)
    for j in cutlass.range_constexpr(kpt):
        p = tid + cutlass.Int32(NT * j)
        k[j] = okey(k[j])
        v = (p < cap) & (p >= init) & (p < fend)  # non-forced, selectable
        incm = incm | _sel(
            v, cutlass.Int32(-2147483648 if j == 31 else 1 << j), cutlass.Int32(0)
        )
        lm = _sel(v & _gt_u32(k[j], lm), k[j], lm)
    # per-warp top-R lane maxima (one lane removed per round): S = R-th largest
    cur = lm
    sv_w = cutlass.Int32(0)
    for r in cutlass.range_constexpr(R):
        m = redux_max_u32(cur)
        sv_w = m
        if cutlass.const_expr(r + 1 < R):
            b = ballot(_sel(cur == m, cutlass.Int32(1), cutlass.Int32(0)))
            cur = _sel(lane == ctz(b), cutlass.Int32(0), cur)
    if lane == 0:
        _st(s.ws, sv_w, idx=warp)
    cute.arch.sync_threads()
    # T = K-th largest S_w, K = ceil(need / R): K warps x R distinct threads hold keys >= T
    kk = (need + cutlass.Int32(R - 1)) // cutlass.Int32(R)
    sv = _sel(lane < W, _ld(s.ws, _min(lane, cutlass.Int32(W - 1))), cutlass.Int32(0))
    c = cutlass.Int32(0)
    for v in cutlass.range_constexpr(W):
        c = c + _sel(_ge_u32(_ld(s.ws, v), sv), cutlass.Int32(1), cutlass.Int32(0))
    tmax = redux_max_u32(_sel((lane < W) & (c >= kk), sv, cutlass.Int32(0)))
    T = _sel(kk > cutlass.Int32(W), cutlass.Int32(0), tmax)
    km = cutlass.Int32(0)
    for j in cutlass.range_constexpr(kpt):
        km = km | _sel(
            (_bit(incm, j) != cutlass.Int32(0)) & _ge_u32(k[j], T),
            cutlass.Int32(-2147483648 if j == 31 else 1 << j),
            cutlass.Int32(0),
        )
    if km != cutlass.Int32(0):
        at = atom_add_shared(smem_addr(s.cnt, 0), popc(km))
        for j in cutlass.range_constexpr(kpt):
            bj = _bit(km, j)
            if (bj != cutlass.Int32(0)) & (at < cutlass.Int32(POOL)):
                _st(s.pk, pack64(k[j], ~(tid + cutlass.Int32(NT * j))), idx=at)
            at = at + bj
    cute.arch.sync_threads()
    n = _ld(s.cnt, 0)
    if (n <= cutlass.Int32(POOL)) & (n >= need):  # certificate (block-uniform)
        if warp == 0:
            e0 = _sel(lane < n, _ld(s.pk, lane), cutlass.Int64(0))
            e1 = _sel(
                lane + cutlass.Int32(32) < n,
                _ld(s.pk, lane + cutlass.Int32(32)),
                cutlass.Int64(0),
            )
            r0 = cutlass.Int32(0)
            r1 = cutlass.Int32(0)
            if n <= cutlass.Int32(32):
                for t in cutlass.range_constexpr(32):
                    r0 = r0 + _sel(
                        _gt_u64(_ld(s.pk, t), e0), cutlass.Int32(1), cutlass.Int32(0)
                    )
            else:
                for t in cutlass.range_constexpr(POOL):
                    x = _ld(s.pk, t)
                    r0 = r0 + _sel(_gt_u64(x, e0), cutlass.Int32(1), cutlass.Int32(0))
                    r1 = r1 + _sel(_gt_u64(x, e1), cutlass.Int32(1), cutlass.Int32(0))
            w0 = (lane < n) & (r0 < need)
            w1 = (lane + cutlass.Int32(32) < n) & (r1 < need)
            b0 = ballot(_sel(w0, cutlass.Int32(1), cutlass.Int32(0)))
            b1 = ballot(_sel(w1, cutlass.Int32(1), cutlass.Int32(0)))
            lt = lanemask_lt()
            if w0:
                _st(s.wsel, ~lo32(e0), idx=popc(b0 & lt))
            if w1:
                _st(s.wsel, ~lo32(e1), idx=popc(b0) + popc(b1 & lt))
            cute.arch.sync_warp()
            if cutlass.const_expr(part_mode):
                _store_partial(s, need, srw, c0, part, lane)
            else:
                _store_page_order(s, need, flo, orw, lane, True, orw)
            _diag(
                diag,
                has_diag,
                _sel(
                    n <= cutlass.Int32(32),
                    cutlass.Int32(DIAG_FAST32),
                    cutlass.Int32(DIAG_FAST64),
                ),
                lane == 0,
            )
    else:
        _radix_select(
            k, incm, need, s, tid, kpt
        )  # exact fallback on the register keys (no reload)
        if warp == 0:
            if cutlass.const_expr(part_mode):
                _store_partial(s, need, srw, c0, part, lane)
            else:
                _store_page_order(s, need, flo, orw, lane, True, orw)
            _diag(diag, has_diag, cutlass.Int32(DIAG_FALLBACK), lane == 0)

_single_tiers(s, srw, orw, cap, init, fend, flo, need, tid, diag, has_diag)

The one-CTA select: 6, 16 or 32 keys per thread.

Source code in vllm/models/minimax_m3/nvidia/specialized_decode/kernels/page_topk_kernel.py
@cute.jit
def _single_tiers(
    s, srw, orw, cap, init, fend, flo, need, tid, diag, has_diag: cutlass.Constexpr
):
    """The one-CTA select: 6, 16 or 32 keys per thread."""
    if cap <= cutlass.Int32(NT * 6):
        _select_body(
            s,
            srw,
            orw,
            cap,
            init,
            fend,
            flo,
            need,
            tid,
            diag,
            has_diag,
            6,
            False,
            cutlass.Int64(0),
            cutlass.Int32(0),
        )
    elif cap <= cutlass.Int32(NT * 16):
        _select_body(
            s,
            srw,
            orw,
            cap,
            init,
            fend,
            flo,
            need,
            tid,
            diag,
            has_diag,
            16,
            False,
            cutlass.Int64(0),
            cutlass.Int32(0),
        )
    else:
        _select_body(
            s,
            srw,
            orw,
            cap,
            init,
            fend,
            flo,
            need,
            tid,
            diag,
            has_diag,
            MAX_CAP // NT,
            False,
            cutlass.Int64(0),
            cutlass.Int32(0),
        )

_stage_rows(s, prw, cap, tid)

s.spid[p] = page_rows[p] for p < cap: SPID_BATCH predicated loads first, then the stores.

Source code in vllm/models/minimax_m3/nvidia/specialized_decode/kernels/page_topk_kernel.py
@cute.jit
def _stage_rows(s, prw: cutlass.Int64, cap, tid):
    """s.spid[p] = page_rows[p] for p < cap: SPID_BATCH predicated loads first, then the stores."""
    for b in cutlass.range(tid, cap, NT * SPID_BATCH):
        v = [
            ldcg_u32(
                prw + cutlass.Int64(b + cutlass.Int32(NT * i)) * cutlass.Int64(4),
                _sel(
                    b + cutlass.Int32(NT * i) < cap, cutlass.Int32(1), cutlass.Int32(0)
                ),
            )
            for i in range(SPID_BATCH)
        ]
        for i in cutlass.range_constexpr(SPID_BATCH):
            if b + cutlass.Int32(NT * i) < cap:
                _st(s.spid, v[i], idx=b + cutlass.Int32(NT * i))

_store_page_order(s, need, base, orw, lane, from_spid, prw)

Winners' pages in s.wsel[0..need) -> page-order store at orw[base + pos] (warp 0; V4: smem reads).

Source code in vllm/models/minimax_m3/nvidia/specialized_decode/kernels/page_topk_kernel.py
@cute.jit
def _store_page_order(
    s,
    need,
    base,
    orw: cutlass.Int64,
    lane,
    from_spid: cutlass.Constexpr,
    prw: cutlass.Int64,
):
    """Winners' pages in s.wsel[0..need) -> page-order store at orw[base + pos] (warp 0; V4: smem reads)."""
    pg = _sel(
        lane < need,
        _ld(s.wsel, _min(lane, cutlass.Int32(TOPK - 1))),
        cutlass.Int32(INT_MAX),
    )
    pos = cutlass.Int32(0)
    for j in cutlass.range_constexpr(TOPK):
        pos = pos + _sel(_ld(s.wsel, j) < pg, cutlass.Int32(1), cutlass.Int32(0))
    if lane < need:
        if cutlass.const_expr(from_spid):
            v = _ld(s.spid, pg)
        else:
            v = ldcg_u32(prw + cutlass.Int64(pg) * cutlass.Int64(4), cutlass.Int32(1))
        st_s32(orw + cutlass.Int64(base + pos) * cutlass.Int64(4), v)

_store_partial(s, need, srw, c0, part, lane)

Chunk CTA: winners in s.wsel[0..need) (chunk-local pages) -> 16 entries key << 32 | ~(c0 + page), zero tail.

Source code in vllm/models/minimax_m3/nvidia/specialized_decode/kernels/page_topk_kernel.py
@cute.jit
def _store_partial(s, need, srw: cutlass.Int64, c0, part: cutlass.Int64, lane):
    """Chunk CTA: winners in s.wsel[0..need) (chunk-local pages) -> 16 entries key << 32 | ~(c0 + page), zero tail."""
    e = cutlass.Int64(0)
    if lane < need:
        pg = _ld(s.wsel, _min(lane, cutlass.Int32(TOPK - 1)))
        key = okey(
            ldcg_u32(srw + cutlass.Int64(pg) * cutlass.Int64(4), cutlass.Int32(1))
        )
        e = pack64(key, ~(c0 + pg))
    if lane < TOPK:
        st_u64(part + cutlass.Int64(lane) * cutlass.Int64(8), e)

ctz(v, *, loc=None, ip=None)

Index of the lowest set bit (__ffs(v) - 1 for v != 0).

Source code in vllm/models/minimax_m3/nvidia/specialized_decode/kernels/page_topk_kernel.py
@dsl_user_op
def ctz(v, *, loc=None, ip=None):
    """Index of the lowest set bit (`__ffs(v) - 1` for v != 0)."""
    return cutlass.Int32(_asm(_i32(), [cutlass.Int32(v)], "{\n\t.reg .b32 t;\n\tbrev.b32 t, $1;\n\tclz.b32 $0, t;\n\t}",
                              "=r,r", False, loc, ip))  # fmt: skip

ldcg_u32(addr, pred, *, loc=None, ip=None)

pred ? ld.global.cg.b32 [addr] : 0 (a predicated load: no access when pred is 0).

Source code in vllm/models/minimax_m3/nvidia/specialized_decode/kernels/page_topk_kernel.py
@dsl_user_op
def ldcg_u32(addr, pred, *, loc=None, ip=None):
    """`pred ? ld.global.cg.b32 [addr] : 0` (a predicated load: no access when pred is 0)."""
    return cutlass.Int32(_asm(_i32(), [cutlass.Int64(addr), cutlass.Int32(pred)],
                              "{\n\t.reg .pred p;\n\tsetp.ne.b32 p, $2, 0;\n\tmov.b32 $0, 0;\n\t"
                              "@p ld.global.cg.b32 $0, [$1];\n\t}", "=r,l,r", True, loc, ip))  # fmt: skip

ldcg_u64(addr, pred, *, loc=None, ip=None)

pred ? ld.global.cg.b64 [addr] : 0.

Source code in vllm/models/minimax_m3/nvidia/specialized_decode/kernels/page_topk_kernel.py
@dsl_user_op
def ldcg_u64(addr, pred, *, loc=None, ip=None):
    """`pred ? ld.global.cg.b64 [addr] : 0`."""
    return cutlass.Int64(_asm(_i64(), [cutlass.Int64(addr), cutlass.Int32(pred)],
                              "{\n\t.reg .pred p;\n\tsetp.ne.b32 p, $2, 0;\n\tmov.b64 $0, 0;\n\t"
                              "@p ld.global.cg.b64 $0, [$1];\n\t}", "=l,l,r", True, loc, ip))  # fmt: skip

okey(raw)

Order-preserving uint32 key of the fp32 bits.

Source code in vllm/models/minimax_m3/nvidia/specialized_decode/kernels/page_topk_kernel.py
def okey(raw):
    """Order-preserving uint32 key of the fp32 bits."""
    return cutlass.select_(
        raw < cutlass.Int32(0), ~raw, raw | cutlass.Int32(-2147483648)
    )

pack64(hi, lo, *, loc=None, ip=None)

hi << 32 | lo of two 32-bit words.

Source code in vllm/models/minimax_m3/nvidia/specialized_decode/kernels/page_topk_kernel.py
@dsl_user_op
def pack64(hi, lo, *, loc=None, ip=None):
    """`hi << 32 | lo` of two 32-bit words."""
    return cutlass.Int64(_asm(_i64(), [cutlass.Int32(lo), cutlass.Int32(hi)], "mov.b64 $0, {$1, $2};", "=l,r,r",
                              False, loc, ip))  # fmt: skip

shr_u32(v, s, *, loc=None, ip=None)

Logical right shift; shift amounts >= 32 give 0 (PTX clamps).

Source code in vllm/models/minimax_m3/nvidia/specialized_decode/kernels/page_topk_kernel.py
@dsl_user_op
def shr_u32(v, s, *, loc=None, ip=None):
    """Logical right shift; shift amounts >= 32 give 0 (PTX clamps)."""
    return cutlass.Int32(_asm(_i32(), [cutlass.Int32(v), cutlass.Int32(s)], "shr.b32 $0, $1, $2;", "=r,r,r", False,
                              loc, ip))  # fmt: skip

smem_addr(arr, idx, *, loc=None, ip=None)

shared::cta address of int32 element idx of arr.

Source code in vllm/models/minimax_m3/nvidia/specialized_decode/kernels/page_topk_kernel.py
@dsl_user_op
def smem_addr(arr, idx, *, loc=None, ip=None):
    """shared::cta address of int32 element `idx` of `arr`."""
    return cutlass.Int32(arr.data_ptr().toint(loc=loc, ip=ip)) + cutlass.Int32(
        idx
    ) * cutlass.Int32(4)