Skip to content

vllm.models.deepseek_v41.nvidia.specialized_decode.ampere.codecs

CuTe DSL device helpers of the SM80 kernels: global loads by address and FP8/FP4 decoding with integer operations.

SM80 has no instruction that converts E4M3 or E2M1 to a wider type (they arrive with SM89 and SM100), so these decoders place a code's bits in an FP32 word and scale it by a power of two:

  • E4M3 s eeee mmm: the FP32 word s | eeeemmm << 20 has exponent field eeee and mantissa mmm; times 2^120 it is the E4M3 value, the subnormals (eeee = 0) included, which are FP32 subnormals before the multiplication. FP32 arithmetic keeps subnormals unless a kernel opts into flush-to-zero, which these kernels do not.
  • E2M1 s eem: the FP32 word s | eem << 22 times 2^126.

Every product is a power-of-two scaling of a value with at most four significant bits, so it is exact, and so is its BF16 rounding.

Kernels that only need BF16 decode two codes at a time into the halves of a 32-bit word (e4m3x2_bf16, e2m1x2_bf16): the same bit placement in BF16, whose exponent field is FP32's, then the power of two and the block scale applied with two fma.rn.bf16x2. Each product is exact, so the result is the bits the FP32 decoders round to.

Functions:

  • bits_from_f32 –

    The 32-bit word of an FP32 value.

  • byte_perm –

    prmt.b32: byte i of the result is the byte of (a, b)

  • ceil_log2 –

    ceil(log2(x)) of a positive normal FP32 value, exactly.

  • e2m1 –

    An E2M1 nibble (the low 4 bits of code) as FP32.

  • e2m1x2_bf16 –

    E2M1 nibbles 2 byte and 2 byte + 1 of word times the

  • e4m3 –

    An E4M3 byte (the low 8 bits of code) as FP32.

  • e4m3_bf16x2 –

    An E4M3 byte as BF16 in both halves.

  • e4m3x2_bf16 –

    E4M3 bytes 2 pair and 2 pair + 1 of word times the BF16x2

  • exp2_to_f32 –

    2^exponent for a normal FP32 exponent.

  • f32_from_bits –

    The FP32 value of a 32-bit word.

  • fma_bf16x2 –

    a * b + c on both BF16 halves of 32-bit words, rounded to

  • issue_s32 –

    load_s32 issued where it is written (see _issue).

  • issue_u32 –

    load_u32 issued where it is written (see _issue).

  • load_s32 –

    One 32-bit global load of a signed index.

  • load_u32 –

    One 32-bit global load.

  • store_u16 –

    Store two bytes (2-byte aligned).

  • store_u32 –

    Store four bytes (4-byte aligned).

  • store_u8 –

    Store one byte.

  • to_e2m1 –

    FP32 to an E2M1 nibble, nearest-even and saturating: the magnitude code

  • to_e4m3 –

    A positive FP32 value in [2^-9, 448] to an E4M3 byte, nearest-even.

  • ue8m0 –

    A UE8M0 scale byte as FP32 (the byte is the exponent field).

  • ue8m0_bf16x2 –

    A UE8M0 scale byte as BF16 in both halves (the byte is the exponent

_asm(result, operands, text, constraints, *, loc, ip)

A side-effect-free inline PTX statement.

Source code in vllm/models/deepseek_v41/nvidia/specialized_decode/ampere/codecs.py
def _asm(result, operands, text, constraints, *, loc, ip):
    """A side-effect-free inline PTX statement."""
    return llvm.inline_asm(
        result,
        operands,
        text,
        constraints,
        has_side_effects=False,
        is_align_stack=False,
        asm_dialect=llvm.AsmDialect.AD_ATT,
        loc=loc,
        ip=ip,
    )

_issue(kind, address, *, loc, ip)

A 32-bit global load that stays where it is written.

load_u32 is free to move: the compiler may sink it into the branch that uses its value, behind the loads of earlier branches. A kernel that issues a batch of these before decoding them keeps the whole batch in flight at once.

Source code in vllm/models/deepseek_v41/nvidia/specialized_decode/ampere/codecs.py
def _issue(kind: str, address: Int64, *, loc, ip):
    """A 32-bit global load that stays where it is written.

    `load_u32` is free to move: the compiler may sink it into the branch
    that uses its value, behind the loads of earlier branches. A kernel that
    issues a batch of these before decoding them keeps the whole batch in
    flight at once.
    """
    return llvm.inline_asm(
        T.i32(),
        [Int64(address).ir_value(loc=loc, ip=ip)],
        f"ld.global.nc.{kind} $0, [$1];",
        "=r,l",
        has_side_effects=True,
        is_align_stack=False,
        asm_dialect=llvm.AsmDialect.AD_ATT,
        loc=loc,
        ip=ip,
    )

_scale_bf16x2(x, power, scale)

BF16x2 x * power * scale, both products exact.

Source code in vllm/models/deepseek_v41/nvidia/specialized_decode/ampere/codecs.py
@cute.jit
def _scale_bf16x2(x: Uint32, power: Constexpr[int], scale: Uint32) -> Uint32:
    """BF16x2 `x * power * scale`, both products exact."""
    negative_zero = Uint32(NEGATIVE_ZERO_BF16X2)
    x = fma_bf16x2(x, Uint32(power), negative_zero)
    return fma_bf16x2(x, scale, negative_zero)

_store(width, address, value, *, loc, ip)

One global store of the low width bits of value.

Source code in vllm/models/deepseek_v41/nvidia/specialized_decode/ampere/codecs.py
def _store(width: str, address: Int64, value: Uint32, *, loc, ip) -> None:
    """One global store of the low `width` bits of `value`."""
    llvm.inline_asm(
        None,
        [
            Int64(address).ir_value(loc=loc, ip=ip),
            Uint32(value).ir_value(loc=loc, ip=ip),
        ],
        f"st.global.{width} [$0], $1;",
        "l,r",
        has_side_effects=True,
        is_align_stack=False,
        asm_dialect=llvm.AsmDialect.AD_ATT,
        loc=loc,
        ip=ip,
    )

bits_from_f32(value, *, loc=None, ip=None)

The 32-bit word of an FP32 value.

Source code in vllm/models/deepseek_v41/nvidia/specialized_decode/ampere/codecs.py
@dsl_user_op
def bits_from_f32(value: Float32, *, loc=None, ip=None) -> Uint32:
    """The 32-bit word of an FP32 value."""
    return Uint32(
        llvm.bitcast(T.i32(), Float32(value).ir_value(loc=loc, ip=ip), loc=loc, ip=ip)
    )

byte_perm(a, b, selector, *, loc=None, ip=None)

prmt.b32: byte i of the result is the byte of (a, b) (bytes 0-3 of a, then 4-7 of b) named by nibble i of selector.

Source code in vllm/models/deepseek_v41/nvidia/specialized_decode/ampere/codecs.py
@dsl_user_op
def byte_perm(a: Uint32, b: Uint32, selector: int, *, loc=None, ip=None) -> Uint32:
    """`prmt.b32`: byte `i` of the result is the byte of `(a, b)`
    (bytes 0-3 of `a`, then 4-7 of `b`) named by nibble `i` of
    `selector`."""
    return Uint32(
        _asm(
            T.i32(),
            [Uint32(a).ir_value(loc=loc, ip=ip), Uint32(b).ir_value(loc=loc, ip=ip)],
            f"prmt.b32 $0, $1, $2, {selector:#x};",
            "=r,r,r",
            loc=loc,
            ip=ip,
        )
    )

ceil_log2(x)

ceil(log2(x)) of a positive normal FP32 value, exactly.

Source code in vllm/models/deepseek_v41/nvidia/specialized_decode/ampere/codecs.py
@cute.jit
def ceil_log2(x: Float32) -> Int32:
    """`ceil(log2(x))` of a positive normal FP32 value, exactly."""
    bits = bits_from_f32(x)
    exponent = Int32((bits >> 23) & 0xFF) - 127
    if (bits & 0x7FFFFF) != 0:
        exponent = exponent + 1
    return exponent

e2m1(code)

An E2M1 nibble (the low 4 bits of code) as FP32.

Source code in vllm/models/deepseek_v41/nvidia/specialized_decode/ampere/codecs.py
@cute.jit
def e2m1(code: Uint32) -> Float32:
    """An E2M1 nibble (the low 4 bits of `code`) as FP32."""
    bits = ((code & 0x8) << 28) | ((code & 0x7) << 22)
    return f32_from_bits(bits) * Float32(E2M1_SCALE)

e2m1x2_bf16(word, byte, scale)

E2M1 nibbles 2 byte and 2 byte + 1 of word times the BF16x2 scale, as BF16x2: code s eem becomes the BF16 s | eem << 6 (the value times 2^-126).

Source code in vllm/models/deepseek_v41/nvidia/specialized_decode/ampere/codecs.py
@cute.jit
def e2m1x2_bf16(word: Uint32, byte: Constexpr[int], scale: Uint32) -> Uint32:
    """E2M1 nibbles `2 byte` and `2 byte + 1` of `word` times the
    BF16x2 `scale`, as BF16x2: code `s eem` becomes the BF16
    `s | eem << 6` (the value times `2^-126`)."""
    x = byte_perm(word, Uint32(0), 0x4440 + byte)
    # The byte at bits 0-7 and 12-19: the low nibble of each half is a code.
    x = x * Uint32(4097)
    x = ((x << 12) & 0x80008000) | ((x << 6) & 0x01C001C0)
    return _scale_bf16x2(x, E2M1_SCALE_BF16X2, scale)

e4m3(code)

An E4M3 byte (the low 8 bits of code) as FP32.

Source code in vllm/models/deepseek_v41/nvidia/specialized_decode/ampere/codecs.py
@cute.jit
def e4m3(code: Uint32) -> Float32:
    """An E4M3 byte (the low 8 bits of `code`) as FP32."""
    bits = ((code & 0x80) << 24) | ((code & 0x7F) << 20)
    return f32_from_bits(bits) * Float32(E4M3_SCALE)

e4m3_bf16x2(code)

An E4M3 byte as BF16 in both halves.

Source code in vllm/models/deepseek_v41/nvidia/specialized_decode/ampere/codecs.py
@cute.jit
def e4m3_bf16x2(code: Uint32) -> Uint32:
    """An E4M3 byte as BF16 in both halves."""
    x = (((code & 0x80) << 8) | ((code & 0x7F) << 4)) * Uint32(0x00010001)
    return fma_bf16x2(x, Uint32(E4M3_SCALE_BF16X2), Uint32(NEGATIVE_ZERO_BF16X2))

e4m3x2_bf16(word, pair, scale)

E4M3 bytes 2 pair and 2 pair + 1 of word times the BF16x2 scale, as BF16x2: code s eeee mmm becomes the BF16 s | eeeemmm << 4 (the value times 2^-120, subnormals included).

Source code in vllm/models/deepseek_v41/nvidia/specialized_decode/ampere/codecs.py
@cute.jit
def e4m3x2_bf16(word: Uint32, pair: Constexpr[int], scale: Uint32) -> Uint32:
    """E4M3 bytes `2 pair` and `2 pair + 1` of `word` times the BF16x2
    `scale`, as BF16x2: code `s eeee mmm` becomes the BF16
    `s | eeeemmm << 4` (the value times `2^-120`, subnormals included)."""
    x = byte_perm(word, Uint32(0), 0x4140 + 0x0202 * pair)
    x = ((x << 8) & 0x80008000) | ((x << 4) & 0x07F007F0)
    return _scale_bf16x2(x, E4M3_SCALE_BF16X2, scale)

exp2_to_f32(exponent)

2^exponent for a normal FP32 exponent.

Source code in vllm/models/deepseek_v41/nvidia/specialized_decode/ampere/codecs.py
@cute.jit
def exp2_to_f32(exponent: Int32) -> Float32:
    """`2^exponent` for a normal FP32 exponent."""
    return f32_from_bits(Uint32(exponent + 127) << 23)

f32_from_bits(bits, *, loc=None, ip=None)

The FP32 value of a 32-bit word.

Source code in vllm/models/deepseek_v41/nvidia/specialized_decode/ampere/codecs.py
@dsl_user_op
def f32_from_bits(bits: Uint32, *, loc=None, ip=None) -> Float32:
    """The FP32 value of a 32-bit word."""
    return Float32(
        llvm.bitcast(T.f32(), Uint32(bits).ir_value(loc=loc, ip=ip), loc=loc, ip=ip)
    )

fma_bf16x2(a, b, c, *, loc=None, ip=None)

a * b + c on both BF16 halves of 32-bit words, rounded to nearest; subnormals are kept.

Source code in vllm/models/deepseek_v41/nvidia/specialized_decode/ampere/codecs.py
@dsl_user_op
def fma_bf16x2(a: Uint32, b: Uint32, c: Uint32, *, loc=None, ip=None) -> Uint32:
    """`a * b + c` on both BF16 halves of 32-bit words, rounded to
    nearest; subnormals are kept."""
    return Uint32(
        _asm(
            T.i32(),
            [
                Uint32(a).ir_value(loc=loc, ip=ip),
                Uint32(b).ir_value(loc=loc, ip=ip),
                Uint32(c).ir_value(loc=loc, ip=ip),
            ],
            "fma.rn.bf16x2 $0, $1, $2, $3;",
            "=r,r,r,r",
            loc=loc,
            ip=ip,
        )
    )

issue_s32(address, *, loc=None, ip=None)

load_s32 issued where it is written (see _issue).

Source code in vllm/models/deepseek_v41/nvidia/specialized_decode/ampere/codecs.py
@dsl_user_op
def issue_s32(address: Int64, *, loc=None, ip=None) -> Int32:
    """`load_s32` issued where it is written (see `_issue`)."""
    return Int32(_issue("s32", address, loc=loc, ip=ip))

issue_u32(address, *, loc=None, ip=None)

load_u32 issued where it is written (see _issue).

Source code in vllm/models/deepseek_v41/nvidia/specialized_decode/ampere/codecs.py
@dsl_user_op
def issue_u32(address: Int64, *, loc=None, ip=None) -> Uint32:
    """`load_u32` issued where it is written (see `_issue`)."""
    return Uint32(_issue("u32", address, loc=loc, ip=ip))

load_s32(address, *, loc=None, ip=None)

One 32-bit global load of a signed index.

Source code in vllm/models/deepseek_v41/nvidia/specialized_decode/ampere/codecs.py
@dsl_user_op
def load_s32(address: Int64, *, loc=None, ip=None) -> Int32:
    """One 32-bit global load of a signed index."""
    return Int32(
        _asm(
            T.i32(),
            [Int64(address).ir_value(loc=loc, ip=ip)],
            "ld.global.nc.s32 $0, [$1];",
            "=r,l",
            loc=loc,
            ip=ip,
        )
    )

load_u32(address, *, loc=None, ip=None)

One 32-bit global load.

Source code in vllm/models/deepseek_v41/nvidia/specialized_decode/ampere/codecs.py
@dsl_user_op
def load_u32(address: Int64, *, loc=None, ip=None) -> Uint32:
    """One 32-bit global load."""
    return Uint32(
        _asm(
            T.i32(),
            [Int64(address).ir_value(loc=loc, ip=ip)],
            "ld.global.nc.u32 $0, [$1];",
            "=r,l",
            loc=loc,
            ip=ip,
        )
    )

store_u16(address, value, *, loc=None, ip=None)

Store two bytes (2-byte aligned).

Source code in vllm/models/deepseek_v41/nvidia/specialized_decode/ampere/codecs.py
@dsl_user_op
def store_u16(address: Int64, value: Uint32, *, loc=None, ip=None) -> None:
    """Store two bytes (2-byte aligned)."""
    _store("u16", address, value, loc=loc, ip=ip)

store_u32(address, value, *, loc=None, ip=None)

Store four bytes (4-byte aligned).

Source code in vllm/models/deepseek_v41/nvidia/specialized_decode/ampere/codecs.py
@dsl_user_op
def store_u32(address: Int64, value: Uint32, *, loc=None, ip=None) -> None:
    """Store four bytes (4-byte aligned)."""
    _store("u32", address, value, loc=loc, ip=ip)

store_u8(address, value, *, loc=None, ip=None)

Store one byte.

Source code in vllm/models/deepseek_v41/nvidia/specialized_decode/ampere/codecs.py
@dsl_user_op
def store_u8(address: Int64, value: Uint32, *, loc=None, ip=None) -> None:
    """Store one byte."""
    _store("u8", address, value, loc=loc, ip=ip)

to_e2m1(x)

FP32 to an E2M1 nibble, nearest-even and saturating: the magnitude code counts the rounding thresholds below |x| (at 0.75, 1.75 and 3.5 the even code is the upper one).

Source code in vllm/models/deepseek_v41/nvidia/specialized_decode/ampere/codecs.py
@cute.jit
def to_e2m1(x: Float32) -> Uint32:
    """FP32 to an E2M1 nibble, nearest-even and saturating: the magnitude code
    counts the rounding thresholds below `|x|` (at 0.75, 1.75 and 3.5 the
    even code is the upper one)."""
    a = cute.math.absf(x)
    code = Uint32(0)
    if a > 0.25:
        code = code + 1
    if a >= 0.75:
        code = code + 1
    if a > 1.25:
        code = code + 1
    if a >= 1.75:
        code = code + 1
    if a > 2.5:
        code = code + 1
    if a >= 3.5:
        code = code + 1
    if a > 5.0:
        code = code + 1
    return code | ((bits_from_f32(x) >> 28) & 0x8)

to_e4m3(x)

A positive FP32 value in [2^-9, 448] to an E4M3 byte, nearest-even.

Normal codes round the FP32 exponent-and-mantissa field to three mantissa bits (a carry moves to the exponent); below 2^-6 the code is the multiple of 2^-9, rounded by the FP32 magic-number addition.

Source code in vllm/models/deepseek_v41/nvidia/specialized_decode/ampere/codecs.py
@cute.jit
def to_e4m3(x: Float32) -> Uint32:
    """A positive FP32 value in `[2^-9, 448]` to an E4M3 byte, nearest-even.

    Normal codes round the FP32 exponent-and-mantissa field to three mantissa
    bits (a carry moves to the exponent); below `2^-6` the code is the
    multiple of `2^-9`, rounded by the FP32 magic-number addition.
    """
    code = Uint32(0)
    if x >= 0.015625:
        bits = bits_from_f32(x)
        code = ((bits + 0x7FFFF + ((bits >> 20) & 1)) >> 20) - ((127 - 7) << 3)
    else:
        magic = Float32(12582912.0)  # 1.5 * 2^23
        code = bits_from_f32(x * 512.0 + magic) - bits_from_f32(magic)
    return code

ue8m0(code)

A UE8M0 scale byte as FP32 (the byte is the exponent field).

Source code in vllm/models/deepseek_v41/nvidia/specialized_decode/ampere/codecs.py
@cute.jit
def ue8m0(code: Uint32) -> Float32:
    """A UE8M0 scale byte as FP32 (the byte is the exponent field)."""
    return f32_from_bits(code << 23)

ue8m0_bf16x2(code)

A UE8M0 scale byte as BF16 in both halves (the byte is the exponent field).

Source code in vllm/models/deepseek_v41/nvidia/specialized_decode/ampere/codecs.py
@cute.jit
def ue8m0_bf16x2(code: Uint32) -> Uint32:
    """A UE8M0 scale byte as BF16 in both halves (the byte is the exponent
    field)."""
    return code * Uint32(0x00800080)