Skip to content

vllm.models.deepseek_v41.nvidia.specialized_decode.ampere

DeepSeek-V4.1 on Ampere datacenter GPUs (A100, SM80).

The stock DeepSeek-V4.1 path needs kernels that A100 lacks: FlashMLA and DeepGEMM build for SM90 and SM100 only, and Triton cannot convert to or from FP8 E4M3 below SM89. This package replaces each of them with a CuTe DSL kernel that runs on SM80, keeping the stock model, scheduler and cache layout around them:

  • the KV caches keep DeepSeek's V4.1 records -- the 528-byte MXFP8 sliding-window record and the 288-byte NVFP4 compressed record -- and the 68-byte MXFP4 index keys, as on SM100;
  • every FP8 and FP4 value is decoded with integer operations into BF16, where it is exact, and multiplied on BF16 tensor cores (mma.sync) with FP32 accumulation; quantizers round to nearest even exactly as the hardware conversions of SM100 would.

The modules:

Module Contents
attention.py The attention layer, its compressor and its metadata backends
indexer.py The indexer: MXFP4 queries and keys, its op (logits, top-k)
indexer_metadata.py The indexer metadata builder (no DeepGEMM schedule)
sparse_mla.py The sparse MLA kernels
mqa_logits.py The indexer logits kernel
topk.py The indexer's decode top-k and candidate-block selection
quantize.py The FP4 quantizers (queries, index keys, compressed records)
mhc.py The mHC layer boundary: post-mix, collapse + RMSNorm, coefficients
all_reduce.py The boundaries' two-shot Lamport all-reduce
all_gather.py The Lamport all-gather (indexer rows; logits and Engram columns)
columns.py The logits and Engram gathers along the last dimension
codecs.py Device helpers: FP8/FP4 decoding and encoding, raw loads/stores

Each kernel is specified by a PyTorch function of tests/models/deepseek_v41/nvidia/specialized_decode/ampere/reference.py, which the docstrings call reference.<function>.

This module holds only the platform predicate, the index K cache choice and a table helper, so that shared DeepSeek-V4.1 code can ask for the Ampere path without importing CuTe DSL.

Modules:

  • all_gather –

    The TP all-gathers of the SM80 decode step, in CuTe DSL.

  • all_reduce –

    The TP all-reduce of DeepSeek-V4.1's decode boundaries on SM80, in CuTe DSL.

  • attention –

    The DeepSeek-V4.1 attention layer on SM80.

  • codecs –

    CuTe DSL device helpers of the SM80 kernels: global loads by address and

  • columns –

    The SM80 and SM90 TP all-gathers along the last dimension: logits and

  • gemv –

    MXFP8 linear layers at decode batch sizes on SM80, in CuTe DSL.

  • indexer –

    The DeepSeek-V4.1 sparse-attention indexer on SM80.

  • indexer_metadata –

    The DeepSeek-V4.1 indexer metadata on SM80.

  • mhc –

    The mHC layer boundary of DeepSeek-V4.1 on SM80, in CuTe DSL.

  • mhc_projection –

    The mHC projection of DeepSeek-V4.1 prefill on SM80, in CuTe DSL.

  • mqa_logits –

    The DeepSeek-V4.1 indexer logits on SM80, in CuTe DSL.

  • quantize –

    The FP4 quantizers of DeepSeek-V4.1 on SM80, in CuTe DSL.

  • row_shard –

    The DeepSeek-V4.1 indexer's prefill rows split over the TP ranks, on SM80

  • sparse_mla –

    DeepSeek-V4.1 sparse MLA on SM80 in CuTe DSL.

  • topk –

    The DeepSeek-V4.1 indexer's decode top-k on SM80, in CuTe DSL.

Functions:

  • indexer_uses_fp4 –

    The index K cache on SM80: always MXFP4, the only one its kernels

  • is_ampere –

    Whether the current device runs DeepSeek-V4.1 through this package.

  • rope_table_f32 –

    The RoPE (cos, sin) table [positions, 64] in FP32, which the

indexer_uses_fp4(vllm_config)

The index K cache on SM80: always MXFP4, the only one its kernels write and score. An explicit FP8 request is rejected.

Source code in vllm/models/deepseek_v41/nvidia/specialized_decode/ampere/__init__.py
def indexer_uses_fp4(vllm_config: VllmConfig) -> bool:
    """The index K cache on SM80: always MXFP4, the only one its kernels
    write and score. An explicit FP8 request is rejected."""
    kv_dtype = vllm_config.attention_config.resolve_indexer_kv_dtype("mxfp4")
    if kv_dtype != "mxfp4":
        raise ValueError(
            "DeepSeek-V4.1 on SM80 scores the MXFP4 indexer cache only; "
            f"got indexer_kv_dtype={kv_dtype!r}."
        )
    return True

is_ampere()

Whether the current device runs DeepSeek-V4.1 through this package.

Source code in vllm/models/deepseek_v41/nvidia/specialized_decode/ampere/__init__.py
def is_ampere() -> bool:
    """Whether the current device runs DeepSeek-V4.1 through this package."""
    if not current_platform.is_cuda():
        return False
    capability = current_platform.get_device_capability()
    return capability is not None and capability.to_int() == AMPERE_CAPABILITY

rope_table_f32(rotary_emb)

The RoPE (cos, sin) table [positions, 64] in FP32, which the kernels read; a table of another dtype is converted once.

Source code in vllm/models/deepseek_v41/nvidia/specialized_decode/ampere/__init__.py
def rope_table_f32(rotary_emb: nn.Module) -> torch.Tensor:
    """The RoPE (cos, sin) table `[positions, 64]` in FP32, which the
    kernels read; a table of another dtype is converted once."""
    table = rotary_emb.cos_sin_cache
    assert isinstance(table, torch.Tensor)
    if table.dtype == torch.float32:
        return table
    converted = getattr(rotary_emb, "_ampere_cos_sin_f32", None)
    if converted is None:
        converted = table.float()
        rotary_emb._ampere_cos_sin_f32 = converted
    return converted