Skip to content

vllm.distributed.device_communicators.nvlink_fabric

Cross-node NVLink fabric detection via NVML.

Functions:

Classify whether a CPU process group can safely rendezvous.

Same-node symmetric-memory rendezvous can reject unsupported GPU connectivity synchronously. Cross-node rendezvous is safe to attempt only when every rank reports the same healthy NVLink fabric.

Source code in vllm/distributed/device_communicators/nvlink_fabric.py
@functools.cache
def get_symmetric_memory_topology(
    group: "ProcessGroup",
) -> SymmetricMemoryTopology:
    """Classify whether a CPU process group can safely rendezvous.

    Same-node symmetric-memory rendezvous can reject unsupported GPU
    connectivity synchronously. Cross-node rendezvous is safe to attempt only
    when every rank reports the same healthy NVLink fabric.
    """
    if all(in_the_same_node_as(group, source_rank=0)):
        return SymmetricMemoryTopology.INTRA_NODE
    if has_cross_node_nvlink(group):
        return SymmetricMemoryTopology.CROSS_NODE_NVLINK
    return SymmetricMemoryTopology.UNSUPPORTED

Return whether all ranks report the same valid NVLink fabric.

Detection fails closed when fabric information is unavailable or incomplete. The caller is responsible for establishing that the group crosses nodes.

Source code in vllm/distributed/device_communicators/nvlink_fabric.py
def has_cross_node_nvlink(group: "ProcessGroup") -> bool:
    """Return whether all ranks report the same valid NVLink fabric.

    Detection fails closed when fabric information is unavailable or incomplete.
    The caller is responsible for establishing that the group crosses nodes.
    """
    if (
        not current_platform.is_cuda()
        or not dist.is_available()
        or not dist.is_initialized()
    ):
        return False

    world_size = dist.get_world_size(group=group)
    if world_size <= 1:
        return False

    fabric_keys: list[_NvlinkFabricKey | None] = [None] * world_size
    dist.all_gather_object(
        fabric_keys,
        _get_local_nvlink_fabric_key(),
        group=group,
    )
    valid_keys = [key for key in fabric_keys if key is not None]
    if len(valid_keys) != world_size:
        return False

    has_shared_fabric = len(set(valid_keys)) == 1
    if has_shared_fabric:
        logger.info_once(
            "Detected multi-node NVLink fabric: %s",
            valid_keys[0],
            scope="local",
        )
    return has_shared_fabric