vllm.utils.sparse_utils
¶
Safe deserialization helpers for client-supplied embedding tensors.
check_sparse_tensor_invariants_threadsafe() is a thread-safe wrapper for
sparse tensor invariant validation. check_torch_archive_size() bounds the
memory torch.load allocates for the tensor storages of a payload.
safe_to_dense() bounds the memory a deserialized tensor may allocate when
densified.
PyTorch's torch.sparse.check_sparse_tensor_invariants() context manager
manipulates a process-global flag (save/enable/restore). When multiple
embedding-load operations run concurrently on a thread-pool executor, one
context can restore the flag to False while another thread is still inside
its guard, bypassing the invariant check.
All call sites MUST use check_sparse_tensor_invariants_threadsafe()
which serializes access behind a lock.
Functions:
-
check_torch_archive_size–Bound the bytes
torch.loadinflates from a client-supplied payload. -
safe_to_dense–Densify a client-supplied embedding tensor within a memory budget.
check_torch_archive_size(payload, *, parameter)
¶
Bound the bytes torch.load inflates from a client-supplied payload.
torch.save writes a zip archive, and torch.load reads each tensor
storage entry into memory at the uncompressed size recorded in the
archive. Entries may be deflate-compressed, so a few MiB of zeros can
declare a storage of many GiB. That allocation happens inside
torch.load, before safe_to_dense() sees the tensor, and a small view
of a large storage keeps the whole storage alive. Reject archives whose
entries add up to more than VLLM_MAX_EMBED_DECODE_BYTES before loading.
Source code in vllm/utils/sparse_utils.py
safe_to_dense(tensor, *, parameter)
¶
Densify a client-supplied embedding tensor within a memory budget.
torch.load(..., weights_only=True) prevents arbitrary code execution but
not memory amplification: a sparse COO tensor carries its own declared
shape, so the dense allocation is bounded by that shape rather than by the
request. A 2.6 KB payload declaring (30000, 30000) materializes 3.4 GiB,
and the shape can be picked to fit whatever host is being targeted. The
invariant checks only reject invalid tensors (indices outside the
declared shape); a single non-zero element inside a (2**20, 2**20) shape
is perfectly valid and passes them.
So reject oversized payloads based on the dense size they declare, before
to_dense() allocates anything.