vllm.model_executor.model_loader.weight_cache.utils
¶
Helpers shared by the weight cache daemon, the IPC loader and the engine.
These live outside protocol so callers that only need to know whether a
draft is cached, or how a daemon group is named, do not have to import the
wire format.
Classes:
-
ExternalWeightCharges–Track a worker's external weight charges and those freed by disk fallback.
Functions:
-
build_warmup_runner–Create a model runner for the daemon's warmup dummy runs.
-
build_weight_cache_key–Fingerprint this rank's shard exactly as
WeightCacheDaemondoes. -
collect_tensors–Collect a model's tensors, preserving tied-parameter aliases.
-
export_entries–Export a model's tensors as
TensorEntrys, preserving aliases. -
exported_weight_bytes–Bytes of the distinct CUDA storages an engine maps from this model.
-
format_daemon_role–Name of a daemon group: the target model or the draft.
-
format_socket_role_suffix–Socket-name suffix keeping the draft group distinct from the target.
-
get_engine_dp_placement–(dp_size, dp_rank)as the engine fingerprints this DP rank. -
is_draft_model_cacheable–Whether the daemon serves the speculative draft as a separate role.
Attributes:
-
DaemonRole–Which daemon group serves a model: the target or the speculative draft.
DaemonRole = Literal['target', 'draft']
module-attribute
¶
Which daemon group serves a model: the target or the speculative draft.
ExternalWeightCharges
dataclass
¶
Track a worker's external weight charges and those freed by disk fallback.
Attributes:
-
current_bytes(int) –External bytes held right now: charges minus fallback removals.
Source code in vllm/model_executor/model_loader/weight_cache/utils.py
current_bytes
property
¶
External bytes held right now: charges minus fallback removals.
build_warmup_runner(vllm_config, local_rank, *, is_draft, model_config)
¶
Create a model runner for the daemon's warmup dummy runs.
Source code in vllm/model_executor/model_loader/weight_cache/utils.py
build_weight_cache_key(vllm_config, model_config, *, is_draft=False)
¶
Fingerprint this rank's shard exactly as WeightCacheDaemon does.
Shared with the daemon so the peer loader requests the shard its own
daemon fingerprinted: TP/PP ranks come from the rank the daemon (or
engine worker) stamped on parallel_config.rank, and the DP placement
follows the engine's per-worker view.
Source code in vllm/model_executor/model_loader/weight_cache/utils.py
collect_tensors(model)
¶
Collect a model's tensors, preserving tied-parameter aliases.
named_parameters/named_buffers are iterated with
remove_duplicate=False so tied weights (e.g. lm_head.weight sharing
storage with embed_tokens.weight) are not silently dropped. Each unique
tensor is kept once; every additional name that refers to the same tensor
object is recorded in the returned alias map so the client can
re-establish the shared identity instead of allocating uninitialized
memory for it.
Returns:
-
dict[str, Tensor]–A
(tensors, kinds, aliases)triple:tensorsmaps a canonical -
dict[str, TensorKind]–name to its tensor,
kindsmaps it to "param" or "buffer", and -
dict[str, str]–aliasesmaps each duplicate name to its canonical name.
Source code in vllm/model_executor/model_loader/weight_cache/utils.py
export_entries(model)
¶
Export a model's tensors as TensorEntrys, preserving aliases.
CUDA reduction arguments must be exported separately for each consumer so that PyTorch registers a reference for each IPC mapping's lifetime.
Returns:
-
dict[str, TensorEntry]–A
(entries, aliases)pair, ascollect_tensorsbut with each -
dict[str, str]–tensor exported.
Source code in vllm/model_executor/model_loader/weight_cache/utils.py
exported_weight_bytes(model)
¶
Bytes of the distinct CUDA storages an engine maps from this model.
This is the memory the engine would otherwise allocate for the weights
itself. memory_allocated() also counts daemon-only state the engine
never maps (communication buffers, warmup workspaces, tensors outside
parameters/buffers that the engine rebuilds itself), so reporting it
would shrink the engine's KV cache below that of a disk load.
Source code in vllm/model_executor/model_loader/weight_cache/utils.py
format_daemon_role(is_draft)
¶
format_socket_role_suffix(is_draft)
¶
Socket-name suffix keeping the draft group distinct from the target.
get_engine_dp_placement(vllm_config, dp_rank)
¶
(dp_size, dp_rank) as the engine fingerprints this DP rank.
The engine runs the DP ranks of a non-MoE target as independent DP=1
replicas, so their workers all report (1, 0).
Source code in vllm/model_executor/model_loader/weight_cache/utils.py
is_draft_model_cacheable(speculative_config)
¶
Whether the daemon serves the speculative draft as a separate role.