standardized_vllm¶

standardized_vllm ¶

StandardizedVLLM: nnsight's VLLM engine renamed to the standard vocabulary.

StandardizedVLLM ¶

StandardizedVLLM(repo_id: str, *args: Any, family: ModuleType | None = None, rename: dict[str, str | list[str]] | None = None, envoys: dict | None = None, **kwargs: Any)

Bases: Standardized, VLLM

A causal language model on the vLLM engine whose modules answer to the standard names.

StandardizedTransformer's counterpart for nnsight's VLLM: the same vocabulary (model.layers[i].self_attn, .mlp, model.norm, model.lm_head), the same values with the same layouts, and the same methods, over the engine's continuous batching. The family is vLLM's own implementation of the checkpoint's model_type, a module under nnterp.families.vllm.

It is nnsight's VLLM otherwise, with vLLM's defaults: a trace is a generation request (max_tokens=16 and temperature=1.0 unless you say), sampling settings go on trace/invoke, one prompt per invoke, and a block runs in the engine's worker, where nnterp has to be importable. What that changes for the values:

A request is one sequence, so every value's batch axis is 1: layer_output is [1, tokens, hidden_size], the whole prompt on the prefill and one token on each decode step (tracer.iter).
logits is this step's, [1, 1, vocab_size]: the engine computes the last position only. logits[:, -1] and next_token_probs read the same on both engines.
Values are private copies, so a saved one stays what it was read as.
The queries, keys, values and head outputs are what goes into and comes out of the engine's attention layer, for this step's tokens. The scores and the pattern are inside its kernel, so they are recomputed from the queries and keys: read-only, and on the prefill only (a decode step raises Unavailable). There is no attention_mask.
project_on_vocab runs the engine's modules, so it works inside a trace only.

PARAMETER	DESCRIPTION
`repo_id`	A HuggingFace repo id. TYPE: `str`
`family`	The family to use instead of the one looked up: a module or any object with `RENAME` and `ENVOYS` written against vLLM's module classes (and any size function or `project_on_vocab` it defines). It applies to this model only; the config is not read for it, and no model type is named or checked. TYPE: `ModuleType \| None` DEFAULT: `None`
`rename`	Extra aliases, merged over the family's; a key given here wins. TYPE: `dict[str, str \| list[str]] \| None` DEFAULT: `None`
`envoys`	Extra `envoys=` entries, merged over the family's `ENVOYS` and nnsight's envoys for vLLM's parallel layers; a key given here wins. TYPE: `dict \| None` DEFAULT: `None`
`**kwargs`	Passed through to `VLLM` (`dispatch`, `mode`, `taps`, and vLLM's engine arguments). TYPE: `Any` DEFAULT: `{}`

ATTRIBUTE	DESCRIPTION
`family`	The vLLM family the checkpoint resolved to, or the `family` passed.

RAISES	DESCRIPTION
`UnsupportedFamily`	when no family is passed and vLLM's implementation of the checkpoint's `model_type` has no family under `nnterp.families.vllm`.

family `instance-attribute` ¶

family = family

config `property` ¶

config: Any

The checkpoint's transformers config, as vLLM's model holds it.

lm_head_output `class-attribute` `instance-attribute` ¶

lm_head_output = unavailable("vLLM never calls lm_head: its logits processor projects with lm_head's weight and applies any softcap and scale in the same call, on the last position only; read logits")

attention_mask `class-attribute` `instance-attribute` ¶

attention_mask = unavailable('a vLLM request is one unpadded sequence; there is no mask')

logits ¶

logits(value: Tensor) -> Tensor

next_token_probs ¶

next_token_probs(value: Any) -> Any

token_embeddings ¶

token_embeddings(value: Tensor) -> Residual

The embedding module's output, [1, tokens, hidden_size]; what a family scales or adds positions to comes after.

final_norm_output ¶

final_norm_output(value: Tensor) -> Residual

The final norm's output, [1, tokens, hidden_size]: every position of the step, of which the engine projects the last.

A family whose blocks hand the stream on in two halves calls the final norm with both, and the norm returns (normed, residual): this is the normed half. Assign to replace it; in-place edits reach the model.

input_ids ¶

input_ids(value: Any) -> Any

input_size ¶

input_size(value: Any) -> Any

project_on_vocab ¶

project_on_vocab(hidden: Tensor) -> Tensor

Logits for a residual-stream tensor: the final norm, lm_head, and vLLM's logits processor.

The logit lens, inside a trace: the modules are the engine's, so there is nothing to call outside one. The processor applies the model's softcapping and scaling where it has them. hidden is [..., hidden]; the leading dimensions are kept. A family whose model projects another way defines def project_on_vocab(model, hidden) in its module.

order ¶

order(layer: int | None = None) -> dict[str, int]

Not supported on the vllm engine: the forward order is measured on StandardizedTransformer only.

rank ¶

rank(name: str, layer: int | None = None) -> tuple[int, int]

Not supported on the vllm engine: the forward order is measured on StandardizedTransformer only.

standardized_vllm¶