standardized¶
Standardized, what every standardized model shares, and StandardizedTransformer, the TransformersModel one.
A read-only value of the model that a family may define instead.
Wraps the standard implementation. On read, a function of the same name
in the model's family module wins (def num_kv_heads(model): ... in
falcon.py), so a family whose config spells a size its own way keeps
that knowledge beside its names and values, and the implementation here
stays the plain case.
Bases: StandardizedProperty
A method of the model that a family may define instead.
StandardizedProperty for a method: on attribute access, a function of
the same name in the model's family module (def project_on_vocab(model,
hidden): ... in cohere.py) is bound in place of the standard
implementation, so a family whose model does something of its own keeps
that beside its names and values, and the implementation here stays the
plain case.
What a standardized model is on any engine: a family, its sizes, its availability, and the methods over its values.
Mixed in ahead of the nnsight model class that runs the checkpoint
(StandardizedTransformer over TransformersModel,
nnterp.StandardizedVLLM over nnsight's VLLM). The leaf resolves
family and hands its RENAME and ENVOYS to nnsight, and defines
the root's values (logits, token_embeddings, ...), which are read
where its engine keeps them. Everything here is written against the
standard names and values only, so it holds on both.
Skip blocks start through end inclusive.
The residual stream entering block start (or skip_with) is
handed straight to block end + 1; the skipped blocks do not run.
Negative indices count from the end. Inside a trace::
with model.trace(prompt):
model.skip_layers(4, 7)
logits = model.logits.save()
steer(layers: int | list[int], vector: Tensor, factor: float = 1.0, token_positions: int | list[int] | slice | None = None, batch_index: int | None = None) -> None
Add factor * vector to the residual stream leaving the given blocks.
vector is [hidden_size] (or broadcastable to the selected slice).
token_positions restricts the positions and batch_index the
row; both default to all. The add is in place on layer_output, so
it reaches the model. Inside a trace, with layers ascending.
The k most likely tokens of one [vocab_size] distribution, as {token: probability}, most likely first.
Keyed by the decoded text. Where two or more of the k decode to
the same text (partial UTF-8 byte tokens all decode to '�'),
those entries are keyed by the tokenizer's raw vocabulary token
instead (convert_ids_to_tokens, unique per id), with the id
appended should even that repeat, so the dict always holds k
entries, each with its own probability.
The k most likely next tokens for each position of hidden, [..., hidden_size].
project_on_vocab then softmax, one {token: probability} per
position, in row-major order over the leading dimensions.
Which standard values this checkpoint has, without running anything.
With layer, that block's values by dotted name ("layer_output",
"self_attn.attention_probabilities", "mlp.mlp_output"): None
when available, else the reason, including "no <module> module"
when the block has no such module at all (a hybrid's blocks have either
self_attn or linear_attn). Without, the root's values
plus every block value: None when available on every block, else
{layer: reason} for the blocks where it is not, so a hybrid reads
as a short dict.
The tree decides what is listed: every child of a block that carries
standard values (a Standard envoy) is walked under its standard
name, so a value installed through envoys= or a registered family
appears here as it does in the envoy's own Standard.support, and a
module no block has (OPT's mlp) has no entry. A standard child of
the root (the vision tower, Vision) is listed the same way, its own
support rows under its standard name ("vision.image_features",
"vision.self_attn.attention_probabilities"); a tower no image
reaches lists none.
Key/value heads: num_key_value_heads under grouped-query attention, else num_heads.
Width of one attention head: the config's head_dim when it says (Qwen3, Gemma), else hidden_size // num_heads.
Width of one head's queries and keys: head_dim, unless the family separates them (DeepSeek's latent attention).
Width of the dense MLP's hidden layer, config.intermediate_size; a mixture of experts' experts are moe_intermediate_size wide.
StandardizedTransformer(repo_id: Any, *args: Any, family: ModuleType | None = None, rename: dict[str, str | list[str]] | None = None, envoys: dict | None = None, tokenizer_kwargs: dict | None = None, **kwargs: Any)
Bases: Standardized, TransformersModel
A causal language model whose modules answer to one set of names.
The checkpoint's config is read first (its model_type), the matching
family toolkit is looked up in nnterp.families.REGISTRY, and the family's
RENAME is handed to nnsight's rename so every envoy in the tree also
answers to the standard name. The original names keep working: an alias is
an extra attribute on the same envoy, not a replacement.
The family's ENVOYS wraps its decoder blocks in its Layer
(layer_output), its attention modules in its Attention
(attention_output, attention_probabilities) and its feed-forwards
in its Mlp (mlp_output). The pattern is read inside the eager
attention forward, so it is unavailable unless the model is loaded with
attn_implementation="eager"; support says which values this
checkpoint has. See nnterp.components.
| PARAMETER | DESCRIPTION |
|---|---|
repo_id
|
A HuggingFace repo id, or an already-loaded
TYPE:
|
family
|
The family to use instead of the one looked up: a module or
any object with
TYPE:
|
rename
|
Extra aliases, merged over the family's; a key given here wins.
TYPE:
|
envoys
|
Extra
TYPE:
|
tokenizer_kwargs
|
Attributes to set on the loaded tokenizer, such as
TYPE:
|
**kwargs
|
Passed through to
TYPE:
|
Over the values, skip_layers, steer, project_on_vocab and
get_topk_closest_tokens do the common things (see each). Inside a
trace input_ids, input_size and attention_mask are what the model
was called with.
The root envoy also answers for the whole model, inside a trace: logits
(the logits the model returns, after any softcap or scale past the head), token_embeddings
(the embedding's output) and next_token_probs; and outside one: the
sizes num_layers, num_heads, num_kv_heads, head_dim,
qk_head_dim, hidden_size, intermediate_size and vocab_size,
read off the config, each a StandardizedProperty the family can define
instead; project_on_vocab is a StandardizedCapability, a method the family
can define the same way.
On an image-text-to-text checkpoint loaded with task="image-text-to-text"
the text names keep their meaning (the language model's), the vision tower
is vision (a Vision, with its own blocks, sizes and values, among them
vision.image_token_mask and vision.image_features) and the last
module before the scatter into the text stream is projector.
| ATTRIBUTE | DESCRIPTION |
|---|---|
family |
The toolkit module the checkpoint resolved to, or the
|
layers |
The decoder blocks, each a
TYPE:
|
embed_tokens, |
The embedding, the final norm, the unembedding.
TYPE:
|
vision |
The vision tower, a
TYPE:
|
projector |
The last module before the wrapper scatters the image features into the text stream.
TYPE:
|
| RAISES | DESCRIPTION |
|---|---|
UnsupportedFamily
|
when no family is passed and none covers the checkpoint's |
property
¶
The checkpoint's tokenizer loaded with add_prefix_space=False, so "word" and " word" differ.
What nnterp.prompt_utils.get_first_tokens uses. Loaded once, on first use.
token_embeddings(value: Tensor) -> Residual
The embedding module's output, [batch, seq, hidden_size].
Positional embeddings and embedding norms (GPT-2's wpe, BLOOM's
word_embeddings_layernorm) are applied after this by the families
that have them. Assign to replace it.
final_norm_output(value: Tensor) -> Residual
The final norm's output, [batch, seq, hidden_size]: the stream the head projects. Assign to replace it; in-place edits reach the model.
lm_head_output(value: Tensor) -> Logits
lm_head's own output, [batch, seq, vocab_size].
logits is what the model returns; on most families the same tensor,
on Gemma-2 the softcapped one. Assign to replace it; in-place edits
reach the model.
Logits for a residual-stream tensor: the final norm, lm_head, and what the model does after the head.
The logit lens: applied to a block's layer_output it reads that
layer's prediction; applied to the last block's, it is logits.
Works inside a trace on a live value and outside on a saved one.
After the head the plain case is the text config's
final_logit_softcapping, when set (Gemma-2's own config; a
multimodal checkpoint's text_config). A family whose model does
something else there (Cohere multiplies by logit_scale, Granite
divides by logits_scaling) defines
def project_on_vocab(model, hidden) in its module, which is bound
in this one's place (StandardizedCapability).
The available values in forward order: name -> rank; equal ranks are one location, either read first.
With layer, that block's values by dotted name (support's), from
0. Without, the root's, numbered across the whole forward. Measured by
a probe run on first use (nnterp.order.probe) and kept on the model,
so route kernels (route_kernels, chunk_per_token) before the first.
Where a value falls in the forward: (side, r), which sorts reads into forward order.
side is -1 or num_layers for a root value served before or
after the blocks, else layer; r is order's rank.
| RAISES | DESCRIPTION |
|---|---|
Unavailable
|
the checkpoint has no such value ( |
KeyError
|
there is no value of that name (a block value needs |