gpt2¶
GPT-2 (GPT2LMHeadModel).
Its tree is transformer.{wte, wpe, drop, h[i].{ln_1, attn, ln_2, mlp}, ln_f}
plus lm_head. The container-level keys are anchored at the root
(transformer.h) so the aliases land on the root envoy: model.layers,
not model.model.layers. wpe (learned absolute positions) and drop
have no standard name; they stay reachable under their own.
A checkpoint with reorder_and_upcast_attn set takes GPT-2's own
_upcast_and_reordered_attn path instead of the shared eager forward, so
none of the attention's interior values is available there.
module-attribute
¶
RENAME = {'transformer.wte': 'embed_tokens', 'transformer.h': 'layers', 'transformer.ln_f': 'norm', 'ln_1': 'input_layernorm', 'attn': 'self_attn', 'ln_2': 'post_attention_layernorm', 'attn.c_proj': 'attn.o_proj', 'mlp.c_proj': 'mlp.down_proj', 'act': 'act_fn'}
intermediate_size(model: StandardizedTransformer) -> int
The MLP width is n_inner, None meaning four times the hidden size; the config's intermediate_size is never read by the model.