mpt¶
MPT (MptForCausalLM).
transformer.{wte, blocks[i].{norm_1, attn, norm_2, ffn, resid_attn_dropout},
norm_f} and lm_head. The block adds the attention's output itself, but
the MLP takes the residual and adds it inside, so its contribution is the
tensor before that add (its dropout's output). The attention does its own
arithmetic whatever attn_implementation says, so the pattern is its
dropout's output and needs no eager load.
module-attribute
¶
RENAME = {'transformer.wte': 'embed_tokens', 'transformer.blocks': 'layers', 'transformer.norm_f': 'norm', 'norm_1': 'input_layernorm', 'norm_2': 'post_attention_layernorm', 'attn': 'self_attn', 'ffn': 'mlp', 'out_proj': 'o_proj', 'act': 'act_fn'}
intermediate_size(model: StandardizedTransformer) -> int
The MLP width is expansion_ratio times the hidden size (transformers builds 4 * hidden_size; equal on the released checkpoints).