xglm¶
XGLM (XGLMForCausalLM).
model.{embed_tokens, embed_positions, layers[i].{self_attn, self_attn_layer_norm,
fc1, fc2, final_layer_norm}, layer_norm} and lm_head. embed_tokens is a
scaled embedding (times sqrt(d_model) under scale_embedding), so
token_embeddings is the scaled lookup; the sinusoidal positions are added
after it, outside any module with a standard name. As on OPT there is no MLP
module: fc1 and fc2 sit on the block, so mlp and mlp_output do not
exist here and StandardizedTransformer.support says so. The block's own
final_layer_norm is the pre-MLP norm and keeps its native name. The block
returns a bare tensor. The attention does its own arithmetic whatever
attn_implementation says (the model has no other): queries scaled by
1/sqrt(head_dim) as they are projected, queries, keys and values flattened to
[batch * num_heads, seq, head_dim] for torch.bmm, the mask added and clamped,
softmax, dropout, bmm with the values; the interior needs no eager load.
module-attribute
¶
RENAME = {'model.embed_tokens': 'embed_tokens', 'model.layers': 'layers', 'model.layer_norm': 'norm', 'self_attn_layer_norm': 'input_layernorm', 'out_proj': 'o_proj'}
intermediate_size(model: StandardizedTransformer) -> int
The width of the block's fc1/fc2 path is ffn_dim.