gpt_neo¶
GPT-Neo (GPTNeoForCausalLM).
transformer.{wte, wpe, drop, h[i].{ln_1, attn.attention, ln_2, mlp}, ln_f} and
lm_head. A sequential block that returns (hidden_states, attn_weights).
attn is a GPTNeoAttention wrapper that forwards to attn.attention, the
GPTNeoSelfAttention holding the projections, and returns its output as is;
self_attn is bound to that inner module, so its input is ln_1's output and
its first output is what the block adds. The inner module does its own arithmetic
in _attn: an fp32 q @ k^T with no 1/sqrt(head_dim) scaling, a causal
mask by torch.where (a sliding window of window_size on the local
layers of attention_layers, which alternate with global ones), the padding
mask added, softmax, a cast to the values' dtype, attn_dropout, then
@ v. The model knows only eager and flash_attention_2; the interior is
read around the _attn call and needs eager. wpe and drop have no
standard name.
module-attribute
¶
RENAME = {'transformer.wte': 'embed_tokens', 'transformer.h': 'layers', 'transformer.ln_f': 'norm', 'ln_1': 'input_layernorm', 'attn.attention': 'self_attn', 'ln_2': 'post_attention_layernorm', 'act': 'act_fn', 'c_proj': 'down_proj', 'out_proj': 'o_proj'}
module-attribute
¶
intermediate_size(model: StandardizedTransformer) -> int
The MLP width is intermediate_size, None meaning four times the hidden size.