gpt_bigcode¶
GPT-BigCode (GPTBigCodeForCausalLM): StarCoder, SantaCoder.
GPT-2's tree and GPT-2's block: transformer.{wte, wpe, h[i].{ln_1, attn, ln_2,
mlp}, ln_f} and lm_head, the residual added in the block, which returns a bare
tensor. The attention projects queries, keys and values with one fused c_attn
and runs the shared eager forward. Under multi_query (every released checkpoint)
the keys and values are one head shared by every query head, so num_kv_heads is
1. Queries, keys and values are split views of the one c_attn tensor: torch
refuses an in-place edit of attention_queries outside torch.no_grad() (assign
instead), while keys and values take one. attention_softmax_in_fp32 and
scale_attention_softmax_in_fp32 are stored on the module but not read: the
eager softmax runs in float32 regardless. The MLP is n_inner wide,
4 * hidden_size when that is None.
module-attribute
¶
RENAME = {'transformer.wte': 'embed_tokens', 'transformer.h': 'layers', 'transformer.ln_f': 'norm', 'ln_1': 'input_layernorm', 'attn': 'self_attn', 'ln_2': 'post_attention_layernorm', 'attn.c_proj': 'attn.o_proj', 'mlp.c_proj': 'mlp.down_proj', 'act': 'act_fn'}
module-attribute
¶
num_kv_heads(model: StandardizedTransformer) -> int
One key/value head under multi_query; else every head.
intermediate_size(model: StandardizedTransformer) -> int
The MLP width is n_inner, 4 * hidden_size when None.