bloom¶
BLOOM (BloomForCausalLM).
transformer.{word_embeddings, word_embeddings_layernorm, h[i].{input_layernorm,
self_attention, post_attention_layernorm, mlp}, ln_f} and lm_head. Both
sublayers take the residual as an argument and add it inside the module
(dropout_add), so the module outputs are residual-stream states; the
contributions are the first argument of each dropout_add call. The
attention does its own arithmetic whatever attn_implementation says, so
the pattern is its dropout's output and needs no eager load.
The embedding norm has no standard name.
module-attribute
¶
RENAME = {'transformer.word_embeddings': 'embed_tokens', 'transformer.h': 'layers', 'transformer.ln_f': 'norm', 'self_attention': 'self_attn', 'dense': 'o_proj', 'gelu_impl': 'act_fn', 'dense_4h_to_h': 'down_proj'}
intermediate_size(model: StandardizedTransformer) -> int
The MLP is four times the hidden size wide; the config has no key for it.