falcon¶
Falcon (FalconForCausalLM): the 7B layout (parallel attention), with or without alibi, and the 40B layout.
transformer.{word_embeddings, h[i].{input_layernorm, self_attention, mlp},
ln_f} and lm_head. One norm feeds both sublayers and the block sums
x + attn + mlp, but it does so by adding the attention output into the
MLP's output tensor in place, so mlp_output reads a copy taken as the
MLP returns, and a transform carries edits to it back into the model.
The attention does its own arithmetic, and config.alibi picks one of two
branches of it with different operations, so each interior value names its
operation by that flag. Without alibi the queries and keys leave
apply_rotary_pos_emb and the pattern is the first softmax, which has no
dropout after it; with alibi there is no rotary, the pattern is the dropout
after the second softmax, and the head outputs are flattened over batch and
heads. The 40B layout (ln_attn / ln_mlp, new_decoder_architecture)
runs the same forward with its key/value heads already broadcast. falcon-11B has
the 40B layout with num_ln_in_parallel_attn 1: one input_layernorm feeds both
sublayers, and 8 key/value heads serve its 32 query heads.
module-attribute
¶
RENAME = {'transformer.word_embeddings': 'embed_tokens', 'transformer.h': 'layers', 'transformer.ln_f': 'norm', 'self_attention': 'self_attn', 'dense': 'o_proj', 'act': 'act_fn', 'dense_4h_to_h': 'down_proj'}
module-attribute
¶
The 40B layout: ln_attn and ln_mlp each norm the block input (num_ln_in_parallel_attn 2).
Whether the block is parallel: every layout but Falcon-RW's (parallel_attn false, not new_decoder_architecture).
A key at one of two operations, chosen by the checkpoint's alibi flag, a config value read at load.
num_kv_heads(model: StandardizedTransformer) -> int
num_kv_heads on the 40B layout (new_decoder_architecture); 1 under multi_query; else every head.
intermediate_size(model: StandardizedTransformer) -> int
The MLP width is ffn_hidden_size.