apertus

apertus

Apertus (ApertusForCausalLM).

Llama's tree and Llama's block under other norm names: model.{embed_tokens, layers[i].{attention_layernorm, self_attn, feedforward_layernorm, mlp}, norm} and lm_head, the residual added in the block, attention through the shared eager forward. attention_layernorm feeds the attention and feedforward_layernorm the MLP, so they are aliased input_layernorm and post_attention_layernorm. The attention norms its queries and keys per head (q_norm, k_norm) before the rotary embedding. The MLP has no gate: down_proj(xielu(up_proj(x))).

RENAME module-attribute

RENAME = {'model.embed_tokens': 'embed_tokens', 'model.layers': 'layers', 'model.norm': 'norm', 'attention_layernorm': 'input_layernorm', 'feedforward_layernorm': 'post_attention_layernorm'}

ENVOYS module-attribute

ENVOYS = {ApertusDecoderLayer: Layer, ApertusAttention: Attention, ApertusMLP: Mlp}