glm

glm

GLM-4 (GlmForCausalLM), the first generation: GLM-4-9B.

Llama's tree and Llama's block: model.{embed_tokens, layers[i].{input_layernorm, self_attn, post_attention_layernorm, mlp}, norm} and lm_head, the residual added in the block, attention through the shared eager forward. The rotary embedding covers partial_rotary_factor of each head, before the interface. The MLP fuses its gate and up projections into one gate_up_proj and returns a bare tensor.

RENAME module-attribute

RENAME = {'model.embed_tokens': 'embed_tokens', 'model.layers': 'layers', 'model.norm': 'norm', 'activation_fn': 'act_fn'}

ENVOYS module-attribute

ENVOYS = {GlmDecoderLayer: Layer, GlmAttention: Attention, GlmMLP: Mlp}