minimax_m2¶
MiniMax-M2 (MiniMaxM2ForCausalLM): MiniMax-M2, M2.5 and M2.7.
Llama's tree and Llama's block: model.{embed_tokens, layers[i].{input_layernorm,
self_attn, post_attention_layernorm, mlp}, norm} and lm_head, the residual
added in the block with no scaling, attention through the shared eager forward.
The attention norms its queries and keys across the whole projection
(q_norm/k_norm over heads * head_dim) before the head split; the
queries and keys are read where they enter the interface, after the norms and
the rotary, so nothing changes. Every block's MLP is a sparse mixture of experts
with a sigmoid router and a correction bias; it returns the routed hidden states
as a bare tensor. intermediate_size is the experts' width and head_dim is
the config's. The released configs give the rotary width as rotary_dim (64 of
128), which MiniMaxM2Config reads as partial_rotary_factor 0.5, so half of
each query and key head is rotated.