dots1

dots1

dots.llm1 (Dots1ForCausalLM).

Llama's tree and Llama's block: model.{embed_tokens, layers[i].{input_layernorm, self_attn, post_attention_layernorm, mlp}, norm} and lm_head, the residual added in the block, attention through the shared eager forward. The attention norms its queries and keys per head (q_norm, k_norm) before the rotary embedding. The first first_k_dense_replace blocks have a dense MLP, the rest a mixture of experts that adds a shared expert (mlp.shared_experts, itself a Dots1MLP) before it returns; the experts are moe_intermediate_size wide.

RENAME module-attribute

RENAME = {'model.embed_tokens': 'embed_tokens', 'model.layers': 'layers', 'model.norm': 'norm', 'gate': 'router'}

ENVOYS module-attribute

ENVOYS = {Dots1DecoderLayer: Layer, Dots1Attention: Attention, Dots1MLP: Mlp, Dots1MoE: Moe}