qwen3_moe

qwen3_moe

Qwen3-MoE (Qwen3MoeForCausalLM).

Llama's tree and Llama's block: model.{embed_tokens, layers[i].{input_layernorm, self_attn, post_attention_layernorm, mlp}, norm} and lm_head, the residual added in the block, attention through the shared eager forward. The MLP is a sparse mixture of experts; head_dim is the config's.

A config with mlp_only_layers set gives those blocks a dense Qwen3MoeMLP that no envoy is keyed to: layers[i].mlp is a plain Envoy there and support() reports "no mlp module on this block". No released checkpoint sets it.

RENAME module-attribute

RENAME = {'model.embed_tokens': 'embed_tokens', 'model.layers': 'layers', 'model.norm': 'norm', 'gate': 'router'}

ENVOYS module-attribute

ENVOYS = {Qwen3MoeDecoderLayer: Layer, Qwen3MoeAttention: Attention, Qwen3MoeSparseMoeBlock: Mlp}

Mlp

Mlp(*args: Any, **kwargs: Any)

Bases: Moe

A mixture of experts: the module returns the routed hidden states (a bare tensor on this transformers), so the base holds.