qwen2_moe¶
Qwen2-MoE (Qwen2MoeForCausalLM).
Llama's tree and Llama's block: model.{embed_tokens, layers[i].{input_layernorm,
self_attn, post_attention_layernorm, mlp}, norm} and lm_head, the residual
added in the block, attention through the shared eager forward. The MLP is a sparse mixture of experts with a shared expert.
A config with mlp_only_layers set gives those blocks a dense Qwen2MoeMLP that no
envoy is keyed to: layers[i].mlp is a plain Envoy there and support() reports
"no mlp module on this block". No released checkpoint sets it.
module-attribute
¶
RENAME = {'model.embed_tokens': 'embed_tokens', 'model.layers': 'layers', 'model.norm': 'norm', 'gate': 'router', 'shared_expert': 'shared_experts'}
module-attribute
¶
ENVOYS = {Qwen2MoeDecoderLayer: Layer, Qwen2MoeAttention: Attention, Qwen2MoeSparseMoeBlock: Mlp}
Bases: Moe
A mixture of experts plus a gated shared expert, returned as one bare tensor, so the base mlp_output holds.
The shared expert's contribution is its output times a sigmoid gate,
F.sigmoid(shared_expert_gate(x)) * shared_expert(x), bound in this
forward after the experts have run, so the forward is instrumented at build.