qwen3_next¶
Qwen3-Next (Qwen3NextForCausalLM).
Llama's tree with a hybrid block: three blocks in four carry a gated DeltaNet
mixer, linear_attn, the fourth ordinary attention, self_attn
(config.layer_types). Each block has one or the other, never both, so on
a linear block every self_attn value is reported missing and the linear
values live at layers[i].linear_attn (see nnterp.LinearAttention). The MLP is a mixture of experts (a dense MLP class exists for the shared expert).
module-attribute
¶
RENAME = {'model.embed_tokens': 'embed_tokens', 'model.layers': 'layers', 'model.norm': 'norm', 'gate': 'router', 'shared_expert': 'shared_experts'}
module-attribute
¶
ENVOYS = {Qwen3NextDecoderLayer: Layer, Qwen3NextAttention: Attention, Qwen3NextGatedDeltaNet: LinearAttention, Qwen3NextMLP: Mlp, Qwen3NextSparseMoeBlock: Moe}