qwen3_5_moe_text¶
Qwen3.5-MoE / 3.6-MoE text (Qwen3_5MoeForCausalLM, model_type qwen3_5_moe_text).
Llama's tree with a hybrid block: three blocks in four carry a gated DeltaNet
mixer, linear_attn, the fourth ordinary attention, self_attn
(config.layer_types). Each block has one or the other, never both, so on
a linear block every self_attn value is reported missing and the linear
values live at layers[i].linear_attn (see nnterp.LinearAttention). The MLP is a mixture of experts.
Every released checkpoint is a qwen3_5_moe wrapper: task="text-generation" (the
default) builds Qwen3_5MoeForCausalLM out of it, task="image-text-to-text" the
wrapper, whose text stack sits at model.language_model; RENAME carries both.
On the wrapper the Qwen ViT model.visual is vision (a QwenVision: packed,
[1, patches, vision_hidden_size], with a learned pos_embed added after patch_embed
and no deepstack), its blocks vision.layers; its merger is projector.
module-attribute
¶
RENAME = {'model.embed_tokens': 'embed_tokens', 'model.layers': 'layers', 'model.norm': 'norm', 'model.language_model.embed_tokens': 'embed_tokens', 'model.language_model.layers': 'layers', 'model.language_model.norm': 'norm', 'model.visual': 'vision', 'model.visual.merger': 'projector', 'blocks': 'layers', 'attn': 'self_attn', 'norm1': 'input_layernorm', 'norm2': 'post_attention_layernorm', 'gate': 'router', 'shared_expert': 'shared_experts', 'attn.proj': 'attn.o_proj', 'linear_fc2': 'down_proj'}
module-attribute
¶
ENVOYS = {Qwen3_5MoeModel: ImageScatter, Qwen3_5MoeDecoderLayer: Layer, Qwen3_5MoeAttention: Attention, Qwen3_5MoeGatedDeltaNet: LinearAttention, Qwen3_5MoeMLP: Mlp, Qwen3_5MoeSparseMoeBlock: Moe, Qwen3_5MoeVisionModel: QwenVision, Qwen3_5MoeVisionBlock: VisionLayer, Qwen3_5MoeVisionAttention: QwenVisionAttention, Qwen3_5MoeVisionMLP: VisionMlp}