glm_moe_dsa¶
GLM-5 (GlmMoeDsaForCausalLM).
DeepSeek-V3.2's tree and block (model.{embed_tokens, layers[i].{input_layernorm,
self_attn, post_attention_layernorm, mlp}, norm} and lm_head, the residual added
in the block): multi-head latent attention with DeepSeek Sparse Attention, the
indexer's top-k selection folded into the attention mask before the shared
attention_interface call under eager and sdpa, so the interior is the base's and means what
it means on deepseek_v32: attention_scores carry the sparse mask,
attention_probabilities is the dense [batch, num_heads, query, key] pattern, zero
outside each query's selected keys. While a prompt is no longer than index_topk
the selection is every causal key.
What differs is that the selection is shared across blocks. config.indexer_types
marks each block "full" (it has an indexer and selects its own keys) or
"shared" (no indexer; it reuses the selection of the block before). The selection
therefore travels between blocks: the attention returns
(attn_output, attn_weights, topk_indices) and the block returns
(hidden_states, topk_indices), so returns_tuple is set; attention_output
and layer_output take the first element. The selection is
self_attn.output[2] on every block (self_attn.indexer.output on a full one);
it has no standard name. skip_layers hands a skipped block's successor None for
the selection, which a shared block refuses (transformers raises ValueError), so
skip up to a full block or to the end. Dense and mixture-of-experts MLPs follow
mlp_layer_types. The sizes are DeepSeek-V2's.