deepseek_v32¶
DeepSeek-V3.2 (DeepseekV32ForCausalLM).
Llama's tree and Llama's block: model.{embed_tokens, layers[i].{input_layernorm,
self_attn, post_attention_layernorm, mlp}, norm} and lm_head, the residual
added in the block, a bare tensor returned. Multi-head latent attention as in V3, with
DeepSeek Sparse Attention on top: each attention module carries an indexer (a
lightning indexer with its own small projections) that scores every key for every
query and keeps the index_topk best. Under eager and sdpa the selection is folded into the
attention mask (every unselected key set to the dtype's minimum) before the shared
attention_interface call, so the interior is the base's, read on the interface:
attention_scoresare the scaled scores with that sparse mask added, so an unselected key reads as the dtype's minimum, like a future one.attention_probabilitiesis the dense[batch, num_heads, query, key]pattern of the sparse attention: exactly zero outside each query's selected keys, rows summing to one over the selection. A written pattern is used as written, so a write can put weight on keys the indexer did not select.
While a prompt is no longer than index_topk (2048 on the released checkpoints) the
indexer selects every causal key and the pattern is the dense causal one. The indexer's
selection is self_attn.indexer.output, the [batch, query, min(index_topk, keys)]
int32 key indices; it has no standard name. On a prompt shorter than index_topk it
holds every key, future ones included (masked in the index scores, so ranked last), and
the causal mask removes those. The first first_k_dense_replace blocks (per
mlp_layer_types) have a dense MLP, the rest a mixture of experts. The sizes are
DeepSeek-V2's.