nnterp · encyclopedia deepseek_v32 · nnterp.families.deepseek_v32

family deepseek_v32

DeepSeek-V3.2

Latent attention over only the 2048 keys a lightning indexer scores best for each query, and a mixture with a shared expert that chooses experts by sigmoid score plus a selection bias and weights them without the bias.

Latent attentionSparse attentionInterleaved rotaryMixture of expertsDense first blocks

The block

01 · the residual stream

One block of the stack. Hover any part for its nnterp name; click to pin the card. The slider picks a block, each tick's color gives its block's shape, and the diagram redraws for the block picked.

embed_tokenstoken_embeddings
→
layers× 61
→
normfinal_norm_output
→
lm_headlm_head_output
→
logitsthe output
i = 0

identity layers[i].input + self_attn.attention_output + mlp.mlp_output == layer_output

The API

02 · support()

Values, by host

⚠︎ marks a value this checkpoint has only under some load: hover it for the condition.

model DeepseekV32ForCausalLMlayoutwhat it isread at
logits Logits [batch seq vocab_size] The logits the model returns, after anything it does past lm_head (a softcap, a scale)

the module's own output
token_embeddings Residual [batch seq hidden_size] The embedding module's output; layers[0].input is what enters the first block

the child module embed_tokens's output
final_norm_output Residual [batch seq hidden_size] The residual stream after the final norm, what lm_head reads

the child module norm's output
lm_head_output Logits [batch seq vocab_size] The head's logits before anything the model does past lm_head (a softcap, a scale)

the child module lm_head's output
next_token_probs NextTokenProbs [batch vocab_size] The next-token distribution at the last position; derived, read-only

the module's own output
input_ids Tokens [batch seq] The token ids the model was called with

the module's own input
attention_mask Tokens [batch seq] The attention mask the model was called with; zeros are padding

the module's own input
input_size scalar [batch, seq] of the current call; read-only

the module's own input
model.layers[i] DeepseekV32DecoderLayerlayoutwhat it isread at
layer_input Residual [batch seq hidden_size] The residual stream entering the block

the module's own input
attention_norm_output Residual [batch seq hidden_size] The normed stream entering the attention

the child module input_layernorm's output
layer_mid Residual [batch seq hidden_size] The residual stream after the attention add

the child module post_attention_layernorm's input
mlp_norm_output Residual [batch seq hidden_size] The normed stream entering the MLP

the child module post_attention_layernorm's output
mlp_input Residual [batch seq hidden_size] What the MLP is called with

the child module mlp's input
layer_output Residual [batch seq hidden_size] The residual stream leaving the block, a tensor even when the block returns a tuple

the module's own output
model.layers[i].self_attn DeepseekV32Attentionlayoutwhat it isread at
⚠︎attention_queries Queries [batch num_heads seq qk_head_dim] The queries entering attention

inside the forward: the arguments of attention_interface_1, element 1
⚠︎attention_keys Keys [batch num_kv_heads seq qk_head_dim] The keys entering attention

inside the forward: the arguments of attention_interface_1, element 2
⚠︎attention_values Values [batch num_kv_heads seq head_dim] The values entering attention

inside the forward: the arguments of attention_interface_1, element 3
⚠︎attention_scores Pattern [batch num_heads query key] The attention scores entering the softmax, masked

inside the forward: the input of nn_functional_softmax_0 inside attention_interface_1
attention_premix HeadOutputs [batch seq num_heads head_dim] Every head's output side by side, as the output projection takes it

the child module o_proj's input
attention_output Residual [batch seq hidden_size] What the attention adds to the residual stream

the module's own output
⚠︎attention_probabilities Pattern [batch num_heads query key] The attention pattern the values are mixed with

inside the forward: the output of nn_functional_dropout_0 inside attention_interface_1
⚠︎attention_head_outputs HeadOutputs [batch seq num_heads head_dim] The per-head outputs before the output projection

inside the forward: the output of attention_interface_1, element 0
model.layers[i].mlp DeepseekV32MLP / DeepseekV32MoElayoutwhat it isread at
⚠︎mlp_activation Neurons [batch seq intermediate_size] The activation function's output

the child module act_fn's output
⚠︎mlp_neurons Neurons [batch seq intermediate_size] What the down projection receives: act(gate)·up on a gated MLP, the activation itself otherwise

the child module down_proj's input
mlp_output Residual [batch seq hidden_size] What the MLP adds to the residual stream

the module's own output
router_logits RouterLogits [batch seq num_experts] The router's logits, one per expert, before the scoring

inside router's forward: the output of F_linear_0
expert_weights ExpertWeights [batch seq top_k] The weight each selected expert's output is scaled by

the child module experts's inputs, element 2
expert_indices ExpertIndices [batch seq top_k] The expert each routing slot sends the token to

the child module experts's inputs, element 1
expert_outputs ExpertOutputs [batch seq top_k hidden_size] Each routing slot's weighted expert output

inside experts's forward: the output of weighted_out_view_0 inside experts_forward_1
routed_output Residual [batch seq hidden_size] The routed experts' combined output

the child module experts's output
shared_expert_output Residual [batch seq hidden_size] The shared expert's output

the child module shared_experts's output

Printout

What print(model) shows on deepseek-ai/DeepSeek-V3.2: the native modules, standard names and every standard value with its layout.

DeepseekV32ForCausalLM(
  (lm_head): Linear(in_features=7168, out_features=129280, bias=False)
  (generator): Generator(
    (streamer): Streamer()
  )
  (embed_tokens): Embedding(129280, 7168)
  (layers): ModuleList(
    (0-2): 3 x DeepseekV32DecoderLayer(
      (self_attn): DeepseekV32Attention(
        (q_a_proj): Linear(in_features=7168, out_features=1536, bias=False)
        (q_a_layernorm): DeepseekV32RMSNorm((1536,), eps=1e-06)
        (q_b_proj): Linear(in_features=1536, out_features=24576, bias=False)
        (kv_a_proj_with_mqa): Linear(in_features=7168, out_features=576, bias=False)
        (kv_a_layernorm): DeepseekV32RMSNorm((512,), eps=1e-06)
        (kv_b_proj): Linear(in_features=512, out_features=32768, bias=False)
        (o_proj): Linear(in_features=16384, out_features=7168, bias=False)
        (indexer): DeepseekV32Indexer(
          (wq_b): Linear(in_features=1536, out_features=8192, bias=False)
          (wk): Linear(in_features=7168, out_features=128, bias=False)
          (k_norm): LayerNorm((128,), eps=1e-06, elementwise_affine=True, bias=True)
          (weights_proj): Linear(in_features=7168, out_features=64, bias=False)
        )
        (attention_queries) -> Queries [batch num_heads seq qk_head_dim]: The queries entering attention
        (attention_keys) -> Keys [batch num_kv_heads seq qk_head_dim]: The keys entering attention
        (attention_values) -> Values [batch num_kv_heads seq head_dim]: The values entering attention
        (attention_scores) -> Pattern [batch num_heads query key]: The attention scores entering the softmax, masked
        (attention_premix) -> HeadOutputs [batch seq num_heads head_dim]: Every head's output side by side, as the output projection takes it
        (attention_output) -> Residual [batch seq hidden_size]: What the attention adds to the residual stream
        (attention_probabilities) -> Pattern [batch num_heads query key]: The attention pattern the values are mixed with
        (attention_head_outputs) -> HeadOutputs [batch seq num_heads head_dim]: The per-head outputs before the output projection
      )
      (mlp): DeepseekV32MLP(
        (gate_proj): Linear(in_features=7168, out_features=18432, bias=False)
        (up_proj): Linear(in_features=7168, out_features=18432, bias=False)
        (down_proj): Linear(in_features=18432, out_features=7168, bias=False)
        (act_fn): SiLUActivation()
        (mlp_activation) -> Neurons [batch seq intermediate_size]: The activation function's output
        (mlp_neurons) -> Neurons [batch seq intermediate_size]: What the down projection receives: act(gate)·up on a gated MLP, the activation itself otherwise
        (mlp_output) -> Residual [batch seq hidden_size]: What the MLP adds to the residual stream
      )
      (input_layernorm): DeepseekV32RMSNorm((7168,), eps=1e-06)
      (post_attention_layernorm): DeepseekV32RMSNorm((7168,), eps=1e-06)
      (layer_input) -> Residual [batch seq hidden_size]: The residual stream entering the block
      (attention_norm_output) -> Residual [batch seq hidden_size]: The normed stream entering the attention
      (layer_mid) -> Residual [batch seq hidden_size]: The residual stream after the attention add
      (mlp_norm_output) -> Residual [batch seq hidden_size]: The normed stream entering the MLP
      (mlp_input) -> Residual [batch seq hidden_size]: What the MLP is called with
      (layer_output) -> Residual [batch seq hidden_size]: The residual stream leaving the block, a tensor even when the block returns a tuple
    )
    (3-60): 58 x DeepseekV32DecoderLayer(
      (self_attn): DeepseekV32Attention(
        (q_a_proj): Linear(in_features=7168, out_features=1536, bias=False)
        (q_a_layernorm): DeepseekV32RMSNorm((1536,), eps=1e-06)
        (q_b_proj): Linear(in_features=1536, out_features=24576, bias=False)
        (kv_a_proj_with_mqa): Linear(in_features=7168, out_features=576, bias=False)
        (kv_a_layernorm): DeepseekV32RMSNorm((512,), eps=1e-06)
        (kv_b_proj): Linear(in_features=512, out_features=32768, bias=False)
        (o_proj): Linear(in_features=16384, out_features=7168, bias=False)
        (indexer): DeepseekV32Indexer(
          (wq_b): Linear(in_features=1536, out_features=8192, bias=False)
          (wk): Linear(in_features=7168, out_features=128, bias=False)
          (k_norm): LayerNorm((128,), eps=1e-06, elementwise_affine=True, bias=True)
          (weights_proj): Linear(in_features=7168, out_features=64, bias=False)
        )
        (attention_queries) -> Queries [batch num_heads seq qk_head_dim]: The queries entering attention
        (attention_keys) -> Keys [batch num_kv_heads seq qk_head_dim]: The keys entering attention
        (attention_values) -> Values [batch num_kv_heads seq head_dim]: The values entering attention
        (attention_scores) -> Pattern [batch num_heads query key]: The attention scores entering the softmax, masked
        (attention_premix) -> HeadOutputs [batch seq num_heads head_dim]: Every head's output side by side, as the output projection takes it
        (attention_output) -> Residual [batch seq hidden_size]: What the attention adds to the residual stream
        (attention_probabilities) -> Pattern [batch num_heads query key]: The attention pattern the values are mixed with
        (attention_head_outputs) -> HeadOutputs [batch seq num_heads head_dim]: The per-head outputs before the output projection
      )
      (mlp): DeepseekV32MoE(
        (experts): DeepseekV32Experts(
          (act_fn): SiLUActivation()
        )
        (router/gate): DeepseekV32TopkRouter()
        (shared_experts): DeepseekV32MLP(
          (gate_proj): Linear(in_features=7168, out_features=2048, bias=False)
          (up_proj): Linear(in_features=7168, out_features=2048, bias=False)
          (down_proj): Linear(in_features=2048, out_features=7168, bias=False)
          (act_fn): SiLUActivation()
          (mlp_activation) -> Neurons [batch seq intermediate_size]: The activation function's output
          (mlp_neurons) -> Neurons [batch seq intermediate_size]: What the down projection receives: act(gate)·up on a gated MLP, the activation itself otherwise
          (mlp_output) -> Residual [batch seq hidden_size]: What the MLP adds to the residual stream
        )
        (mlp_activation) -> Neurons [batch seq intermediate_size]: The activation function's output
        (mlp_neurons) -> Neurons [batch seq intermediate_size]: What the down projection receives: act(gate)·up on a gated MLP, the activation itself otherwise
        (router_logits) -> RouterLogits [batch seq num_experts]: The router's logits, one per expert, before the scoring
        (expert_weights) -> ExpertWeights [batch seq top_k]: The weight each selected expert's output is scaled by
        (expert_indices) -> ExpertIndices [batch seq top_k]: The expert each routing slot sends the token to
        (expert_outputs) -> ExpertOutputs [batch seq top_k hidden_size]: Each routing slot's weighted expert output
        (routed_output) -> Residual [batch seq hidden_size]: The routed experts' combined output
        (shared_expert_output) -> Residual [batch seq hidden_size]: The shared expert's output
        (mlp_output) -> Residual [batch seq hidden_size]: What the MLP adds to the residual stream
      )
      (input_layernorm): DeepseekV32RMSNorm((7168,), eps=1e-06)
      (post_attention_layernorm): DeepseekV32RMSNorm((7168,), eps=1e-06)
      (layer_input) -> Residual [batch seq hidden_size]: The residual stream entering the block
      (attention_norm_output) -> Residual [batch seq hidden_size]: The normed stream entering the attention
      (layer_mid) -> Residual [batch seq hidden_size]: The residual stream after the attention add
      (mlp_norm_output) -> Residual [batch seq hidden_size]: The normed stream entering the MLP
      (mlp_input) -> Residual [batch seq hidden_size]: What the MLP is called with
      (layer_output) -> Residual [batch seq hidden_size]: The residual stream leaving the block, a tensor even when the block returns a tuple
    )
  )
  (norm): DeepseekV32RMSNorm((7168,), eps=1e-06)
  (logits) -> Logits [batch seq vocab_size]: The logits the model returns, after anything it does past lm_head (a softcap, a scale)
  (token_embeddings) -> Residual [batch seq hidden_size]: The embedding module's output; layers[0].input is what enters the first block
  (final_norm_output) -> Residual [batch seq hidden_size]: The residual stream after the final norm, what lm_head reads
  (lm_head_output) -> Logits [batch seq vocab_size]: The head's logits before anything the model does past lm_head (a softcap, a scale)
  (next_token_probs) -> NextTokenProbs [batch vocab_size]: The next-token distribution at the last position; derived, read-only
  (input_ids) -> Tokens [batch seq]: The token ids the model was called with
  (attention_mask) -> Tokens [batch seq]: The attention mask the model was called with; zeros are padding
  (input_size): [batch, seq] of the current call; read-only
)

Config

model.num_layers61
model.hidden_size7168
model.num_heads128
model.num_kv_heads128
model.head_dim128
model.qk_head_dim192
model.vocab_size129280
model.intermediate_size18432
architecturesDeepseekV32ForCausalLM
num_hidden_layers61
hidden_size7168
intermediate_size18432
num_attention_heads128
num_key_value_heads128
head_dim64
vocab_size129280
max_position_embeddings163840
hidden_actsilu
rms_norm_eps1e-06
tie_word_embeddingsFalse
dtypebfloat16
q_lora_rank1536
kv_lora_rank512
qk_nope_head_dim128
qk_rope_head_dim64
v_head_dim128
first_k_dense_replace3
moe_intermediate_size2048
n_routed_experts256
num_experts_per_tok8
n_shared_experts1
routed_scaling_factor2.5
norm_topk_probTrue
attention_biasFalse
n_group8
topk_group4

The Notes

03 · what to know before you run

The block, in order

h   = x + self_attn(input_layernorm(x))      # sparse latent attention
out = h + mlp(post_attention_layernorm(h))   # dense on blocks 0-2, a mixture after

Llama's pre-norm block and Llama's names. The block adds each sublayer's output to the stream and returns a tensor, so the identity is the plain sum on a dense block and on a mixture block:

with model.trace(prompt):
    x = model.layers[1].input.save()
    attn = model.layers[1].self_attn.attention_output.save()
    mlp = model.layers[1].mlp.mlp_output.save()
    out = model.layers[1].layer_output.save()

torch.testing.assert_close(x + attn + mlp, out)

config.mlp_layer_types says which blocks are dense: blocks 0 to 2 on every checkpoint (first_k_dense_replace 3), a mixture on the other 58. The configs carry one multi-token-prediction block (num_nextn_predict_layers 1), stored after the last block; transformers does not build it, so model.layers has 61 blocks.

Latent attention: every head's key comes from one latent

attention_queries and attention_keys are qk_head_dim wide (qk_nope_head_dim + qk_rope_head_dim, 128 + 64 = 192), attention_values and attention_head_outputs head_dim (v_head_dim, 128). transformers sets the config's head_dim to qk_rope_head_dim, the rotary width (64); no served value is that wide. Queries go through a compressed rank (q_lora_rank 1536): q_a_proj, q_a_layernorm, then q_b_proj, with no q_proj module. Keys and values are expanded for every head from one latent per token (kv_lora_rank 512), so attention_keys has num_heads (128) heads. The last 64 dimensions of every head's key are one rotary key that all heads share.

attn = model.layers[1].self_attn
with model.trace(prompt):
    q = attn.attention_queries.save()          # [1, 128, seq, 192]
    k = attn.attention_keys.save()             # [1, 128, seq, 192]
    v = attn.attention_values.save()           # [1, 128, seq, 128]

assert torch.equal(k[:, 0, :, -64:], k[:, 1, :, -64:])   # one rotary key

The rotary part is served de-interleaved

The attention rotates pair (2i, 2i + 1) of the projection into dimensions i and i + 32 of the rotary part. The forward always does this; the config has no rope_interleave key to switch it. attention_queries[..., -64:] and attention_keys[..., -64:] are in that rotate-half order: at position 0, where rotary is the identity, the served rotary part is q_b_proj's even dimensions followed by its odd ones.

with model.trace(prompt):
    proj = attn.q_b_proj.output.save()
with model.trace(prompt):
    q = attn.attention_queries.save()

rope = proj.view(1, -1, model.num_heads, model.qk_head_dim)[0, 0, 0, -64:]
torch.testing.assert_close(q[0, 0, 0, -64:], torch.cat([rope[0::2], rope[1::2]]))

The query scale carries YaRN's factor

Every checkpoint extends the context with YaRN (rope_scaling factor 40, mscale_all_dim 1.0), and the attention multiplies the scores by qk_head_dim ** -0.5 times 1.87 (self_attn.scaling, 0.1352), not by qk_head_dim ** -0.5 (0.0722) alone; recompute attention_scores from the queries and keys with self_attn.scaling.

Up to 2048 tokens the sparse attention is plain causal attention

A lightning indexer (self_attn.indexer: index_n_heads 64 heads of index_head_dim 128) scores every key for every query and keeps the index_topk (2048) best. The attention sets the other keys' mask entries to the dtype's minimum before it calls the attention function. It does this under eager and under the default sdpa load alike, so both loads run the same sparse computation; the interior values need attn_implementation="eager". attention_scores carry that mask, so a dropped key reads like a future one. attention_probabilities is the dense [batch, num_heads, query, key] pattern, exactly zero outside each query's selection.

The selection is self_attn.indexer.output, int32 key indices of shape [batch, query, min(index_topk, seq)]; it has no standard name. The indexer ranks future keys last but does not drop them. On a prompt of up to 2048 tokens each row therefore lists every position, the causal mask removes the future keys, and the pattern is plain causal: row t has t + 1 nonzero entries.

with model.trace(prompt):
    sel = attn.indexer.output.save()           # [1, seq, min(2048, seq)]
    probs = attn.attention_probabilities.save()

rows = (probs[0, 0] != 0).sum(-1)
assert torch.equal(rows, torch.arange(1, rows.numel() + 1))   # plain causal

A written pattern is used as written, so a write can give weight to a key the indexer dropped. The indexer runs under torch.no_grad(): no gradient reaches its projections, and the selection does not move with a gradient-based attribution.

The mixture: sigmoid scores, a selection bias, renormalized weights

256 routed experts, 8 per token, and one shared expert every token runs through. router_logits are the float32 logits before the sigmoid, and before the selection bias (router.e_score_correction_bias). The bias is added to the sigmoids only to choose: the experts sit in n_group 8 groups, each group scored by the sum of its two best biased scores, and a token takes its 8 experts from its best topk_group 4 groups. expert_weights are the chosen experts' sigmoids without the bias, renormalized to sum to one (norm_topk_prob) and multiplied by routed_scaling_factor, so they sum to 2.5 at every token. A written router_logits moves both the choice and the weights; a written bias moves only the choice.

moe = model.layers[3].mlp
with model.trace(prompt):
    logits = moe.router_logits.save()
    w = moe.expert_weights.save()
    idx = moe.expert_indices.save()
    routed = moe.routed_output.save()
    shared = moe.shared_expert_output.save()
    out = moe.mlp_output.save()

chosen = logits.sigmoid().gather(-1, idx)
scale = model.config.routed_scaling_factor
torch.testing.assert_close(w, chosen / chosen.sum(-1, keepdim=True) * scale)
torch.testing.assert_close(routed + shared, out)

The shared expert runs after the routed ones, so shared_expert_output is read after routed_output. Ablating an expert is zeroing the weight of every slot that chose it; the other slots keep their weights, so the token's weights then sum to less than 2.5:

with model.trace(prompt):
    moe.expert_weights = moe.expert_weights.masked_fill(moe.expert_indices == 3, 0)
    ablated = model.logits.save()

Blocks 0 to 2 have a dense MLP: support() reports every mixture value missing there.

From the family module

04 · nnterp/families/deepseek_v32.py

DeepSeek-V3.2 (DeepseekV32ForCausalLM).

Llama's tree and Llama's block: model.{embed_tokens, layers[i].{input_layernorm, self_attn, post_attention_layernorm, mlp}, norm} and lm_head, the residual added in the block, a bare tensor returned. Multi-head latent attention as in V3, with DeepSeek Sparse Attention on top: each attention module carries an indexer (a lightning indexer with its own small projections) that scores every key for every query and keeps the index_topk best. Under eager and sdpa the selection is folded into the attention mask (every unselected key set to the dtype's minimum) before the shared attention_interface call, so the interior is the base's, read on the interface:

  • attention_scores are the scaled scores with that sparse mask added, so an unselected key reads as the dtype's minimum, like a future one.
  • attention_probabilities is the dense [batch, num_heads, query, key] pattern of the sparse attention: exactly zero outside each query's selected keys, rows summing to one over the selection. A written pattern is used as written, so a write can put weight on keys the indexer did not select.

While a prompt is no longer than index_topk (2048 on the released checkpoints) the indexer selects every causal key and the pattern is the dense causal one. The indexer's selection is self_attn.indexer.output, the [batch, query, min(index_topk, keys)] int32 key indices; it has no standard name. On a prompt shorter than index_topk it holds every key, future ones included (masked in the index scores, so ranked last), and the causal mask removes those. The first first_k_dense_replace blocks (per mlp_layer_types) have a dense MLP, the rest a mixture of experts. The sizes are DeepSeek-V2's.