nnterp · encyclopedia deepseek_v3 · nnterp.families.deepseek_v3

family deepseek_v3

DeepSeek-V3DeepSeek-R1Moonlight

Latent attention with queries and keys wider than the values, and a mixture with a shared expert, whose router chooses experts by sigmoid scores plus a selection bias and weights them without it.

Latent attentionInterleaved rotaryMixture of expertsDense first blocks

The block

01 · the residual stream

One block of the stack. Hover any part for its nnterp name; click to pin the card. The slider picks a block, each tick's color gives its block's shape, and the diagram redraws for the block picked.

embed_tokenstoken_embeddings
→
layers× 61
→
normfinal_norm_output
→
lm_headlm_head_output
→
logitsthe output
i = 0

identity layers[i].input + self_attn.attention_output + mlp.mlp_output == layer_output

The API

02 · support()

Values, by host

⚠︎ marks a value this checkpoint has only under some load: hover it for the condition.

model DeepseekV3ForCausalLMlayoutwhat it isread at
logits Logits [batch seq vocab_size] The logits the model returns, after anything it does past lm_head (a softcap, a scale)

the module's own output
token_embeddings Residual [batch seq hidden_size] The embedding module's output; layers[0].input is what enters the first block

the child module embed_tokens's output
final_norm_output Residual [batch seq hidden_size] The residual stream after the final norm, what lm_head reads

the child module norm's output
lm_head_output Logits [batch seq vocab_size] The head's logits before anything the model does past lm_head (a softcap, a scale)

the child module lm_head's output
next_token_probs NextTokenProbs [batch vocab_size] The next-token distribution at the last position; derived, read-only

the module's own output
input_ids Tokens [batch seq] The token ids the model was called with

the module's own input
attention_mask Tokens [batch seq] The attention mask the model was called with; zeros are padding

the module's own input
input_size scalar [batch, seq] of the current call; read-only

the module's own input
model.layers[i] DeepseekV3DecoderLayerlayoutwhat it isread at
layer_input Residual [batch seq hidden_size] The residual stream entering the block

the module's own input
attention_norm_output Residual [batch seq hidden_size] The normed stream entering the attention

the child module input_layernorm's output
layer_mid Residual [batch seq hidden_size] The residual stream after the attention add

the child module post_attention_layernorm's input
mlp_norm_output Residual [batch seq hidden_size] The normed stream entering the MLP

the child module post_attention_layernorm's output
mlp_input Residual [batch seq hidden_size] What the MLP is called with

the child module mlp's input
layer_output Residual [batch seq hidden_size] The residual stream leaving the block, a tensor even when the block returns a tuple

the module's own output
model.layers[i].self_attn DeepseekV3Attentionlayoutwhat it isread at
⚠︎attention_queries Queries [batch num_heads seq qk_head_dim] The queries entering attention

inside the forward: the arguments of attention_interface_1, element 1
⚠︎attention_keys Keys [batch num_kv_heads seq qk_head_dim] The keys entering attention

inside the forward: the arguments of attention_interface_1, element 2
⚠︎attention_values Values [batch num_kv_heads seq head_dim] The values entering attention

inside the forward: the arguments of attention_interface_1, element 3
⚠︎attention_scores Pattern [batch num_heads query key] The attention scores entering the softmax, masked

inside the forward: the input of nn_functional_softmax_0 inside attention_interface_1
attention_premix HeadOutputs [batch seq num_heads head_dim] Every head's output side by side, as the output projection takes it

the child module o_proj's input
attention_output Residual [batch seq hidden_size] What the attention adds to the residual stream

the module's own output
⚠︎attention_probabilities Pattern [batch num_heads query key] The attention pattern the values are mixed with

inside the forward: the output of nn_functional_dropout_0 inside attention_interface_1
⚠︎attention_head_outputs HeadOutputs [batch seq num_heads head_dim] The per-head outputs before the output projection

inside the forward: the output of attention_interface_1, element 0
model.layers[i].mlp DeepseekV3MLP / DeepseekV3MoElayoutwhat it isread at
⚠︎mlp_activation Neurons [batch seq intermediate_size] The activation function's output

the child module act_fn's output
⚠︎mlp_neurons Neurons [batch seq intermediate_size] What the down projection receives: act(gate)·up on a gated MLP, the activation itself otherwise

the child module down_proj's input
mlp_output Residual [batch seq hidden_size] What the MLP adds to the residual stream

the module's own output
router_logits RouterLogits [batch seq num_experts] The router's logits, one per expert, before the scoring

inside router's forward: the output of F_linear_0
expert_weights ExpertWeights [batch seq top_k] The weight each selected expert's output is scaled by

the child module experts's inputs, element 2
expert_indices ExpertIndices [batch seq top_k] The expert each routing slot sends the token to

the child module experts's inputs, element 1
expert_outputs ExpertOutputs [batch seq top_k hidden_size] Each routing slot's weighted expert output

inside experts's forward: the output of weighted_out_view_0 inside experts_forward_1
routed_output Residual [batch seq hidden_size] The routed experts' combined output

the child module experts's output
shared_expert_output Residual [batch seq hidden_size] The shared expert's output

the child module shared_experts's output

Printout

What print(model) shows on deepseek-ai/DeepSeek-V3: the native modules, standard names and every standard value with its layout.

DeepseekV3ForCausalLM(
  (lm_head): Linear(in_features=7168, out_features=129280, bias=False)
  (generator): Generator(
    (streamer): Streamer()
  )
  (embed_tokens): Embedding(129280, 7168)
  (layers): ModuleList(
    (0-2): 3 x DeepseekV3DecoderLayer(
      (self_attn): DeepseekV3Attention(
        (q_a_proj): Linear(in_features=7168, out_features=1536, bias=False)
        (q_a_layernorm): DeepseekV3RMSNorm((1536,), eps=1e-06)
        (q_b_proj): Linear(in_features=1536, out_features=24576, bias=False)
        (kv_a_proj_with_mqa): Linear(in_features=7168, out_features=576, bias=False)
        (kv_a_layernorm): DeepseekV3RMSNorm((512,), eps=1e-06)
        (kv_b_proj): Linear(in_features=512, out_features=32768, bias=False)
        (o_proj): Linear(in_features=16384, out_features=7168, bias=False)
        (attention_queries) -> Queries [batch num_heads seq qk_head_dim]: The queries entering attention
        (attention_keys) -> Keys [batch num_kv_heads seq qk_head_dim]: The keys entering attention
        (attention_values) -> Values [batch num_kv_heads seq head_dim]: The values entering attention
        (attention_scores) -> Pattern [batch num_heads query key]: The attention scores entering the softmax, masked
        (attention_premix) -> HeadOutputs [batch seq num_heads head_dim]: Every head's output side by side, as the output projection takes it
        (attention_output) -> Residual [batch seq hidden_size]: What the attention adds to the residual stream
        (attention_probabilities) -> Pattern [batch num_heads query key]: The attention pattern the values are mixed with
        (attention_head_outputs) -> HeadOutputs [batch seq num_heads head_dim]: The per-head outputs before the output projection
      )
      (mlp): DeepseekV3MLP(
        (gate_proj): Linear(in_features=7168, out_features=18432, bias=False)
        (up_proj): Linear(in_features=7168, out_features=18432, bias=False)
        (down_proj): Linear(in_features=18432, out_features=7168, bias=False)
        (act_fn): SiLUActivation()
        (mlp_activation) -> Neurons [batch seq intermediate_size]: The activation function's output
        (mlp_neurons) -> Neurons [batch seq intermediate_size]: What the down projection receives: act(gate)·up on a gated MLP, the activation itself otherwise
        (mlp_output) -> Residual [batch seq hidden_size]: What the MLP adds to the residual stream
      )
      (input_layernorm): DeepseekV3RMSNorm((7168,), eps=1e-06)
      (post_attention_layernorm): DeepseekV3RMSNorm((7168,), eps=1e-06)
      (layer_input) -> Residual [batch seq hidden_size]: The residual stream entering the block
      (attention_norm_output) -> Residual [batch seq hidden_size]: The normed stream entering the attention
      (layer_mid) -> Residual [batch seq hidden_size]: The residual stream after the attention add
      (mlp_norm_output) -> Residual [batch seq hidden_size]: The normed stream entering the MLP
      (mlp_input) -> Residual [batch seq hidden_size]: What the MLP is called with
      (layer_output) -> Residual [batch seq hidden_size]: The residual stream leaving the block, a tensor even when the block returns a tuple
    )
    (3-60): 58 x DeepseekV3DecoderLayer(
      (self_attn): DeepseekV3Attention(
        (q_a_proj): Linear(in_features=7168, out_features=1536, bias=False)
        (q_a_layernorm): DeepseekV3RMSNorm((1536,), eps=1e-06)
        (q_b_proj): Linear(in_features=1536, out_features=24576, bias=False)
        (kv_a_proj_with_mqa): Linear(in_features=7168, out_features=576, bias=False)
        (kv_a_layernorm): DeepseekV3RMSNorm((512,), eps=1e-06)
        (kv_b_proj): Linear(in_features=512, out_features=32768, bias=False)
        (o_proj): Linear(in_features=16384, out_features=7168, bias=False)
        (attention_queries) -> Queries [batch num_heads seq qk_head_dim]: The queries entering attention
        (attention_keys) -> Keys [batch num_kv_heads seq qk_head_dim]: The keys entering attention
        (attention_values) -> Values [batch num_kv_heads seq head_dim]: The values entering attention
        (attention_scores) -> Pattern [batch num_heads query key]: The attention scores entering the softmax, masked
        (attention_premix) -> HeadOutputs [batch seq num_heads head_dim]: Every head's output side by side, as the output projection takes it
        (attention_output) -> Residual [batch seq hidden_size]: What the attention adds to the residual stream
        (attention_probabilities) -> Pattern [batch num_heads query key]: The attention pattern the values are mixed with
        (attention_head_outputs) -> HeadOutputs [batch seq num_heads head_dim]: The per-head outputs before the output projection
      )
      (mlp): DeepseekV3MoE(
        (experts): DeepseekV3Experts(
          (act_fn): SiLUActivation()
        )
        (router/gate): DeepseekV3TopkRouter()
        (shared_experts): DeepseekV3MLP(
          (gate_proj): Linear(in_features=7168, out_features=2048, bias=False)
          (up_proj): Linear(in_features=7168, out_features=2048, bias=False)
          (down_proj): Linear(in_features=2048, out_features=7168, bias=False)
          (act_fn): SiLUActivation()
          (mlp_activation) -> Neurons [batch seq intermediate_size]: The activation function's output
          (mlp_neurons) -> Neurons [batch seq intermediate_size]: What the down projection receives: act(gate)·up on a gated MLP, the activation itself otherwise
          (mlp_output) -> Residual [batch seq hidden_size]: What the MLP adds to the residual stream
        )
        (mlp_activation) -> Neurons [batch seq intermediate_size]: The activation function's output
        (mlp_neurons) -> Neurons [batch seq intermediate_size]: What the down projection receives: act(gate)·up on a gated MLP, the activation itself otherwise
        (router_logits) -> RouterLogits [batch seq num_experts]: The router's logits, one per expert, before the scoring
        (expert_weights) -> ExpertWeights [batch seq top_k]: The weight each selected expert's output is scaled by
        (expert_indices) -> ExpertIndices [batch seq top_k]: The expert each routing slot sends the token to
        (expert_outputs) -> ExpertOutputs [batch seq top_k hidden_size]: Each routing slot's weighted expert output
        (routed_output) -> Residual [batch seq hidden_size]: The routed experts' combined output
        (shared_expert_output) -> Residual [batch seq hidden_size]: The shared expert's output
        (mlp_output) -> Residual [batch seq hidden_size]: What the MLP adds to the residual stream
      )
      (input_layernorm): DeepseekV3RMSNorm((7168,), eps=1e-06)
      (post_attention_layernorm): DeepseekV3RMSNorm((7168,), eps=1e-06)
      (layer_input) -> Residual [batch seq hidden_size]: The residual stream entering the block
      (attention_norm_output) -> Residual [batch seq hidden_size]: The normed stream entering the attention
      (layer_mid) -> Residual [batch seq hidden_size]: The residual stream after the attention add
      (mlp_norm_output) -> Residual [batch seq hidden_size]: The normed stream entering the MLP
      (mlp_input) -> Residual [batch seq hidden_size]: What the MLP is called with
      (layer_output) -> Residual [batch seq hidden_size]: The residual stream leaving the block, a tensor even when the block returns a tuple
    )
  )
  (norm): DeepseekV3RMSNorm((7168,), eps=1e-06)
  (logits) -> Logits [batch seq vocab_size]: The logits the model returns, after anything it does past lm_head (a softcap, a scale)
  (token_embeddings) -> Residual [batch seq hidden_size]: The embedding module's output; layers[0].input is what enters the first block
  (final_norm_output) -> Residual [batch seq hidden_size]: The residual stream after the final norm, what lm_head reads
  (lm_head_output) -> Logits [batch seq vocab_size]: The head's logits before anything the model does past lm_head (a softcap, a scale)
  (next_token_probs) -> NextTokenProbs [batch vocab_size]: The next-token distribution at the last position; derived, read-only
  (input_ids) -> Tokens [batch seq]: The token ids the model was called with
  (attention_mask) -> Tokens [batch seq]: The attention mask the model was called with; zeros are padding
  (input_size): [batch, seq] of the current call; read-only
)

Config

model.num_layers61
model.hidden_size7168
model.num_heads128
model.num_kv_heads128
model.head_dim128
model.qk_head_dim192
model.vocab_size129280
model.intermediate_size18432
architecturesDeepseekV3ForCausalLM
num_hidden_layers61
hidden_size7168
intermediate_size18432
num_attention_heads128
num_key_value_heads128
head_dim64
vocab_size129280
max_position_embeddings163840
hidden_actsilu
rms_norm_eps1e-06
tie_word_embeddingsFalse
dtypebfloat16
q_lora_rank1536
kv_lora_rank512
qk_nope_head_dim128
qk_rope_head_dim64
v_head_dim128
first_k_dense_replace3
moe_intermediate_size2048
n_routed_experts256
num_experts_per_tok8
n_shared_experts1
routed_scaling_factor2.5
norm_topk_probTrue
attention_biasFalse
n_group8
topk_group4

The Notes

03 · what to know before you run

The block, in order

h   = x + self_attn(input_layernorm(x))      # multi-head latent attention
out = h + mlp(post_attention_layernorm(h))   # dense on blocks 0-2, a mixture after

Llama's pre-norm block and Llama's names. The first first_k_dense_replace blocks have a dense MLP and the rest a mixture: blocks 0 to 2 on the DeepSeek checkpoints, block 0 on Moonlight. The block adds each sublayer's output to the stream and returns a tensor, so the identity is the plain sum on a dense block and on a mixture block:

with model.trace(prompt):
    x = model.layers[3].input.save()
    attn = model.layers[3].self_attn.attention_output.save()
    mlp = model.layers[3].mlp.mlp_output.save()
    out = model.layers[3].layer_output.save()

torch.testing.assert_close(x + attn + mlp, out)

The DeepSeek configs carry one multi-token-prediction block (num_nextn_predict_layers 1), stored after the last block. transformers does not build it, so model.layers has 61 blocks.

Latent attention: queries and keys are wider than values

attention_queries and attention_keys are qk_head_dim wide (qk_nope_head_dim + qk_rope_head_dim, 128 + 64 = 192), attention_values and attention_head_outputs head_dim (v_head_dim, 128). transformers sets the config's head_dim to qk_rope_head_dim (64), which no served value has. Every DeepSeek checkpoint compresses the queries (q_lora_rank 1536: q_a_proj, q_a_layernorm and q_b_proj, with no q_proj module). Moonlight has q_lora_rank null and one q_proj.

Keys and values are expanded from one latent per token (kv_lora_rank 512) for every head, so attention_keys has num_heads (128) heads, and the last 64 dimensions of every head's key are one shared rotary key. The attention interior needs attn_implementation="eager".

attn = model.layers[3].self_attn
with model.trace(prompt):
    q = attn.attention_queries.save()          # [1, 128, seq, 192]
    k = attn.attention_keys.save()             # [1, 128, seq, 192]
    v = attn.attention_values.save()           # [1, 128, seq, 128]

assert torch.equal(k[:, 0, :, -64:], k[:, 1, :, -64:])   # one rotary key

The rotary part is served de-interleaved

The checkpoints store the rotary dimensions in interleaved pairs (rope_interleave, which defaults to true and no checkpoint changes), and the attention rotates pair (2i, 2i + 1) of the projection into dimensions i and i + 32 of the rotary part. attention_queries[..., -64:] and attention_keys[..., -64:] are in that rotate-half order, not the order of q_b_proj's output: at position 0, where rotary is the identity, the served rotary part is the projection's even dimensions followed by its odd ones.

with model.trace(prompt):
    proj = attn.q_b_proj.output.save()
with model.trace(prompt):
    q = attn.attention_queries.save()

rope = proj.view(1, -1, model.num_heads, model.qk_head_dim)[0, 0, 0, -64:]
torch.testing.assert_close(q[0, 0, 0, -64:], torch.cat([rope[0::2], rope[1::2]]))

The query scale carries YaRN's factor

The DeepSeek checkpoints extend the context with YaRN (rope_scaling factor 40, mscale_all_dim 1.0), and the attention multiplies the scores by qk_head_dim ** -0.5 times 1.87 (self_attn.scaling, 0.1352), not by qk_head_dim ** -0.5 alone. Recompute attention_scores from the queries and keys with self_attn.scaling. Moonlight has no rope_scaling, and its scale is qk_head_dim ** -0.5.

The mixture: sigmoid scores, a selection bias, renormalized weights

256 routed experts, 8 per token, and one shared expert every token runs through (on Moonlight 64 routed experts, 6 per token and two shared experts). router_logits are the logits before the sigmoid, and before the selection bias (router.e_score_correction_bias). The bias is added to the sigmoids only to choose: the experts sit in n_group 8 groups, each group scored by the sum of its two best biased scores, and a token takes its 8 experts from its best topk_group 4 groups. Moonlight has one group, so every expert is a candidate. expert_weights are the chosen experts' sigmoids without the bias, renormalized to sum to one (norm_topk_prob) and multiplied by routed_scaling_factor, so they sum to 2.5 at every token (2.446 on Moonlight). A written router_logits moves both the choice and the weights; a written bias moves only the choice.

moe = model.layers[3].mlp
with model.trace(prompt):
    logits = moe.router_logits.save()
    w = moe.expert_weights.save()
    idx = moe.expert_indices.save()
    routed = moe.routed_output.save()
    shared = moe.shared_expert_output.save()
    out = moe.mlp_output.save()

chosen = logits.sigmoid().gather(-1, idx)
scale = model.config.routed_scaling_factor
torch.testing.assert_close(w, chosen / chosen.sum(-1, keepdim=True) * scale)
torch.testing.assert_close(routed + shared, out)

The shared expert runs after the routed experts. Ablating an expert is zeroing the weight of every slot that chose it; the other slots keep their weights, so the token's weights then sum to less than 2.5:

with model.trace(prompt):
    moe.expert_weights = moe.expert_weights.masked_fill(moe.expert_indices == 3, 0)
    ablated = model.logits.save()

Blocks 0 to 2 have a dense MLP: support() reports every mixture value missing there.

From the family module

04 · nnterp/families/deepseek_v3.py

DeepSeek-V3 (DeepseekV3ForCausalLM).

Llama's tree and Llama's block: model.{embed_tokens, layers[i].{input_layernorm, self_attn, post_attention_layernorm, mlp}, norm} and lm_head, the residual added in the block, attention through the shared eager forward. Multi-head latent attention as in V2; dense MLPs for the first first_k_dense_replace blocks, a mixture of experts after. Moonshot's Kimi K2.5 / K2.6 / K2.7-Code (Kimi_K25ForConditionalGeneration) wrap this text model under model.language_model with lm_head at the root, so RENAME carries both spellings; their text_config says kimi_k2, which kimi_k2.py maps to this family.