Skip to content

deepseek_v4

deepseek_v4

DeepSeek-V4 (DeepseekV4ForCausalLM): a hyper-connection residual, compressed attention, all-MoE.

model.{embed_tokens, layers[i].{attn_hc, input_layernorm, self_attn, ffn_hc, post_attention_layernorm, mlp}, hc_head, norm} and lm_head.

The residual is hc_mult parallel streams, [batch, seq, streams, hidden_size], between every pair of blocks (the model copies the embedding into each stream before block 0). Each sublayer reads one collapse of the streams and writes back into all of them through a hyper-connection (attn_hc before the attention, ffn_hc before the MLP), which returns (post, comb, collapsed): collapsed is the pre-weighted sum of the streams that the sublayer's norm reads, post [batch, seq, streams] how much of the sublayer's output each stream receives, comb [batch, seq, streams, streams] a doubly stochastic matrix mixing the streams, consumed transposed. The block, per token::

h   = attention_combᵀ · x  + attention_post ⊗ self_attn(input_layernorm(collapsed_a))
out = mlp_combᵀ       · h  + mlp_post       ⊗ mlp(post_attention_layernorm(collapsed_f))
  • layer_output is the block's own stream tensor, under the Streams layout, and so is layers[i].input; writes land natively, and skip_layers and steer work unchanged (a [hidden_size] steering vector broadcasts over the streams).
  • The contributions are the sublayers' own outputs, [batch, seq, hidden_size], unscaled: attention_output is self_attn.output[0], mlp_output the mixture's return (routed plus shared experts). The weights that put them into the streams are the block's attention_post, attention_comb, mlp_post and mlp_comb, read off the hyper-connections' outputs. The identity is the block's formula above, not a sum; its stream mean is additive (comb is doubly stochastic) up to the Sinkhorn projection's residual, about 2e-6 relative in float32: ``layer_output.mean(2) == input.mean(2) + attention_post.mean(-1, keepdim=True)
  • attention_output + mlp_post.mean(-1, keepdim=True) * mlp_output``.
  • The readout collapses the streams once more: norm(hc_head(streams)), hc_head a learned, content-dependent weighting. The family's project_on_vocab does the same, so the lens on the last block is logits exactly.

The attention runs the shared interface, with a per-head sink (s_aux), so the scores are GPT-OSS's binding. It is multi-query (one key/value head), and the keys and values are one tensor: the same object is passed as both, so an in-place edit of one edits the other. On a compressed block (compressed_sparse_attention, heavily_compressed_attention) the compressor's entries are concatenated after the token keys once the prompt is compress_rates[type] tokens long, so the key axis is longer than the sequence there. The interface's output keeps a rotation on its rotary slice (the values are the rotated keys); the module rotates it back before the grouped output projection, and attention_head_outputs is that de-rotated tensor, what the projection reads. Not latent attention: the plain root sizes hold.

Every MLP is a mixture (DeepseekV4SparseMoeBlock, a Moe; its router, gate, is aliased router); on the hash_moe blocks the router picks experts from the token ids (tid2eid[input_ids]), so a run needs input_ids. There the logits only weight the experts the ids chose: writing router_logits changes expert_weights, not expert_indices.

RENAME module-attribute 

RENAME = {'model.embed_tokens': 'embed_tokens', 'model.layers': 'layers', 'model.norm': 'norm', 'gate': 'router'}

STREAMS_BOUNDARY module-attribute 

STREAMS_BOUNDARY = "hyper-connection streams: the sublayers read a mix of the streams, not the stream; the block's inner boundaries are not mapped for this family yet"

ENVOYS module-attribute 

ENVOYS = {DeepseekV4DecoderLayer: Layer, DeepseekV4Attention: Attention, DeepseekV4SparseMoeBlock: Mlp}

Mlp 

Mlp(*args: Any, **kwargs: Any)

Bases: Moe

DeepSeek-V4's mixture of experts returns routed plus shared experts as a bare tensor, so the base holds.

SCORING property 

SCORING: str

"hash" on a hash_moe block (the token id picks the experts), else the config's scoring_func ("sqrtsoftplus", per expert).

project_on_vocab 

project_on_vocab(model: StandardizedTransformer, hidden: Tensor) -> Tensor

The logit lens as the model makes its logits: hc_head over the streams, the final norm, lm_head.

A tensor with the stream axis, [batch, seq, streams, hidden_size] (rank 4) or one position of it, [streams, hidden_size] (rank 2 with hc_mult rows), is collapsed by the model's hc_head first; the result drops the stream axis, [..., vocab_size]. Any other tensor is a plain [..., hidden_size] stream (a sublayer's output, one stream layer_output[:, :, k]) and goes through norm and lm_head alone.