deepseek_v2¶
DeepSeek-V2 (DeepseekV2ForCausalLM).
Llama's tree and Llama's block: model.{embed_tokens, layers[i].{input_layernorm,
self_attn, post_attention_layernorm, mlp}, norm} and lm_head, the residual
added in the block, attention through the shared eager forward. Multi-head latent attention: queries and keys are qk_head_dim wide (qk_nope_head_dim + qk_rope_head_dim), values v_head_dim, and the interface sees num_heads key/value heads whatever num_key_value_heads says. The first first_k_dense_replace blocks have a dense MLP, the rest a mixture of experts.
module-attribute
¶
RENAME = {'model.embed_tokens': 'embed_tokens', 'model.layers': 'layers', 'model.norm': 'norm', 'gate': 'router'}
module-attribute
¶
ENVOYS = {DeepseekV2DecoderLayer: Layer, DeepseekV2Attention: Attention, DeepseekV2MLP: Mlp, DeepseekV2Moe: Moe}
head_dim(model: StandardizedTransformer) -> int
Width of one head's values and outputs: v_head_dim (the config's head_dim is set to qk_rope_head_dim, the rotary slice's width, which no served value has).
qk_head_dim(model: StandardizedTransformer) -> int
Width of one head's queries and keys: the non-rotary part plus the rotary part.