kimi_k25¶
kimi_k25
¶
Kimi K2.5 (model_type kimi_k25): the multimodal wrapper around Kimi K2's text model.
Its text_config is a DeepseekV3Config whose model_type says kimi_k2
in the checkpoint's files and deepseek_v3 where transformers rewrites it at
load, so the wrapper's own type names the family (nnterp.families.resolve): Kimi
K2's, which is DeepSeek-V3's tree, envoys and sizes (kimi_k2.py).
ENVOYS
module-attribute
¶
ENVOYS = {DeepseekV3DecoderLayer: Layer, DeepseekV3Attention: Attention, DeepseekV3MLP: Mlp, DeepseekV3MoE: Moe}
RENAME
module-attribute
¶
RENAME = {'model.embed_tokens': 'embed_tokens', 'model.layers': 'layers', 'model.norm': 'norm', 'model.language_model.embed_tokens': 'embed_tokens', 'model.language_model.layers': 'layers', 'model.language_model.norm': 'norm', 'gate': 'router'}
Attention
¶
Mlp
¶
Bases: Mlp
DeepSeek's dense MLP or mixture of experts; both return the hidden states, and the residual is added in the block.
Moe
¶
head_dim
¶
head_dim(model: StandardizedTransformer) -> int
Width of one head's values and outputs: v_head_dim (the config's head_dim is set to qk_rope_head_dim, the rotary slice's width, which no served value has).
qk_head_dim
¶
qk_head_dim(model: StandardizedTransformer) -> int
Width of one head's queries and keys: the non-rotary part plus the rotary part.