Skip to content

kimi_k25

kimi_k25

Kimi K2.5 (model_type kimi_k25): the multimodal wrapper around Kimi K2's text model.

Its text_config is a DeepseekV3Config whose model_type says kimi_k2 in the checkpoint's files and deepseek_v3 where transformers rewrites it at load, so the wrapper's own type names the family (nnterp.families.resolve): Kimi K2's, which is DeepSeek-V3's tree, envoys and sizes (kimi_k2.py).

ENVOYS module-attribute

ENVOYS = {DeepseekV3DecoderLayer: Layer, DeepseekV3Attention: Attention, DeepseekV3MLP: Mlp, DeepseekV3MoE: Moe}

RENAME module-attribute

RENAME = {'model.embed_tokens': 'embed_tokens', 'model.layers': 'layers', 'model.norm': 'norm', 'model.language_model.embed_tokens': 'embed_tokens', 'model.language_model.layers': 'layers', 'model.language_model.norm': 'norm', 'gate': 'router'}

Attention

Bases: Attention

DeepSeek-V3's latent attention: V2's, rotating its rope parts with apply_rotary_pos_emb or, where the config sets rope_interleave, its interleaved twin.

rope_call

rope_call() -> str

Layer

Bases: Layer

DeepSeek-V3's decoder block; returns a bare tensor, so the base holds.

Mlp

Bases: Mlp

DeepSeek's dense MLP or mixture of experts; both return the hidden states, and the residual is added in the block.

Moe

Bases: Moe, Mlp

DeepSeek-V3's mixture of experts: a sigmoid router with a selection bias and group-limited top-k, routed experts and shared experts.

SCORING class-attribute instance-attribute

SCORING = 'sigmoid'

head_dim

head_dim(model: StandardizedTransformer) -> int

Width of one head's values and outputs: v_head_dim (the config's head_dim is set to qk_rope_head_dim, the rotary slice's width, which no served value has).

qk_head_dim

qk_head_dim(model: StandardizedTransformer) -> int

Width of one head's queries and keys: the non-rotary part plus the rotary part.