mimo_v2_flash¶
MiMo-V2-Flash on vLLM (vllm.model_executor.models.mimo_v2).
Llama's names and Llama's fused block: it takes and returns the stream as
(hidden_states, residual). vLLM reads the configuration under the names
Xiaomi's released config.json gives it (hybrid_layer_pattern,
moe_layer_freq, swa_*, layernorm_epsilon), beside the ones
transformers reads; both describe the same model.
The attention is transformers' (see the transformers family): queries and
keys head_dim wide, values and head outputs v_head_dim wide, the
values scaled by attention_value_scale before the attention layer, and on
a sliding-window block twice the key/value heads and a learned sink logit per
head that joins the softmax as one extra key and is dropped after it. The
values and head outputs are split into heads v_head_dim wide here, and
the recomputed pattern is the softmax with the block's sink column, dropped,
so a sliding block's rows sum to less than one, as on transformers; the
scores are the masked scores before the sink joins. mlp is a dense MLP or
a mixture of experts (moe_layer_freq), whose output is what the block
adds; the routing is inside vLLM's fused MoE kernel and has no values here.
module-attribute
¶
module-attribute
¶
ENVOYS = {MiMoV2FlashDecoderLayer: Layer, MiMoV2Attention: Attention, MiMoV2MLP: Mlp, MiMoV2MoE: Mlp}
ValueHeads(*args: Any, heads: str | None = None, batch: bool | Callable[[Envoy], bool] = False, factor: Callable[[Envoy], float] | None = None, **kwargs: Any)
Bases: Flat
A Flat over the values or the head outputs: split into heads by the attention layer's value width, head_size_v.
Bases: FusedLayer
The decoder block: forward(positions, hidden_states, residual) -> (hidden_states, residual).