flex_olmo¶
FlexOlmo on vLLM (vllm.model_executor.models.flex_olmo).
Llama's names and OLMo-2's post-norm block around a mixture of experts. The
block is not fused, whatever its signature says: it is called
forward(positions, hidden_states, residual), ignores the residual it is
handed, and returns (hidden_states, None) with the whole stream first.
The None is what tells the model's final norm there is no residual to add.
What the block adds is each sublayer's post-norm's output, as on
transformers, so the contributions point at the sibling norms. The attention
norms its queries and keys over all heads at once before the rotary
embedding. Every block's mlp is the mixture of experts; the routing is
inside vLLM's fused MoE kernel and has no values here.