llama4_text¶
Llama 4, text (Llama4ForCausalLM) on vLLM (vllm.model_executor.models.llama4).
Llama's names and Llama's fused block: it takes and returns the stream as
(hidden_states, residual). The feed-forward is feed_forward, aliased to
mlp as on transformers: a dense MLP on some blocks, on the others
(every interleave_moe_layer_step-th) a mixture of experts whose output,
the shared expert's plus the routed experts', is what the block adds. The
routing is inside vLLM's fused MoE kernel and has no values here. The shared
expert (shared_expert, aliased to shared_experts) is the dense MLP's
class, so it is an Mlp too; its mlp_output is unavailable.
The attention is iRoPE's, as on transformers: a NoPE block
(no_rope_layers[i] == 0) has no rotary embedding and no query and key
norm and attends over the whole prefix; the others rotate, then L2-norm their
queries and keys, then attend within chunks of attention_chunk_size
tokens (vLLM's ChunkedLocalAttention), so the recomputed scores mask the
keys of other chunks too. The queries and keys served are the ones the
attention layer receives, after all of that, with one change: vLLM permutes
each head's query and key channels at load (q_proj and k_proj rows,
so that its rotary embedding can rotate halves where transformers rotates
adjacent pairs), and the values put them back in transformers' order, and an
edit back in vLLM's.
One difference from transformers is in vLLM's model, not here: vLLM applies
the NoPE blocks' attention temperature tuning only when the generation
config says attn_temperature_tuning or max_model_len exceeds 32K,
whatever the model config says (pass
override_generation_config={"attn_temperature_tuning": True} to match
transformers). The tuning leaves the queries of the first 8191 positions
unscaled either way.
module-attribute
¶
RENAME = {'model.embed_tokens': 'embed_tokens', 'model.layers': 'layers', 'model.norm': 'norm', 'feed_forward': 'mlp', 'shared_expert': 'shared_experts'}
module-attribute
¶
ENVOYS = {Llama4DecoderLayer: Layer, Llama4Attention: Attention, LlamaMLP: Mlp, Llama4MoE: Mlp}
Interleaved(*args: Any, heads: str | None = None, batch: bool | Callable[[Envoy], bool] = False, factor: Callable[[Envoy], float] | None = None, **kwargs: Any)
Bases: Flat
A Flat over queries or keys whose channels vLLM permuted at load: served in transformers' order, handed back in vLLM's.
Bases: FusedLayer
The decoder block: forward(positions, hidden_states, residual) -> (hidden_states, residual).
The base's recomputed scores, with the keys of another chunk at -inf on a chunked block.
A head's channels from vLLM's load-time order (the even channels, then the odd ones) back to transformers' order.
pairs_first's inverse: transformers' order of a head's channels to vLLM's.