llama

llama

Llama on vLLM (vllm.model_executor.models.llama).

The names are transformers' (model.layers[i].{input_layernorm, self_attn, post_attention_layernorm, mlp}, model.norm, lm_head). The block fuses each residual add into the norm that follows, so it takes and returns the stream as (hidden_states, residual).

RENAME module-attribute

RENAME = {'model.embed_tokens': 'embed_tokens', 'model.layers': 'layers', 'model.norm': 'norm'}

ENVOYS module-attribute

ENVOYS = {LlamaDecoderLayer: Layer, LlamaAttention: Attention, LlamaMLP: Mlp}

Layer

Layer(*args: Any, **kwargs: Any)

Bases: FusedLayer

Llama's decoder block: forward(positions, hidden_states, residual) -> (hidden_states, residual).