gemma¶
Gemma on vLLM (vllm.model_executor.models.gemma).
Llama's names and Llama's fused block. Two things are not transformers':
there is no lm_head module (the logits processor unembeds with
embed_tokens' weight, which the checkpoint ties), so project_on_vocab
is defined here; and the sqrt(hidden_size) scaling of the embeddings is
applied by the model after embed_tokens, not inside it as on
transformers, so token_embeddings is the unscaled lookup and
layers[0].layer_input the scaled stream.
module-attribute
¶
module-attribute
¶
ENVOYS = {GemmaDecoderLayer: Layer, GemmaAttention: Attention, GemmaMLP: Mlp}
Bases: FusedLayer
The decoder block: forward(positions, hidden_states, residual) -> (hidden_states, residual).
project_on_vocab(model: StandardizedVLLM, hidden: Tensor) -> Tensor
The final norm, then vLLM's logits processor over embed_tokens, which is the unembedding here.