granite

granite

Granite on vLLM (vllm.model_executor.models.granite).

Llama's names and a plain block: it is called with the positions and the residual stream and returns the stream. Granite's four scalars are where they are on transformers, and the values carry the same meanings:

• residual_multiplier: the block adds each sublayer's output times it, so attention_output and mlp_output are the module's output times the multiplier, the term that reaches the stream. That product is a computed copy (a Flat with a factor): an assignment or an in-place edit is divided back into the module's output, and a read that edits nothing hands the output back untouched.
• embedding_multiplier: the model multiplies the embedding module's output by it (in place) before the first block, so token_embeddings is the lookup before the multiplier, as on transformers, and layers[0].layer_input the multiplied stream.
• attention_multiplier is the attention layer's scale, which the recomputed scores use.
• logits_scaling: vLLM's logits processor divides by it (its scale), so the default project_on_vocab through that processor already does.

RENAME module-attribute

RENAME = {'model.embed_tokens': 'embed_tokens', 'model.layers': 'layers', 'model.norm': 'norm'}

ENVOYS module-attribute

ENVOYS = {GraniteDecoderLayer: Layer, GraniteAttention: Attention, GraniteMLP: Mlp}

residual_multiplier

residual_multiplier(envoy: Envoy) -> float

What the block multiplies each sublayer's output by before adding it: the parent block's own scalar.