opt¶
OPT on vLLM (vllm.model_executor.models.opt).
Its tree is transformers' (model.decoder.{embed_tokens, embed_positions,
layers[i].{self_attn_layer_norm, self_attn, final_layer_norm, fc1, fc2},
final_layer_norm}). The block is plain: called with the residual stream
alone, it returns the stream. As on transformers there is no MLP module:
fc1 and fc2 sit on the block, so mlp and mlp_output do not
exist here either, and what the feed-forward path adds is fc2's output
(vLLM's linear layers return (output, bias), the bias already added: read
layers[i].fc2.output[0]).
transformers scales the queries by head_dim**-0.5 at q_proj's output;
vLLM leaves them unscaled and its attention layer applies the scale. So
attention_queries is served times that scale, the pre-scaled queries
transformers serves, as a computed copy divided back on a write, and the
recomputed scores are queries @ keys.T with no further scale.
When the checkpoint ties its embeddings (OPT-125m and most of the family),
vLLM's lm_head is embed_tokens: the same module under both names.
On a checkpoint with do_layer_norm_before false (OPT-350m) the block
norms after each add, and the stream is not layer_input plus the two
contributions.
module-attribute
¶
RENAME = {'model.decoder.embed_tokens': 'embed_tokens', 'model.decoder.layers': 'layers', 'model.decoder.final_layer_norm': 'norm', 'self_attn_layer_norm': 'input_layernorm', 'out_proj': 'o_proj'}
head_dim**-0.5, which transformers applies to the queries and vLLM's attention layer to the scores.