mpt¶
MPT on vLLM (vllm.model_executor.models.mpt).
Its tree is transformers' (transformer.{wte, blocks[i].{norm_1, attn,
norm_2, ffn}, norm_f}); lm_head is the embedding itself, which the
checkpoint ties. The block is called with the positions and the residual
stream and returns the stream, adding both contributions itself; vLLM's
attention and feed-forward modules return their contributions alone, where
transformers' feed-forward adds the residual inside. The attention is
ALiBi's: no rotary embedding, a per-head slope on the scores.
module-attribute
¶
RENAME = {'transformer.wte': 'embed_tokens', 'transformer.blocks': 'layers', 'transformer.norm_f': 'norm', 'norm_1': 'input_layernorm', 'attn': 'self_attn', 'norm_2': 'post_attention_layernorm', 'ffn': 'mlp', 'out_proj': 'o_proj', 'act': 'act_fn'}