phi

phi

Phi 1 / 1.5 / 2 (PhiForCausalLM).

Llama's containers with a parallel block: one input_layernorm feeds both sublayers and x + attn + mlp is summed at the end (resid_dropout on each, inert in eval). The final norm is final_layernorm. There is no post_attention_layernorm.

RENAME module-attribute

RENAME = {'model.embed_tokens': 'embed_tokens', 'model.layers': 'layers', 'model.final_layernorm': 'norm', 'dense': 'o_proj', 'activation_fn': 'act_fn', 'fc2': 'down_proj'}

ENVOYS module-attribute

ENVOYS = {PhiDecoderLayer: Layer, PhiAttention: Attention, PhiMLP: Mlp}