starcoder2

starcoder2

StarCoder2 (Starcoder2ForCausalLM).

Llama's tree and Llama's block: model.{embed_tokens, layers[i].{input_layernorm, self_attn, post_attention_layernorm, mlp}, norm} and lm_head, the residual added in the block, attention through the shared eager forward. The norms are nn.LayerNorm; the MLP is c_fc/c_proj with no gate; the attention and MLP outputs pass a residual dropout inside their module, the identity in eval.

RENAME module-attribute

RENAME = {'model.embed_tokens': 'embed_tokens', 'model.layers': 'layers', 'model.norm': 'norm', 'act': 'act_fn', 'c_proj': 'down_proj'}

ENVOYS module-attribute

ENVOYS = {Starcoder2DecoderLayer: Layer, Starcoder2Attention: Attention, Starcoder2MLP: Mlp}