gptj

gptj

GPT-J (GPTJForCausalLM).

transformer.{wte, drop, h[i].{ln_1, attn, mlp}, ln_f} and lm_head. A parallel block: ln_1 feeds both sublayers and x + attn + mlp is summed in the block, which returns a tuple. The attention does its own arithmetic in _attn rather than the shared eager forward, so the pattern and the rest of the interior are read around that method call.

RENAME module-attribute

RENAME = {'transformer.wte': 'embed_tokens', 'transformer.h': 'layers', 'transformer.ln_f': 'norm', 'ln_1': 'input_layernorm', 'attn': 'self_attn', 'out_proj': 'o_proj', 'act': 'act_fn', 'fc_out': 'down_proj'}

ENVOYS module-attribute

ENVOYS = {GPTJBlock: Layer, GPTJAttention: Attention, GPTJMLP: Mlp}

intermediate_size

intermediate_size(model: StandardizedTransformer) -> int

The MLP width is n_inner, None meaning four times the hidden size.