cohere¶
Cohere (CohereForCausalLM, Command-R).
model.{embed_tokens, layers[i].{input_layernorm, self_attn, mlp}, norm} and
lm_head: Llama's names. The block is parallel: one LayerNorm (not RMS)
feeds both sublayers and the block returns x + attn + mlp, so the
contributions are the two modules' outputs and the base holds. There is no
post_attention_layernorm. The attention runs the shared interface, after
an optional per-head q_norm / k_norm (use_qk_norm) and an
interleaved rotary.
The model multiplies the head's output by config.logit_scale to make the
logits, so the family defines project_on_vocab with that step, and the
logit lens on the last block equals logits.
Aya Vision 32B (aya_vision with a cohere text config) loads through this
family. It binds no tower keys, so that load has no vision or projector.
module-attribute
¶
module-attribute
¶
project_on_vocab(model: StandardizedTransformer, hidden: Tensor) -> Tensor
The logit lens as the model makes its logits: the final norm, lm_head, then times logit_scale.