cohere2¶
Cohere 2 (Cohere2ForCausalLM, Command-R7B / Command-A).
Cohere's tree and parallel block (x + attn + mlp from one LayerNorm), with
sliding-window and full attention layers interleaved (config.layer_types).
Only the sliding layers apply the rotary; the full layers use no position
encoding. Both kinds make the same attention_interface call, so every
interior value resolves on every block, and on a full layer
attention_queries / attention_keys are the unrotated projections.
The model multiplies the head's output by config.logit_scale, as Cohere's
does, so the family takes Cohere's project_on_vocab.
Aya Vision and Cohere2-Vision put a SigLIP tower at model.vision_tower, which is
vision (its post_layernorm over the patches is vision.norm), and a
pixel-shuffling projector model.multi_modal_projector, which is projector;
vision.image_features is read at the scatter (ImageScatter). Aya Vision's
projector reads the last block's layer_output before vision.norm
(vision_feature_layer -1, strategy "full") and norms after its pixel shuffle;
Cohere2-Vision's reads vision.tower_output and has no norm. Aya Vision 32B's text
config is cohere, so it loads through that family, which binds no tower.
module-attribute
¶
RENAME = {'model.embed_tokens': 'embed_tokens', 'model.layers': 'layers', 'model.norm': 'norm', 'model.language_model.embed_tokens': 'embed_tokens', 'model.language_model.layers': 'layers', 'model.language_model.norm': 'norm', 'model.vision_tower': 'vision', 'model.multi_modal_projector': 'projector', 'embeddings.patch_embedding': 'patch_embed', 'encoder.layers': 'layers', 'post_layernorm': 'norm', 'layer_norm1': 'input_layernorm', 'layer_norm2': 'post_attention_layernorm', 'out_proj': 'o_proj', 'activation_fn': 'act_fn', 'fc2': 'down_proj'}
module-attribute
¶
ENVOYS = {Cohere2DecoderLayer: Layer, Cohere2Attention: Attention, Cohere2MLP: Mlp, SiglipVisionModel: Vision, SiglipEncoderLayer: VisionLayer, SiglipAttention: VisionAttention, SiglipMLP: VisionMlp, AyaVisionModel: ImageScatter, Cohere2VisionModel: ImageScatter}