hyperclovax¶
HyperCLOVA X (HyperCLOVAXForCausalLM, NAVER's HyperCLOVA X SEED Think).
Llama's tree with a sandwich block and Granite's µP multipliers::
h = x + post_norm1(self_attn(input_layernorm(x))) * residual_multiplier
out = h + post_norm2(mlp(post_attention_layernorm(h))) * residual_multiplier
post_norm1 / post_norm2 are RMS norms under use_post_norm and
identities otherwise; post_attention_layernorm is the pre-MLP norm (the Llama
meaning). What the block adds is each post-norm's output times the multiplier, so
attention_output and mlp_output are that product: a computed copy, divided
by the multiplier on assignment, with a transform carrying an in-place edit back
into the post-norm's output, as on Granite; both modules keep the config, where
the multiplier is read. attention_multiplier is the
softmax scale passed to the shared interface; embedding_multiplier scales the
embedding module's output before the first block (token_embeddings times it is
layers[0].input); logits_scaling multiplies the head's output (Granite
divides), so the family's project_on_vocab does the same.
HyperCLOVA X Vision V2 (HyperCLOVAXVisionV2ForConditionalGeneration, model
type hyperclovax_vision_v2, needs transformers 5.18 or later) wraps this text model
at model.language_model. Its tower is the Qwen2.5-VL ViT at model.vision_model,
vision (a QwenVision: packed, windowed, its own merger inside), and a linear
model.projector after the tower, projector, maps the merger's output onto
the text width; vision.image_features is read at the scatter in
HyperCLOVAXVisionV2Model's forward (ImageScatter).
module-attribute
¶
RENAME = {'model.embed_tokens': 'embed_tokens', 'model.layers': 'layers', 'model.norm': 'norm', 'model.language_model.embed_tokens': 'embed_tokens', 'model.language_model.layers': 'layers', 'model.language_model.norm': 'norm', 'model.vision_model': 'vision', 'model.projector': 'projector', 'blocks': 'layers', 'attn': 'self_attn', 'norm1': 'input_layernorm', 'norm2': 'post_attention_layernorm', 'attn.proj': 'attn.o_proj'}
module-attribute
¶
ENVOYS = {HyperCLOVAXDecoderLayer: Layer, HyperCLOVAXAttention: Attention, HyperCLOVAXMLP: Mlp, Qwen2_5_VisionTransformerPretrainedModel: QwenVision, Qwen2_5_VLVisionBlock: VisionLayer, Qwen2_5_VLVisionAttention: QwenVisionAttention, Qwen2_5_VLMLP: VisionMlp}
project_on_vocab(model: StandardizedTransformer, hidden: Tensor) -> Tensor
The logit lens as the model makes its logits: the final norm, lm_head, then times logits_scaling.