nemotron_h¶

nemotron_h ¶

Nemotron-H (NemotronHForCausalLM; Nemotron-H, Nemotron-Nano-2, Nemotron-3 Nano, Super, Ultra and Lightning).

Every block holds one norm and one sublayer, the mixer, whose class the block's entry in config.layers_block_type picks::

model.embeddings
model.layers[i]          NemotronHBlock
    .norm                RMSNorm
    .mixer               one of:
                           NemotronHMamba2Mixer   ("linear_attention", an SSD mixer)
                           NemotronHAttention     ("full_attention", no rotary embedding)
                           NemotronHMoE           ("moe": routed experts plus a shared expert)
                           NemotronHMLP           ("mlp")
model.norm_f
lm_head

Each block is hidden + mixer(norm(hidden)). One native name, four meanings, so the standard name is keyed on the mixer's class in RENAME: linear_attn on a Mamba-2 block, self_attn on an attention block, mlp on an MoE or MLP block. The block's norm keeps its name (its output is the sublayer's .input). A block has exactly one of the three, so support reports the other two missing on it, per block, and the contribution identity is input + <the one sublayer's output> == layer_output.

The attention runs the shared eager forward (no rotary embedding), so the base Attention holds; the Mamba-2 mixer is transformers' Mamba-2 code, so the base StateSpace holds; the MoE and the MLP return what the block adds. The model makes its logits in float32 (lm_head(...).float()), so project_on_vocab does too. Sizes: the attention's heads are the config's num_attention_heads / num_key_value_heads / head_dim; the dense MLP is intermediate_size wide, an MoE block's experts moe_intermediate_size.

NemotronH Omni (NemotronH_Omni_Reasoning_V3, model type nemotron_h_omni, transformers 5.18 and later) holds this model whole at language_model (language_model.model.*, language_model.lm_head) beside a RADIO tower at vision_model, vision (a RadioVision), and multi_modal_projector, projector (after the pixel shuffle). The root scatters the image features in its own forward (ROOT_SCATTER), so vision.image_features is read there. RADIO packs every image of an invoke into one row behind its CLS and register tokens, so its blocks' values are Patches with 1 in the images axis and prefix + patches tokens per image; patch_embeddings is the patch projection's output, before those tokens and the position embeddings. A block scales what each sublayer adds (layer_scale1, layer_scale2), so attention_output and mlp_output are the scales' outputs. The attention makes one interface call over the row for one image and one per image for several, so its queries, keys and values are served whole and its scores, pattern and head outputs are Unavailable (RADIO_PER_IMAGE).

MIXER_NAMES `module-attribute` ¶

MIXER_NAMES = {NemotronHMamba2Mixer: 'linear_attn', NemotronHAttention: 'self_attn', NemotronHMoE: 'mlp', NemotronHMLP: 'mlp'}

RENAME `module-attribute` ¶

RENAME = {'model.embeddings': 'embed_tokens', 'model.layers': 'layers', 'model.norm_f': 'norm', **MIXER_NAMES, 'gate': 'router', 'language_model.model.embeddings': 'embed_tokens', 'language_model.model.layers': 'layers', 'language_model.model.norm_f': 'norm', 'language_model.lm_head': 'lm_head', 'vision_model': 'vision', 'multi_modal_projector': 'projector', 'embeddings.patch_projection': 'patch_embed', 'encoder.layer': 'layers', 'attention': 'self_attn', 'norm1': 'input_layernorm', 'norm2': 'post_attention_layernorm', 'activation_fn': 'act_fn', 'fc2': 'down_proj'}

ROOT_SCATTER `module-attribute` ¶

ROOT_SCATTER = 'inputs_embeds_masked_scatter_0'

RADIO_PER_IMAGE `module-attribute` ¶

RADIO_PER_IMAGE = "RADIO's attention makes one interface call over the packed row for one image (attention_interface_1) and one per image for several (attention_interface_3), so no one call holds the block's pattern for every input; read attention_queries and attention_keys, whole, and split them at the attention's cu_seqlens argument (self_attn.inputs[1]['cu_seqlens'], absent for one image)"

ENVOYS `module-attribute` ¶

ENVOYS = {NemotronHBlock: Layer, NemotronHAttention: Attention, NemotronHMamba2Mixer: StateSpace, NemotronHMoE: Moe, NemotronHMLP: Mlp, modeling_radio.RadioModel: RadioVision, modeling_radio.RadioLayer: VisionLayer, modeling_radio.RadioAttention: RadioAttention, modeling_radio.RadioMLP: RadioMlp, modeling_radio.RadioSwiGLUFFN: RadioMlp}

RadioVision ¶

RadioVision(*args: Any, **kwargs: Any)

Bases: Vision

RADIO, NemotronH Omni's tower: packed, every image's CLS and register tokens then its patches, in one row.

hidden_size, num_heads and patch_size are the config's; intermediate_size is hidden_size * mlp_ratio; image_size is Unavailable (the processor sizes each image's grid; image_grid_hw). No final norm: tower_output is the last block's stream.

image_size `class-attribute` `instance-attribute` ¶

image_size = property(variable_resolution)

intermediate_size `property` ¶

intermediate_size: int

RadioAttention ¶

RadioAttention(*args: Any, **kwargs: Any)

Bases: VisionAttention

RADIO's attention: what it adds is layer_scale1's output; the interior is read whole around the calls (RADIO_PER_IMAGE).

off_interface ¶

off_interface() -> str | None

attention_output ¶

attention_output(value: Tensor) -> Tensor

attention_queries ¶

attention_queries(value: Tensor) -> Queries

Every image's queries, [1, num_heads, tokens, head_dim], before any per-image split; assign or edit in place.

attention_keys ¶

attention_keys(value: Tensor) -> Keys

Every image's keys, [1, num_heads, tokens, head_dim], before any per-image split; assign or edit in place.

attention_values ¶

attention_values(value: Tensor) -> Values

Every image's values, [1, num_heads, tokens, head_dim], before any per-image split; assign or edit in place.

RadioMlp ¶

RadioMlp(*args: Any, **kwargs: Any)

Bases: VisionMlp

RADIO's MLP: what it adds is layer_scale2's output.

mlp_output ¶

mlp_output(value: Tensor) -> Tensor

project_on_vocab ¶

project_on_vocab(model, hidden: Tensor) -> Tensor

The logit lens as the model makes its logits: the final norm, lm_head, then float32.