inif.models

Core Pydantic models — InifDocument, Sample, TokenOrSeqRef, Sequence, Text, and friends.

The classes here define the on-disk shape of an INIF document. They are re-exported from the top-level inif package, so most code does from inif import Sample, TokenOrSeqRef, ... rather than reaching into this module directly.

The public verbs live on InifDocument and Sample as methods:

IO and dict round-trips — InifDocument.save, InifDocument.load (classmethod), InifDocument.to_dict, InifDocument.from_dict (classmethod).
Sub-document construction — InifDocument.subset, InifDocument.filter_samples, InifDocument.filter_samples_by_score, InifDocument.get_sample.
Sequence (de)duplication — InifDocument.deduplicate_sequences(min_length=5), InifDocument.expand_sequences().
Tagging — Sample.tag_by_regex, Sample.tag_by_regexes, Sample.tag_by_text_regex, Sample.tag_by_predicate, Sample.tag_by_predicates, Sample.tag_chat_roles, Sample.tag_special_tokens, Sample.create_span_from_tag, Sample.annotate, Sample.annotate_positions, Sample.annotation_positions, Sample.remove_annotation. The same tag methods are also available on InifDocument for whole-document fan-out.
Selection — Sample.select_by_position, Sample.select_by_annotation, Sample.select_by_sequence_id, Sample.select_by_span.
Sequence handling — Sample.materialize_position, Sample.get_expanded_tokens, Sample.get_tokens_by_positions, TokenOrSeqRef.expanded_tokens.
Token extras — TokenOrSeqRef.set_extra, .get_extra, .has_extra, .pop_extra, .extras.
Viewer — InifDocument.render_html, InifDocument.save_html, InifDocument.show.

Documents

InifDocument

The top-level container — metadata, deduplicated sequences, samples.

total_samples is a computed property (len(samples)); there is no stored field for it. Use :meth:subset to derive a self-contained sub-document with sequences pruned to those referenced by the kept samples.

class InifDocument(BaseModel)

Methods

subset

Return a new InifDocument with only samples matching predicate.

Sequences not referenced by any retained sample are dropped, so the output stays self-contained and minimal. Metadata is copied as-is.

The returned document is independent from this one: samples, sequences, metadata, tokens, and nested extras are deep-copied.

def subset(self, predicate: Callable[[Sample], bool]) -> InifDocument

predicate Callable[[Sample], bool]

to_dict

Convert this document to a JSON-ready dict.

Uses pydantic’s mode="json" so datetimes serialize as ISO-8601 strings. With compact=True (default), default-valued and None fields are stripped — for example, sequence-ref tokens (id is None) serialize to a single-key {"token": "<seq_id>"} dict.

def to_dict(self, compact: bool = True) -> dict

compact bool

save

Save this document to path.

.inif paths are written as indexed compressed archives; .inif.json / .json paths are written as plain JSON. indent controls pretty-printing for plain JSON (None produces a single-line dump).

def save(
    self,
    path: str | Path,
    compress: bool | None = None,
    compact: bool = True,
    indent: int | None = 4,
) -> None

path str | Path
compress bool | None
compact bool
indent int | None

from_dict

Build an :class:InifDocument from a JSON-ready dict.

@classmethod
def from_dict(cls, data: dict) -> InifDocument

data dict

load

Load an :class:InifDocument from path.

.inif paths are read as indexed archives; .inif.json / .json paths are read as plain JSON.

@classmethod
def load(cls, path: str | Path, compress: bool | None = None) -> InifDocument

path str | Path
compress bool | None

deduplicate_sequences

Find token runs common to ALL samples and replace them with refs.

Returns a new document; this one is not modified. Common runs of length min_length or more are extracted into :class:Sequence objects and the tokens carrying them in each sample are swapped for a sequence-ref :class:TokenOrSeqRef.

def deduplicate_sequences(self, min_length: int = 5) -> InifDocument

min_length int

expand_sequences

Expand all sequence references back to flat vocab tokens.

Returns a new document whose tokens are independent from this one and whose sequences list is empty. Re-running :meth:deduplicate_sequences rediscovers the same shared runs.

def expand_sequences(self) -> InifDocument

filter_samples_by_score

Return samples whose scorer value satisfies predicate.

Compose with the per-sample selection methods (e.g. :meth:Sample.select_by_annotation) on each returned sample to drill down to specific tokens.

def filter_samples_by_score(
    self,
    scorer: str,
    predicate: Callable[[str | int | float | bool | list | dict], bool],
) -> list[Sample]

scorer str
predicate Callable[[str | int | float | bool | list | dict], bool]

tag_by_regex

Tag every token across all samples whose string matches pattern.

def tag_by_regex(self, pattern: str | re.Pattern[str], tag: str) -> None

pattern str | re.Pattern[str]
tag str

tag_by_regexes

Apply multiple regex taggers across all samples in one pass per sample.

def tag_by_regexes(
    self,
    regex_tags: list[tuple[str | re.Pattern[str], str]],
) -> None

regex_tags list[tuple[str | re.Pattern[str], str]]

tag_by_text_regex

Apply text-based regex tagging across all samples.

def tag_by_text_regex(
    self,
    pattern: str,
    tag: str,
    mode: "TextTagMode | str" = "all",
) -> None

pattern str
tag str
mode 'TextTagMode | str'

tag_by_predicates

Apply multiple Python predicate taggers across all samples.

def tag_by_predicates(self, predicate_tags: list["PredicateTag"]) -> None

predicate_tags list['PredicateTag']

tag_chat_roles

Tag chat roles for every sample in this document.

messages_per_sample must be a list with one message list per sample, in the same order as self.samples.

def tag_chat_roles(
    self,
    messages_per_sample: list[list[dict[str, str]]],
    tokenizer: Any,
) -> None

messages_per_sample list[list[dict[str, str]]]
tokenizer Any

remove_annotation

Remove every annotation named name across all samples.

def remove_annotation(self, name: str) -> None

name str

render_html

Render this document as a self-contained HTML string.

When tokenizer is provided, the tokenizer’s byte-level representation of newlines (e.g. Ċ for GPT-2 family) is detected automatically so that visual line breaks are inserted after newline tokens.

def render_html(
    self,
    compact: bool = False,
    title: str | None = None,
    tokenizer: Any = None,
) -> str

compact bool
title str | None
tokenizer Any

show

Display this document as HTML in a Jupyter notebook.

def show(
    self,
    compact: bool = False,
    title: str | None = None,
    tokenizer: Any = None,
) -> Any

compact bool
title str | None
tokenizer Any

save_html

Save this document as a self-contained HTML file.

When title is not given, the source filename is used if available, otherwise falls back to the model name.

def save_html(
    self,
    path: str | Path,
    compact: bool = False,
    title: str | None = None,
    source: str | Path | None = None,
    tokenizer: Any = None,
) -> None

path str | Path
compact bool
title str | None
source str | Path | None
tokenizer Any

Samples

Sample

One self-contained tokenized generation trace.

The required field is id; everything else defaults to an empty list or None. The first-class fields target, references, choices, interaction_type, error, and sample_hash are aligned with the every_eval_ever schema so filters and viewers can rely on them without reaching into metadata.

Construction validates that every span.positions index and every annotation.ranges window lies within len(tokens); out-of-range values raise ValidationError.

class Sample(BaseModel)

Methods

annotate

Add ranges to the annotation called name.

Each annotation name maps to exactly one entry on Sample.annotations. If the entry already exists, the new ranges are appended and _merge_ranges collapses any overlapping or adjacent intervals into single half-open spans (so adding [6, 10) to existing [5, 7) yields a single [5, 10)). The existing entry’s metadata is preserved — passing a different metadata on a subsequent call to the same name is a silent no-op for the metadata field; only the ranges are merged in.

def annotate(
    self,
    name: str,
    ranges: Iterable[tuple[int, int]],
    metadata: dict | None = None,
) -> TokenAnnotation

name str
ranges Iterable[tuple[int, int]]
metadata dict | None

select_by_position

Select tokens at the given position(s).

Accepts an int, list of int, or slice. Returns a :class:~inif.selectors.TokenSelection with the matching tokens and their positions in this sample.

def select_by_position(
    self, positions: int | list[int] | slice
) -> "TokenSelection"

positions int | list[int] | slice

select_by_annotation

Select tokens covered by annotation_name on this sample.

def select_by_annotation(self, annotation_name: str) -> "TokenSelection"

annotation_name str

select_by_sequence_id

Select sequence-ref tokens that point at seq_id.

A sequence ref is identified by id is None and carries the target :class:Sequence id in its token field. Useful for finding where a shared run is referenced in a sample without expanding it.

def select_by_sequence_id(self, seq_id: str) -> "TokenSelection"

seq_id str

select_by_span

Select tokens whose positions fall inside the named span.

def select_by_span(self, span_name: str) -> "TokenSelection"

span_name str

tag_by_regex

Tag every token whose string matches pattern.

When sequences is provided, the search runs over the expanded view of the sample so tokens currently compressed inside a sequence ref are inspected too. Matches inside a ref cause the containing ref to be materialized in this sample (per-token information attaches to real Tokens; other refs and other samples are untouched).

def tag_by_regex(
    self,
    pattern: str | re.Pattern[str],
    tag: str,
    sequences: list[Sequence] | None = None,
) -> None

pattern str | re.Pattern[str]
tag str
sequences list[Sequence] | None

tag_by_regexes

Apply multiple regex taggers in one token pass.

The preferred entry point when several regex strategies are known up front. If sequences is provided, sequence refs are materialized only when at least one expanded token actually matches.

def tag_by_regexes(
    self,
    regex_tags: list[tuple[str | re.Pattern[str], str]],
    sequences: list[Sequence] | None = None,
) -> None

regex_tags list[tuple[str | re.Pattern[str], str]]
sequences list[Sequence] | None

tag_by_text_regex

Tag tokens whose concatenated text matches a regex.

See :func:inif.tagging._tag_by_text_regex for the matching algorithm and the meaning of mode.

def tag_by_text_regex(
    self,
    pattern: str,
    tag: str,
    mode: "TextTagMode | str" = "all",
) -> None

pattern str
tag str
mode 'TextTagMode | str'

tag_by_predicate

Tag tokens that satisfy predicate.

def tag_by_predicate(
    self,
    predicate: Callable[["TokenOrSeqRef"], bool],
    tag: str,
) -> None

predicate Callable[['TokenOrSeqRef'], bool]
tag str

tag_by_predicates

Apply multiple Python predicate taggers in one token pass.

def tag_by_predicates(
    self,
    predicate_tags: list["PredicateTag"],
    sequences: list[Sequence] | None = None,
) -> None

predicate_tags list['PredicateTag']
sequences list[Sequence] | None

tag_chat_roles

Tag tokens with their chat-template role.

Works with any HuggingFace chat template. Content tokens get the role of their enclosing message; everything else (delimiters, role names, auto-generated text) is tagged "template". Must be called AFTER :meth:InifDocument.deduplicate_sequences if the document was deduplicated.

def tag_chat_roles(
    self,
    messages: list[dict[str, str]],
    tokenizer: Any,
    sequences: list[Sequence] | None = None,
) -> None

messages list[dict[str, str]]
tokenizer Any
sequences list[Sequence] | None

tag_special_tokens

Tag tokens whose id appears in tokenizer.all_special_ids.

def tag_special_tokens(self, tokenizer: Any, tag: str = "special") -> None

tokenizer Any
tag str

create_span_from_tag

Build a :class:Span whose positions are everywhere tag is set.

The new span is appended to self.spans and returned. Existing spans are left untouched.

def create_span_from_tag(self, tag: str, span_name: str) -> Span

tag str
span_name str

materialize_position

Ensure expanded_position is a real vocab token in self.tokens.

If the position falls inside a sequence ref, the entire ref is expanded in place into its constituent tokens (with their original ids preserved). Other refs are left untouched. This is the primitive that lets callers attach interpretability outputs (logit lens scores, probe activations, etc.) to specific positions even when those positions are currently compressed inside a shared sequence.

Returns (actual_index_in_self_tokens, the_token).

def materialize_position(
    self,
    expanded_position: int,
    sequences: list[Sequence],
) -> tuple[int, TokenOrSeqRef]

expanded_position int
sequences list[Sequence]

SampleScore

One scorer’s evaluation result for a sample.

value is whatever the scorer reports — bool for pass/fail, float for graded scorers, dict / list for structured outputs. answer is the extracted answer string (when applicable).

class SampleScore(BaseModel)

Tokens and text

TokenOrSeqRef

A token entry — either a vocabulary token or a reference to a Sequence.

A vocabulary token has an integer id (the model’s vocab id) and a token string (the decoded piece). A sequence reference has id is None and uses the token field to carry the target :class:Sequence id (a string). The presence / absence of id is the discriminator — there is no separate sequence_id field.

Tokens accept arbitrary extra fields via model_config["extra"] = "allow". Use the :meth:set_extra / :meth:get_extra helpers (not direct attribute assignment) so values stay in sync between __dict__ and model_extra.

class TokenOrSeqRef(BaseModel)

Attributes

sequence_id str | None

Convenience accessor returning the target Sequence id for refs.

Returns the value of self.token when this is a sequence ref (id is None); None for vocabulary tokens. Provided so the "is this a ref to seq X?" check reads naturally without callers having to remember the id is None invariant.

Methods

set_extra

Set an extra field, keeping attribute access and serialization in sync.

def set_extra(self, key: str, value: Any) -> None

key str
value Any

expanded_tokens

Materialize this entry into vocab tokens (no-op for vocab tokens).

For a sequence ref, self.token names the target sequence; the method looks it up in sequences and returns fresh vocab tokens with the original ids preserved.

def expanded_tokens(self, sequences: list[Sequence]) -> list[TokenOrSeqRef]

sequences list[Sequence]

Sequence

A token run shared across every sample in a document.

Both the token strings and their vocabulary ids are stored, so a deduplicate_sequences → expand_sequences round-trip preserves the exact ids. n_tokens must equal len(tokens) (enforced by a validator).

class Sequence(BaseModel)

Text

A named text segment on a :class:Sample.

A sample’s texts list carries one entry per source segment — for chat inputs that’s one per message (including the system prompt); for plain-text inputs it’s one per input string. The default naming scheme is role-based for chat ("system_0", "user_0", "assistant_0", "user_1", …) and index-based for plain text ("text_0", "text_1", …).

start and end are half-open token offsets covering the tokens that render this text in the chat-template output ([start, end)): for chat messages this captures the content plus any chat-template delimiters assigned to that message; for plain text it covers the whole token stream. They are None when the converter cannot map the text to a token range (e.g. lossy decode round-trip).

children lets a message carry sub-segments — used for assistant turns whose body splits into reasoning / content / tool-calls (each becomes one child :class:Text with its own value, start / end, and metadata). When children are present the parent’s own value is typically empty and the renderer iterates the children. Each child’s range must sit inside the parent’s range.

metadata is a free-form dict for caller-supplied context (e.g. turn index, tool-call payload — the tool-call children carry {"id", "type", "function"} here).

class Text(BaseModel)

TokenExtras

Conventional extra fields recognized by inif tooling.

Tokens accept arbitrary extras (model_config["extra"] = "allow"); this model documents the well-known names so external validators, viewers, and converters know what to expect. Not used at runtime — it only contributes to the JSON schema as a $defs entry.

Repeated token labels such as chat roles, "generated", and "reasoning" are stored in :class:TokenAnnotation ranges on :class:Sample, not repeated on every token.

class TokenExtras(BaseModel)

Attributes

logprob float | None: Per-token log-probability from the model
logit_lens dict | None: Per-layer logit-lens output keyed by layer name

Annotations and spans

TokenAnnotation

A named region on a sample expressed as half-open token ranges.

ranges are [start, end); start < end is enforced. Use the sample-level helpers (:meth:Sample.annotate, :meth:Sample.annotate_positions) to add annotations — they merge adjacent / overlapping ranges with identical metadata into a single record.

class TokenAnnotation(BaseModel)

Span

An ad-hoc named position list on a sample.

Unlike :class:TokenAnnotation, spans never auto-merge — adding a second span with the same name produces two Span objects on the sample. Use spans for free-form bookmarks and answer locations where the named-range model doesn’t fit.

class Span(BaseModel)

Metadata

Metadata

Document-level metadata: model, source eval, packages, timestamps.

Required on every :class:InifDocument. created_at is a real datetime in memory; pydantic auto-parses ISO strings on load and re-emits ISO on save.

class Metadata(BaseModel)

ModelInfo

Identifying info for the model whose tokens populate this document.

name is the only required field. revision and huggingface_id aid reproducibility; generation_config and loading_config are free-form bags of inference / loading parameters as recorded by the upstream framework.

class ModelInfo(BaseModel)

SourceEval

Provenance metadata for a document built from an evaluation framework.

Set by the Inspect AI / evaleval converters, absent for ad-hoc text inputs.

class SourceEval(BaseModel)