inif.flat

Flat parallel-array view of an InifDocument for high-throughput scans.

FlatTokenStore is an analysis representation, not a replacement file format. It exposes flat parallel arrays of token ids, strings, sequence provenance, and sparse annotations — useful when you need to scan tokens across samples without reconstructing Pydantic objects per token.

Store

FlatTokenStore

Flat token arrays for high-throughput scans over INIF documents.

This is an internal analysis representation, not a replacement file format. Tokens are addressed by global position; sample_offsets maps sample-local ranges back to sample_ids. Annotations are sparse: each annotation name stores the global positions carrying it.

Source

class FlatTokenStore(BaseModel)

Methods

from_document

Build a flat view from doc.

With expand_sequences=True (default), sequence refs are expanded into their constituent vocab tokens. The sample-flat sequence_ids array still records which shared run each materialised position came from, even though the expanded :class:TokenOrSeqRef objects themselves don’t carry that provenance.

With expand_sequences=False, ref positions land as token_ids[i] is None and token_texts[i] carries the target :class:Sequence id (mirroring how refs are stored on disk).

Source

@classmethod
def from_document(
    cls,
    doc: InifDocument,
    expand_sequences: bool = True,
) -> FlatTokenStore

doc InifDocument

expand_sequences bool

find_regexes

Return global token positions matching each regex tag spec.

Source

def find_regexes(
    self,
    regex_tags: list[RegexAnnotation],
) -> dict[str, list[int]]

regex_tags list[RegexAnnotation]

annotate_regexes

Apply regex annotations over flat arrays without constructing Tokens.

Source

def annotate_regexes(self, regex_tags: list[RegexAnnotation]) -> None

regex_tags list[RegexAnnotation]

to_document

Materialize a flat analysis document.

The result keeps sample ids, token ids/text, and sparse annotations. Sequence-ref provenance lives only on the flat store; the emitted :class:TokenOrSeqRef objects are plain vocab tokens (or refs when token_ids[pos] is None). It intentionally does not recreate shared :class:Sequence objects.

Source

def to_document(self, metadata: Metadata | None = None) -> InifDocument

metadata Metadata | None