inif.converters.text

Build INIF documents from raw text strings or text files on disk.

The fastest path from “I have some strings” to “I have an InifDocument”. Handles plain strings and chat-style message lists; chat detection is per-element. See the text converter guide for options and conventions.

Entry points

from_texts

Converts one or more inputs into Sample objects in an InifDocument.

Source

def from_texts(
    texts: list[str] | list[list[dict[str, str]]],
    tokenizer: Any,
    sample_ids: list[str] | None = None,
    min_sequence_length: int = 5,
    deduplicate: bool = True,
    tag_chat_roles: bool = False,
) -> InifDocument

texts list[str] | list[list[dict[str, str]]]: Either a list of plain strings, or a list of chat message lists (list[dict[str, str]] with "role"/"content" keys) — detected per-element. Chat inputs are tokenized via tokenizer.apply_chat_template; plain strings via tokenizer.encode.
tokenizer Any: A HuggingFace tokenizer, or a model identifier string (e.g. "openai/gpt-oss-20b") that will be loaded via AutoTokenizer.from_pretrained.
sample_ids list[str] | None: Optional list of sample IDs (defaults to “sample_0”, …).
min_sequence_length int: Minimum length for common sequence detection.
deduplicate bool: Whether to run sequence deduplication. Default: True.
tag_chat_roles bool: Whether to tag chat-input tokens with roles after dedup. No-op for plain-string inputs.

from_text_files

Each file becomes one Sample. Finds common sequences across samples.

Source

def from_text_files(
    paths: list[str | Path],
    tokenizer: Any,
    min_sequence_length: int = 5,
    deduplicate: bool = True,
) -> InifDocument

paths list[str | Path]: List of file paths to process.
tokenizer Any: A HuggingFace tokenizer, or a model identifier string (e.g. "openai/gpt-oss-20b") that will be loaded via AutoTokenizer.from_pretrained.
min_sequence_length int: Minimum length for common sequence detection.
deduplicate bool: Whether to run sequence deduplication.