Converters

Building INIF documents from raw text and from eval-framework outputs.

INIF is a sink for tokenized generation traces. The converters in inif.converters translate from the format produced by your generation pipeline into an InifDocument.

Source	Module	Entry point
Raw text / chat messages	inif.converters.text	from_texts, from_text_files
Inspect AI eval logs	inif.converters.inspect_ai	from_eval_log, from_eval_file
every_eval_ever records	inif.converters.evaleval	from_instance_records, from_eval_json, from_hf_dataset

All of them follow the same conventions:

• Tokenization is mandatory. Every Sample carries a tokens list. There is no opt-out — see Tokenizer resolution for how the tokenizer is chosen.
• Sequence deduplication runs by default. Pass deduplicate=False to skip it.
• Chat role tagging is on by default for the eval converters (tag_chat_roles=True). The text converter has it off by default (chat inputs are detected per element and tagged when present, but the default text input is plain string).

Picking the right converter

• You have raw strings or chat messages produced by something other than an eval framework — use the text converter.
• You have Inspect AI .eval log files — use the Inspect AI converter. Per-token logprobs and generated-output annotations are attached automatically when present.
• You have evaleval JSON / JSONL or want to pull straight from the evaleval/EEE_datastore HF dataset — use the evaleval converter. Reasoning-trace spans, multi-turn / agentic interactions, tool calls, and aggregate metric metadata are preserved.

Common pitfalls

WarningClosed-source model ids

Auto-loading the tokenizer from eval_log.eval.model fails for closed models (openai/gpt-4, anthropic/claude-3-...) because there is no HF tokenizer to fetch. Pass tokenizer="<hf-stand-in>" explicitly. The warning that gets emitted on mismatch is informational — token-level data will reflect the supplied tokenizer, not the original model.

WarningTokenizers that are lossy on per-token decode

A few tokenizers (notably old SentencePiece slow tokenizers, and Qwen, which replaces U+2028 with U+FFFD) cannot reliably reconstruct per-token strings. The converters silently skip role / generated / reasoning tagging for those samples rather than raise. The tokens themselves are still correct.