inif.converters.evaleval

Convert every_eval_ever instance-level records into INIF documents.

Targets the instance_level_eval_0.2.2 schema used by the evaleval/EEE_datastore HuggingFace dataset. The HF entry point requires the evaleval extra (pip install "inif[evaleval]"); local JSON / JSONL conversion does not. See the evaleval converter guide for the field-mapping table, scoring, and metadata promotion rules.

Entry points

from_instance_records

Convert a list of EEE instance records into an :class:InifDocument.

Source

def from_instance_records(
    records: list[dict],
    aggregate: dict | None = None,
    *,
    tokenizer: Any = "auto",
    include_messages: bool = True,
    deduplicate: bool = True,
    min_sequence_length: int = 5,
    tag_chat_roles: bool = True,
    tag_generated: bool = True,
    tag_reasoning: bool = True,
) -> InifDocument

records list[dict]

Instance-level records conforming to instance_level_eval_0.2.2.

aggregate dict | None

Optional aggregate record (eval.schema.json). Contributes model developer info, eval library version, and metric config to document-level metadata.

tokenizer Any

One of:

• "auto" (default) or None — auto-load AutoTokenizer.from_pretrained based on each record’s model_id (with aggregate fallback). Raises if the id is missing or the tokenizer cannot be loaded (closed-source ids like openai/gpt-4 will hit this).
• a model id string — loaded via AutoTokenizer.from_pretrained; warns if the resolved id disagrees with the source records’ model_id.
• a tokenizer instance — used as-is; warns if its name_or_path disagrees with the source records’ model_id.

Tokens are a load-bearing invariant of every INIF sample, so there is no way to opt out of tokenization — every option here yields a usable tokenizer or raises.

include_messages bool

Include message texts on each Sample.

deduplicate bool

Run sequence deduplication over the produced samples.

min_sequence_length int

Minimum length for common-sequence detection.

tag_chat_roles bool

Add role annotations via character-span matching.

tag_generated bool

Annotate the last assistant message’s tokens with "generated".

tag_reasoning bool

Annotate tokens belonging to each turn’s reasoning_trace span. Matching tokens get reasoning + assistant; tokens within the span whose id appears in tokenizer.all_special_ids additionally get template. Best-effort: silently skipped when the token stream doesn’t round-trip cleanly to the formatted chat template or the reasoning text can’t be located.

from_eval_json

Load EEE JSON files from disk and convert.

aggregate_path is optional; instances_path can be either a single JSON file containing a list of records, or a .jsonl file with one record per line.

Source

def from_eval_json(
    aggregate_path: str | Path | None,
    instances_path: str | Path,
    **kwargs: Any,
) -> InifDocument

aggregate_path str | Path | None

instances_path str | Path

**kwargs Any

from_hf_dataset

Convert an EEE config from the HuggingFace datastore.

Source

def from_hf_dataset(
    config: str,
    split: str = "samples",
    *,
    aggregate_config: str | None = None,
    aggregate_split: str = "train",
    repo: str = "evaleval/EEE_datastore",
    limit: int | None = None,
    revision: str | None = None,
    **kwargs: Any,
) -> InifDocument

config str

A config name from evaleval/EEE_datastore, typically one ending in _samples (e.g. "theory_of_mind_samples").

split str

Dataset split for instance records (usually "samples").

aggregate_config str | None

Optional paired aggregate config (without _samples suffix). When provided, the first row is used as the aggregate.

aggregate_split str

Split name for the aggregate config (defaults to "train").

repo str

HuggingFace repo id (override for testing).

limit int | None

Maximum number of records to convert (None = all).

revision str | None

HuggingFace dataset revision / commit.

**kwargs Any

Forwarded to :func:from_instance_records.