inif.tagging

Type aliases and modes for the tagging methods on Sample / InifDocument.

Tagging is exposed as methods on Sample and InifDocument — tag_by_regex, tag_by_regexes, tag_by_text_regex, tag_by_predicate, tag_by_predicates, tag_chat_roles, tag_special_tokens, create_span_from_tag. This module re-exports the supporting types so callers building up batched tagger lists can spell them out.

See Tagging tokens for usage patterns and the difference between token-string and concatenated-text matching.

Modes

TextTagMode

Which tokens to tag when a text-level regex match spans multiple tokens.

Source

class TextTagMode(str, Enum)

Type aliases