iSSKrSSKJr SSKJr SSKJrJrJrJ r J r J r J r J r Jr SSKrSSKJr SSKJrJr \"\5rSrS rSq"S S 5rg) N) getLogger)Path) AbstractSetcast CollectionIteratorListLiteralOptionalSequenceUnion)load_tiktoken_bpe)CL100K_PAT_STRLLAMA_SPECIAL_TOKENSiiac\rSrSrSr\S5r\\4S\ S\ S\ \ 4Sjjr SS S .S \ S \ S \ S\ \\S\\ 4S\\S\\ 4S\ \4 SjjrSS\\S\ S\ 4SjjrS\S\ 4Sjr\S \ S\S\\ 4Sj5rSrg)TiktokenTokenizer,z Tokenizing and encoding/decoding text using the Tiktoken tokenizer. WARNING: The regex and special tokens are hardcoded from Llama 3+. c[cJ[[RR [RR [ 5S55q[$)Nztokenizer.model) _INSTANCErospathjoindirname__file__)clss Z/var/www/html/ai-image-ml/venv/lib/python3.13/site-packages/pytorch_tokenizers/tiktoken.py get_instanceTiktokenTokenizer.get_instance2s<  ) RWW__X68IJI model_pathpat_strspecial_tokensc[RRU5(dU5e[U5n[ U5n[ U5VVs0sH upgXuU-_M snnUl[R"[U5RUUUR S9Ul U[ U5-Ul UR SUl UR SUlgs snnf)zt Initializes the Tokenizer with a Tiktoken model. Args: model_path (str): The path to the Tiktoken model file. )namer"mergeable_ranksr#z<|begin_of_text|>z<|end_of_text|>N)rrisfilerlen enumerater#tiktokenEncodingrr%modeln_wordsbos_ideos_id)selfr!r"r#r&num_base_tokensitokens r__init__TiktokenTokenizer.__init__<sww~~j))5:5)+J7o.8A7P 7P81EQ& &7P &&j!&&+..   ,c..AA ../BC ../@A  sCNallowed_specialdisallowed_specialsboseosr8allr9returnc ^^Uc [5n[T5[LdeUU4Sj[S[ T5[ 55n/nUH-nUR TRRUUUS95 M/ U(aURSTR5 U(aURTR5 U$)a0 Encodes a string into a list of token IDs. Args: s (str): The input string to be encoded. bos (bool): Whether to prepend the beginning-of-sequence token. eos (bool): Whether to append the end-of-sequence token. allowed_special ("all"|set[str]): allowed special tokens in string disallowed_special ("all"|set[str]): special tokens that raise an error when in string Returns: list[int]: A list of token IDs. By default, setting disallowed_special=() encodes a string by ignoring special tokens. Specifically: - Setting `disallowed_special` to () will cause all text corresponding to special tokens to be encoded as natural text (insteading of raising an error). - Setting `allowed_special` to "all" will treat all text corresponding to special tokens to be encoded as special tokens. c3v># UH.nTRTX[-[5HnUv M M0 g7f)N)$_split_whitespaces_or_nonwhitespacesTIKTOKEN_MAX_ENCODE_CHARSMAX_NO_WHITESPACES_CHARS).0r2substrr:r0s r +TiktokenTokenizer.encode..~sC @CC!3346N  @s69rr7) settypestrranger(rBextendr,encodeinsertr.appendr/) r0r:r;r<r8r9substrstrEs `` rrMTiktokenTokenizer.encode\s<  "!eOAw#~~ 1c!f&?@ F HH !!$3'9"   HHQ $  HHT[[ !r rQskip_special_tokenscbURR[[[U55$)z Decodes a list of token IDs into a string. Args: t (List[int]): The list of token IDs to be decoded. Returns: str: The decoded string. )r,decoderr int)r0rQrSs rrUTiktokenTokenizer.decodes$zz  d3i!344r cVURRU5RS5$)z Decodes a single token ID into a string. Args: t (int): The token ID to be decoded. Returns: str: The decoded string. zutf-8)r,decode_single_token_bytesrU)r0rQs r decode_tokenTiktokenTokenizer.decode_tokens$zz33A6==gFFr max_consecutive_slice_lenc# # Sn[U5S:aUSR5OSnSn[[U55H:nXR5nX6- (aSnUnM$US- nX!:dM0XUv UnSnM< XSv g7f)z Splits the string `s` so that each substring contains no more than `max_consecutive_slice_len` consecutive whitespaces or consecutive non-whitespaces. rFrN)r(isspacerK)r:r\current_slice_lencurrent_slice_is_space slice_startr2 is_now_spaces rrA6TiktokenTokenizer._split_whitespaces_or_nonwhitespacess36q6A:15 s1vA4<<>L%4$%!)5&!Q&!$@**"#K()% os A)B/B)r.r/r,r-r#)F)__name__ __module__ __qualname____firstlineno____doc__ classmethodrrrrJr r4boolr r r rrrVrMr rUrZ staticmethodrrA__static_attributes__r6r rrr,s) &$8 BBBS BLNREG6 6 6  6 "% C8H(H"IJ 6"'%.*S/"AB6 c6p 5 5D 5S 5 Gc Gc G +. #r r)rloggingrpathlibrtypingrrrrr r r r r r* tiktoken.loadr constantsrrrdloggerrBrCrrr6r rrssW    +; 8  $ "  YYr