iHSSKrSSKrSSKrSSKJr SSKJr "SS5rg)N)List)SentencePieceProcessorc \rSrSrS\4SjrS\S\S\S\\4Sjr S \\S\4S jr S \S\4S jr S S .S\S\SS4Sjjr Sr g)Llama2cTokenizer model_pathc,[RRU5(d SU35e[US9UlXlURR 5UlURR5UlURR5Ul [R"SURSURSUR35 URR 5URR5:Xdeg)Nz*Need a valid tokenizer model path but got ) model_filez#words: z - BOS ID: z - EOS ID: ) ospathisfilersp_modelr vocab_sizen_wordsbos_ideos_idlogginginfoget_piece_size)selfrs Y/var/www/html/ai-image-ml/venv/lib/python3.13/site-packages/pytorch_tokenizers/llama2c.py__init__Llama2cTokenizer.__init__sww~~j)) 8 E )/*E $!MM446 ==//1 ==//1  t||nK }K } U }}'')T]]-I-I-KKKKsboseosreturnc[U5[LdeURRU5nU(aUR/U-nU(aX@R /-nU$N)typestrrencoderr)rrrrts rr#Llama2cTokenizer.encode#sOAw#~~ MM  #  !A [[M!Arr$c8URRU5$r rdecoderr$s rr(Llama2cTokenizer.decode-}}##A&&rc8URRU5$r r'r)s r decode_tokenLlama2cTokenizer.decode_token1r+rF)prepend_padding output_pathr/Nc //pCU(a1URSRS55 URS5 [UR5HnURR U5nURR U5nXPR:XaSnOXPR:XaSnURSS5nURS5nURU5 URU5 M U(dSO[S U55n [US 5n U R[R"S URURURU 55 [X45HFupU R[R"S U [!U 555 U RU 5 MH S S S 5 ["R$"SU35 g !,(df  N(=f)a  Export tokenizer.model to another serialization format. Here we did some lightweight processing such as supporting prepend padding token, prepend max token length and replace '_' back to empty space. The binary format is: 1. vocab size: int32 2. bos token id: int32 3. eos token id: int32 4. max token length: int32 5. score: float32, len of bytes: int32, token bytes: [byte] for each token :param output_path: output path of the new binary. :param prepend_padding: a boolean to control if we want to prepend a padding token. :return: None zzutf-8zzu▁ rc38# UHn[U5v M g7fr )len).0r$s r *Llama2cTokenizer.export..bs3KFqCFFFswbIIIIfINzWrote tokenizer to )appendr#rangerr id_to_piece get_scorerrreplacemaxopenwritestructpackzipr5rr) rr0r/tokensscoresir$rbmax_token_lengthfbytesscores rexportLlama2cTokenizer.export5st(R  MM'..1 2 MM" t||$A ))!,A ''*AKKkk! %%A!A MM!  MM! %$%+13KF3K0K+t $ GG DLL$++t{{DT  !$F 3  D%U<=!4%  *;-89% $s BG G*)rrrrr)__name__ __module__ __qualname____firstlineno__r"rboolrintr#r(r-rO__static_attributes__rrrrsL3L$$Td3i'S 'c''c'c'CH::#::4::D::::rr)rr rDtypingr sentencepiecerrrXrrr[s" J_:_:r