9i"SrSSKJrJrJrJr SSKrSSKrSSKrSSK r SSK J r SSK J r JrJrJr SSKrSSKrSSKJrJr SSKJr \"5rSS 0r"S S \5rg) zTokenization classes for QWen.)absolute_importdivisionprint_functionunicode_literalsN)open)ListOptionalTupleUnion) AddedTokenPreTrainedTokenizer) get_logger vocab_file qwen.tiktokencD^\rSrSrSr\rSU4SjjrSrSr Sr S\ S\ \ 4S jr S \ S\\ 4S jrS \\ S\ 4S jr\S5rS\S\ 4SjrS\ S\4Sjr\S\\ 4Sj5r\S\\4Sj5rSrSS\\\\4S\S\ 4SjjrSrU=r$) QWenTokenizerzQWen tokenizer.c >[U[5(a [USSS9OUn[U[5(a [USSS9OUn[U[5(a [USSS9OUn[U[5(a [USSS9OUn[TU]UUUUUUU S9 XlUbUO [ S5UlX lSn Sn SnSnU (a5U UUS S S S S 4[[S5Vs/sH nSUS3PM sn5-nOXU4nSnS[SS4SjnU"U5n[U[U5S9VVs0sH unnUU_M nnnUUl [R"U UUUS9n[U5[U5-UR :Xd,[U5[U5-SUR S35eUUlUR"UlUR$R'5VVs0sH unnUU_M snnUlUUlUR*R,UlUUUlUUUlgs snfs snnfs snnf)NF)lstriprstrip)errors unk_token bos_token eos_token pad_tokenadd_prefix_space add_bos_tokengmBQwen <|endoftext|>z <|im_start|>z <|im_end|>zzzzzzzn(?i:'s|'t|'re|'ve|'m|'ll|'d)|[^\r\n\p{L}\p{N}]?\p{L}+|\p{N}| ?[^\s\p{L}\p{N}]+[\r\n]*|\s*[\r\n]+|\s+(?!\S)|\s+tiktoken_bpe_filereturnzdict[bytes, int]c[US5R5nSUR55VVs0sH%up#[R"U5[ U5_M' snn$s snnf)Nrbc3R# UHo(dM UR5v M g7fN)split).0lines g/var/www/html/land-doc-ocr/venv/lib/python3.13/site-packages/modelscope/models/nlp/qwen/tokenization.py DQWenTokenizer.__init__..load_tiktoken_bpe..cs$$O0E%1DJJLL0Es '')rread splitlinesbase64 b64decodeint)r"contentstokenranks r+load_tiktoken_bpe1QWenTokenizer.__init__..load_tiktoken_bpe_sn-t499;H$O080C0C0E$O$OKE  'T2$O s,A%)start)pat_strmergeable_ranksspecial_tokensz != z in encoding) isinstancestrr super__init__rr2max_lenrtuplerange enumeratelenr;tiktokenEncodingn_vocabr:encoderitemsdecoder tokenizer eot_tokeneod_id im_start_id im_end_id)selfrrr@rrrrrradd_more_sp_tokenskwargsname ENDOFTEXTIMSTARTIMENDir;PAT_STRr6r:indexr4enckv __class__s r+r?QWenTokenizer.__init__!sBL3B B Jyu =%. BL3B B Jyu =%. BL3B B Jyu =%. BL3B B Jyu =%.  -'  +")"5w3t9  #    uSz:z!1~z:; )=A1)=> nn.. )'2'.O;" $?s(H: 3H?*Ic.URR$r'rKrGrPs r+__len__QWenTokenizer.__len__s~~%%%cUR$r')r:ras r+ get_vocabQWenTokenizer.get_vocabs###rdc /n[U[5(a9XR;aURU$URR U5$UH\nX0R;a UR URU5 M2UR URR U55 M^ [ U5UR:a8[RSR[ U5UR55 U$)NzToken indices sequence length is longer than the specified maximum sequence length for this model ({} > {}). Running this sequence through the model will result in indexing errors) r<r=r;rHgetappendrDr@loggerwarningformat)rPtokensidsr4s r+convert_tokens_to_ids#QWenTokenizer.convert_tokens_to_idss fc " ",,,**622||''//E+++ 4..u56 4<<++E23  s8dll " NNMs3x.  0  rdsave_directoryr#c l[RRUS5n[USSS9nURR 5HMupV[ R"U5RS5S-[U5-S-nURU5 MO SSS5 U4$!,(df  U4$=f)z~ Save only the vocabulary of the tokenizer (vocabulary + added tokens). Returns: `Tuple(str)`: Paths to the files saved. rwutf8)encoding  N) ospathjoinrr:rIr0 b64encodedecoder=write)rPrrrR file_pathrtr[r\r*s r+save_vocabularyQWenTokenizer.save_vocabularysGGLLA )S6 2a,,224''*11&9C?#a&H4O 53} 3 2}s A,B## B3textc /n[R"SU5nURRU5H!nUR UR U5 M# U$)a Converts a string in a sequence of tokens, replacing unknown tokens with the `unk_token`. Args: text (`str`): The sequence to be encoded. kwargs (additional keyword arguments, *optional*): Will be passed to the underlying model specific encode method. See details in [`~PreTrainedTokenizerBase.__call__`] Returns: `List[str]`: The list of tokens. NFC) unicodedata normalizerKencode_ordinaryrjrJ)rPrrRrnts r+tokenizeQWenTokenizer.tokenizesM$$UD1//5A MM$,,q/ *6 rdrncSRU5n[UVs/sHo0RUPM sn5RSURS9nU$s snf)z Converts a sequence of tokens in a single string. The most simple way to do it is `" ".join(tokens)` but we often want to remove sub-word tokenization artifacts at the same time. zutf-8)r)r{ bytearray byte_decoderr}r)rPrnrcs r+convert_tokens_to_string&QWenTokenizer.convert_tokens_to_stringsW wwv=1++A.=>EE DKKF) >sAc.URR$r'r`ras r+ vocab_sizeQWenTokenizer.vocab_sizes~~%%%rdrYcXRR:a UR$URRU/5$r')rKrGrr})rPrYs r+_convert_id_to_token"QWenTokenizer._convert_id_to_tokens3 NN** *>> !~~$$eW--rdr4cURRURS5URRURSS9S5$)z*Converts a token to an id using the vocab.zUTF-8all)allowed_specialr)rHriencoderKr)rPr4s r+_convert_token_to_id"QWenTokenizer._convert_token_to_idsE|| LL ! NN ! !$..% ! H K  rdcvURR5Vs/sHn[U5PM nnU$s snf)z `List[str]`: All the special tokens (`''`, `''`, etc.) mapped to class attributes. Convert tokens of `tokenizers.AddedToken` type to string. )r;keysr=)rPsall_tokss r+all_special_tokens QWenTokenizer.all_special_tokenss8%)$7$7$<$<$>?$>qCF$>?@s6cbURR5Vs/sHoPM nnU$s snf)zi `List[int]`: List the ids of the special tokens(`''`, `''`, etc.) mapped to class attributes. )r;values)rPr\all_idss r+all_special_idsQWenTokenizer.all_special_idss3 #1188:;:1:;rsV %**  // 8. !?3a0'a0rd