9i`ZSSKJrJrJrJr SSKrSSKrSSKrSSKrSSK r SSK r SSK r Sr "SS\5r\R "\5rSrSr"SS \5r"S S \5r"S S \5rSrSrSrSSKJr \"5S5rSr"SS\5r g!\a SrN'f=f))absolute_importdivisionprint_functionunicode_literalsNc pSSSSSSSSS .nUR5Hup#URX#5nM U$) N-'zn'tz'mz don'tz'sz'vez're)z - z ' z n'tz 'mz do notz 'sz 'vez 're)itemsreplace)string replace_mpkvs l/var/www/html/land-doc-ocr/venv/lib/python3.13/site-packages/modelscope/preprocessors/nlp/space/tokenizer.py clean_stringrsK J  "%# Mc@\rSrSr/S4SjrSrSrSr/4SjrSr g ) TokenizerBertc^UTlUS:XGaWSSS.TlUHCnUTR;dMUS;dMS[TR5S3TRU'ME TRR5VVs0sHupVXe_M snnTlUVs/sHnTRR Xw5PM nnSTlT=R [U4S jU55- sl[UTR S 9Tl TR H(nUTRR;aMS US 35e [TRR5Tl gUS :XaSS0TlTRR5VVs0sHupVXe_M snnTlUVs/sHnUTR;dMUPM nn[RRUS5n[RRUS5n [XUS9Tl [U5Tl[TR5Tl g["es snnfs snfs snnfs snf)Nrz [unused0]z [unused1])z[BOS]z[EOS])[PAD][UNK]z[unused]rz[SEP]rz[CLS]z[MASK]c3L># UHnUTR;dMUv M g7fNspecial_tokens).0xselfs r %Tokenizer.__init__..7s')JNq,-T5H5H,H*+Ns$ $) never_splitzspecial token 'z' is not in the vocabularyGPT2rzz vocab.jsonz merges.txtr)tokenizer_typespec_convert_dictlenr spec_revert_dictgetrtuple BertTokenizer _tokenizervocab vocab_sizeospathjoin GPT2Tokenizer num_specials ValueError) r" vocab_pathrr'tokenrrtok vocab_file merges_files ` r__init__Tokenizer.__init__!sb, V #$$&D "( 6 665I<$+3t/E/E+F*Gq!I** (!2288:%:DA:%D ! AO@N&&**34 #-D    5)JN)J$J J ,(;(;=DO**doo333fseKe5ff3+!$//"7"78DO v %&-w%7D "!2288:%:DA:%D ! .-d444~ j,?J'',,z<@K+HDO #N 3D !$//2DO E%%sI%II.IIc8URRU5$r)r.tokenizer"texts rr?Tokenizer.tokenizeRs''--rcURS:XaGUVs/sHo RRX"5PM nnURR U5nU$UVs/sHo RRX"5PM nnURR U5nUVs/sHoDUR -UR -PM nnU$s snfs snfs snfNr)r'r(r+r.convert_tokens_to_idsr5r0)r"tokensr9idsis rrETokenizer.convert_tokens_to_idsUs   & (FLMfs,,00:fFM//77?CJFLMfs,,00:fFM//77?CFIJc)))T__ {}). Running this sequence through BERT will result in indexing errors)rr/r)r}loggerwarningru)r"rFrGr8s rrE#BertTokenizer.convert_tokens_to_idssgE JJtzz%( ) s8dll " NNHHNHdllI, -  rcX/nUH!nURURU5 M# U$)z?Converts a sequence of ids in wordpiece tokens using the vocab.)rrv)r"rGrFrHs rrK#BertTokenizer.convert_ids_to_tokenss-A MM$,,Q/ 0 r)ryrwrvr}r/r{)TNTr) rXrYrZr[__doc__r<r?rErKr\r]rrr-r-s)I $#'K "EH rr-cJ\rSrSrSrS SjrSrSrSrSr Sr S r S r g ) rxzDRuns basic tokenization (punctuation splitting, lower casing, etc.).cXlX lg)zVConstructs a BasicTokenizer. Args: do_lower_case: Whether to lower case the input. Nrq)r"rrr%s rr<BasicTokenizer.__init__s+&rclURU5nURU5n[U5n/nUHdnUR(a0X@R;a!UR 5nUR U5nURURU55 Mf [SRU55nU$)zTokenizes a piece of text.rN) _clean_text_tokenize_chinese_charsrmrrr%lower_run_strip_accentsextend_run_split_on_puncr3)r"rA orig_tokensrr8 output_tokenss rr?BasicTokenizer.tokenizes%++D1)$/   E!!e3C3C&C //6    7 7 > ? ! ,CHH\,BC rc[R"SU5n/nUH2n[R"U5nUS:XaM!URU5 M4 SR U5$)z$Strips accents from a piece of text.NFDMnrO) unicodedata normalizecategoryrr3)r"rAoutputcharcats rr!BasicTokenizer._run_strip_accentssY$$UD1D&&t,Cd{ MM$   wwvrcXR;aU/$[U5nSnSn/nU[U5:amX#n[U5(aUR U/5 SnO.U(aUR /5 SnUSR U5 US- nU[U5:aMmUVs/sHnSR U5PM sn$s snf)z&Splits punctuation on a piece of text.rTFrbrO)r%listr)_is_punctuationrr3)r"rAcharsrHstart_new_wordrrr!s rr!BasicTokenizer._run_split_on_puncs ## #6MT  #e*n8Dt$$ tf%!%!MM"%!&r !!$' FA#e*n%++Fq F+++s$Cc/nUHjn[U5nURU5(a5URS5 URU5 URS5 MYURU5 Ml SRU5$)z)Adds whitespace around any CJK character.rNrO)ord_is_chinese_charrr3r"rArrcps rr&BasicTokenizer._tokenize_chinese_chars/skDTB$$R(( c" d# c" d#wwvrcUS:=(a US:*nU=(d US:=(a US:*nU=(d US:=(a US:*nU=(d US:=(a US:*nU=(d US :=(a US :*nU=(d US :=(a US :*nU=(d US :=(a US:*nU=(d US:=(a US:*nU(agg)z6Checks whether CP is the codepoint of a CJK character.iNii4iMiiߦii?i@ii iiiiiTFr])r"rtmps rrBasicTokenizer._is_chinese_char<sV|,f 4bFl3rV|6bGm5g 6bGm5g 6bGm5g 6bGm5g 4bFl3rV|6bGm5g  rc/nUH`n[U5nUS:XdUS:Xd[U5(aM,[U5(aURS5 MOURU5 Mb SR U5$)zBPerforms invalid character removal and whitespace cleanup on text.rirNrO)r _is_control_is_whitespacerr3rs rrBasicTokenizer._clean_textSsgDTBQw",+d*;*;d## c" d#wwvrrqN)Tr) rXrYrZr[rr<r?rrrrrr\r]rrrxrxs0N $K '* ,, . rrxc(\rSrSrSrSSjrSrSrg)rziazRuns WordPiece tokenization.c(XlX lX0lgr)r/ unk_tokenmax_input_chars_per_word)r"r/rrs rr<WordpieceTokenizer.__init__ds "(@%rc8/n[U5GHn[U5n[U5UR:aUR UR 5 MESnSn/nU[U5:ax[U5nSn Xh:a<SR XFU5n US:aSU -n XR;aU n O US-nXh:aM<U cSnO$UR U 5 UnU[U5:aMxU(aUR UR 5 MURU5 GM U$)aTokenizes a piece of text into its word pieces. This uses a greedy longest-match-first algorithm to perform tokenization using the given vocabulary. For example: >>> input = "unaffable" >>> output = ["un", "##aff", "##able"] Args: text: A single token or whitespace separated tokens. This should have already been passed through `BasicTokenizer`. Returns: A list of wordpiece tokens. FrNrOz##rbT) rmrr)rrrr3r/r) r"rArr8ris_badstart sub_tokensend cur_substrsubstrs rr?WordpieceTokenizer.tokenizeis$ (.EKE5zD999$$T^^4FEJ#e*$%j! kWWU%56Fqy!%+%+ 1HCk%!F!!*-#e*$"$$T^^4$$Z0;/<r)rrr/N)rd)rXrYrZr[rr<r?r\r]rrrzrzas&A 1rrzcpUS:XdUS:Xd US:XdUS:Xag[R"U5nUS:Xagg)z1Checks whether `chars` is a whitespace character.rN   TZsF)rrrrs rrrs= s{ddlddlddl   t $C d{ rcUS:Xd US:XdUS:Xag[R"U5nURS5(agg)z.Checks whether `chars` is a control character.rrrFCT)rr startswithrs rrrs? t|tt|tt|   t $C ~~c rcB[U5nUS:=(a US:*nU=(d US:=(a US:*nU=(d US:=(a US:*nU=(d US:=(a US:*nU(ag [R"U5nURS 5(ag g ) z2Checks whether `chars` is a punctuation character.!/:@[`{~TPF)rrrr)rrrrs rrrs TB 8 bC ("('rRxC ("('rRxC *"))c C    t $C ~~c r) lru_cachecS$)NcU$rr])funcs rlru_cache..sDrr]r]rrrrs  rc <[RSS:Xa[O[n[ [ [ S5[ S5S-55[ [ [ S5[ S5S-55-[ [ [ S5[ S 5S-55-nUS S nSn[ S 5H4nXA;dM URU5 URS U-5 US- nM6 UVs/sH o0"U5PM nn[[X55$s snf) a Returns list of utf-8 byte and a corresponding list of unicode strings. The reversible bpe codes work on unicode strings. This means you need a large # of unicode characters in your vocab if you want to avoid UNKs. When you're at something like a 10B token dataset you end up needing around 5K for decent coverage. This is a significant percentage of your normal, say, 32K bpe vocab. To avoid that, we want lookup tables between utf-8 bytes and unicode strings. And avoids mapping to whitespace/control characters the bpe code barfs on. r!~rb¡¬®ÿN) sys version_infounichrchrrrangerrdictzip)_chrbscsnbs rbytes_to_unicoders%%a(A-63D eCHHqL" #%)%D D A +'&( (+/uSY47IM0C+D DB AB A 4[ ; IIaL IIdQh  FA   2a$q'2B  B  s2Dcd[5nUSnUSSHnURX#45 UnM U$)zxReturn set of symbol pairs in a word. Word is represented as tuple of symbols (symbols being variable-length strings). rrbN)rQadd)wordpairs prev_charrs r get_pairsrs? EEQIQR 9#$  Lrc\\rSrSrSrSSjrSrSrSrSr S r SS jr S r S r S rg)r4i z: GPT-2 BPE tokenizer. Peculiarities: - Byte-level BPE Nc UbUO [S5Ul[R"[ USS95UlUR R 5VVs0sHupgXv_M snnUlX0l[5Ul URR 5VVs0sHupgXv_M snnUl [ USS9R5RS5SSnUV s/sHn [U R55PM n n [[!U [#[%U 5555Ul0Ul[*R,"S5Ul0Ul0UlUR5U5 gs snnfs snnfs sn f)NrsrPr`rrbrzJ's|'t|'re|'ve|'m|'ll|'d| ?\p{L}+| ?\p{N}+| ?[^\s\p{L}\p{N}]+|\s+(?!\S)|\s+)r|r}jsonloadreencoderr decodererrorsr byte_encoderrSreadrlr,rrrr) bpe_rankscacherecompilepatrspecial_tokens_decoderset_special_tokens) r"r:r;rrr}rrbpe_datamerge bpe_mergess rr<GPT2Tokenizer.__init__s> #*"5w3t9 yyj7!CD )-););)=>)=)=>  ,..2.?.?.E.E.GH.GdaQT.GH g6;;=CCDI!BO8@AueEKKM* Ac*eC O.DEF :: ] !&(# /!?IAsE7E=$#FcX[UR5[UR5-$r)r)rr)r"s r__len__GPT2Tokenizer.__len__)s!4<< 3t':':#;;;rcH^U(d0Tl0Tlg[U4Sj[U555TlTRR 5VVs0sHup#X2_M snnTl[ R SRTR55 gs snnf)zAdd a list of additional tokens to the encoder. The additional tokens are indexed starting from the last index of the current vocabulary in the order of the `special_tokens` list. Nc3\># UH!upU[TR5U-4v M# g7fr)r)r)r rHr9r"s rr#3GPT2Tokenizer.set_special_tokens..5s.#L1Jvq%(T\\):Q)>#?1Js),zSpecial tokens {})rrr enumerater rinforu)r"rrrs` rr  GPT2Tokenizer.set_special_tokens,s "$D *,D ' "#L1:>1J#LL++113' 3 D3' #  '..t/B/BCD ' sBc^UTR;aTRU$[U5n[U5nU(dU$[UU4SjS9nUTR;aOUupV/nSnU[ U5:aUR XX5n URX(U 5 U nX(U:Xa7U[ U5S- :a%UUS-U:XaURXV-5 US- nOURX(5 US- nU[ U5:aM[U5nUn[ U5S:XaO [U5nMSRU5nUTRU'U$![a URX(S5 Mlf=f)NcN>TRRU[S55$)Ninf)rr+float)pairr"s rr#GPT2Tokenizer.bpe..Hs(:(:4u(Nr)keyrrbrrN) rr,rminrr)rhr Exceptionrr3) r"r8rrbigramfirstsecondnew_wordrHjs ` rbpeGPT2Tokenizer.bpe=s} DJJ ::e$ $U|$LNPFT^^+"MEHAc$i- 5,AOOD1I.A 7e#CIM(9dA?"(?)OOEN3FAOODG,FAc$i- XHD4yA~!$;<xx~  5 '!OODH-s8&EE32E3c^/n[R"TRU5H\nSRU4SjU55nUS:XaM&UR STR U5R S555 M^ U$)zTokenize a string. rOc3># UH8n[U5TR;dMTR[U5v M: g7fr)rr)r rr"s rr#)GPT2Tokenizer.tokenize..ls9 {}). Running this sequence through the model will result in indexing errors) isinstancestrrrunicoderrr+rr)r}rrru)r"rFrGpython_version_3python_version_2r8s rrE#GPT2Tokenizer.convert_tokens_to_idsts%fc2   Q 1 $ DFG)D  /,,,**622||''22E+++ 4..u56 4<<++E156  s8dll " NNMs3x.  0  rc/nUHYnX@R;a)U(d URURU5 M9M;URURU5 M[ U$)z9Converts a sequence of ids in BPE tokens using the vocab.)rrr)r"rGskip_special_tokensrFrHs rrK#GPT2Tokenizer.convert_ids_to_tokenssYA///*MM$"="=a"@A+ dll1o.   rcBURURU55$r)rEr?r@s rencodeGPT2Tokenizer.encodes))$--*=>>rcSRUVs/sHo RUPM sn5n[UVs/sHo@RUPM sn5R SUR S9nU$s snfs snf)NrOrP)r)r3rrRrSrTr)r"rFr8rArVs rrTGPT2Tokenizer.decodespww@ U+@A=1++A.=>EE DKKF) A=s A1A6) rrSrrrrrr}rrr)r NN)F)rXrYrZr[rr<rr r%r?rErKr:rTr\r]rrr4r4 sA" $ 04<E")V 0 ?rr4)! __future__rrrrrcloggingr1rrrregexrrobjectr getLoggerrXrrjrmr-rxrzrrr functoolsr ImportErrorrrr4r]rrrEs**    YYR   8 $ CFCLsVsl99x  @+!#  6 UFUa!!!s<B B*)B*