9iGSrSSKrSSKrSSKrSSKJr SSKrSSKrSSK r \"5S5r \"5S5r Sr Sr Sr"S S \5rS S jrg) zCLIP Tokenizer.N) lru_cachec[RR[RR[RR [ 55S5$)Nzbpe_simple_vocab_16e6.txt.gz)ospathjoindirnameabspath__file__l/var/www/html/land-doc-ocr/venv/lib/python3.13/site-packages/modelscope/models/multi_modal/gemm/tokenizer.py default_bpers7 77<< 12& ((r c [[[S5[S5S-55[[[S5[S5S-55-[[[S5[S5S-55-nUSSnS n[S 5H4nX0;dM URU5 URS U-5 US- nM6 UVs/sHn[ U5PM nn[ [ X55$s snf) a Returns list of utf-8 byte and a corresponding list of unicode strings. The reversible bpe codes work on unicode strings. This means you need a large # of unicode characters in your vocab if you want to avoid UNKs. When you're at something like a 10B token dataset you end up needing around 5K for decent coverage. This is a significant percentage of your normal, say, 32K bpe vocab. To avoid that, we want lookup tables between utf-8 bytes and unicode strings. And avoids mapping to whitespace/control characters the bpe code barfs on. !~¡¬®ÿNr)listrangeordappendchrdictzip)bscsnbs r bytes_to_unicoder#s eCHHqL" #%)%D D A +'&( (+/uSY47IM0C+D DB AB A 4[ ; IIaL IIdQh  FA   "Q#a&"B  B  sC:cd[5nUSnUSSHnURX#45 UnM U$)zwReturn set of symbol pairs in a word. Word is represented as tuple of symbols (symbols being variable-length strings). rrN)setadd)wordpairs prev_charchars r get_pairsr+5s? EEQIQR 9#$  Lr c[R"U5n[R"[R"U55nUR 5$N)ftfyfix_texthtmlunescapestriptexts r basic_cleanr5As3 == D ==t, -D ::<r cV[R"SSU5nUR5nU$)Nz\s+ )resubr2r3s r whitespace_cleanr:Gs$ 66&#t $D ::<|startoftext|> <|endoftext|>)rDrEz[<\|startoftext\|>|<\|endoftext\|>|'s|'t|'re|'ve|'m|'ll|'d|[\p{L}]+|[\p{N}]|[^\s\p{L}\p{N}]+)r# byte_encoderitems byte_decodergzipopenreaddecodesplittuplervaluesrrextendrrrlenencoderdecoder bpe_rankscacher8compile IGNORECASEpat)selfr>kvmergesmergevocabs r __init__SimpleTokenizer.__init__Os,..2.?.?.E.E.GH.GdaQT.GH8$))+227;AA$G-.4:;F5% &F;%'..01U3UVU33E LL ( '9:CuSZ'89: )-););)=>)=)=> c&%F *<=>0,  :: n MM!I<3 ?sG)#G/G41G9cz^UTR;aTRU$[USS5USS-4-n[U5nU(dUS-$/n[UU4SjS9nUTR;aOUupg/nSn U [ U5:aUR Xi5n URX)U 5 U n X)U:Xa7U [ U5S- :a%UU S-U:XaURXg-5 U S- n OURX)5 U S- n U [ U5:aM[U5nUn[ U5S:XaO [U5nM[ U5S:a[US5 S RU5nUTRU'U$![a/n URU 5 URX)S5 Sn A MSn A ff=f) NrBcN>TRRU[S55$)Ninf)rTgetfloat)pairrYs r %SimpleTokenizer.bpe..qs(:(:4u(Nr )keyrrdr7) rUrNr+minrTrQindexrP Exceptionrprintr) rYtokenr'r( error_listbigramfirstsecondnew_wordijerrs ` r bpeSimpleTokenizer.bpees DJJ ::e$ $U3BZ E"I$6#99$6> ! NPFT^^+"MEHAc$i- 5,AOOD1I.A 7e#CIM(9dA?"(?)OOEN3FAOODG,FA!c$i-"XHD4yA~!$=> z?S *R. !xx~  5 -!%%c*OODH-s &F F: $F55F:cf^/n[[U55R5n[R"TR U5HfnSR U4SjURS555nURU4SjTRU5RS555 Mh U$)NrCc3B># UHnTRUv M g7fr-)rF).0r"rYs r )SimpleTokenizer.encode..s#<%:!--a0%:r@c3B># UHnTRUv M g7fr-)rR)r~ bpe_tokenrYs r rrs"K/I)#ll95/Irr7) r:r5lowerr8findallrXrencoderPrzrM)rYr4 bpe_tokensrqs` r rSimpleTokenizer.encodes  D 1288:ZZ$/EGG<%*\\'%:<EE IF''.wvs';  A=s A7A<)rTrHrFrUrSrRrXN) __name__ __module__ __qualname____firstlineno__rstrr_rzrrL__static_attributes__r r r r<r<Ms#'2},-^r r<c[U[5(aU/nURSnURSnUVs/sHnU/URU5-U/-PM nn[R "[ U5U[RS9n[U5H]up[ U 5U:a$U(a U SUn XZS'O[SXSU35e[R"U 5XS[ U 524'M_ U$s snf)a\ Returns the tokenized representation of given input string(s) Parameters ---------- texts : Union[str, List[str]] An input string or a list of input strings to tokenize context_length : int The context length to use; all CLIP models use 77 as the context length truncate: bool Whether to truncate the text in case its encoding is longer than the context length Returns ------- A two-dimensional tensor containing the resulting tokens, shape = [number of input strings, context_length]. We return LongTensor when torch version is <1.8.0, since older index_select requires indices to be long. rDrE)dtypeNrbzInput z is too long for context length ) isinstancerrRrtorchzerosrQint enumerate RuntimeErrortensor) tokenizertextscontext_lengthtruncate sot_token eot_tokenr4 all_tokensresultrwrs r clip_tokenizers  %!!"34I!!/2I#%#d+ 0 0 66)D#% [[Z. JFz*  v; '0&r "UXJ&F~FVW#(,,v"6,3v;,+ M%s#C;)r=T)__doc__rIr0r functoolsrr.regexr8rrr#r+r5r:objectr<rr r r rsn    ( (   4   UfUp$r