9iDSrSSKrSSKJrJrJrJr SSKrSSK r SSK J r SSK JrJr SSKJr SSKJr \R*"5rSS 0r"S S 5r"S S 5r"SS\ 5rg)z!Tokenization classes for ChatGLM.N)DictListOptionalUnion)PreTrainedTokenizer) BatchEncoding EncodedInput)PaddingStrategy)loggerzTHUDM/chatglm-6bicR\rSrSrSrSrS\\4SjrSr Sr Sr S r S r S rg ) TextTokenizerc[R"5UlURRU5 URR 5UlgN)spmSentencePieceProcessorspLoad vocab_size num_tokens)self model_paths j/var/www/html/land-doc-ocr/venv/lib/python3.13/site-packages/modelscope/models/nlp/chatglm/tokenization.py__init__TextTokenizer.__init__s6,,.  Z '',,.c8URRU5$r)r EncodeAsIdsrtexts rencodeTextTokenizer.encodesww""4((ridsc8URRU5$r)r DecodeIds)rr#s rdecodeTextTokenizer.decodeww  %%rc8URRU5$r)rEncodeAsPiecesrs rtokenizeTextTokenizer.tokenize!sww%%d++rcbUVs/sHo RRU5PM sn$s snfrr PieceToId)rtokenstokens rconvert_tokens_to_ids#TextTokenizer.convert_tokens_to_ids$s'6<=fU!!%(f===s$,c8URRU5$rr.rr1s rconvert_token_to_id!TextTokenizer.convert_token_to_id'sww  ''rc8URRU5$r)r IdToPiece)ridxs rconvert_id_to_token!TextTokenizer.convert_id_to_token*r(rcUR$r)rrs r__len__TextTokenizer.__len__-s r)rrN)__name__ __module__ __qualname____firstlineno__rr!rintr&r+r2r6r;r?__static_attributes__rrr r s5/ )&$s)&,>(&rr c\rSrSrSSjrSr\S\4Sj5r\S5r \ S5r \ S5r \SS \ S \4S jj5rSS \ 4S jjrSS \ S \\4SjjrS\\S \ 4SjrSS \ S \\ 4SjjrS\\\ 44SjrSrg) SPTokenizer1cpUceXlX l/SQUlX0lX@l[ U5Ulg)N)[MASK][gMASK]z[sMASK]z zz) vocab_filenum_image_tokensspecial_tokensmax_blank_length byte_fallbackr text_tokenizer)rrPrQrSrTs rrSPTokenizer.__init__3s@%%%$ 0 !1*+J7rcUR$r)rUr>s r_get_text_tokenizerSPTokenizer._get_text_tokenizerEs"""rlengthcUS:deSUS3$)Nz<|blank_z|>rG)rZs rget_blank_tokenSPTokenizer.get_blank_tokenHs{{&$$rcg)Nz<|tab|>rGrGrr get_tab_tokenSPTokenizer.get_tab_tokenMsrc.URR$r)rUrr>s rnum_text_tokensSPTokenizer.num_text_tokensQs""---rc4URUR-$r)rQrcr>s rrSPTokenizer.num_tokensUs$$t';';;;rr max_lencURS[R55n[USS5H+nURSU-[R U55nM- U$)N  )replacerIr`ranger])r rgis r_encode_whitespacesSPTokenizer._encode_whitespacesYsR||D+";";"=>w2&A<<a)D)DQ)GHD' rcxU(aURSS5nU(aURXRS9nU$)N )rg)rmrprS)rr linebreak whitespacess r _preprocessSPTokenizer._preprocess`s; <<e,D ++33,5D rreturncURXU5nU(dSU-nUR5RU5nUVs/sHofUR-PM nnU(aU$USS$s snfa @param text: Text to encode. @param linebreak: Whether to encode newline ( ) in text. @param whitespaces: Whether to encode multiple whitespaces or tab in text, useful for source code encoding. @param special_tokens: Whether to encode special token ([MASK], [gMASK], etc.) in text. @param add_dummy_prefix: Whether to add dummy blank space in the beginning. rtr\N)rwrXr!rQ)rr rurvadd_dummy_prefixtmpxr0s rr!SPTokenizer.encodehsp=4q$//!34A<< 4 4Q 7qAD5 E.s"C C"C"cURXU5nU(dSU-nUR5RU5nU(aU$USS$r{)rwrXr+)rr rurvr|r0s rr+SPTokenizer.tokenizesN=4zrkzThe key should be str or int.) isinstancerErQformatrUr;str startswithendswithisdigitr6 ValueError)rr~s r __getitem__SPTokenizer.__getitem__s a  (((#**1--**>>---// 3  ||I&&1::,,qW__..1Qr7|#**>>..//<= =r)rTrSrQrRrUrPN) NPT)r)TT)TTT)rArBrCrDrrX staticmethodrEr]r`propertyrcrrrprwrr!r&r+rrrFrGrrrIrI1s  8$#%%%..<<#  $ ::*.c :&tCyS !"& ::,09 :$>U38_>rrIc ^\rSrSrSrSS0r\r/SQrS S!U4Sjjjr \ S\ \ 4S j5r \ S\ \ 4S j5r\ S 5rS rS rSrS"S\\ \\ 4S\S\S\4SjjrSrSrS#SjrS#S\\ S\ \\ S\\ 4SjjrS\R:SS4S\\\\4\ 4S\ \ S\S\ \ S\ \S\!4 Sjjr"Sr#U=r$$)$ChatGLMTokenizera Construct a ChatGLM tokenizer. Based on byte-level Byte-Pair-Encoding. Args: vocab_file: Path to the vocabulary file. do_lower_case: Use lower case letters. remove_space: Remove spaces. bos_token: The bos token eos_token: The Eos Token end_token: The end token mask_token: The mask token gmask_token: The gmask token padding_side: The padding side num_image_tokens: The `num_image_tokens` in `SPTokenizer` rPzice_text.model) input_idsattention_mask position_idsryNc >[XS9Ul[T U] "SUUU UUUUUU S. U D6 X lX0lXlX@lXPlX`l Xpl Xl g)N)rQ) do_lower_case remove_space padding_side bos_token eos_token end_token mask_token gmask_tokenrQrG) rI sp_tokenizersuperrrrrPrrrrr) rrPrrrrrrrrrQkwargs __class__s rrChatGLMTokenizer.__init__s~( ;  '%%!#-  +($"""$&rcTURcgURUR5$r)rr2r>s rgmask_token_idChatGLMTokenizer.gmask_token_ids(    #))$*:*:;;rcTURcgURUR5$)zr `Optional[int]`: Id of the end of context token in the vocabulary. Returns `None` if the token has not been set. N)rr2r>s r end_token_idChatGLMTokenizer.end_token_ids& >> !))$..99rc.URR$)zReturns vocab size )rrr>s rrChatGLMTokenizer.vocab_sizes  +++rc[UR5Vs0sHnURU5U_M nnURUR5 U$s snf)zReturns vocab as a dict )rnr_convert_id_to_tokenupdateadded_tokens_encoder)rrovocabs r get_vocabChatGLMTokenizer.get_vocabs[4??+ +  % %a (! ++    T../  sAcUR(a.SRUR5R55nOUnUR(aUR 5nU$)Nrl)rjoinstripsplitrlower)rinputsoutputss rpreprocess_text ChatGLMTokenizer.preprocess_textsF   hhv||~3356GG   mmoGrc ^URU5nURRU5nU$)zReturns a tokenized string. )rrr+)rr rseqs r _tokenizeChatGLMTokenizer._tokenizes-##D)((. r token_idsskip_special_tokensclean_up_tokenization_spacesc [U[5(aU/n[U5S:XagURU;a)[ [ URR U55nURRU5$)Nr) rrElen pad_token_idlistfilter__ne__rr&)rrrrrs r_decodeChatGLMTokenizer._decodesi i % %" I y>Q     )VT%6%6$>$> JKI  '' 22rc URU$)z1Converts a token (str) in an id using the vocab. rr5s r_convert_token_to_id%ChatGLMTokenizer._convert_token_to_id#  ''rc URU$)z=Converts an index (integer) in a token (str) using the vocab.r)rindexs rr%ChatGLMTokenizer._convert_id_to_token'rrc[RRU5(a.[RRUURS5nOUn[ UR S5nUR5nSSS5 [ US5nURW5 SSS5 U4$!,(df  N7=f!,(df  U4$=f)a9 Save the vocabulary and special tokens file to a directory. Args: save_directory (`str`): The directory in which to save the vocabulary. filename_prefix (`str`, *optional*): An optional prefix to add to the named of the saved files. Returns: `Tuple(str)`: Paths to the files saved. rPrbNwb) ospathisdirrvocab_files_namesopenrPreadwrite)rsave_directoryfilename_prefixrPfin proto_strwriters rsave_vocabulary ChatGLMTokenizer.save_vocabulary+s 77== ( (n&*&<&<\&JLJ(J $//4 (C I)*d #v LL #$~ ) ($ #~s+B-B>- B;> C token_ids_0 token_ids_1cURURnURURnURURnX1;a XA;aX/- nUSU:wa%USU:waXRUR/- nXRUR /- nUbU(a USU:waX%/- nX- nU$)a6 Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and adding special tokens. A BERT sequence has the following format: - single sequence: `[CLS] X [SEP]` - pair of sequences: `[CLS] A [SEP] B [SEP]` Args: token_ids_0 (`List[int]`): List of IDs to which the special tokens will be added. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. Returns: `List[int]`: List of [input IDs](../glossary#input-ids) with the appropriate special tokens. rk)rrrrrr)rrrmask_ids gmask_idseos_ids r build_inputs_with_special_tokens1ChatGLMTokenizer.build_inputs_with_special_tokensFs($$T__5%%d&6&67 ""4>>2  &9+G ; &K r?h &;r?i+G --dnn=> >K))$..9::  "+b/V";x'  &Krencoded_inputs max_lengthpadding_strategypad_to_multiple_ofreturn_attention_maskc URURnURURnURURnURS:XdeXR Sn [ U 5n U[R:Xa [ U 5nUbUbX$-S:wa X$-S-U-nU[R:g=(a [ U 5U:gn UGb?SU;arXi;aU RU5n OU n [R"SX45n [R"U 5n SU SS2SS2SU 24'[R"U S:5n XS'SU;a[R"U [R S9nXy;aUOUnX;aU RU5nUUW S&[R""[R$"W [R S9[R"SX- S-[R S9/5n[R&"UU/SS 9US'U (aU[ U 5- nSU;a$[R("USS US4US4/S S S 9US'SU;aUR*/U-US-US'SU;aS/U-US-US'SU;a[R("USS US4/S9US'UR,/U-U -XR S'U$)a Pad encoded inputs (on left/right and up to predefined length or max length in the batch) Args: encoded_inputs: Dictionary of tokenized inputs (`List[int]`) or batch of tokenized inputs (`List[List[int]]`). max_length: maximum length of the returned list and optionally padding length (see below). Will truncate by taking into account the special tokens. padding_strategy: PaddingStrategy to use for padding. - PaddingStrategy.LONGEST Pad to the longest sequence in the batch - PaddingStrategy.MAX_LENGTH: Pad to the max length (default) - PaddingStrategy.DO_NOT_PAD: Do not pad The tokenizer padding sides are defined in self.padding_side: - 'left': pads on the left of the sequences - 'right': pads on the right of the sequences pad_to_multiple_of: (optional) Integer if set will pad the sequence to a multiple of the provided value. This is especially useful to enable the use of Tensor Core on NVIDIA hardware with compute capability `>= 7.5` (Volta). return_attention_mask: (optional) Set to False to avoid returning attention mask (default: set to model specifics) leftrNrjrg?r)dtype)axis)rrconstantT) pad_widthmodeconstant_valuestoken_type_idsspecial_tokens_mask)r)rrrrrmodel_input_namesrr LONGEST DO_NOT_PADrnponestrilbool_arangeint64 concatenatezerosstackpadpad_token_type_idr)rrrrrr bos_token_id mask_token_idrrequired_input seq_lengthneeds_to_be_paddedcontext_lengthrrr mask_positionblock_position_ids differences r_padChatGLMTokenizer._padls7@((8 ))$//: **4+;+;<  F***'(>(>q(AB( 66 6^,J  !&8&D/141Q6:LMJ.1K1KK*PS Q)Q*  !~51%3%9%9,%GN%/N!#!Z)D!E!#!889q!_n_45!#.3*>!?3A/0^3!yy288D .;.M]Sa /$2$8$8$DM4AL1%'^^HH^288<II:6:"((L5&" 24!#56Q2@~. #c.&99J>13566"#34% AQH#$( 4*/0  >1**44!/0@!A4B/0%699!/0E!F9G45/13">2% A729~.(()J6G 11 r) rrrrrrrrrP) FFrNrOzrLrMrr)ryN)FTr)%rArBrCrD__doc__r&PRETRAINED_POSITIONAL_EMBEDDINGS_SIZESmax_model_input_sizesrrrrrErrrrrrrrboolrrrrrrr rrr rdictrrF __classcell__)rs@rrrs &'78BG %#""!$&$"'%#%%N< << :hsm::,, -259 3 d3i0 3%) 3/3 3! 3((<04$c$"$s),$9=S $R%),;,F,F,004 fd3 #45}DEfSMf* f %SM f (~ f ffrr)rrtypingrrrrnumpyr sentencepiecertransformers.tokenization_utilsr$transformers.tokenization_utils_baserr transformers.utilsr modelscope.utilsr logging get_loggerrr rIrrGrrr*sf' ..?L..    *& :u>u>pi*ir