9iSrSSKJrJrJrJrJrJr SSKJ r SSK J r SSK J r Jr SSKJr SSKJr \"5rS\R(0rS00rS S S .rS S 0S S 0S .r"S S\ 5rg)zTokenization classes for PoNet ) TYPE_CHECKINGAnyDictListOptionalUnion)PaddingStrategy) BertTokenizer) BatchEncoding EncodedInput) ModelFile) get_logger vocab_filei)z nlp_ponet_fill-mask_chinese-basez nlp_ponet_fill-mask_english-base do_lower_caseTc \rSrSrSr\r\r\ r \ r S\ RSS4S\\\\4\4S\\S\ S\\S\\S \4 S jjrS rg) PoNetTokenizer0am Construct an PoNet tokenizer. Based on BertTokenizer. This tokenizer inherits from :class:`~transformers.BertTokenizer` which contains most of the main methods. Users should refer to this superclass for more information regarding those methods. Refer to superclass :class:`~transformers.BertTokenizer` for usage examples and documentation concerning parameters. Nencoded_inputs max_lengthpadding_strategypad_to_multiple_ofreturn_attention_maskreturncLUcSUR;nXRSnU[R:Xa [U5nUbUbX$-S:wa X$-S-U-nU[R:g=(a [U5U:gnU(GazU[U5- nUR S:XaU(aS/[U5-S/U--US'SU;aUSUR /U--US'SU;aUSS/U--US'SU;aUSUSSS-/U--US'X`R/U--XRS'U$UR S :XaU(aS/U-S/[U5--US'SU;aUR /U-US-US'SU;aUSSS-/U-US-US'SU;aS/U-US-US'UR/U-U-XRS'U$[S [UR 5-5eU(aSU;aS/[U5-US'U$) a Pad encoded inputs (on left/right and up to predefined length or max length in the batch) Args: encoded_inputs: Dictionary of tokenized inputs (`List[int]`) or batch of tokenized inputs (`List[List[int]]`). max_length: maximum length of the returned list and optionally padding length (see below). Will truncate by taking into account the special tokens. padding_strategy: PaddingStrategy to use for padding. - PaddingStrategy.LONGEST Pad to the longest sequence in the batch - PaddingStrategy.MAX_LENGTH: Pad to the max length (default) - PaddingStrategy.DO_NOT_PAD: Do not pad The tokenizer padding sides are defined in self.padding_side: - 'left': pads on the left of the sequences - 'right': pads on the right of the sequences pad_to_multiple_of: (optional) Integer if set will pad the sequence to a multiple of the provided value. This is especially useful to enable the use of Tensor Core on NVIDIA hardware with compute capability >= 7.5 (Volta). return_attention_mask: (optional) Set to False to avoid returning attention mask (default: set to model specifics) attention_maskrrighttoken_type_idsspecial_tokens_mask segment_idsleftzInvalid padding strategy:) model_input_namesr LONGESTlen DO_NOT_PAD padding_sidepad_token_type_id pad_token_id ValueErrorstr) selfrrrrrrequired_inputneeds_to_be_padded differences h/var/www/html/land-doc-ocr/venv/lib/python3.13/site-packages/modelscope/models/nlp/ponet/tokenization.py_padPoNetTokenizer._pad@s2@ ! ($48N8N$N !'(>(>q(AB 66 6^,J  !&8&D/141Q6:LMJ.1K1KK*PS Q)Q* #c.&99J  G+(89sS&>(8(+,# *:8;N#34#~5&'78112Z?@##34)N:+KK55 41""f,(89sZ7GKN+K,8,N#34$~5..8"8#%34D%E8FN#34!N25CM5RSU5VYZ5Z4[^h4h4B=4Q5RN=1(N:="=#%34I%J=KN#89 ,,- :^K55  !!<#&t'8'8#9":;; "'7~'M01sS5H/HN+ ,)__name__ __module__ __qualname____firstlineno____doc__VOCAB_FILES_NAMESvocab_files_namesPRETRAINED_VOCAB_FILES_MAPpretrained_vocab_files_map&PRETRAINED_POSITIONAL_EMBEDDINGS_SIZESmax_model_input_sizesPRETRAINED_INIT_CONFIGURATIONpretrained_init_configurationr r&rrr+r r rintbooldictr1__static_attributes__r4r3r0rr0s*!;B$A! %),;,F,F,004 \d3 #45}DE\SM\* \ %SM \ (~ \ \\r3rN)r9typingrrrrrrtransformers.file_utilsr *transformers.models.bert.tokenization_bertr transformers.tokenization_utilsr r modelscope.utils.constantr modelscope.utils.loggerrlogger VOCAB_FILEr:r<r>r@rr4r3r0rNs &BB3DG/. !9#7#78*B/),(+*& ) ) !l]lr3