9i)bJSSKrSSKrSSKrSSKrSSKrSSKr"SS\5rg)Nc\rSrSrSrSSjrSrSrSrSr S r S r S r S r \S 5r\S5rSSjr\S5r\SSj5r\SSj5r\SSj5rSrSrSrSrSrSrg)Vocaba1 The class used to convert between tokens and ids. It also includes some store/load functions. Args: counter (collections.Counter, optional): A Counter instance describes the tokens and their frequencies. Its keys will be indexed according to the order of frequency sorting to construct mapping relationship. If None, `token_to_idx` must be provided as the mapping relationship. Default: None. max_size (int, optional): Max size of vocab, not including special tokens. Default: None. min_freq (int, optional): Ignore tokens whose frequencies are less than `min_freq`. Default: 1. token_to_idx (dict, optional): A dict specifies the mapping relationship between tokens and indices to be used. If provided, adjust the tokens and indices mapping according to it. If None, counter must be provided. Default: None. unk_token (str, optional): Special token for unknown token. If no need, it also could be None. Default: None. pad_token (str, optional): Special token for padding token. If no need, it also could be None. Default: None. bos_token (str, optional): Special token for bos token. If no need, it also could be None. Default: None. eos_token (str, optional): Special token for eos token. If no need, it lso could be None. Default: None. kwargs (dict): Keyword arguments ending with `_token`. It can be used to specify further special tokens that will be exposed as attribute of the vocabulary and associated with an index. Nc  :^^^ST4SU4SU4SU44n U H upXU 'M /n U R5n[U5nUHUnURS5(d[SR U55eXnUcM<UU ;dMDU R U5 MW UcU(dS5eU HnUU;aM SR U55e UTlUR5VVs0sH unnUU_M snnTlT(aITR Tm[R"U4S j5TlTR RU5 O[U 5VVs0sH unnUU_M snnTl[R"5TlTR RS TRR555 TRXX#5 U(aTRU5 T(aUU4S jTR lU TlU R5H`unnUR#S 5(a [S 5e[%TU5(a[SR UU55e['TUU5 Mb gs snnfs snnf)N unk_token pad_token bos_token eos_token_tokenzc{} is invalid. Only keyword arguments that end in '_token' are supported to declare special tokens.z4token_to_idx should not be None when counter is Nonez{} is not in token_to_idxc>T$N) unk_indexso/var/www/html/land-doc-ocr/venv/lib/python3.13/site-packages/paddlex/inference/models/common/tokenizer/vocab.py Vocab.__init__..jsYc3,# UH upX!4v M g7fr r).0idxtokens r !Vocab.__init__..qs&/I /Isc">TRT$r  _token_to_idx)selfrsrrrxsT=O=O>r_zIt is not allowed to use identifiers starting with underscore. In Python identifier names beginning with underscore are internal.zJvocab.{} already exists. Please choose a different identifier for token {})keyssortedendswith ValueErrorformatappendritems _idx_to_token collections defaultdictupdate enumerate_index_counter_keys+_sort_index_according_to_user_specificationdefault_factory_identifiers_to_tokens startswithhasattrsetattr)rcountermax_sizemin_freq token_to_idxrrr r kwargscombsnamevaluespecial_tokens special_iterspecial_token_name special_tokenrr identifierrs` ` @r__init__Vocab.__init__9s) $ ) $ ) $ ) $   !KD 4L!{{} l+ ". %..x88 1178J1K #6M(].-P%%m4#/ ? W!W WG~=V"=V'9sM]"=V"D "-!8!8!:D     % %&/3/A/A/G/G/I&   $ $Wh Q@@N6""2 '-#!' J$$S)) / tZ(( HHN"EI D*e ,"0/"U "s )J!Jc [UR5SS9nURSSS9 [U5nUcSO U[ U5-nUHupgXt:d[ UR 5U:Xa gXb;dM,UUR [ [UR R55S/-5S-'[ UR R55URU'M g)Nc US$)Nrrxs rr+Vocab._index_counter_keys..sAaDr)keyc US$)NrrCs rrrEsqtrT)rFreverserH) r r%sortsetlenr&maxlistrr)rr2r:r3r4 token_freqsrfreqs rr+Vocab._index_counter_keyssW]]_.A ^T:^,#+4C,>,C,C,E'F"'M#NQR#RS-00B0B0G0G0I,J""5)'rc[UR55RURR55(d [ S5e[ [UR 555[ U5:wa [ S5e[UR 55S:d0[UR 55[ UR5:a-[ SR[ UR555eUR5H[up#URUnURUnX0RU'X@RU'XPRU'X RU'M] g)Nz`User-specified token_to_idx mapping can only contain tokens that will be part of the vocabulary.z3User-specified indices must not contain duplicates.rzUser-specified indices must not be < 0 or >= the number of tokens that will be in the vocabulary. The current vocab contains {}tokens.) rLrissubsetr5r"rMvaluesminrNr#r% idx_to_token)rr5rnew_idxold_idx ousted_tokens rr,1Vocab._sort_index_according_to_user_specificationsD<$$&'001B1B1G1G1IJJ>  s<&&() *c,.? ?RS S |""$ % )S1D1D1F-G3   L .  &T%6%6!78 +002NE''.G,,W5L'.  e $.5  l +)5  g &).  g &3rcSn[U[[[R45(dU/nSn[U[[45(a[R "U5n[U[R5(aF[ UR5S:a-[SR[ UR555e/nUHhn[U[[R45(d![R"S5 [U5nURURU5 Mj U(aUS$U$![ a [SRU55ef=f)u Maps the input indices to token list. Args: indices (int|list[int]|tuple[int]|numpy.ndarray): The input indice(s) for mapping. Must be an `int` or 1D `list[int]`|`tuple[int]`|`numpy.ndarray`. Returns: str|list[str]: Obtained token(s). If `indices` is an integer, it will return a str. If `indices` is a list/tuple of integers, it will return a list of str. Example: .. code-block:: python from paddlenlp.data import Vocab # The vocab file. The sample file can be downloaded firstly. # wget https://bj.bcebos.com/paddlenlp/data/senta_word_dict.txt vocab_file_path = './senta_word_dict.txt' # Initialize the Vocab vocab = Vocab.load_vocabulary( vocab_file_path, unk_token='[UNK]', pad_token='[PAD]') tokens = vocab.to_tokens([0, 1, 2, 3]) print(tokens) # ['[PAD]', '[UNK]', '一斤三', '意面屋'] FTrHzEToken indices is invalid. Expected 1D array, but received {}D array. zfThe type of `to_tokens()`'s input `indices` is not `int` which will be forcibly transferred to `int`. z4Token index {} in the provided `indices` is invalid.r) isinstancerOtuplenpndarrayasarrayrMshaper"r#intintegerwarningswarnr$r&KeyError)rindices to_reducetokensrs r to_tokensVocab.to_tokenss1: 'D%#<==iGI ge} - -jj)G g , ,W]]1Ca1GW^^ &  CcC#455 |#h  d0056&vay161   JQQRUV s E%E+c X$)uc Maps the input tokens into indices. Args: tokens (str|list[str]|tuple[str], optional): The input token(s) for mapping. Returns: int|list[int]: Obationed indice(s). If `tokens` is a str, it will return an integer. If `tokens` is a list/tuple of str, it will return a list of integers. Example: .. code-block:: python from paddlenlp.data import Vocab # The vocab file. The sample file can be downloaded firstly. # wget https://bj.bcebos.com/paddlenlp/data/senta_word_dict.txt vocab_file_path = './senta_word_dict.txt' # Initialize the Vocab vocab = Vocab.load_vocabulary( vocab_file_path, unk_token='[UNK]', pad_token='[PAD]') tokens = vocab.to_indices(['[PAD]', '[UNK]', '一斤三', '意面屋']) print(tokens) # [0, 1, 2, 3] rrrjs r to_indicesVocab.to_indicess :|rc@[U[[45(d7XR;aURU$URUR$UVs/sH:nX R;aURUOURURPM< sn$s snfr )r]rOr^rr)rrjrs r __getitem__Vocab.__getitem__s&4-00///""6* ''7 $  $E 2 22&&u-++DNN;<$  sABc,[UR5$r )rMr&rs r__len__ Vocab.__len__'s4%%&&rcXR;$r r)rrs r __contains__Vocab.__contains__*s****rc X$)z} Maps the input tokens into indices. Its function is the same as the :meth:`to_indices` method. See detail at `to_indices`. rrns r__call__Vocab.__call__-s |rcUR$r )r&rus rrWVocab.idx_to_token6!!!rcUR$r rrus rr5Vocab.token_to_idx;rrch0n[UR5US'[UR5US'URUS'URUS'[ R "U5nU(a2[R"USSS9nURU5 SSS5 U$U$!,(df  U$=f) a Summarizes some information of vocab as JSON string. If path is gaven, the JSON string will be saved into files. The JSON string and the saved file all can be used to reconstruct the :class:`Vocab` by calling :meth:`from_json` method. Args: path (str, optional): The path to save JSON string. If None, the JSON will not be saved. Default: None. Returns: str: The JSON string including information of vocab. Example: .. code-block:: python from paddlenlp.data import Vocab # The vocab file. The sample file can be downloaded firstly. # wget https://bj.bcebos.com/paddlenlp/data/senta_word_dict.txt vocab_file_path = './senta_word_dict.txt' # Initialize the Vocab vocab = Vocab.load_vocabulary( vocab_file_path, unk_token='[UNK]', pad_token='[PAD]') json_str = vocab.to_json(path='./vocab.json') rWr5ridentifiers_to_tokenswutf-8encodingN) dictrWr5rr.jsondumpsioopenwrite)rpath vocab_dictjson_strfs rto_json Vocab.to_json@s8 %)$*;*;%< >"%)$*;*;%< >""&.. ;.2.I.I *+::j) sW5!6x65s B"" B1c[RRU5(a6[R"USSS9n[ R "U5nSSS5 O[ R"U5nWRS5nURS5nURS[55nSU;aUS U"S SUUS.UD6nU$!,(df  Na=f) a Loads :class:`Vocab` from JSON string or JSON file, which is gotten by calling :meth:`to_json` method. Args: json_str (str): JSON string or file path of JSON string. Returns: Vocab: An instance of :class:`Vocab` generated from information contained in JSON string. Example: .. code-block:: python from paddlenlp.data import Vocab # The vocab file. The sample file can be downloaded firstly. # wget https://bj.bcebos.com/paddlenlp/data/senta_word_dict.txt vocab_file_path = './senta_word_dict.txt' # Initialize the Vocab vocab = Vocab.load_vocabulary( vocab_file_path, unk_token='[UNK]', pad_token='[PAD]') json_str = vocab.to_json(path='./vocab.json') vocab1 = Vocab.from_json(json_str) vocab2 = Vocab.from_json('./vocab.json') print(len(vocab), len(vocab1), len(vocab2)) # 1256608 1256608 1256608 rrrNr5rr)r2r5rr) osrisfilerrrloadloadsgetr)clsrrrr5rrvocabs r from_jsonVocab.from_jsongs@ 77>>( # #39Q!YYq\ :9H-J!~~n5 NN;/ */F O / /%k2 % $    :9s C Cc $U"SSUUUUUS.UD6nU$)a Builds the :class:`Vocab` from a dict. Args: token_to_idx (dict): A dict describes the mapping relationship between tokens and indices. unk_token (str, optional): The special token for unknown token. If no need, it also could be None. Default: None. pad_token (str, optional): The special token for padding token. If no need, it also could be None. Default: None. bos_token (str, optional): The special token for bos token. If no need, it also could be None. Default: None. eos_token (str, optional): The special token for eos token. If no need, it also could be None. Default: None. kwargs (dict): Keyword arguments ending with `_token`. It can be used to specify further special tokens that will be exposed as attribute of the vocabulary and associated with an index. Returns: Vocab: An instance of :class:`Vocab` generated from the given dict and special tokens. Example: .. code-block:: python from paddlenlp.data import Vocab # The vocab file. The sample file can be downloaded firstly. # wget https://bj.bcebos.com/paddlenlp/data/senta_word_dict.txt vocab_file_path = './senta_word_dict.txt' # Initialize the Vocab vocab = Vocab.load_vocabulary( vocab_file_path, unk_token='[UNK]', pad_token='[PAD]') vocab1 = Vocab.from_dict(vocab.token_to_idx) print(len(vocab), len(vocab.token_to_idx), len(vocab1)) # 1256608 1256608 1256608 N)r2r5rrr r rr)rr5rrr r r6rs r from_dictVocab.from_dicts7d %    rc [R"5n UHn U RU 5 M [U 4UUUUUUUS.UD6n U $)a Builds the :class:`Vocab` according to given iterator and other information. Firstly, iterate over the `iterator` to construct a :class:`collections.Counter` and used to init the as :class:`Vocab`. Args: iterator (collections.Iterable): Iterator of tokens. Each element should be a list of tokens if wordlevel vocab is needed. max_size (int, optional): The max size of vocab, not including special tokens. Default: None. min_freq (int, optional): Ignore tokens whose frequencies are less than `min_freq`. Default: 1. token_to_idx (dict, optional): A dict specifies the mapping relationship between tokens and indices to be used. If provided, adjust the tokens and indices mapping according to it. If None, counter must be provided. Default: None. unk_token (str, optional): The special token for unknown token ''. If no need, it also could be None. Default: None. pad_token (str, optional): The special token for padding token ''. If no need, it also could be None. Default: None. bos_token (str, optional): The special token for bos token ''. If no need, it also could be None. Default: None. eos_token (str, optional): The special token for eos token ''. If no need, it also could be None. Default: None. kwargs (dict): Keyword arguments ending with `_token`. It can be used to specify further special tokens that will be exposed as attribute of the vocabulary and associated with an index. Returns: Vocab: An instance of :class:`Vocab` generated from given iterator and other informations. Example: .. code-block:: python from paddlenlp.data import Vocab # The vocab file. The sample file can be downloaded firstly. # wget https://bj.bcebos.com/paddlenlp/data/senta_word_dict.txt vocab_file_path = './senta_word_dict.txt' # Initialize the Vocab vocab = Vocab.load_vocabulary( vocab_file_path, unk_token='[UNK]', pad_token='[PAD]') vocab1 = Vocab.build_vocab([list(vocab.token_to_idx.keys())]) print(len(vocab), len(vocab1)) # 1256608 1256608 )r3r4r5rrr r )r'Counterr)r) iteratorr3r4r5rrr r r6r2rjrs r build_vocabVocab.build_vocabsa|%%'F NN6 "   %     rc 0n[R"USSS9n[U5H#upU RS5n [ U5Xj'M% SSS5 [ R "U4UUUUS.UD6n U $!,(df  N,=f)a Builds the :class:`Vocab` from a file reserving all tokens by calling :meth:`Vocab.from_dict` method. The file contains a token per line, and the line index would be the index of corresponding token. Args: filepath (str): the path of file to construct vocabulary. unk_token (str, optional): special token for unknown token. If no need, it also could be None. Default: None. pad_token (str, optional): special token for padding token. If no need, it also could be None. Default: None. bos_token (str, optional): special token for bos token. If no need, it also could be None. Default: None. eos_token (str, optional): special token for eos token. If no need, it also could be None. Default: None. kwargs (dict): Keyword arguments ending with `_token`. It can be used to specify further special tokens that will be exposed as attribute of the vocabulary and associated with an index. Returns: Vocab: An instance of :class:`Vocab` generated from the given file. Example: .. code-block:: python from paddlenlp.data import Vocab # The vocab file. The sample file can be downloaded firstly. # wget https://bj.bcebos.com/paddlenlp/data/senta_word_dict.txt vocab_file_path = './senta_word_dict.txt' # Initialize the Vocab vocab = Vocab.load_vocabulary( vocab_file_path, unk_token='[UNK]', pad_token='[PAD]') print(len(vocab)) # 1256608 rrr N)rrr r )rrr*rstriprcrr) filepathrrr r r6r5rindexlinerrs rload_vocabularyVocab.load_vocabulary$s^ WWXsW 5(|  D)&)%j # ,6       6 5s 3A22 Bc[US5n[[UR55H$nUR URUS-5 M& SSS5 g!,(df  g=f)z Save the :class:`Vocab` to a specific file. Can be reloaded by calling `load_vocabulary`. Args: filepath (str): the path of file to save vocabulary. rrN)rrangerMr&r)rrrrs rsave_vocabularyVocab.save_vocabularybsR(C AS!3!345**3/$676! s AA A+cfURbURUR$UR$r )rrrus rget_unk_token_idVocab.get_unk_token_idm5~~)   t~~ .  rcfURbURUR$UR$r )r rrus rget_bos_token_idVocab.get_bos_token_idtrrcfURbURUR$UR$r )r rrus rget_eos_token_idVocab.get_eos_token_id{rrcfURbURUR$UR$r )rrrus rget_pad_token_idVocab.get_pad_token_idrr)r.r&r)NNrHNNNNNr )NNNN)NrHNNNNN)__name__ __module__ __qualname____firstlineno____doc__r?r+r,rkrorrrvryr|propertyrWr5r classmethodrr staticmethodrrrrrrr__static_attributes__rrrrrs#DS-jK"/8:2x>"'+""""%N//b ::xKKZ ;;z 8    rr) r'rrrrenumpyr_objectrrrrrs' o  Fo  r