9iZJSSKJrJr SSKrSSKJr SSKJr "SS\5rg))AnyDictN)ModeKeys)OfaBasePreprocessorc^\rSrSrSr\R 4U4SjjrS\\ \ 4S\\ \ 44Sjr S\\ \ 4S\\ \ 44U4Sjjr S\\ \ 4S\\ \ 44U4Sjjr S rS rU=r$) OfaSummarizationPreprocessor z+ OFA preprocessor for summarization tasks. c:>[[U] "XU/UQ70UD6 g)zpreprocess the data Args: cfg(modelscope.utils.config.ConfigDict) : model config model_dir (str): model path, mode: preprocessor mode (model mode) N)superr __init__)selfcfg model_dirmodeargskwargs __class__s j/var/www/html/land-doc-ocr/venv/lib/python3.13/site-packages/modelscope/preprocessors/ofa/summarization.pyr %OfaSummarizationPreprocessor.__init__s8 *   T D48 DURU5nUSR5n[TU] X0RS9nUR SS5R SS5nUR USS9US'URUSS S R55n[R"URU/5US 'U$) aQ Building training samples. step 1. Preprocess the data using the logic of `_build_infer_sample` and make sure the label data in the result. step 2. Preprocess the label data. Contains: - Get the lower case of label, and using `pre_caption` function to do the str preprocessing as new input label. - Tokenize the new input label as `target` for model input. - Add noise to the `target` - Calculate the `prev_output_tokens` from noise `target` for model input. Args: data (`Dict[str, Any]`): Input data, should contains the key of `image`, `prompt` and `label`, `image` refers the image input data, `prompt` refers the text input data and the `label` is the supervised data for training. Return: A dict object, contains source, image, mask, label, target tokens, and previous output tokens data. label max_words[unk]unkF)add_bostargetNprev_output_tokens) rlowerr pre_captionmax_tgt_lengthreplace tokenize_textadd_noise_to_tgtclonetorchcatbos_item)rrsample target_strr(noise_target_itemrs rr0OfaSummarizationPreprocessor._build_train_sample%s*))$/G_**, $Z;N;N$O/77G--fe-Dx 11 8 Sb ! ' ' )+',yy ]]- .(0#$ rc>[TU]XRSURS9nUR SS5R SS5nUR R RSS5nURU5nURU5nURS:Xa URnOURS :XaUS S nO[eUUS .nS UR;a'URS U;aXRS US'U$)a Building inference samples. step 1. Preprocessing the input text via `pre_cation` function, see more details from the doc of `pre_cation`. step 2. Uniform the unknown token, such as `` -> `unk` and `` -> `unk`. step 3. Get the prompt from input, concatenate with the input text, as new input. step 4. Tokenize the input text and generate the decoder prompt. step 5. Determine Whether or not to add labels to the sample. Args: data (`Dict[str, Any]`): Input data, should contains the key of `image` and `prompt`, the former refers the image input data, and the later refers the text input data. Return: A dict object, contains text, decoder prompt and label data. textr"r$r%r&promptz, " {} " Summarize the article with a title: none prev_outputNr))sourcedecoder_promptsummaryr!) r r, column_mapmax_src_lengthr.rmodelgetformatr/ prompt_typer4NotImplementedError) rrr>r;r:inputsr?r5rs rr0OfaSummarizationPreprocessor._build_infer_sampleEs"$ ( )T5H5H%J/77G## DF}}V$##D)   v %!]]N    .#CR[N% %,   'DOOI,F$,N"??9#=>F7O rc[R"URS55R5URR R SS5:n[R"S[UR5URR R SS5- URR R SS5- UR54S 9X'U$) a5 Add noise token to the target sentence. step 1. Sampling from uniform distribution to randomly select the noise indices. step 2. Sampling from normal distribution as noise token to replace the relative token in the target. Args: target: A sequence of tokens. Returns: A sequence of tokens. r noise_ratiog num_codesi num_binsi)size) r2 FloatTensorrOuniform_rrCrDrandintlensrc_dictsum)rr( noise_indicess rr0-OfaSummarizationPreprocessor.add_noise_to_tgtks)) KKN$HJ););s*$$ !&  !3!3K!F Fhhnn  T2 3##%( !*  r)__name__ __module__ __qualname____firstlineno____doc__r INFERENCEr rstrrrrrr0__static_attributes__ __classcell__)rs@rr r s((D 2T#s(^2S#X2 S#X4S>@$S#X$4S>$Lrr ) typingrrr2modelscope.utils.constantrbaserr rXrrres  .%w#6wr