9i/ASSKrSSKrSSKrSSKrSSKrSSKrSSKrSSKrSSKJ r SSK J r SSK Jr SSKJr SSKJr SSKJr SSKJr SSKJrJr SSKJrJr SS KJrJrJ r SS K!J"r" \""5r#S /r$"S S \%5r&"SS\5r'\RP"\ RR\RTS9"SS \55r+SSjr,g)N) roi_align)Models) TorchModel)MODELS)FPNTrans)LayoutRobertaModelLayoutRobertaPreTrainedModel)TransformerDecoderTransformerDecoderLayer)ModeKeys ModelFileTasks) get_loggerVLDocForDocVLEmbeddingc\rSrSrSrSrg)GeoVLDocModelOutputs c@XlX lX0lX@lXPlg)N text_featurestext_mm_featuresblock_vis_featuresblock_vis_mm_featuresimage_mm_features)selfrrrrrs i/var/www/html/land-doc-ocr/venv/lib/python3.13/site-packages/modelscope/models/multi_modal/vldoc/model.py__init__GeoVLDocModelOutputs.__init__"s"+ 0"4%:"!2)rrrrrN)__name__ __module__ __qualname____firstlineno__r__static_attributes__rrrr s3rrch^\rSrSrSU4SjjrS\4SjrSSjrSrU=r $) GeoVLDocModel6cv>[TU]U5 XlX l[ URSS5(a?URR SS:Xa[ U5UlO![ U5UlO[ U5Ul[URRSS9Ul [R"SS/5Ul [R"SURR5Ul[#URRURR$URR&SS 9n[)US5Ul[#URRURR$URR&SS 9n[)US5UlUR/5 g) N architecturesrrF)img_size inner_vitT) self_attn)superrconfighard_negtive_samplinggetattrr*r text_encoderr image_sizevisual_encodernnAdaptiveAvgPool2dpoolLinear hidden_size vis_linearr num_attention_headsintermediate_sizer cross_modal_textcross_modal_visual init_weights)rr1r2cross_modal_text_layercross_modal_visual_layer __class__s rrGeoVLDocModel.__init__8sZ   %:" 4;; 6 6{{((+/CC$6v$>!$6v$>! 26 :D &[[++u>((!Q0 ))C)@)@A!8 KK # # KK + + KK ) ) " !33I1 M#: KK # # KK + + KK ) ) $ #55M56#8 r ckpt_pathc[R"USS9n0nUR5HupEURSS5nXSU'M UR U5 g)Ncpu map_locationgeo_vl_doc_model.)torchloaditemsreplaceload_state_dict)rrF state_dictstate_dict_newkvs rfrom_pretrainedGeoVLDocModel.from_pretrainedZsUZZ > $$&DA -r2A !1 ' ^,rc hURunnUbUOURRnUUS'URRSS:XaUR"U4UUUU U U UUUS. UD6nOUR"U4UUUU U U UUUS. UD6nUSSunnURunn[ R "SUURS9RUS5RUU5nUUU4nURunnnURU5n[ R "SUURS9RUS5RUU5RS5n[ R"UU4S5RUU-S 5RUS RS 9n US R[ R:Xa{[!US R[ R"5U R[ R"5SUS R%S5S - S 9n!U!RUS RS 9n!OA[!US U R[ R"5SUS R%S5S - S 9n!U!R'S5R'S5RUUS5n!UR)U!5n!U!URS5-n!UR+US 5R'S5R'S5n"UR)U"5RS5n"[ R"U"U!4S5n#[ R,"US45RUR5n$[ R"U$U4S5n%SU- S:n&SU%- S:n'UR/UR1SS5U#R1SS5U&U'S9n(UR3U#R1SS5UR1SS5U'U&S9n)U(R1SS5n(U)R1SS5n)U)SS2SS24n*[5UU(U!U*U)S9$)N line_bboxrr) bboxattention_masktoken_type_ids position_ids head_mask inputs_embedsoutput_attentionsoutput_hidden_states return_dict)devicer-feat_ms)dtypeg@@) spatial_scaler.)tgtmemorytgt_key_padding_maskmemory_key_padding_maskr)shaper1use_return_dictr*r4rMarangerdreshapeexpandr6 unsqueezecattorhfloat16rfloat32sizesqueezer<r9onesr? transposer@r)+r input_idsimagerZbbox_4p_normalizedr[first_token_idxesfirst_token_idxes_maskr\r]r^r_encoder_hidden_statesencoder_attention_maskpast_key_values use_cacher`rarbkwargs batch_sizeseq_lenoutputssequence_output pooled_output_ num_first B_batch_dim feature_bbox block_num visual_out batch_idxsbatch_idx_with_bboxblk_vis_featuresfull_img_featuresvis_inps glb_feat_attnvis_masknew_attention_mask new_vis_mask text_mm_feat vis_mm_featrs+ rforwardGeoVLDocModel.forwardbs*(oo G& ,0KK,G,G #{ ;; $ $Q '+? ?'' '--)#+"3%9'  G'' '--)#+"3%9'  G*1!&).. 9ll z##%%,WZ-.&006z90M  K)::; &,,9a((/ \\ z/"8"8::A'A;%vj)#>"?  i & &%-- 7(9%((7#&&u}}5(388>qAII  3(  ??+;<+.D.N.N /!II y !##*71:ggaj !OO,=>HHK 99/1ABAF J?366y7G7GH 99m-CDaH  .0A5H ) ,,))!Q/%%a+!3$0 -2 --""1a(",,Q2!-$6 . $--a3 !++Aq1 !,AqrE 2#))/"7)   r)r1r?r@r2r9r4r<r6)FNNNNNNNNNNNNNNNNNN) r r!r"r#rstrrVrr$ __classcell__rDs@rr'r'6sY D--#'#"&'+#!"&*'+ $"&%) %Z Z rr') module_namecb^\rSrSrSrS\4U4SjjrSSjrSrU=r $)rz Generate multi-modal document embeddings in segment-level and token-level. Args: model_dir: the path in model hub, e.g., 'damo/multi-modal_convnext-roberta-base_vldoc-embedding' model_dirc n>[T U]"USU0UD6 SSKJn [R R US5n[RSRU55 [R RU5(deURU5Ul [UR5Ul[R R U[R 5n[R RU5(deURR#U5 [RSRU55 SSKJn [R R U[R(5nUR#U5Ul[,R.R15(a8SR[3[R4R7S S555OS Ul[,R.R15(aqURR;UR85 [RS R[3[R4R7S S5555 gURR=5 [RS 5 g) Nrr)LayoutRobertaConfigz config.jsonzLoading config file from {}zLoading model from {})VLDocXLMTokenizerzcuda:{} LOCAL_RANKrHz%Use GPU {} for finetuning & inferencez"Use CPU for finetuning & inference)r0r;modelscope.models.multi_modal.vldoc.modeling_layout_robertarospathjoinloggerinfoformatexistsfrom_json_filer1r' doc_modelr TORCH_MODEL_FILErV0modelscope.models.multi_modal.vldoc.tokenizationrTOKENIZER_FOLDER tokenizerrMcuda is_availableintenvirongetrdrufloat) rrargsrrmodel_cfg_path model_pathrtokenizer_pathrDs rrVLDocForDocVLEmbedding.__init__ s t>9>v> di? 188HIww~~n----)88H &t{{3WW\\)Y-G-GH ww~~j)))) &&z2 +22:>? Wi1K1KL*::>J#(**"9"9";"; &&s2::>> !,(AF :: " " $ $ NN  dkk * KK?FFBJJNN<346 7 NN " KK< =rc UR"S0SU_SU_SU_SU_SU_SU_SU_SU_S U _S U _S U _S U _S U _SU_SU_SU_SU_SU_UD6n[URURS9$)a Args: - input_ids: :math:`(B, T, E)`, the input tokens, where B is the batch size, T is the max token size, E is the embedding dimension. - image: :math:`(B, C, H, W)`, normalized images. - bbox: :math:`(B, T, 4)`, segment boxes denoted by top-left and bottom-right vertexes whose values are normalized to [0, 1000). - bbox_4p_normalized: :math:`(B, T, 8)`, word boxes denoted by 4 vertexes, whose values are normalized to [0, 1). - attention_mask: :math:`(B, T)`, mask for input tokens, where 0 means masked. - first_token_idxes: :math:`(B, S)`, indexes of the corresponding first tokens of all segments, where S is the max segment size. - first_token_idxes_mask: :math:`(B, S)`, mask for segments, where 0 means masked. Optional: - line_rank_id: :math:`(B, T)`, orders of segments. - line_rank_inner_id: :math:`(B, T)`, BIE-like tags. To be more specific, please refer to the class `TextLayoutSerializer` in `modelscope/models/multi_modal/vldoc/processing.py`. r|r}rZr~r[rrr\r]r^r_rrrrr`rarb) img_embeddingtext_embeddingr%)rdictrr)rr|r}rZr~r[rrr\r]r^r_rrrrr`rarbr vldoc_outputss rrVLDocForDocVLEmbedding.forward*sR 2  *  0 $:*& (#8$:,  0!""6#$$' *'99(99  r)r1rdrrr) r r!r"r#__doc__rrrr$rrs@rrrsY>#>D#'#"&'+#!"&*'+ $"&%) %A A rc^^ ^^^^Tc[R"USS9m/n/n[TR55nUS:Xa[ [ U55HtnXxn Sn U R S5(a(U RSS5n [R"U 5n U (dMPURXx5 URU 5 Mv [XV5HupTRU 5TU 'M /m/m/m [TSS5mTR5mTbTTlSU UUUUU4SjjmSn [US 5(d'[!S TR555(aS n T"X S 9 [ T5S :a9["R%SR'UR(R*T55 [ T5S :a9["R%SR'UR(R*T55 [ T 5S :a>[-SR'UR(R*SR/T 555eU$)NrHrIrobertazroberta.zgeo_vl_doc_model.text_encoder. _metadatarLc >Tc0OTRUSS05nURT XSTT T5 URR5Hup4UcM T"XAU-S-5 M g)NreT.)r_load_from_state_dict_modulesrO) moduleprefixlocal_metadatanamechild error_msgsrNmetadata missing_keysrRunexpected_keyss rrN$init_pretrained_weight..loadso'/X\\ 3BK6$$Z%1?J P!??002KD UTMC/03rgeo_vl_doc_modelc3B# UHoRS5v M g7f)rKN) startswith).0ss r )init_pretrained_weight..s 6J7H!LL, - -7HsrK)rrz7Weights of {} not initialized from pretrained model: {}z0Weights from pretrained model not used in {}: {}z*Error(s) in loading state_dict for {}: {}z )rL)rMrNlistkeysrangelenrrPcopydeepcopyappendzippopr3rhasattranyrrrrDr RuntimeErrorr)modelpretrained_model_pathrR cache_dir init_backboneold_keysnew_keysstate_dict_keysikeynew_keyold_key start_prefixrrNrrrs ` @@@@@rinit_pretrained_weightrnsZZ 5EJ HH:??,-O !s?+,A!$CG~~j))++j&FHmmG,w 23(- 3(nnW5 74LOJz;5H"J' 11L 5, - -#6J7A7H6J3J3J* $ <1 E L L((, 8 9 ?a FMM OO $ $o7 8 : : A A((&++j*A CD D Lr)NNr)-rloggingmathrresysjsonrMtorch.distributed distributeddisttorch.nnr7torchvision.opsrmodelscope.metainformodelscope.modelsrmodelscope.models.builderr2modelscope.models.multi_modal.vldoc.conv_fpn_transrrrr 5modelscope.models.multi_modal.vldoc.transformer_localr r modelscope.utils.constantr r rmodelscope.utils.loggerrr__all__objectrr'register_moduledocument_vl_embeddingvldocrrr%rrrs  %&(,G61@@.  # $363,F 0F R33Nk Zk Ok b Cr