9i7SSKrSSKrSSKrSSKrSSKJr SSKJrJr SSK r SSK J s J r SSKr SSKrSSK JrJrJrJ r SSKJrJr SSKJr SSKJrJrJrJrJr SSKJrJ r J!r!J"r"J#r#J$r$J%r%J&r&J'r' SS K(J)r) SS K*J+r+ SS K,J-r- SS K.J/r/ SS K0J1r1J2r2 SSK3J4r4 SSK5J6r6 SSK7J8r8 SSK9J:r: SSK;Jr> \RZR5 \-R"5rASrBSrCSrD"SS\ R5rF"SS\ R5rG"SS\ R5rH"SS\ R5rI"SS\ R5rJ"S S!\ R5rK"S"S#\ R5rL"S$S%\ R5rM"S&S'\ R5rN"S(S)\ R5rO"S*S+\ R5rP"S,S-\ R5rQ"S.S/\ R5rR"S0S1\ R5rS"S2S3\ R5rT"S4S5\)5rU\"S6S7\55rV"S8S9\U5rW"S:S;\U5rX"S<S=\U5rY"S>S?\ R5rZ"S@SA\U5r["SBSC\U5r\"SDSE\U5r]"SFSG\U5r^"SHSI\U5r_"SJSK\U5r`"SLSM\U5ra"SNSO\2\)5rb\4R"\8R\/RSP9"SQSR\b55rfg)SN) dataclass)OptionalTuple)Tensordevicedtypenn)CrossEntropyLossMSELoss)ACT2FN) ModelOutputadd_code_sample_docstringsadd_start_docstrings%add_start_docstrings_to_model_forwardreplace_return_docstrings) )BaseModelOutputWithPastAndCrossAttentions,BaseModelOutputWithPoolingAndCrossAttentions!CausalLMOutputWithCrossAttentionsMaskedLMOutputMultipleChoiceModelOutputNextSentencePredictorOutputQuestionAnsweringModelOutputSequenceClassifierOutputTokenClassifierOutput)PreTrainedModel) BertConfig)logging)Models)Model TorchModel)MODELS)AttentionBackboneModelOutput)Tasks)parse_labels_in_order)apply_chunking_to_forward find_pruneable_heads_and_indicesprune_linear_layerr BertTokenizerc SSKnSSKnSSKn[ RRU5n[RSRU55 URRU5n/n/n UHfup[RSRX55 URRXj5n URU 5 U RU 5 Mh [X5GHupU R!S5n [#SU 55(a5[RSRSR%U 555 McUn U HnUR'S U5(aUR!S U5nOU/nUSS :Xd USS :Xa [)U S 5n OZUSS:Xd USS:Xa [)U S5n O;USS:Xa [)U S 5n O%USS:Xa [)U S5n O[)XS5n [-U5S:dM[/US5nU Un M WSSS:Xa [)U S 5n OUS :XaUR1U 5n U R2U R2:Xd"SU R2SU R2S35e[RSRU 55 [8R:"U 5U lGM U$![a [R S5 ef=f![*a8 [RSRSR%U 555 GMf=f![4a1nU=R6U R2U R24- sleSnAff=f)z'Load tf checkpoints in a pytorch model.rNzLoading a TensorFlow model in PyTorch, requires TensorFlow to be installed. Please see https://www.tensorflow.org/install/ for installation instructions.z(Converting TensorFlow checkpoint from {}z"Loading TF weight {} with shape {}/c3,# UH nUS;v M g7f))adam_vadam_mAdamWeightDecayOptimizerAdamWeightDecayOptimizer_1 global_stepN).0ns k/var/www/html/land-doc-ocr/venv/lib/python3.13/site-packages/modelscope/models/multi_modal/mgeo/backbone.py *load_tf_weights_in_bert..Ms%a  sz Skipping {}z [A-Za-z]+_\d+z_(\d+)kernelgammaweight output_biasbetabiasoutput_weightssquad classifieri _embeddingszPointer shape z and array shape z mismatchedzInitialize PyTorch weight {})renumpy tensorflow ImportErrorloggererrorospathabspathinfoformattrainlist_variables load_variableappendzipsplitanyjoin fullmatchgetattrAttributeErrorlenint transposeshapeAssertionErrorargstorch from_numpydata)modelconfigtf_checkpoint_pathrCnptftf_path init_varsnamesarraysnamer\arraypointerm_name scope_namesnumes r4load_tf_weights_in_bertrr0s& ggoo01G KK:AA'JK''0I E F  8??LM&&w5 T e ! 5) zz#    KK ,,SXXd^< = F||,f55 hhy&9 %h 1~)[^w-F!'84Q=0KNf4L!'62Q#33!'84Q7*!'<8%g1~>G;1$+a.)!#,+, #$<= (gx0G x LL'E  , Y .? }KX Y,  299$?@''. [*\ LA   Q   Z&KK 4 4SXXd^ DE  FFw}}ekk2 2F  s5 K)K4 [TU]5 [R"URUR SS9Ul[R"URUR 5Ul[R"URUR SS9Ul [R"URUR SS9Ul [R"URUR SS9Ul[R"URUR SS9Ul[R"UR"UR SS9Ul[R"UR&UR SS9Ul[+US5(a[R"UR,UR SS9Ul[R"UR0UR SS9Ul[R"UR4UR SS9Ul[R8"UR UR:S9Ul[R<"UR>5Ul URCS[DRF"UR5RIS55 [KUSS5Ul&Xl'g) Nr padding_idxprov_vocab_sizeeps position_idsrAposition_embedding_typeabsolute)(super__init__r Embedding vocab_size hidden_sizeword_embeddingsmax_position_embeddingsposition_embeddingstype_vocab_sizetoken_type_embeddingsrel_type_vocab_sizerel_type_embeddingsabsolute_x_vocab_sizeabsolute_x_embeddingsabsolute_y_vocab_sizeabsolute_y_embeddingsrelative_x_vocab_sizerelative_x_embeddingsrelative_y_vocab_sizerelative_y_embeddingshasattrryprov_embeddingscity_vocab_sizecity_embeddingsdist_vocab_sizedist_embeddings LayerNormlayer_norm_epsDropouthidden_dropout_probdropoutregister_bufferr_arangeexpandrWrrcselfrc __class__s r4rGisEmbeddings.__init__}s9 !||   v11q B#%<<0N0N060B0B$D %'\\  " "F$6$6A&G"#%<<  & &(:(:$K %'\\  ( (&*<* J":  ' ': 5"&":":<"H  - -J..|<< 001Fq!GHHI2JK K 001Fq!GHHI2JK K 001Fq!GHHI2JK K 001Fq!GHHI2JK K 001Fq!GHHI2JK K 001Fq!GHHI2JK K 001Fq!GHHI2JK K 001Fq!GHHI2JK K   ..x8 8J ..x8 8J ..x8 8J^^J/ \\*- r)rrrrrcrrrrrrrrrr) NNNNrNNNNNN __name__ __module__ __qualname____firstlineno____doc__rr__static_attributes__ __classcell__rs@r4rtrtzs9Q(V#!"'(!&*&*==rrtcF^\rSrSrSrU4SjrSSjrSrU=r$)BertEmbeddingszGConstruct the embeddings from word, position and token_type embeddings.c>[TU]5 [R"URUR UR S9Ul[R"URUR 5Ul [R"URUR 5Ul [R"UR URS9Ul [R"UR5UlUR#S[$R&"UR5R)S55 [+USS5UlXlg)Nrwrzr|r}rr)rrr rrr pad_token_idrrrrrrrrrrrr_rrrWrrcrs r4rBertEmbeddings.__init__s  !||      ++ -$&<<0N0N060B0B$D %'\\&2H2H282D2D&F"    F$9$9;zz&"<"<=    LL77 8 ? ? H J(/v/H/9(;$ rc UbUR5n OUR5SSn U Sn UcURSS2XZU-24nUc8[R"U [RURR S9nUcUR U5nURU5n XK-n URS:XaURU5n X- n URU 5n URU 5n U $)Nr~rArr) rr|r_rrrrrrrrr)rrrr|rrrrrrrrrrs r4rBertEmbeddings.forwards  #..*K',,.s3K ^  ,,Q-C/EEF.F.FGL  !"[[5::d6G6G6N6NPN   00;M $ : :> J":  ' ': 5"&":":<"H   -J^^J/ \\*- r)rrcrrrrr)NNNNrNNNrrs@r4rrs18#!"'(!&*&*$$rrc\^\rSrSrU4SjrSrSrSrSrSr S Sjr S r U=r $) BertSelfAttentioni-c>[TU]5 XlURUR-S:wa5[ US5(d$[ SURUR4-5eURUl[URUR- 5UlURUR-Ul [R"URUR5Ul U(aa[R"URUR5Ul[R"URUR5UlO`[R"URUR5Ul[R"URUR5Ul[R "UR"5Ul['USS5UlUR(S:XdUR(S:XaGUR*Ul[R,"SUR*-S - UR5UlS Ulg) Nrembedding_sizezLThe hidden size (%d) is not a multiple of the number of attention heads (%d)rr relative_keyrelative_key_queryr@rAF)rrrcrnum_attention_headsr ValueErrorrZattention_head_size all_head_sizer Linearquery encoder_widthkeyvaluerattention_probs_dropout_probrrWrrrdistance_embeddingsave_attentionrrcis_cross_attentionrs r4rBertSelfAttention.__init__/s     : : :a ?(I*I*##V%?%?@AB B $*#=#= #&v'9'9)/)C)C(D$E !558P8PPYYv1143E3EF yy!5!5t7I7IJDH6#7#79K9KLDJyy!3!3T5G5GHDH6#5#5t7I7IJDJzz&"E"EF '.v/H/9(;$  ' '> 9T=Y=Y]q=q+1+I+ID (&(llF222Q6(('*D #$rcXlgNattn_gradients)rrs r4save_attn_gradients%BertSelfAttention.save_attn_gradientsQs,rcUR$rrrs r4get_attn_gradients$BertSelfAttention.get_attn_gradientsTs"""rcXlgr attention_map)rrs r4save_attention_map$BertSelfAttention.save_attention_mapWs*rcUR$rrrs r4get_attention_map#BertSelfAttention.get_attention_mapZs!!!rcUR5SSURUR4-nUR"U6nUR SSSS5$)Nr~rr@rAr)rrrviewpermute)rx new_x_shapes r4transpose_for_scores&BertSelfAttention.transpose_for_scores]sSffhsmt'?'?'+'?'?'AA FFK yyAq!$$rcURU5nUSLn U (aCURURU55n URURU55n UnOUbuURURU55n URURU55n [R "USU /SS9n [R "USU /SS9n O@URURU55n URURU55n URU5n X4n[R "U U RSS55n URS:XdURS:XGaDUR5Sn[R"U[RURS 9RSS5n[R"U[RURS 9RSS5nUU- nURUUR-S- 5nUR!U R"S 9nURS:Xa[R$"S U U5nU U-n OHURS:Xa8[R$"S U U5n[R$"S U U5nU U-U-n U [&R("UR*5- n UbX-n [,R."SS9"U 5nU (a=UR0(a,UR3U5 UR5UR65 UR9U5nUbUU-n[R "UU 5nUR;SSSS 5R=5nUR5SSUR>4-nUR"U6nU(aUU4OU4nUU4-nU$)Nrr@dimrAr~rrrrzbhld,lrd->bhlrzbhrd,lrd->bhlrr) rrrrr_catmatmulr[rrrrrrrrtoreinsummathsqrtrr Softmaxrr register_hookrrr contiguousr)r hidden_statesattention_mask head_maskencoder_hidden_statesencoder_attention_maskpast_key_valueoutput_attentionsmixed_query_layerr key_layer value_layer query_layerattention_scoresrposition_ids_lposition_ids_rdistancepositional_embeddingrelative_position_scoresrelative_position_scores_queryrelative_position_scores_keyattention_probsattention_probs_dropped context_layernew_context_layer_shapeoutputss r4rBertSelfAttention.forwardcs!JJ}5 3$> 11./1I33 013K3N  '11$((=2IJI33DJJ}4MNK >!#4i"@aHI))^A%6 $D!LK11$((=2IJI33DJJ}4MNK//0AB #1!<< (1(;(;B(CE  ' '> 9T=Y=Y]q=q&++-a0J"\\%**$++--1T"a[ #\\%**$++--1T!R[ &6H#'#:#:4777!;$= #7#:#:!''$;$) ++~=+0<<$k3G,I(#36N#N --1EE16$k3G2I./4||$i1E0G,#36T#TWs#s +dii  $ $/&&  %/@ **,-=> $"5"5  # #O 4  ) )$*B*B C#',,"?  &= &I # %O ^..r)rrrrrcrrrrrrrrrNNNNNF) rrrrrrrrrrrrrrs@r4rr-sB $D-#+"%"#aarrc.^\rSrSrU4SjrSrSrU=r$)BertSelfOutputic(>[TU]5 [R"URUR5Ul[R "URURS9Ul[R"UR5Ul gNrz) rrr rrdenserrrrrrs r4rBertSelfOutput.__init__sc YYv1163E3EF    F$9$9;zz&"<"<= rcpURU5nURU5nURX-5nU$rr(rrrr  input_tensors r4rBertSelfOutput.forward5 =1  ]3 }'CD rrr(rrrrrrrrrrs@r4r%r%>rr%cH^\rSrSrSU4SjjrSrSSjrSrU=r$) BertAttentionic>[TU]5 [X5Ul[ U5Ul[ 5Ulgr)rrrrr%outputset pruned_headsrs r4rBertAttention.__init__s0 %fA $V, Erc6[U5S:Xag[XRRURRUR 5up[ URRU5URl[ URRU5URl[ URRU5URl [ URRUSS9URl URR[U5- URlURRURR-URl UR RU5Ulg)NrrAr)rYr&rrrr8r'rrrr6r(runion)rheadsindexs r4 prune_headsBertAttention.prune_headss u:? 7 9900 II ) )4+<+<>  -TYY__eD *499==%@ ,TYY__eD .t{{/@/@%QO )- (E(E I) %"&))"?"?$))B_B_"_  --33E:rc pURUUUUUUU5nURUSU5n U 4USS-n U $)NrrA)rr6) rr r r r rrr self_outputsattention_outputr!s r4rBertAttention.forwards\yy    ! "    ;;|A F##AB'(r)r6r8rFr#) rrrrrr>rrrrs@r4r4r4s+" ;,"#rr4c.^\rSrSrU4SjrSrSrU=r$)BertIntermediatei c>[TU]5 [R"URUR 5Ul[UR[5(a[URUl gURUl gr) rrr rrintermediate_sizer( isinstance hidden_actstrr intermediate_act_fnrs r4rBertIntermediate.__init__ s` YYv1163K3KL f'' - -'-f.?.?'@D $'-'8'8D $rcJURU5nURU5nU$rr(rLrr s r4rBertIntermediate.forwards& =1 00? rrOr1rs@r4rFrF s9rrFc.^\rSrSrU4SjrSrSrU=r$) BertOutputic(>[TU]5 [R"URUR 5Ul[R"UR URS9Ul[R"UR5Ul gr') rrr rrHrr(rrrrrrs r4rBertOutput.__init__sc YYv779K9KL    F$9$9;zz&"<"<= rcpURU5nURU5nURX-5nU$rr+r,s r4rBertOutput.forward$r/rr0r1rs@r4rSrSr2rrScD^\rSrSrU4SjrSSjrSrSrU=r$) BertLayeri+c0>[TU]5 XlURUlSUl[ U5UlX!R:UlUR(aX l [ USS9Ul [U5Ul [U5Ulg)NrAT)r)rrrcchunk_size_feed_forward seq_len_dimr4 attention fusion_layerhas_cross_attention layer_numcrossattentionrF intermediaterSr6)rrcr`rs r4rBertLayer.__init__-s  '-'E'E$&v.$-1D1D$D  # #&N"/4#1D ,V4 ( rc fUbUSSOSnURUUUUUS9n U Sn U SSn U Sn UR(aUcS5e[U5[:XaUR U UUX@R UR R- [U5-XPR UR R- [U5-US9n U Sn XSS-n O UR U UUUUUS9n U Sn XSS-n [URURURU 5nU4U -n X4-n U $)Nr@)rrrrAr~z>encoder_hidden_states must be given for cross-attention layers)r) r]r_typelistrar`rcr^rYr%feed_forward_chunkr[r\)rr r r r rrrself_attn_past_key_valueself_attention_outputsrBr!present_key_valuecross_attention_outputs layer_outputs r4rBertLayer.forward<s9G8R$23423$5X\ !!%   /3 "0" 2!4(2.226  # #(4 v6v v4)*d2*.*=*=$")>>-1[[-E-E,F,/0E,F+GH+NN.2kk.F.F-G-01F-G,HI'8+> +'$;1#= !Ab$AA+/*=*=$")*&7 +>+'$;1#= !%11H1H151M1M151A1A1AC  "W,11rcJURU5nURX!5nU$r)rbr6)rrBintermediate_outputrls r4rgBertLayer.feed_forward_chunks)"//0@A{{#6I r) r]r[rcrar_rbr`r6r\r#) rrrrrrrgrrrs@r4rYrY+s- )$"#BHrrYcF^\rSrSrU4SjrSSjrSrU=r$) BertEncoderic >[TU]5 Xl[R"[ UR 5Vs/sHn[X5PM sn5Ulgs snfr) rrrcr ModuleListrangenum_hidden_layersrYlayer)rrcirs r4rBertEncoder.__init__sK  ]]+01I1I+J K+JaYv !+J KM KsAc ^^U (aSOSn T(aSOSn T(aURR(aSOSnU(aSOSnU S:XaSnURRnOpU S:XaSnURRnOQU S:Xa-URRnURRnOU S:XaSnURRn[ WW5HnUR UnU (aX4-n UbUUOSnUbUUOSm[ URSS5(akUR(aZU(a[RS 5 SnUU4S jn[RRRU"U5UUUUU5nOU"UUUUUTT5nUSnU(a UUS 4- nT(dMU US 4-n M U (aX4-n U (d[S UUU U U455$[UUU U US9$)Nr1textrrfusion multi_modalgradient_checkpointingFzh`use_cache=True` is incompatible with `config.gradient_checkpointing=True`. Setting `use_cache=False`...c>^UUU4SjnU$)Nc>T"/UQTPTP76$rr1)inputsmodulerrs r4custom_forwardJBertEncoder.forward..create_custom_forward..custom_forwards&% 9v 9~ 9&7 99rr1)rrrrs` r4create_custom_forward2BertEncoder.forward..create_custom_forwards9*)rr~rAc30# UH nUcMUv M g7frr1)r2vs r4r5&BertEncoder.forward..s" %q  %s )last_hidden_statepast_key_valuesr  attentionscross_attentions)rcadd_cross_attentionr^rvrurwrWtrainingrGwarnr_utils checkpointtupler)rr r r r rr use_cacheroutput_hidden_states return_dictmodeall_hidden_statesall_self_attentionsall_cross_attentionsnext_decoder_cache start_layer output_layerrx layer_modulelayer_head_maskr layer_outputsrs ` @r4rBertEncoder.forwardsF#7BD$5b4 4;;#B#B HL $-R$ 6>K;;33L W_K;;88L X ++22K;;88L ] "K;;88L{L1A::a=L#$58I$I!.7.CilO%1-7; t{{$<"&--KK/0!&I*!& 6 6 A A),7!"#)* ! !-!"#)*"%! *!,M"}R'8&;;"  &9!!$=('(#c2h 14E E  "!#$ %   9+.+*1   r)rcrw) NNNNNNFFTr}r1rs@r4rrrrs5M"#" j j rrrc.^\rSrSrU4SjrSrSrU=r$) BertPooleric>[TU]5 [R"URUR5Ul[R "5Ulgr)rrr rrr(Tanh activationrs r4rBertPooler.__init__s9 YYv1163E3EF '')rc\USS2S4nURU5nURU5nU$Nr)r(r)rr first_token_tensor pooled_outputs r4rBertPooler.forwards6+1a40 #56  6 r)rr(r1rs@r4rrs$ rrc.^\rSrSrU4SjrSrSrU=r$)BertPredictionHeadTransformi cp>[TU]5 [R"URUR5Ul[ UR[5(a[URUl OURUl [R"URURS9Ul gr') rrr rrr(rIrJrKr transform_act_fnrrrs r4r$BertPredictionHeadTransform.__init__ s YYv1163E3EF f'' - -$*6+<+<$=D !$*$5$5D !   F$9$9;rclURU5nURU5nURU5nU$r)r(rrrPs r4r#BertPredictionHeadTransform.forwards4 =1 --m< }5 r)rr(rr1rs@r4rr s;rrc.^\rSrSrU4SjrSrSrU=r$)BertLMPredictionHeadicH>[TU]5 [U5Ul[R "UR URSS9Ul[R"[R"UR55Ul URURl gNF)r<) rrr transformr rrrdecoder Parameterr_rr<rs r4rBertLMPredictionHead.__init__ sr 4V<yy    1 1? LLV->->!?@ !II rcJURU5nURU5nU$rrrrPs r4rBertLMPredictionHead.forward.$}5  ]3 rr<rrr1rs@r4rrs &rrc.^\rSrSrU4SjrSrSrU=r$)BertOnlyMLMHeadi4cB>[TU]5 [U5Ulgr)rrr predictionsrs r4rBertOnlyMLMHead.__init__6s /7rc(URU5nU$rr)rsequence_outputprediction_scoress r4rBertOnlyMLMHead.forward:s ,,_=  rrr1rs@r4rr4s8!!rrc.^\rSrSrU4SjrSrSrU=r$)BertOnlyNSPHeadi?cn>[TU]5 [R"URS5UlgNr@)rrr rrseq_relationshiprs r4rBertOnlyNSPHead.__init__As'  " &*<*[TU]5 [U5Ul[R "UR S5Ulgr)rrrrr rrrrs r4rBertPreTrainingHeads.__init__Ls4 /7 " &*<*)-D(5$$ %,+/u((/15U..58[TU]U5 XlURS:Xa[ U5UlO[ U5Ul[U5UlU(a [U5OSUl UR5 gr) rrrc gis_embeddingrrrtrrencoderrpooler init_weights)rrcadd_pooling_layerrs r4rBertModel.__init__sc     1 $,V4DO+F3DO"6* ,=j(4  rc.URR$rrrrs r4get_input_embeddingsBertModel.get_input_embeddingss...rc$XRlgrr)rrs r4set_input_embeddingsBertModel.set_input_embeddingss */'rcUR5H7up#URRURR U5 M9 g)z Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base class PreTrainedModel N)itemsrrwr]r>)rheads_to_prunerwr<s r4 _prune_headsBertModel._prune_headss< +002LE LL  u % / / ; ;E B3rr rr is_decoderreturncUR5S:XaUSS2SSS2SS24nGO:UR5S:XGaU(aUupg[R"XsS9nUSSSS24RXgS5USSS2S4:*n U R UR 5n U R SUR S:aVUR SU R S- n [R"[R"XgU 4UU R S9U /SS9n U SS2SSS2SS24USS2SSSS24-nO2USS2SSSS24nO$[S RX!R 55eUR UR S 9nS U- S -nU$) a  Makes broadcastable attention and causal masks so that future and masked tokens are ignored. Arguments: attention_mask (:obj:`torch.Tensor`): Mask with ones indicating tokens to attend to, zeros for tokens to ignore. input_shape (:obj:`Tuple[int]`): The shape of the input to the model. device: (:obj:`torch.device`): The device of the input to the model. Returns: :obj:`torch.Tensor` The extended attention mask, with a the same dtype as :obj:`attention_mask.dtype`. rNr@rrA)rrr~)axiszAWrong shape for input_ids (shape {}) or attention_mask (shape {})rrg) rr_rrepeatrrr\ronesrrM) rr rrr extended_attention_mask batch_sizerseq_ids causal_maskprefix_seq_lens r4get_extended_attention_mask%BertModel.get_extended_attention_masks.    1 $&4Qa]&C #    !Q & )4& ,,zA%dD!m4;;A/29$4-2HI *nn^-A-AB $$Q'.*>*>q*AA%3%9%9&(..q1&2N"'))!JJ!+ H'-&1&7&79(   #K+6a6:Aq7A+BDRSTSWSWYZT[E\+\' +9D$9I*J'S %9%9:< <#:"<"<**#=##&)@#@H"L&&rcU bU OURRn U bU OURRn UbUOURRnU(aU bU OURRn OSn UbUb [ S5eUb"UR 5nUunnURnO[Ub%UR 5SSnUunnURnO3Ub%UR 5SSnUunnURnO [ S5eU bU SSRSOSnUc[R"UUU-4US9nUc$[R"U[RUS 9nURUUUU5nUb[U5[:XaUSR 5unnnOUR 5unnnUU4n [U 5[:Xa"U V!s/sHn!UR!U!5PM n"n!O>U c'[R"U US9n UR!U 5n"OUR!U 5n"OSn"UR#UURR$5nUcUR'UUUUUUUUUUUS 9 n#OUn#UR)U#UUUU"U U U U UUS 9 n$U$Sn%UR*bUR+U%5OSn&U(d U%U&4U$S S-$[-U%U&U$R.U$R0U$R2U$R4S 9$s sn!f)a Args: input_ids (`torch.LongTensor` of shape `((batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`BertTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) attention_mask (`torch.FloatTensor` of shape `((batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) token_type_ids (`torch.LongTensor` of shape `((batch_size, sequence_length)`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0, 1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. [What are token type IDs?](../glossary#token-type-ids) position_ids (`torch.LongTensor` of shape `((batch_size, sequence_length)`, *optional*): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.max_position_embeddings - 1]`. [What are position IDs?](../glossary#position-ids) head_mask (`torch.FloatTensor` of shape `(num_heads,)` or `(num_layers, num_heads)`, *optional*): Mask to nullify selected heads of the self-attention modules. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. inputs_embeds (`torch.FloatTensor` of shape `((batch_size, sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~file_utils.ModelOutput`] instead of a plain tuple. encoder_hidden_states (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*): Sequence of hidden-states at the output of the last layer of the encoder. Used in the cross-attention if the model is configured as a decoder. encoder_attention_mask (`torch.FloatTensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on the padding token indices of the encoder input. This mask is used in the cross-attention if the model is configured as a decoder. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. past_key_values (`tuple(tuple(torch.FloatTensor))` of length `config.n_layers` with each tuple having 4 tensors of shape `(batch_size, num_heads, sequence_length - 1, embed_size_per_head)`): Contains precomputed key and value hidden states of the attention blocks. Can be used to speed up decoding. If `past_key_values` are used, the user can optionally input only the last `decoder_input_ids` (those that don't have their past key value states given to this model) of shape `(batch_size, 1)` instead of all `decoder_input_ids` of shape `(batch_size, sequence_length)`. use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). Others (**kwargs) some additional parameters might passed in from upstream pipeline, which not influence the results. NFzDYou cannot specify both input_ids and inputs_embeds at the same timer~zGYou have to specify either input_ids or inputs_embeds or encoder_embedsrr@rr) rr|rrrrrrrrr) r r r rrrrrrrrAr pooler_outputrr rr)rcrruse_return_dictrrrrr\r_rrrrrerfinvert_attention_mask get_head_maskrvrrrrrr rr)'rrr rr|r rencoder_embedsr rrrrrrr rrrrrrrrrrrrrencoder_batch_sizeencoder_sequence_length_encoder_hidden_shapemaskencoder_extended_attention_maskembedding_outputencoder_outputsrrs' r4rBertModel.forward)s^2C1N-TXT_T_TqTq$8$D KK , , &1%V  "#..*K%0 "J %%F  &',,.s3K%0 "J "))F  '(--/4K%0 "J #**FY  "-"1!3A!6!N 150P0P K1= ! ,)*d2AVBtv?"$;QBWA[A[B>"$;Q$6$;$= *+t3!73 6..t4 603/(/).(*9&262L2L*3,/372L2L*3,//3 +&&y'+{{'D'DF   !##)-+'=)&;&;!!! /   . ,, 2"7#B+/!5#'  *!, $ 7 =A #]3oab6II I;-'+;;)77&11,==   y3sK=)rcrrr)T)NNNNNNNNNNNNNNFr}NNNNNN)rrrrrrrrr rrrZrboolrrrrrs@r4rrs%N /0CJ'&J'16sJ'EKJ'04J'9?J'\"#! ""/r r rrcT^\rSrSrU4SjrSrSrSSjrSrU=r $)BertForPreTrainingic>[TU]U5 [U5Ul[ U5UlUR 5 gr)rrrrrclsrrs r4rBertForPreTraining.__init__ s4  f% '/ rcBURRR$rr/rrrs r4get_output_embeddings(BertForPreTraining.get_output_embeddings(xx##+++rc8XRRlgrr2rnew_embeddingss r4set_output_embeddings(BertForPreTraining.set_output_embeddings+'5$rc  U bU OURRn URUUUUUUU U U S9 n U SSupURX5unnSnUbvUbs[ 5nU"UR SURR 5UR S55nU"UR SS5UR S55nUU-nU (dUU4U SS-nUbU4U-$U$[UUUU RU RS9$)aU labels (:obj:`torch.LongTensor` of shape ``(batch_size, sequence_length)``, `optional`): Labels for computing the masked language modeling loss. Indices should be in ``[-100, 0, ..., config.vocab_size]`` (see ``input_ids`` docstring) Tokens with indices set to ``-100`` are ignored (masked), the loss is only computed for the tokens with labels in ``[0, ..., config.vocab_size]`` next_sentence_label (``torch.LongTensor`` of shape ``(batch_size,)``, `optional`): Labels for computing the next sequence prediction (classification) loss. Input should be a sequence pair (see :obj:`input_ids` docstring) Indices should be in ``[0, 1]``: - 0 indicates sequence B is a continuation of sequence A, - 1 indicates sequence B is a random sequence. kwargs (:obj:`Dict[str, any]`, optional, defaults to `{}`): Used to hide legacy arguments that have been deprecated. Returns: Example: >>> from transformers import BertTokenizer, BertForPreTraining >>> import torch >>> tokenizer = BertTokenizer.from_pretrained('bert-base-uncased') >>> model = BertForPreTraining.from_pretrained('bert-base-uncased') >>> inputs = tokenizer("Hello, my dog is cute", return_tensors="pt") >>> outputs = model(**inputs) >>> prediction_logits = outputs.prediction_logits >>> seq_relationship_logits = outputs.seq_relationship_logits Nr rr|r rrrrr@r~)rrrr r) rcrrr/r rrrr r)rrr rr|r rlabelsnext_sentence_labelrrrr!rrrr total_lossloss_fctmasked_lm_lossnext_sentence_lossr6s r4rBertForPreTraining.forward.sZJ&1%[TU]U5 [USS9Ul[ U5UlUR 5 gNFrrrrrrr/rrs r4rBertLMHeadModel.__init__6  f> "6* rcBURRR$rr2rs r4r3%BertLMHeadModel.get_output_embeddingsr5rc8XRRlgrr2r7s r4r9%BertLMHeadModel.set_output_embeddingsr;rcUbUOURRnU bSn URUUUUUUUUU U U U UUUS9nUSnURU5nU(aUSS2SS2SS24R 5$SnU bUSS2SS2SS24R 5nU SS2SS24R 5n [ US9nU"UR SURR5U R S55nUR URS5S5RS5nUbQ[R"[R"WSS9U-SS9*nUU S :g-RS5nSU- U-UU--nU(dU4US S-nUbU4U-$U$[UUURURUR UR"S 9$) a encoder_hidden_states (:obj:`torch.FloatTensor` of shape :obj:`(batch_size, sequence_length, hidden_size)`, `optional`): Sequence of hidden-states at the output of the last layer of the encoder. Used in the cross-attention if the model is configured as a decoder. encoder_attention_mask (:obj:`torch.FloatTensor` of shape :obj:`(batch_size, sequence_length)`, `optional`): Mask to avoid performing attention on the padding token indices of the encoder input. This mask is used in the cross-attention if the model is configured as a decoder. Mask values selected in ``[0, 1]``: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. labels (:obj:`torch.LongTensor` of shape :obj:`(batch_size, sequence_length)`, `optional`): Labels for computing the left-to-right language modeling loss (next word prediction). Indices should be in ``[-100, 0, ..., config.vocab_size]`` (see ``input_ids`` docstring) Tokens with indices set to ``-100`` are ignored (masked), the loss is only computed for the tokens with labels n ``[0, ..., config.vocab_size]`` past_key_values (:obj:`tuple(tuple(torch.FloatTensor))` of length :obj:`config.n_layers` with each tuple having 4 tensors of shape :obj:`(batch_size, num_heads, sequence_length - 1, embed_size_per_head)`): Contains precomputed key and value hidden states of the attention blocks. Can be used to speed up decoding. If :obj:`past_key_values` are used, the user can optionally input only the last :obj:`decoder_input_ids` (those that don't have their past key value states given to this model) of shape :obj:`(batch_size, 1)` instead of all :obj:`decoder_input_ids` of shape :obj:`(batch_size, sequence_length)`. use_cache (:obj:`bool`, `optional`): If set to :obj:`True`, :obj:`past_key_values` key value states are returned and can be used to speed up decoding (see :obj:`past_key_values`). Returns: Example: >>> from transformers import BertTokenizer, BertLMHeadModel, BertConfig >>> import torch >>> tokenizer = BertTokenizer.from_pretrained('bert-base-cased') >>> config = BertConfig.from_pretrained("bert-base-cased") >>> model = BertLMHeadModel.from_pretrained('bert-base-cased', config=config) >>> inputs = tokenizer("Hello, my dog is cute", return_tensors="pt") >>> outputs = model(**inputs) >>> prediction_logits = outputs.logits NF)r rr|r rr rrrrrrr rrr~rA) reductionrr@)rlogitsrr rr)rcrrr/r r rrrsumr_F log_softmaxrrr rr)rrr rr|r rr rr>rrrrrr rUr soft_labelsalpha return_logitsr!rrlm_lossshifted_prediction_scoresrA loss_distillr6s r4rBertLMHeadModel.forwardsP&1%=:<>=>?;@)AAK &AqrE]--/F')# UHnURST5v M g7f)rN) index_select)r2 past_statebeam_idxs r4r51BertLMHeadModel._reorder_cache..7s(%.",J''844",s")r)rrermreordered_past layer_pasts ` r4_reorder_cacheBertLMHeadModel._reorder_cache4s=J u%.",%. .1 1NrrE)NNNNNNNNNNNNNNTrr}NrF)NN)rrrr"_keys_to_ignore_on_load_unexpectedrrr3r9rrgrqrrrs@r4rHrH~s*3&4'#,6 "#! +B L,059 :rrHc.^\rSrSrU4SjrSrSrU=r$)GisBertLMPredictionHeadi=c >[TU]5 [U5Ul[R "UR USS9Ul[R"[R"U55Ul URURl gr) rrrrr rrrrr_rr<)rrcrrs r4r GisBertLMPredictionHead.__init__?sa 4V<yy!3!3ZeL LLZ!89 !II rcJURU5nURU5nU$rrrPs r4rGisBertLMPredictionHead.forwardLrrrr1rs@r4ruru=s &rruc~^\rSrSrS/rSS/rU4SjrS SjrS SjrSr U=r $) BertForGisMaskedLMiRrr|rIc>[TU]U5 [USS9Ul[ XR 5Ul[ UUR5Ul[ UUR5Ul [ XR5Ul [ XR5Ul [ XR5Ul[ XR5Ul[ XR"5Ul[ XR"5Ul[ XR(5Ul[ XR(5UlXlUR15 grK)rrrrrur cls_geom_idr cls_geom_typer cls_rel_typercls_absolute_position_x1cls_absolute_position_x2rcls_absolute_position_y1cls_absolute_position_y2rcls_relative_position_x1cls_relative_position_x2rcls_relative_position_y1cls_relative_position_y2rcrrs r4rBertForGisMaskedLM.__init__Ys$  f> 26;L;LM4V5;5K5KM3F4:4N4NP(? 00)2%(? 00)2%(? 00)2%(? 00)2%(? 00)2%(? 00)2%(? 00)2%(? 00)2%  rcU bU OURRn URUUUUUUUUU U U U UUUUUS9nUSnURU5n[ 5nU"UR SURR 5U R S55nURURURURURURURURURUR / nUUUSS2SS2S4USS2SS2S4USS2SS2S4USS2SS2S4USS2SS2S4USS2SS2S4USS2SS2S4USS2SS2S4/ n URR"URR$URR&URR&URR(URR(URR*URR*URR,URR,/ n![/UU U!5H?un"n#n$U#cM U""U5n%UU"U%R SU$5U#R S55- nMA U (dU4USS-n&UbU4U&-$U&$[1UUUR2UR4S9$) a labels (:obj:`torch.LongTensor` of shape :obj:`(batch_size, sequence_length)`, `optional`): Labels for computing the masked language modeling loss. Indices should be in ``[-100, 0, ..., config.vocab_size]`` (see ``input_ids`` docstring) Tokens with indices set to ``-100`` are ignored (masked), the loss is only computed for the tokens with labels in ``[0, ..., config.vocab_size]`` Nr rr|r rr!r rrrrr rrrrrr~r@rArrrWr r)rcrrr}r rrr~rrrrrrrrrrrrrrrrRrr r)'rrr rr|r rr!r rr>rrrr rr[r\r]rrrtoken_type_ids_labelrel_type_ids_labelabsolute_position_ids_labelrelative_position_ids_labelr!rrrArB positions_clspositions_labelpositions_sizemyclsmylabelsmysizemyprediction_scoresr6s' r4rBertForGisMaskedLM.forwardusL&1% &IIr) rrrrrr}r~rrrrrrc)NNNNNNNNNNNNNFr}NrFNNNNNNNr) rrrrrsrrrrgrrrs@r4r{r{Rs*3&4'#<"#! ""!$($(5v t6:JJrr{c^\rSrSrS/rSS/rU4SjrSrSrS Sjr S S jr S r U=r $) BertForMaskedLMirr|rIc>[TU]U5 [USS9Ul[ U5UlUR 5 grKrMrs r4rBertForMaskedLM.__init__rOrcBURRR$rr2rs r4r3%BertForMaskedLM.get_output_embeddingsr5rc8XRRlgrr2r7s r4r9%BertForMaskedLM.set_output_embeddingsr;rcfU bU OURRn URUUUUUUUUU U U U UUUUUS9nUSnURU5nU(aU$SnU bF[ 5nU"UR SURR 5U R S55nUbP[R"[R"USS9U-SS9*nUU S:gR5nSU- U-UU--nU (dU4USS-nUbU4U-$U$[UUURURS 9$) a labels (:obj:`torch.LongTensor` of shape :obj:`(batch_size, sequence_length)`, `optional`): Labels for computing the masked language modeling loss. Indices should be in ``[-100, 0, ..., config.vocab_size]`` (see ``input_ids`` docstring) Tokens with indices set to ``-100`` are ignored (masked), the loss is only computed for the tokens with labels in ``[0, ..., config.vocab_size]`` Nrrr~rrVrAr@r)rcrrr/r rrr_rXrYrZrrr r)rrr rr|r rr!r rr>rrrr rr[r\r]rrrr!rrrBrAr`r6s r4rBertForMaskedLM.forwards>&1%L %&)78:?,:NON'*WQR[8F!/!;$& HAG H$!//))   rc URnUSnURRcS5eURURSS45n[R "X&/SS9n[R "US4URR[RURS9n[R "X/SS9nXS.$rrrs r4rg-BertForMaskedLM.prepare_inputs_for_generationks oo *1~{{''3e5ee3%//1E1Ea1H!0LM N#ArJjj"6!:!%!9!9',zz(1(8(8: IIy6A> &IIrrE)NNNNNNNNNNNNNFr}NrFNNNr) rrrrrsrrr3r9rrgrrrs@r4rrs*3&4'#,6 "#! ""-S n6:JJrrcF^\rSrSrU4SjrSSjrSrU=r$)BertForNextSentencePredictionic>[TU]U5 [U5Ul[ U5UlUR 5 gr)rrrrrr/rrs r4r&BertForNextSentencePrediction.__init__s4  f% "6* rc vU bU OURRn URUUUUUUUU U S9 n U Sn URU 5nSnUb2[ 5nU"UR SS5UR S55nU (dU4U SS-nUbU4U-$U$[ UUU RU RS9$)u7 labels (:obj:`torch.LongTensor` of shape :obj:`(batch_size,)`, `optional`): Labels for computing the next sequenåce prediction (classification) loss. Input should be a sequence pair (see ``input_ids`` docstring). Indices should be in ``[0, 1]``: - 0 indicates sequence B is a continuation of sequence A, - 1 indicates sequence B is a random sequence. Returns: Example: >>> from transformers import BertTokenizer, BertForNextSentencePrediction >>> import torch >>> tokenizer = BertTokenizer.from_pretrained('bert-base-uncased') >>> model = BertForNextSentencePrediction.from_pretrained('bert-base-uncased') >>> prompt = "In Italy, pizza served in formal settings, such as at a restaurant, is presented unsliced." >>> next_sentence = "The sky is blue due to the shorter wavelength of blue light." >>> encoding = tokenizer(prompt, next_sentence, return_tensors='pt') >>> outputs = model(**encoding, labels=torch.LongTensor([1])) >>> logits = outputs.logits >>> assert logits[0, 0] < logits[0, 1] # next sentence was random Nr=rAr~r@r) rcrrr/r rrr r)rrr rr|r rr>rrrkwargsr!rseq_relationship_scoresrCrAr6s r4r%BertForNextSentencePrediction.forwardsB&1%F!3!?(* LEK L+#*!//))   rrE NNNNNNNNNNr1rs@r4rrs4##!""&%) C C rrcF^\rSrSrU4SjrSSjrSrU=r$)BertForSequenceClassificationic0>[TU]U5 URUl[U5Ul[ R "UR5Ul[ R"URUR5Ul UR5 gr rr num_labelsrrr rrrrrr?rrs r4r&BertForSequenceClassification.__init__si   ++f% zz&"<"<= ))F$6$68I8IJ rc 0U bU OURRn URUUUUUUUU U S9 n U Sn URU 5n UR U 5n SnUb~UR S:Xa2[ 5nU"U RS5URS55nO<[5nU"U RSUR 5URS55nU (dU 4U SS-nUbU4U-$U$[UU U RU RS9$)a~ labels (:obj:`torch.LongTensor` of shape :obj:`(batch_size,)`, `optional`): Labels for computing the sequence classification/regression loss. Indices should be in :obj:`[0, ..., config.num_labels - 1]`. If :obj:`config.num_labels == 1` a regression loss is computed (Mean-Square loss), If :obj:`config.num_labels > 1` a classification loss is computed (Cross-Entropy). Nr=rAr~r@r) rcrrrr?rr rr rr r)rrr rr|r rr>rrrr!rrWrrAr6s r4r%BertForSequenceClassification.forwards+&&1%[TU]U5 [U5Ul[R "UR 5Ul[R"URS5Ul UR5 g)NrA) rrrrr rrrrrr?rrs r4rBertForMultipleChoice.__init__sV  f% zz&"<"<= ))F$6$6: rc ZU bU OURRn UbURSOURSn Ub!URSUR S55OSnUb!URSUR S55OSnUb!URSUR S55OSnUb!URSUR S55OSnUb1URSUR S5UR S55OSnUR UUUUUUUU U S9 n U Sn UR U 5n URU 5nURSU 5nSnUb[5nU"X5nU (dU4U SS-nUbU4U-$U$[UUU RU RS9$)a< labels (:obj:`torch.LongTensor` of shape :obj:`(batch_size,)`, `optional`): Labels for computing the multiple choice classification loss. Indices should be in ``[0, ..., num_choices-1]`` where :obj:`num_choices` is the size of the second dimension of the input tensors. (See :obj:`input_ids` above) NrAr~rr=r@r) rcrr\rrrrr?r rr r)rrr rr|r rr>rrr num_choicesr!rrWreshaped_logitsrrAr6s r4rBertForMultipleChoice.forward$s*&1%[TU]U5 URUl[USS9Ul[ R "UR5Ul[ R"URUR5Ul UR5 grKrrs r4r#BertForTokenClassification.__init__ssk   ++f> zz&"<"<= ))F$6$68I8IJ rc U bU OURRn URUUUUUUUU U S9 n U Sn URU 5n UR U 5n SnUb[ 5nUbUR S5S:HnU R SUR5n[R"UUR S5[R"UR5RU55nU"UU5nO2U"U R SUR5UR S55nU (dU 4U SS-nUbU4U-$U$[UU U RU RS9$)z labels (:obj:`torch.LongTensor` of shape :obj:`(batch_size, sequence_length)`, `optional`): Labels for computing the token classification loss. Indices should be in ``[0, ..., config.num_labels - 1]``. Nr=rr~rAr@r)rcrrrr?r rrr_wheretensor ignore_indextype_asrr r)rrr rr|r rr>rrrr!rrWrrA active_loss active_logits active_labelsr6s r4r"BertForTokenClassification.forward}sr&&1% > rrcN^\rSrSrS/rU4SjrSSjrSrU=r$)BertForQuestionAnsweringirc>[TU]U5 URUl[USS9Ul[ R "URUR5UlUR5 grK) rrrrrr rr qa_outputsrrs r4r!BertForQuestionAnswering.__init__sU   ++f> ))F$6$68I8IJ rc U bU OURRn URUUUUUUU U U S9 n U Sn URU 5nUR SSS9unnUR S5nUR S5nSnUbUb[ UR55S:aUR S5n[ UR55S:aUR S5nURS5nURSU5 URSU5 [US9nU"X5nU"UU5nUU-S- nU (dUU4U SS-nUbU4U-$U$[UUUU RU RS 9$) a start_positions (:obj:`torch.LongTensor` of shape :obj:`(batch_size,)`, `optional`): Labels for position (index) of the start of the labelled span for computing the token classification loss. Positions are clamped to the length of the sequence (:obj:`sequence_length`). Position outside of the sequence are not taken into account for computing the loss. end_positions (:obj:`torch.LongTensor` of shape :obj:`(batch_size,)`, `optional`): Labels for position (index) of the end of the labelled span for computing the token classification loss. Positions are clamped to the length of the sequence (:obj:`sequence_length`). Position outside of the sequence are not taken into account for computing the loss. Nr=rrAr~r)rr@)r start_logits end_logitsr r) rcrrrrSsqueezerYrclamp_r rr r)rrr rr|r rstart_positions end_positionsrrrr!rrWrrr@ ignored_indexrA start_lossend_lossr6s r4r BertForQuestionAnswering.forwards<&1%[TU]"UR40UD6 [[U]U5 gr)rr name_or_pathr)rrcrrs r4rMGeoPreTrainedModel.__init__( s* ,,77 eT#F+rc[U[R5(akURRR SUR RS9 URb%URRR5 gg[U[R5(axURRR SUR RS9 URb2URRURR5 gg[U[R5(aJURRR5 URRRS5 gg)zInitialize the weightsrrNr)rIr rr9rarrcrr<rrrxrrrs r4r!MGeoPreTrainedModel._init_weights, s fbii ( ( MM   & &dkk;; ' ={{&   &&('  - - MM   & &dkk;; ' =!!- ""6#5#56<<>.  - - KK   " " $ MM   $ $S ).rc<[U[5(aX!lggr)rIrrr~)rrrs r4_set_gradient_checkpointing/MGeoPreTrainedModel._set_gradient_checkpointing> s fk * *,1 ) +rc >URSS5nURSS5n[X#40UD6nUc[S0UD6nU"U5nO[[U]"SSU0UD6nX&lU$)ahInstantiate the model. Args: kwargs: Input args. model_dir: The model dir used to load the checkpoint and the label information. num_labels: An optional arg to tell the model how many classes to initialize. Method will call utils.parse_label_mapping if num_labels not supplied. If num_labels is not found, the model will use the default setting (2 classes). Returns: The loaded model, which is initialized by transformers.PreTrainedModel.from_pretrained model_dirNcfgpretrained_model_name_or_pathr1)popr$rrrfrom_pretrainedr)r/rrr model_argsrcrbrs r4 _instantiate MGeoPreTrainedModel._instantiateB s~&JJ{D1 jj%*9DVD  -*-FKE%5G.7G;EGE# rr1rD)rrrrrrrrsupports_gradient_checkpointingrrrr classmethodrrrrs@r4rr sG L&*#'6&7#,*$2rr) module_namec~^\rSrSrS S\S\S\4U4SjjjrS SjrSr Sr S r U=r $) MGeoib rc finetune_modegis_numc >[TU]U5 X lXl[ XS9UlURS:Xa[ R"UR5n[ USS9UlURR5H nSUl M [R"URURR5Ul[R"X6R5UlUR#5 g)NrL multi-modalF)rrrrcr text_encoderr from_dict gis_encoder parameters requires_gradr rrgis2textrgis_typer) rrcrrrr gis_configparamrs r4r MGeo.__init__e s  * % 9    .#--f.@.@AJ(uMD ))446&+#7IIj&<&<&*kk&=&=?DMLL2H2HIDM rc URS:XaaUb^[U5S:aO/n/nUHEnURUR"SSSS.UD6R5 URUS5 MG U(aUR R UUUS9nOUR R US9nURS:XaUb[U5S:aU/nU/nUVs/sHnURU5PM nn[WWU5H;unnnURURUU-55 URU5 M= [R"US S 9n[R"US S 9n OUnUn UR U UU U USS 9n!U(dU!$[U!RU!RU!RU!RU!R U!R"S 9$s snf)NrrTr{)rrr )rr|r)rrArr~)r r!rrrrrr1)rrYrQrrrrrrRrr_rr"rrr rrr6)"rrr rr|r rr!r rrrrrrr rgis_listgis_tpuse_token_typegis_embsgis_attsgisr(embsattsgtptp_embgegagt merge_embmerge_attentionr)s" r4r MGeo.forward~ s(    .83GCMMHH$$,F,'*,,=,=?$4 56  #00;;#)- <   $00;;# < '     .83GCMM$%D"#D4:;FSdmmC(FF;!(Hf= B DMM"r'23 B> $A.I#ii"5O(I,O++*$/!5# ," "+-??)77+;;)77&11,==   )rCs !" ++.+@@8B&&/,;+<>> ((*    $GTjBIIjZCRYYCLW Wt RYY  1BII1h ryy    X Xvr "))r j   "))&299,!bii!&bii& 9299 9%/%2 $:{$: $:NC #C L ] ,] @|)|~bii*lJ,lJ^|J)|J~M $7M `F $7F RS /S lL !4L ^\ 2\ ~B*oBJFKK@h( h(Ah(r