9i~SrSSKJr SSKJrJrJr SSKrSSKrSSKJ r SSK J r SSK J r SSKJr SS KJr SS KJr SS KJrJr SS KJr SS KJr SSKJr SSKJr SSK J!r! \RD"5r"SS\ RF5r$"SS\RJRL5r'"SS\(5r)Sr*"SS\RJRL5r+"SS\ RF5r,"SS\ RF5r-"SS \ RF5r."S!S"\ RF5r/"S#S$\ RF5r0"S%S&\ RF5r1"S'S(\ RF5r2"S)S*\ RF5r3S+r4S9S,jr5\RlRnS-5r8\RlRnS.5r9\RlRnS/5r:"S0S1\ RF5r;"S2S3\ RF5r<"S4S5\\5r=\R|"\R~\RS69"S7S8\=55rAg):zPyTorch DeBERTa-v2 model.)Sequence)OptionalTupleUnionN)nn) LayerNorm)ACT2FN)PreTrainedModel)softmax_backward_data)Models)Model TorchModel)MODELS)AttentionBackboneModelOutput)logger)Tasks)DebertaV2Configc>^\rSrSrU4SjrSr\S5rSrU=r $) ContextPooler(c>[TU]5 [R"URUR5Ul[ UR5UlXl gN) super__init__rLinearpooler_hidden_sizedense StableDropoutpooler_dropoutdropoutconfigselfr" __class__s i/var/www/html/land-doc-ocr/venv/lib/python3.13/site-packages/modelscope/models/nlp/deberta_v2/backbone.pyrContextPooler.__init__*sF YYv88%88: $V%:%:;  cUSS2S4nURU5nURU5n[URR"U5nU$Nr)r!rr r"pooler_hidden_act)r$ hidden_states context_token pooled_outputs r&forwardContextPooler.forward1sM&ad+  ]3  =1 t{{<<=mL r(c.URR$r)r" hidden_sizer$s r& output_dimContextPooler.output_dim;s{{&&&r()r"rr!) __name__ __module__ __qualname____firstlineno__rr/propertyr4__static_attributes__ __classcell__r%s@r&rr(s!''r(rcH\rSrSrSr\S5r\S5r\S5rSr g)XSoftmaxAae Masked Softmax which is optimized for saving memory Args: input (`torch.tensor`): The input tensor that will apply softmax. mask (`torch.IntTensor`): The mask matrix where 0 indicate that element will be ignored in the softmax calculation. dim (int): The dimension that will apply softmax Example: >>> import torch >>> from transformers.models.deberta_v2.modeling_deberta_v2 import XSoftmax >>> # Make a tensor >>> x = torch.randn([4, 20, 100]) >>> # Create a mask >>> mask = (x > 0).int() >>> # Specify the dimension to apply softmax >>> dim = -1 >>> y = XSoftmax.apply(x, mask, dim) ctX0lUR[R5)nUR U[R "[R "UR5R55n[R"XPR5nURUS5 URU5 U$r*) dimtotorchbool masked_filltensorfinfodtypeminsoftmax masked_fill_save_for_backward)r$inputmaskrBrmaskoutputs r&r/XSoftmax.forward\s''%**%&""5#(<< EKK0H0L0L#MOvxx0E1% v& r(cVURun[XX RU5nUSS4$r) saved_tensorsr rB)r$ grad_outputrQ inputGrads r&backwardXSoftmax.backwardhs1'' )$VXX*02 $$$r(cjSSKJs Jn SSKJnJn UR SX$RSS9nUR SUR SUR S[R"S[RS 9S 9U5URS S9nU"XUUR S[R"[R"UR5R55R5S 95n U"X U5n U"X UUR S[R"S[RS 9S 95$) Nr)rFrKCastLong)to_iSubConstantrrI)value_tByte)torch.onnx.symbolic_helperonnxsymbolic_helpertorch.onnx.symbolic_opset9rFrKopcast_pytorch_to_onnxrDrGint64rHtyperIrJuint8) gr$rOrBsym_helprFrKmask_cast_valuer_maskrQs r&symbolicXSoftmax.symbolicos55C$$ D< 0 i 0 0!))I '')CCr(rsN)r6r7r8r9rqrrr/rWrD_CGraphValuerfloatruror;rsr(r&rrsv%%DEHHNND588>>D!%"78D=BXX^^DDr(rcH^\rSrSrSrU4SjrSrSrS SjrSr Sr U=r $) rzo Optimized dropout module for stabilizing the training Args: drop_prob (float): the dropout probabilities cJ>[TU]5 XlSUlSUlgr*)rr drop_probcount context_stack)r$rr%s r&rStableDropout.__init__s" " !r(cUR(a3URS:a#[RXR 55$U$)zR Call the module Args: x (`torch.tensor`): The input tensor to apply dropout r)trainingrrapply get_context)r$xs r&r/StableDropout.forwards3 ==T^^a/>>!%5%5%78 8r(c SUlSUlgr*)rrr3s r& clear_contextStableDropout.clear_contexts !r(cxURc/UlSUlURHnXlX#lM gr*)rrryrx)r$ryrxcs r& init_contextStableDropout.init_contexts7    %!#D  ##A%LG$r(cBURbUR[UR5:a#URR[ 55 URURnUR UlU=RS- slU$UR $)Nr)rrlenappendrurr!)r$rs r&rStableDropout.get_contextsw    )zzS!3!344"")).*:;$$TZZ0C..CK JJ!OJJ>> !r()rrr)Tr) r6r7r8r9rqrr/rrrr;r<r=s@r&rrs&" " " "r(rc.^\rSrSrU4SjrSrSrU=r$)DebertaV2SelfOutputic>[TU]5 [R"URUR5Ul[ URUR5Ul[UR5Ul gr) rrrrr2rrlayer_norm_epsrhidden_dropout_probr!r#s r&rDebertaV2SelfOutput.__init__sX YYv1163E3EF "6#5#5v7L7LM$V%?%?@ r(cpURU5nURU5nURX-5nU$rrr!rr$r, input_tensors r&r/DebertaV2SelfOutput.forward 5 =1  ]3 }'CD r()rrr!r6r7r8r9rr/r;r<r=s@r&rrsA r(rc:^\rSrSrU4SjrSSjrSrU=r$)DebertaV2Attentionicn>[TU]5 [U5Ul[ U5UlXlgr)rrDisentangledSelfAttentionr$rrQr"r#s r&rDebertaV2Attention.__init__s+ -f5 )&1  r(c URUUUUUUS9nU(aUupxUcUnURXt5n U(aU W4$U $)N) query_states relative_posrel_embeddings)r$rQ) r$r,attention_maskoutput_attentionsrrr self_output att_matrixattention_outputs r&r/DebertaV2Attention.forwardsdii   %%)    &1 #K  (L;;{A $j1 1# #r()r"rQr$FNNNrr=s@r&rrs  $$r(rcb^\rSrSrU4SjrS\R S\R 4SjrSrU=r $)DebertaV2Intermediatei9c>[TU]5 [R"URUR 5Ul[UR[5(a[URUl gURUl gr) rrrrr2intermediate_sizerr| hidden_actstrr intermediate_act_fnr#s r&rDebertaV2Intermediate.__init__;s` YYv1163K3KL f'' - -'-f.?.?'@D $'-'8'8D $r(r,rcJURU5nURU5nU$rrr)r$r,s r&r/DebertaV2Intermediate.forwardCs& =1 00? r(r) r6r7r8r9rrDTensorr/r;r<r=s@r&rr9s(9U\\ellr(rc.^\rSrSrU4SjrSrSrU=r$)DebertaV2OutputiJc >[TU]5 [R"URUR 5Ul[UR UR5Ul[UR5Ul Xl gr) rrrrrr2rrrrrr!r"r#s r&rDebertaV2Output.__init__Ls] YYv779K9KL "6#5#5v7L7LM$V%?%?@  r(cpURU5nURU5nURX-5nU$rrrs r&r/DebertaV2Output.forwardSrr()rr"rr!rr=s@r&rrJsr(rc:^\rSrSrU4SjrSSjrSrU=r$)DebertaV2Layeri[c>[TU]5 [U5Ul[ U5Ul[ U5Ulgr)rrr attentionr intermediaterrQr#s r&rDebertaV2Layer.__init__]s3 +F31&9%f- r(c URUUUUUUS9nU(aUupxURU5n URX5n U(aU W4$U $)NrrrrrrrQ) r$r,rrrrrrrintermediate_output layer_outputs r&r/DebertaV2Layer.forwardcsm >>  /%%) *  +; ( "//0@A{{#6I  *- - r(r)NNNFrr=s@r&rr[s .  r(rc.^\rSrSrU4SjrSrSrU=r$) ConvLayeri~cr>[TU]5 [USS5n[USS5n[USS5Ul[R "UR UR UUS- S-US9Ul[UR UR5Ul[UR5Ul Xl g) Nconv_kernel_size conv_groupsrconv_acttanh)paddinggroups)rrgetattrrrConv1dr2convrrrrr!r")r$r" kernel_sizerr%s r&rConvLayer.__init__s f&8!< 2 F; II        1_*  #6#5#5v7L7LM$V%?%?@  r(cURURSSS5R55RSSS5R5nSU- R5nUR UR S5R UR55S5 [UR"URU55nX$-nURU5RU5nUcUnU$UR5UR5:waEUR5S:Xa URS5RS5nUR S5nURUR5nXs-nU$)Nrrr)rpermute contiguousrErL unsqueezeexpandsizer rr!rrCrBsqueezerI) r$r,residual_states input_maskoutrPlayer_norm_inputrQ output_statess r&r/ConvLayer.forwardsHii --aA6AACDLL q!Z\ Z%%' ,33CHHJ?CT]]#DLL$56*0 01445EF  "M~~#3#7#7#99>>#q(!+!3!3A!6!>!>q!AJ'11!4 #v||4J"/Mr()rr"rrr!rr=s@r&rr~s r(rcV^\rSrSrSrU4SjrSrSrS SjrS Sjr Sr U=r $) DebertaV2Encoderiz8Modified BertEncoder with relative position bias supportc>[TU]5 [R"[ UR 5Vs/sHn[ U5PM sn5Ul[USS5Ul UR(a[USS5Ul URS:aURUl [USS5Ul URS-nURS:aURS-n[R"UUR5Ul[US S 5R!5R#S 5Vs/sHnUR%5PM snUlS UR&;a$[)URUR*S S9Ul[USS5S:a [-U5OSUlSUlgs snfs snf)Nrelative_attentionFmax_relative_positionsrrposition_bucketsrr norm_rel_ebdnone| layer_normT)elementwise_affiner)rrr ModuleListrangenum_hidden_layersrlayerrr r max_position_embeddingsr  Embeddingr2rlowersplitstripr rrrrgradient_checkpointing)r$r"_ pos_ebd_sizerr%s r&rDebertaV2Encoder.__init__s ]]-263K3K-L M-L^F #-L MO ")&2F"N  " "*1&2JB+PD '**Q..4.L.L+$+F4F$KD !66:L$$q(#44q8 "$,,|/5/A/A#CD  V^V<BBDJJ3O O GGIO  4,, ,&""%%#')DN *19K12*467*8If%=A &+#? N$ s GGcUR(aURROSnUb!SUR;aUR U5nU$)Nr)r rweightr r)r$rs r&get_rel_embedding"DebertaV2Encoder.get_rel_embeddingsE7;7N7N,,33TX  %<4;L;L+L!^^N;Nr(c UR5S::aTURS5RS5nX"RS5RS5-nUR5nU$UR5S:XaURS5nU$)Nrrrr)rBrrbyte)r$rextended_attention_masks r&get_attention_mask#DebertaV2Encoder.get_attention_masks    1 $&4&>&>q&A&K&KA&N #47V7V8IbM"N+002N   !Q &+55a8Nr(cUR(aWUcTUbURS5OURS5n[UURS5URURS9nU$)Nr$ bucket_size max_position)r rbuild_relative_positionr r )r$r,rrqs r& get_rel_posDebertaV2Encoder.get_rel_possq  " "|';#/!!5B5G5G5K 2""2& 11!88 :L r(c ^UR5S::aUnO"URS5S:R5nURU5nUR XU5nU(aSOSn T(aSOSn [ U[ 5(aUSn OUn UR5n U n [UR5HupU(aX4-n UR(aLUR(a;U4Sjn[RRRU"U5U UUUU 5n O U"U UUUU TS9n T(aU un nUS:XaURbURXU5n Ub=U n[ U[ 5(a%US-[!UR5:aXS-OSn OU n T(dMU W4-n M U(aX4-n U(d[#SXU 455$[%U U U S 9$) Nrr$rrsc>^UU4SjnU$)Nc>T"/UQTP76$rrs)inputsmodulers r&custom_forwardODebertaV2Encoder.forward..create_custom_forward..custom_forwards%AvA/@AAr(rs)r5r6rs` r&create_custom_forward7DebertaV2Encoder.forward..create_custom_forward sB*)r()rrrrrc30# UH nUcMUv M g7frrs).0vs r& +DebertaV2Encoder.forward..<s"MaMs last_hidden_stater, attentions)rBsumr%r'r/r|rr! enumeraterrrrDutils checkpointrrtupler)r$r,routput_hidden_statesrrr return_dictrall_hidden_statesall_attentionsnext_kvrri layer_moduler8att_ms ` r&r/DebertaV2Encoder.forwards     1 $'J(,,R014:: > W^\-A-A'-J$^-J r(c.[R"SU5n[R"SU5nUSS2S4USSS24- nUS:aUS:a [XbU5nUR[R5nUSU2SS24nUR S5nU$)a6 Build relative position according to the query and key We assume the absolute position of query \(P_q\) is range from (0, query_size) and the absolute position of key \(P_k\) is range from (0, key_size), The relative positions from query to key is \(R_{q \rightarrow k} = P_q - P_k\) Args: query_size (int): the length of query key_size (int): the length of key bucket_size (int): the size of position bucket max_position (int): the maximum allowed absolute position Return: `torch.LongTensor`: A tensor with shape [1, query_size, key_size] rN)rDaranger[rClongr) query_sizekey_sizer+r,q_idsk_ids rel_pos_idss r&r-r-Ws* LLJ 'E LLH %E4.5q>1KQ>"  r(cURURS5URS5URS5URS5/5$)Nrrr$re)rfrg key_layers r&p2c_dynamic_expandrksN >>rr  r(cURUR5SSURS5URS54-5$)Nrr$re) pos_indexp2c_attrjs r&pos_dynamic_expandrosH   GLLN2A.(~~b19>>"3EFG HHr(cJ^\rSrSrSrU4SjrSrSSjrSrSr U=r $) ria Disentangled self-attention module Parameters: config (`DebertaV2Config`): A model config class instance with the configuration to build a new model. The schema is similar to *BertConfig*, for more details, please refer [`DebertaV2Config`] c>[TU]5 URUR-S:wa&[ SURSURS35eURUlURUR-n[ USU5UlURUR -Ul[R"URURSS9Ul [R"URURSS9Ul [R"URURSS9Ul [ USS 5Ul URb URO/Ul[ US S 5UlUR(Ga![ US S 5Ul[ US S 5UlUR"S:aUR$UlUR"UlUR S:aUR Ul[)UR*5UlUR(dSUR;a/[R"URURSS9UlSUR;a0[R"URUR5Ul[)UR25Ulg)NrzThe hidden size (z6) is not a multiple of the number of attention heads ()attention_head_sizeTbias share_att_keyFr r rr rc2pp2c)rrr2num_attention_heads ValueErrorrrs all_head_sizerr query_projkey_proj value_projrv pos_att_typer r r rrrr pos_dropout pos_key_projpos_query_projattention_probs_dropout_probr!)r$r"_attention_head_sizer%s r&r"DisentangledSelfAttention.__init__sg     : : :a ?#F$6$6#78 445Q89 9$*#=#= %11V5O5OO#*63H+?$A !558P8PP))    2 2?    2 2? ))    2 2?%V_eD393F3F3RF//XZ")&2F"N  " " "$+F4F$KD !*1&2JB+PD '**Q..4.L.L+ $ ; ;D $$q($($9$9!,V-G-GHD %%D---(* **D,>,>T)KD%D---*,))F4F4F484F4F+HD'%V%H%HI r(cUR5SSUS4-nURU5nURSSSS5R5RSURS5URS55$)Nrrrrr)rviewrr)r$rattention_heads new_x_shapes r&transpose_for_scores.DisentangledSelfAttention.transpose_for_scoressoffhsm&;; FF; yyAq!$//166r166!978vvbzC Cr(c tUcUnURURU5UR5nURURU5UR5nURUR U5UR5n Sn Sn SUR ;aU S- n SUR ;aU S- n [ R"[ R"URS5[ RS9U -5n [ R"XxRSS55[ R"XRS9- n UR(a%URU5nUR!XxUUU 5n U bX-n U n U R#SURU RS5U RS55n [$R'XS5nUR)U5n[ R"UR#SURS5URS55U 5nUR#SURURS5URS55R+SS SS 5R-5nUR5SSS -nUR#U5nU(aX4$U$) a- Call the module Args: hidden_states (`torch.FloatTensor`): Input states to the module usually the output from previous layer, it will be the Q,K and V in *Attention(Q,K,V)* attention_mask (`torch.ByteTensor`): An attention mask matrix of shape [*B*, *N*, *N*] where *B* is the batch size, *N* is the maximum sequence length in which element [i,j] = *1* means the *i* th token in the input can attend to the *j* th token. output_attentions (`bool`, optional): Whether return the attention matrix. query_states (`torch.FloatTensor`, optional): The *Q* state in *Attention(Q,K,V)*. relative_pos (`torch.LongTensor`): The relative position encoding between the tokens in the sequence. It's of shape [*B*, *N*, *N*] with values ranging in [*-max_relative_positions*, *max_relative_positions*]. rel_embeddings (`torch.FloatTensor`): The embedding of relative distances. It's a tensor of shape [\(2 \times \text{max_relative_positions}\), *hidden_size*]. Nrrwrxrr_r$rrr)r)rr|ryr}r~rrDsqrtrGrrbmm transposerIr rdisentangled_attention_biasrr?rr!rr)r$r,rrrrrrgrj value_layerrel_att scale_factorrxattention_scoresattention_probs context_layernew_context_layer_shapes r&r/!DisentangledSelfAttention.forwardsL  (L// OOL )4+C+CE -- MM- ($*B*BD // OOM *D,D,DF  D%% % A L D%% % A L LL))"-U[[ A !99[2E2E 3||..00  " "!--n=N66{7C7E7CEG  /9 ++00T5M5M1A1F1Fr1J1A1F1Fr1JL #..)92N,,7  _%9%9"%=!0!5!5b!9 ;wq!Q?@8BBL*,  #0"4"4"6s";f"D%**+BC !3 3 r(c P Uc@URS5n[UURS5URURS9nUR 5S:Xa!UR S5R S5nOVUR 5S:XaUR S5nO0UR 5S:wa[ SUR 535eURnUR5RUR5nUSUS-2SS24R S5nUR(aURURU5UR5RURS5UR-SS5nURUR!U5UR5RURS5UR-SS5n OS UR";aXURUR%U5UR5RURS5UR-SS5n S UR";aXURUR'U5UR5RURS5UR-SS5nSn S UR";Ga[(R*"[(R,"W RS 5[(R.S 9U-5n [(R0"XR3S S55n [(R4"X7-SUS-S- 5n [(R6"U S U R9S5R;URS5URS5URS 5/5S 9n X[(R,"XR<S 9- - n S UR";Ga[(R*"[(R,"WRS 5[(R.S 9U-5n URS5URS5:wai[URS5URS5URURS9RUR5nUR S5nOUn[(R4"U*U-SUS-S- 5n[(R0"X(R3S S55n[(R6"US UR9S5R;URS5URS5URS5/5S 9R3S S5nU U[(R,"U UR<S 9- - n U $)Nr$r*rrrrrz2Relative position ids must be of dim 2 or 3 or 4. rwrxrr_)rBindex)rr-r r rBrrzrr^rCdevicervrr|ryrepeatr}rrrrDrrGrrrclampgatherrrrI)r$rgrjrrrr.att_spanpos_query_layer pos_key_layerscorerxc2p_attrfr_posp2c_posrns r&r5DisentangledSelfAttention.disentangled_attention_bias.s    $A2r" 11!88 :L     "'11!4>>qAL    1 $'11!4L    1 $D\EUEUEWDXY $$#((*--k.@.@A '(Q,(9:DDQG   "77/((**0&$$Q'4+C+CCQ+K !55 n-((**0&$$Q'4+C+CCQ+K  ))) $ 9 9%%n5,,!..4f#((+t/G/GG/ )))"&";";''7,,#..4f#((+t/G/GG/  D%% %JJ ]//35;;G Eii -D-DR-LMGkk,"91hlQ>NOGllooa(//$$Q'$$Q' %%b)1G u||EGG GE D%% %JJ _11"5U[[I E~~b![%5%5b%99/NN2&NN2& $ 5 5!%![TU]5 [USS5n[USUR5Ul[ R "URURUS9Ul[USS5Ul UR(dSUl O0[ R "URUR5Ul URS:a0[ R "URUR5Ul URUR:wa/[ R"URURSS9Ul[!URUR"5Ul[%UR&5UlXlUR-S [.R0"UR5R3S 55 g) N pad_token_idrembedding_size) padding_idxposition_biased_inputTFrt position_ids)rr)rrrr2rrr vocab_sizeword_embeddingsrposition_embeddingsrtype_vocab_sizetoken_type_embeddingsr embed_projrrrrr!r"register_bufferrDr]r)r$r"rr%s r&rDebertaV2Embeddings.__init__sv v~q9 %f.>&,&8&8:!||   t22  N&-V5L-1&3"))'+D $')||..0C0C(ED $  ! !A %)+f6L6L6:6I6I*KD &   &"4"4 4 ii##V%7%7eEDO"6#5#5v7L7LM$V%?%?@     LL77 8 ? ? H Jr(cUbUR5nOUR5SSnUSnUcURSS2SU24nUc8[R"U[RURR S9nUcUR U5nURb URUR 55nO[R"U5nUn UR(aX- n URRS:aURU5n X- n URURR:waURU 5n UR!U 5n UbUR#5U R#5:waEUR#5S:Xa UR%S5R%S5nUR'S5nUR)U R*5nX-n UR-U 5n U $)NrrrIrrrr)rrrDzerosr^rrr zeros_likerr"rrrr2rrrBrrrCrIr!) r$ input_idstoken_type_idsrrO inputs_embeds input_shape seq_lengthr embeddingsrs r&r/DebertaV2Embeddings.forwards  #..*K',,.s3K ^  ,,Q  ^$>~$N !  /J   $++"9"9 94J^^J/  xxzZ^^--88:?<<?2215D~~a(77:++,D#*J\\*- r() rr"r!rrrrrr)NNNNN) r6r7r8r9rqrr/r;r<r=s@r&rrs(QJB#!" 22r(rcj^\rSrSrSr\rSrS/rS/r Sr U4Sjr Sr S S jr \U4S j5rS rU=r$) DebertaV2PreTrainedModelizz An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models. debertarrTc b>[TU]"UR40UD6 [[U]U5 gr)rr name_or_pathr r$r"kwargsr%s r&r!DebertaV2PreTrainedModel.__init__s* ,,77 eT#F+r(cF[U[R5(akURRR SUR RS9 URb%URRR5 gg[U[R5(axURRR SUR RS9 URb2URRURR5 ggg)zInitialize the weights.g)meanstdN) r|rrr datanormal_r"initializer_rangeruzero_rr)r$r5s r& _init_weights&DebertaV2PreTrainedModel._init_weightss fbii ( ( MM   & &dkk;; ' ={{&   &&('  - - MM   & &dkk;; ' =!!- ""6#5#56<<>..r(c<[U[5(aX!lggr)r|rr)r$r5values r&_set_gradient_checkpointing4DebertaV2PreTrainedModel._set_gradient_checkpointings f. / /,1 ) 0r(c ~>URSS5nUc[S0UD6nU"U5nU$[[U]US9nU$)N model_dir)pretrained_model_name_or_pathrs)poprrr from_pretrained)clsrr ponet_configmodelr%s r& _instantiate%DebertaV2PreTrainedModel._instantiate s[JJ{D1  *4V4L %E  %CL_M  r(rs)F)r6r7r8r9rqr config_classbase_model_prefix_keys_to_ignore_on_load_missing"_keys_to_ignore_on_load_unexpectedsupports_gradient_checkpointingrrr classmethodrr;r<r=s@r&rrsP #L!'5&6#*?)@&&*#, ?2  r(r) module_namec^\rSrSrSrU4SjrSrSrSrSS\ \ RS\ \ RS \ \ RS \ \ RS \ \ RS \ \ S \ \ S\ \ S\ \\44SjjrSrU=r$)DebertaV2ModeliaThe bare DeBERTa_v2 Model transformer outputting raw hidden-states without any specific head on top. The DeBERTa model was proposed in [DeBERTa: Decoding-enhanced BERT with Disentangled Attention](https://arxiv.org/abs/2006.03654) by Pengcheng He, Xiaodong Liu, Jianfeng Gao, Weizhu Chen. It's build on top of BERT/RoBERTa with two improvements, i.e. disentangled attention and enhanced mask decoder. With those two improvements, it out perform BERT/RoBERTa on a majority of tasks with 80GB pretraining data. This model is also a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior. Parameters: config (`DebertaV2Config`): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. c >[TU]U5 [U5Ul[ U5UlSUlXlUR5 gr*) rrrrrencoderz_stepsr" post_initrs r&rDebertaV2Model.__init__-s>  -f5'/    r(c.URR$rrrr3s r&get_input_embeddings#DebertaV2Model.get_input_embeddings7s...r(c$XRlgrr)r$new_embeddingss r&set_input_embeddings#DebertaV2Model.set_input_embeddings:s *8'r(c[S5e)z Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base class PreTrainedModel z7The prune function is not implemented in DeBERTa model.)NotImplementedError)r$heads_to_prunes r& _prune_headsDebertaV2Model._prune_heads=s " EG Gr(rrrrrrrGrHrc rUbUOURRnUbUOURRnUbUOURRnUbUb [ S5eUbUR 5n O"UbUR 5SSn O [ S5eUb UR O UR n Uc[R"XS9nUc$[R"U [RU S9nURUUUUUS9n URU USUUS 9n U S n URS :aU S n[UR5Vs/sHoRRSPM nnU SnURR!5nURR#U5nURR%U 5nUS SHnU"UUS UUUS 9nU R'U5 M! U SnU(dU4U U(aS S-$SS-$[)UU(a U R*OSU R,S9$s snf)u Args: input_ids (`torch.LongTensor` of shape `('batch_size, sequence_length')`): Indices of input sequence tokens in the vocabulary. attention_mask (`torch.FloatTensor` of shape `('batch_size, sequence_length')`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. token_type_ids (`torch.LongTensor` of shape `('batch_size, sequence_length')`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0, 1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. position_ids (`torch.LongTensor` of shape `('batch_size, sequence_length')`, *optional*): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0,config.max_position_embeddings - 1]`. inputs_embeds (`torch.FloatTensor` of shape `('batch_size, sequence_length', hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert *input_ids* indices into associated vectors than the model's internal embedding lookup matrix. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a dataclass instead of a plain tuple. Returns: Returns `modelscope.outputs.AttentionBackboneModelOutput` Examples: >>> from modelscope.models import Model >>> from modelscope.preprocessors import Preprocessor >>> model = Model.from_pretrained('damo/nlp_debertav2_fill-mask_chinese-lite', task='backbone') >>> preprocessor = Preprocessor.from_pretrained('damo/nlp_debertav2_fill-mask_chinese-lite') >>> print(model(**preprocessor('这是个测试'))) NzDYou cannot specify both input_ids and inputs_embeds at the same timerz5You have to specify either input_ids or inputs_embeds)rr)rrrrOrT)rGrrHrr$Frrr?)r"rrGuse_return_dictrzrrrDonesrr^rrrrrr!r'r/rrr,rA)r$rrrrrrrGrHrrembedding_outputencoder_outputsencoded_layersr,rlayersrrrel_posrsequence_outputs r&r/DebertaV2Model.forwardEsr2C1N-TXT_T_TqTq$8$D KK , , &1%V  "#..*K  &',,.s3KGI I&/%:!! @T@T  !"ZZ CN  !"[[5::f>N ??)%' + ,,  !%/# ' )+ <A 'tnA d^A  u22 3A A r(r)rr)Brqcollections.abcrtypingrrrrDtorch.utils.checkpointrtorch.nnrtransformers.activationsr transformers.modeling_utilsr transformers.pytorch_utilsr modelscope.metainfor modelscope.modelsr rmodelscope.models.builderrmodelscope.outputsrmodelscope.utilsrloggingmodelscope.utils.constantr configurationr get_loggerModulerautogradFunctionr?objectrurrrrrrrrrrr[r-jitscriptrhrkrorrrregister_modulebackbone deberta_v2rrsr(r&rs!$)) +7<&/,;.+*    'BII'2DJu~~&&DJPV*%Du~~&&%DR."BII."d "))  "$"$L BII " bii "  RYY  F& &R['ryy['|()+)+BHH w wvU"))Ur-z?-`F4E4EFk -k Gk r(