9i >SSKrSSKJrJrJrJrJr SSKrSSKrSSK J r SSK J r SSK Jr SSKJrJrJrJrJr SSKJrJrJrJrJrJrJrJrJr SSKJ r SSK!J"r# SS KJ$r$ S r%S r&\'"S 5r("S S \5r)"SS\5r*"SS\5r+Sr,SSjr-Sr.Sr/g)N)AnyDictListOptionalTuple)utils) fsdp_wrap)FairseqEncoderFairseqEncoderDecoderModelFairseqIncrementalDecoderregister_modelregister_model_architecture) AdaptiveSoftmax BaseLayerFairseqDropoutLayerDropModuleList LayerNormPositionalEmbeddingSinusoidalPositionalEmbeddingTransformerDecoderLayerTransformerEncoderLayer)checkpoint_wrapper) quant_noise)TensorigחAc &^\rSrSrSrU4Sjr\S5r\S5r \SSj5r \S5r \S5r SS \ S \ S \\S \\4S jjr\R$R&SS\\\\\\\\44S\ S\\\\44Sjj5rSS\\\\4S\\\\\\44S\4SjjrSS\\\\4S\\\\\\44S\4SjjrS\\\44SjrS\\\\\44SjrS\\\\\\444SjrSr U=r!$) CanmtModela Args: encoder (TransformerEncoder): the encoder decoder (TransformerDecoder): the decoder The CanmtModel provides the following named architectures and command-line arguments: .. argparse:: :ref: fairseq.models.transformer_parser :prog: cb>[TU]X#5 XlSUlX lX0lX@lg)NT)super__init__argssupports_align_argsencoderdecodersecond_decoder)selfr!r#r$r% __class__s g/var/www/html/land-doc-ocr/venv/lib/python3.13/site-packages/modelscope/models/nlp/canmt/canmt_model.pyr CanmtModel.__init__-s- * #'   ,c&URS[R"5SS9 URS[SSS9 URS[SS S9 URS S [SS S9 URS [SSS9 URS[ SSS9 URS[ SSS9 URS[ SSS9 URS[ SSS9 URSSSS9 URSSSS9 URS[SS S9 URS![ SS"S9 URS#[ SS$S9 URS%[ SS&S9 URS'[ SS(S9 URS)SS*S9 URS+SS,S9 URS-[ SS.S9 URS/SS0S9 URS1SS2S9 URS3S4SS5S69 URS7S8S9S:94 URS;[SSS9 URS?SS@S9 URSASSBS9 URSCSSDS9 URSES4SSFS69 URSGS4SSHS69 URSI[SSJSKSL9 URSM[SSJSNSL9 URSOSPSQSR9 URSSSPSQSR9 URST[SSJSUSL9 URSV[ SSWSXSL9 URSY[SSJSZSL9 URS[[ S[ S\SL9 gP)]z+Add model-specific arguments to the parser.z--activation-fnzactivation function to use)choiceshelpz --dropoutDzdropout probability)typemetavarr-z--attention-dropoutz)dropout probability for attention weightsz--activation-dropoutz--relu-dropoutz,dropout probability after activation in FFN.z--encoder-embed-pathSTRz%path to pre-trained encoder embeddingz--encoder-embed-dimNzencoder embedding dimensionz--encoder-ffn-embed-dimz#encoder embedding dimension for FFNz--encoder-layersznum encoder layersz--encoder-attention-headsznum encoder attention headsz--encoder-normalize-before store_truez)apply layernorm before each encoder block)actionr-z--encoder-learned-posz0use learned positional embeddings in the encoderz--decoder-embed-pathz%path to pre-trained decoder embeddingz--decoder-embed-dimzdecoder embedding dimensionz--decoder-ffn-embed-dimz#decoder embedding dimension for FFNz--decoder-layersznum decoder layersz--decoder-attention-headsznum decoder attention headsz--decoder-learned-posz0use learned positional embeddings in the decoderz--decoder-normalize-beforez)apply layernorm before each decoder blockz--decoder-output-dimzPdecoder output dimension (extra linear layer if different from decoder embed dimz"--share-decoder-input-output-embedz)share decoder input and output embeddingsz--share-all-embeddingszWshare encoder, decoder and output embeddings (requires shared dictionary and embed dim)z --no-token-positional-embeddingsFz?if set, disables positional embeddings (outside self attention))defaultr4r-z--adaptive-softmax-cutoffEXPRzacomma separated list of adaptive softmax cutoff points. Must be used with adaptive_loss criterion)r0r-z--adaptive-softmax-dropoutz6sets adaptive softmax dropout for the tail projectionsz--layernorm-embeddingzadd layernorm to embeddingz--no-scale-embeddingzif True, dont scale embeddingsz--checkpoint-activationszicheckpoint activations at each layer, which saves GPU memory usage at the cost of some additional computez--offload-activationszUcheckpoint activations at each layer, then save to gpu.Sets --checkpoint-activations.z--no-cross-attentionzdo not perform cross-attentionz--cross-self-attentionzperform cross+self-attentionz--encoder-layerdroprz!LayerDrop probability for encoder)r/r0r5r-z--decoder-layerdropz!LayerDrop probability for decoderz--encoder-layers-to-keepNz=which layers to *keep* when pruning as a comma-separated list)r5r-z--decoder-layers-to-keepz--quant-noise-pqz0iterative PQ quantization noise at training timez--quant-noise-pq-block-sizez1block size of quantization noise at training timez--quant-noise-scalarzBscalar quantization noise and scalar quantization at training timez--min-params-to-wrapadminimum number of params for a layer to be wrapped with FSDP() when training with --ddp-backend=fully_sharded. Smaller values will improve memory efficiency, but may make torch.distributed communication less efficient due to smaller input sizes. This option is set to 0 (i.e., always wrap) when --checkpoint-activations or --offload-activations are passed.) add_argumentrget_available_activation_fnsfloatstrintDEFAULT_MIN_PARAMS_TO_WRAP)parsers r(add_argsCanmtModel.add_args5s  668-  /  eS7L  N !<  >  " ?  A  "8  :  !.  0  %6  8  %  '  '.  0  (<  >  #C  E  "8  :  !.  0  %6  8  %  '  '.  0  #C  E  (<  >  "2  3  0<  >  $:  ;  . M  O  '8  9 :  (I  K  #-  /  "1  3  &B  C  #-  .  "1  3  $/  1  !4  6  !4  6  &P   &P   C  E  )D  F  " P    ".1  3r*c[U5 UR(a)[URRS55UlUR (a)[UR RS55Ul[USS5c [Ul [USS5c [Ul URURpCUR(aX4:wa [S5eUR UR":wa [S5eUR$(a%UR$UR&:wa [S5eUR)XUR UR&5nUnSUlONUR)XUR UR&5nUR)XUR"UR$5n[US S 5(aSUlUR/XU5nUR1XU5nUR1XU5n UR(d#[US [25n [5XzS 9n[5XS 9nU"XX5$) zBuild a new model instance.,max_source_positionsNmax_target_positionsz3--share-all-embeddings requires a joined dictionaryzP--share-all-embeddings requires --encoder-embed-dim to match --decoder-embed-dimz?--share-all-embeddings not compatible with --decoder-embed-pathToffload_activationsFmin_params_to_wrapmin_num_params)base_architectureencoder_layers_to_keeplensplitencoder_layersdecoder_layers_to_keepdecoder_layersgetattrDEFAULT_MAX_SOURCE_POSITIONSrCDEFAULT_MAX_TARGET_POSITIONSrD vocab_src vocab_tgtshare_all_embeddings ValueErrorencoder_embed_dimdecoder_embed_dimdecoder_embed_pathencoder_embed_pathbuild_embedding share_decoder_input_output_embedcheckpoint_activations build_encoder build_decoderr=r ) clsr!tasksrc_dicttgt_dictencoder_embed_tokensdecoder_embed_tokensr#r$r%rFs r( build_modelCanmtModel.build_models) $  & &"%d&A&A&G&G&L"MD   & &"%d&A&A&G&G&L"MD  4/ 6 >(DD % 4/ 6 >(DD %!^^T^^(  $ $# IKK%%)?)?? f&&,,0G0GG U$'#6#6t7;7M7M7;7N7N$P $8 48D 1##6t7;7M7M7;7N7N$P $'#6#6t7;7M7M7;7N7N$P  4. 6 6*.D '##D4HI##D4HI**4+?A((!(/C)C"E  KGKG4'::r*c[U5nUR5n[XSU5nU(a-[R"U5n[R "XU5 U$N)rKpad Embeddingrparse_embeddingload_embedding) r`r! dictionary embed_dimpathnum_embeddings padding_idxemb embed_dicts r(r[CanmtModel.build_embedding9sKZ nn& ;? ..t4J   = r*c[XU5$ri)TransformerEncoder)r`r!rb embed_tokenss r(r^CanmtModel.build_encoderEs!$,??r*c 0[UUU[USS5S9$)Nno_cross_attentionF)no_encoder_attn)TransformerDecoderrP)r`r!rcrxs r(r_CanmtModel.build_decoderIs%!   #D*>F   r*return_all_hiddens features_onlyalignment_layeralignment_headsc URUUUS9n URUU UUUUUS9n URUSUSUUUUS9n U SSn U SSn U /U /S .nURUUUUUSUS9nXU4$) z Run the forward pass for an encoder-decoder model. Copied from the base class, but without ``**kwargs``, which are not supported by TorchScript. ) src_lengthsr) encoder_outrrrrrNT)rrfull_context_alignmentrrrr last_layerself_attn_padding_mask)rencoder_padding_mask)r#r$r%)r& src_tokensrprev_output_tokensprev_src_tokensrrrrr decoder_outdecoder_out_redecoder_out_tensordecoder_padding decoder_kvssrc_outs r(forwardCanmtModel.forwardRs$ll #1#3 ll #'++#1#  '#'++#1&  ,A.|<(+,DE//%4$5 %% #'++1& [00r* net_output log_probssamplec&URXU5$)z@Get normalized probabilities (or log probs) from a net's output.)get_normalized_probs_scriptable)r&rrrs r(get_normalized_probsCanmtModel.get_normalized_probss33J4:< " ll** #0+ "&+& ?{1~9+a.&11"1~)!nV4 k622&D ,&q>DAr1H~ N1bc19 % * *; 71$D+a. )) d*<aQh(^L9...r*cSnUnURRXS9nSn[U5nUS:a[USbU[US[5(aUSnO(USSn [U [5(aU nOU bU SnUb USS2SSS24nUSSS2SS2SS24R U5US::aSOUS4n UR U SSS9n U SS2SSS24n USSn XX4$) N)rrrrrTrr)r%rrKrrrrrs r(forward_decoder_srcCanmtModel.forward_decoder_srcs,:> " ))11 2- "&+& ?{1~9+a.&11"1~)!nV4 k622&D ,&q>DAr1H~ N1bc19 % * *; 71$D+a. )) d*<aQh(^L9.;;r* net_inputcUR5VVs0sHup#US:wdM US:wdMUS:wdMX#_M! nnnURRU5$s snnf)Nrrsources)itemsr#forward_torchscript)r&rkv encoder_inputs r(forward_encoderCanmtModel.forward_encoderso") )Q2F-F && +, > AD)  ||// >>  s AAAAcBUceURRX5$) Reorder encoder output according to *new_order*. Args: encoder_out: output from the ``forward()`` method new_order (LongTensor): desired order Returns: *encoder_out* rearranged according to *new_order* )r#reorder_encoder_out)r&r new_orders r(rCanmtModel.reorder_encoder_outs%'''||// HHr*c:URRX5 gri)r$#reorder_incremental_state_scripting)r&rrs r(reorder_incremental_state$CanmtModel.reorder_incremental_states 88  +r*)r!r$r#r%r"ri)TFNN)?)"__name__ __module__ __qualname____firstlineno____doc__r staticmethodr? classmethodrfr[r^r_boolrr<rtorchjitexportrrrr;rrr:rrrrr__static_attributes__ __classcell__r's@r(rrs6 -F3F3R7;7;r  @@  $(#)-)-:1 ! :1:1"#:1"#:1x YY /3 <&(4T(6:J5K0K+L"MMN<<c6k*+ <<! &/3V ,-&/!d30@+@&A!AB &/  &/Z! #<3V ,-#<!d30@+@&A!AB #<  #B6l?K:L1MI + d30@+@&A!AB++r*rcd^\rSrSrSrU4SjrSrSS\\R4Sjjr SS\\RS\ S \\R4S jjr SS\\RS\ S \\R4S jjr \RRS \\\\ 44S j5rSrSrSrU=r$)rwia- Transformer encoder consisting of *args.encoder_layers* layers. Each layer is a :class:`TransformerEncoderLayer`. Args: args (argparse.Namespace): parsed command-line arguments dictionary (~fairseq.data.Dictionary): encoding dictionary embed_tokens (torch.nn.Embedding): input embedding c>Xl[TU] U5 URS[R "S/55 [ URURRS9Ul URUl URnURUl URUlX0lUR (aSO["R$"U5UlUR((d*[+URUURUR,S9OSUl[1USS5n[1USS5(a[3XES 9UlOSUlUR6(dJUR8S :a:[;[<R>"XDSS 9UR8UR@5Ul!OSUl!URS :a[EURS 9Ul#O[<RH"/5Ul#URFRK[MURN5Vs/sHo`RQU5PM sn5 [SURF5Ul*URV(a[3XES 9Ul,gSUl,gs snf)Nversion module_namerlearnedrFlayernorm_embeddingrrbiasp)-r!rr register_bufferrrrdropoutr'rdropout_moduleencoder_layerdrop embedding_dimrrrCrxno_scale_embeddingmathsqrt embed_scaleno_token_positional_embeddingsrencoder_learned_posembed_positionsrPrradaptive_inputquant_noise_pqapply_quant_noise_nnLinearquant_noise_pq_block_sizerrlayers ModuleListextendrangerMbuild_encoder_layerrK num_layersencoder_normalize_before layer_norm)r&r!rnrxrorir's r(r TransformerEncoder.__init__s  $ Y aS(9:, LLdnn&=&=?!%!7!7 .. '33$($=$=!("&"9"93tyy @88 ))  00   ?C x/ 4. 6 6'0'JD $'+D $""t':':Q'>1 )U;##.. D  $D   ! !C '-0F0FGDK--+DK 49$:M:M4N 4Nq $ $T *4N  dkk*  ( (' ADO"DO s'J c[U5n[USS5nU(a[USS5n[X$S9nU(d[US[5OSn[ X%S9nU$Nr]FrE)offload_to_cpurFrrG)rrPrr=r )r&r!layer checkpointrrFs r(r&TransformerEncoder.build_encoder_layerGse'-T#;UC $T+@%HN&uLE D.0J K#$ %C r*token_embeddingc*UcURU5nURU-=p4URbX@RU5-nURbURU5nUR U5nUR bUR U5nX44$ri)rxrrrrr)r&rrxembeds r(forward_embedding$TransformerEncoder.forward_embeddingSs  ""// ;O$$66    +,,Z88A  # # /((+A    "    '  #Axr*rrtoken_embeddingsc&URXX45$) Args: src_tokens (LongTensor): tokens in the source language of shape `(batch, src_len)` src_lengths (torch.LongTensor): lengths of each source sentence of shape `(batch)` return_all_hiddens (bool, optional): also return all of the intermediate hidden states (default: False). token_embeddings (torch.Tensor, optional): precomputed embeddings default `None` will recompute embeddings Returns: dict: - **encoder_out** (Tensor): the last encoder layer's output of shape `(src_len, batch, embed_dim)` - **encoder_padding_mask** (ByteTensor): the positions of padding elements of shape `(batch, src_len)` - **encoder_embedding** (Tensor): the (scaled) embedding lookup of shape `(batch, src_len, embed_dim)` - **encoder_states** (List[Tensor]): all intermediate hidden states of shape `(src_len, batch, embed_dim)`. Only populated if *return_all_hiddens* is True. )forward_scriptable)r&rrrr s r(rTransformerEncoder.forwardcs<&&z'9M Mr*cJURUR5nURRS:H=(d UR 5nUR UU5upxU(a&USUR S5RU5- -nURSS5n/n U(aU RU5 URH2n U "UU(aUOSS9nU(dMU ceU RU5 M4 URbURU5nU/U/U/U //S.$)r xlarrrN)rrrencoder_embeddingencoder_statesrr) eqrrdevicer/anyr  unsqueezetype_as transposeappendrr) r&rrrr rhas_padsrrrrs r(r%TransformerEncoder.forward_scriptables8> *}}T-=-=>$$))U2 6J6N6N7 #55j6F H Q-77;CCAFFGA KK1    ! !! $[[E&:!%'A"!%111%%a(! ?? &"A3%9$:"3!4,   r*rcX[US5S:Xa/nOUSSRSU5/n[US5S:Xa/nOUSSRSU5/n[US5S:Xa/nOUSSRSU5/n[US5S:Xa/nOUSSRSU5/n[US5S:Xa/nOUSSRSU5/nUSn[U5S:a([U5HupU RSU5X'M UUUUUUS .$) rrrrrrrrrr)rK index_select enumerate) r&rrnew_encoder_outnew_encoder_padding_masknew_encoder_embeddingrrridxstates r(r&TransformerEncoder.reorder_encoder_outs {=) *a / OM*1-::1iHO {12 3q 8') $23A6CCy"( $ {./ 0A 5$& !/03@@IN% ! {<( )Q .J\*1-;;AyIJ {=) *a /K]+A.<    ',, ,4,,''557 7r*c[UR[5(aZSRU5nX1;a[ SRU55 X [ R "S5USRU5'[UR5H1nURURUSRX$55 M3 SRU5n[R"URU[ R"S/55S5S:a'S UlS Ul[ R"S/5X'U$) @Upgrade a (possibly old) state dict for new versions of fairseq.{}.embed_positions.weightsz deleting {0}r {}.embed_positions._float_tensorz {}.layers.{} {}.versionrNF)rrrformatprintr FloatTensorrrrupgrade_state_dict_namedritemgetrr normalize)r& state_dictname weights_keyr version_keys r(r6+TransformerEncoder.upgrade_state_dict_nameds  d**,I J J6==dCK(n++K89+**1- 9@@ t'A KKN 3 3N11$: <( #))$/ ::jnn[%,,s2CDQG H1 L"DO"DN&+llA3&7J #r*)r!rrrrxrrrrrCr9rrrrri)NFN)rrrrrr rrrrr rrrrrrr;rrr)r6rrrs@r(rwrws3#j EI+3ELL+A&/3#(37 Mell+M! M #5<<0 MH/3#(37 E ell+E ! E #5<<0 E N YY: tCf4E/F: : x7r*rwc^\rSrSrSrSU4SjjrSrSSjrSS\\ \ \ \ 4S\\ \ \ \ \\ 44S\ S \ S \\S \\S \\S \ 4SjjrSS\\ \ \ \ 4S\\ \ \ \ \\ 44S \ S \\S \\4 SjjrSS\\ \ \ \ 4S\\ \ \ \ \\ 44S \ S \\S \\4 SjjrSrSrSrSrSrU=r$)r}i&a Transformer decoder consisting of *args.decoder_layers* layers. Each layer is a :class:`TransformerDecoderLayer`. Args: args (argparse.Namespace): parsed command-line arguments dictionary (~fairseq.data.Dictionary): decoding dictionary embed_tokens (torch.nn.Embedding): output embedding no_encoder_attn (bool, optional): whether to attend to encoder outputs (default: False). c >Xl[T U] U5 URS[R "S/55 [R "S5Ul[URURRS9Ul URUl URUlUR nUR"nXplUR&UlUR*UlUR,UlX0lUR0(aSO[2R4"U5UlUR8(dJUR:S:a:[=[>R@"XwSS9UR:URB5Ul"OSUl"Xv:wa [AXgSS9OSUl#URH(d*[KUR,UUR*URLS9OSUl'[QUS S5n[QUS S5(a[SXxS 9Ul*OSUl*[QUS S5Ul+URS :a[YURS9Ul-O[>R\"/5Ul-URZR_[aURb5V s/sHn UReX5PM sn 5 [gURZ5Ul4URj(a![QUSS5(d[SXxS 9Ul6OSUl6XpR(:wa%URn(d[AXpR(SS9OSUl8SUl9XPl:URtcURwXU5 ggs sn f)NrrrrrFrrrrrcross_self_attentionrrno_decoder_final_norm)1 )U;##.. D  $D + ?E :15 88 ))  00   ?C x/ 4. 6 6'0'JD $'+D $$+D2H,1%3!  ! !C '-0F0FGDK--+DK 4../ /  $ $T ;/  dkk*  ( (-u2626' ADO"DO111-- 933% @37  !%!2  ! ! )  ( (< H *' sM<c URb{[[U5UR[R "UR[ S9URUR(aUOSURURS9Ul GOUR(a[R"URR R"SURR R"SSS9UlURR UR$lOn[R"UR[U5SS9Ul[R&R)UR$R SURS-S9 [+US S5n[-U5H>nUR.R1US-UR2-US--[5U55 M@ g) N)r/)radaptive_inputsfactortie_projrrFr࿩meanstd base_layers)adaptive_softmax_cutoffrrKrHr eval_str_listr<adaptive_softmax_dropoutrMadaptive_softmax_factortie_adaptive_projrOrFrrrxweightshaperPinitnormal_rPrrinsertrOr)r&r!rnrxnum_base_layersrs r(rQ*TransformerDecoder.build_output_projections  ' ' 3$3J%%##D$@$@sK55,,!-2633// %D ! * *%'YY!!((..q1!!((..q1&D " -1,=,=,D,DD " " )%'YY%%s:U&DD " GGOO&&--))4/  1"$ q9'A KK  a%4...Oa4GH$ (r*c[X5n[USS5nU(a[USS5n[X5S9nU(d[US[5OSn[ X6S9nU$r)rrPrr=r )r&r!r|rrrrFs r(rK&TransformerDecoder.build_decoder_layerse'>T#;UC $T+@%HN&uLE D.0J K#$ %C r*rrrrrrrrc dURUUUUUUS9upU(dURU 5n X4$)a Args: prev_output_tokens (LongTensor): previous decoder outputs of shape `(batch, tgt_len)`, for teacher forcing encoder_out (optional): output from the encoder, used for encoder-side attention, should be of size T x B x C incremental_state (dict): dictionary used for storing state during :ref:`Incremental decoding` features_only (bool, optional): only return features without applying output layer (default: False). full_context_alignment (bool, optional): don't apply auto-regressive mask to self-attention (default: False). Returns: tuple: - the decoder's output of shape `(batch, tgt_len, vocab)` - a dictionary with any model-specific outputs )rrrrr)extract_features output_layer) r&rrrrrrrrrrextras r(rTransformerDecoder.forwardsJ@(( #/#9++ ) !!!$Axr*c.URUUUUUU5$ri)extract_features_scriptable)r&rrrrrrs r(rm#TransformerDecoder.extract_featuress)//    "     r*cUR5upxUcURS- nSn Sn UbH[US5S:a6USSn U R5SU:XdSUSU R35eUb[US5S:aUSSn Sn URbUR XS9n UbUSS2S S24nU b U SS2S S24n UR UR U5-n URbURU 5n URbURU 5n U bX- n URbURU 5n URU 5n U RSS5n Sn UR(d.URUR5R5(aURUR5n SnU /n[!UR"5HunnUcU(dUR%U 5nOSnU"U U U UUU ['UU:H5['UU:H5S 9un nnUR)U 5 UcMcUU:XdMkUR+5R-U 5nM UbUbUSUnUR/SS 9nUR0bUR1U 5n U nU RSS5n UR2bUR3U 5n U U/UUU S .4$) a Similar to *forward* but only return features. Includes several features from "Jointly Learning to Align and Translate with Transformer Models" (Garg et al., EMNLP 2019). Args: full_context_alignment (bool, optional): don't apply auto-regressive mask to self-attention (default: False). alignment_layer (int, optional): return mean alignment over heads at this layer (default: last layer). alignment_heads (int, optional): only average alignment over this many heads (default: all heads). Returns: tuple: - the decoder's features of shape `(batch, tgt_len, embed_dim)` - a dictionary with any model-specific outputs NrrrzExpected enc.shape == (t, z , c) got r)rr)self_attn_maskr need_attnneed_head_weights)dim)r inner_statesrr)sizerrKrdrrrxrrIrrrrArrrrr rbuffered_future_maskrrr:tor[rrN)r&rrrrrrbsslenenc padding_mask positionsrrrryr$rru layer_attnself_attn_hiddenrs r(rr.TransformerDecoder.extract_features_scriptablesI:&**,  ""oo1O $)-  "s;}+E'F'Jm,Q/CHHJqMR' M3B4y L M'  "s23(578(9&'=>qAL    +,,"-II  (!3ArsF!; $%af-    t001CD D    '  #A    *##A&A  NA  # # /((+A    " KK1 37  $ $(:(=(=  )""%#%)(%7%:%:4;K;K%L ""&01s #DKK0JC (1G!%!:!:1!=!%.3!-'= 68"&(>"@ / +Az+    "%#*@!''),,Q/%1(  *,_-999#D ?? &"A KK1     +$$Q'AF($&<    r*cBURcURU5$U$)z(Project features to the vocabulary size.)rOrP)r&featuress r(rnTransformerDecoder.output_layerts$  ())(3 3Or*cURc UR$[URURR5$)z/Maximum output length supported by the decoder.)rrDr(r)r*s r(r) TransformerDecoder.max_positions|r,r*cURS5nURRS5S:XdCURRUR:XaURRS5U:aE[R"[ R "[R"X"/55S5UlURRU5UlURSU2SU24$)Nrr) rzrDrrtriurfill_with_neg_infzerosr|)r&tensorrxs r(r{'TransformerDecoder.buffered_future_maskskk!n    " "1 % *))00FMMA$$))!,s2 % '' SJ(?@!!ED  --008  #tt,,r*c 2[UR[5(a@SRU5nX1;aX [R "S5USRU5'US3U;a>UR (aUS3nOUS3nXA;aUUXS3'UR (dX [UR5H]nSSS S .nUR5H@upxS H5n S RX%Xy5n X;dMXUS RX%X5'X M7 MB M_ SRU5n [R"URU [R"S/55S5S::a'SUlSUl[R"S/5X'U$)r.r/rr0z.output_projection.weightz.embed_tokens.weightz .embed_outself_attn_layer_normencoder_attn_layer_normfinal_layer_norm)012)rcrz{}.layers.{}.layer_norms.{}.{}z{}.layers.{}.{}.{}r1rr2NF)rrrr3rr5rFrrrrr7r8rrr9) r&r:r;r< embed_out_keyrlayer_norm_mapoldnewmrr=s r(r6+TransformerDecoder.upgrade_state_dict_nameds d**,I J J6==dCK(+**1- 9@@ V, -Z ?,,#'&(< = #'& 3 *AK!B# V#<=>44"1t'A,.'N +002+A8??)A0: ##7#>#> S$-.&M ,3( #))$/ ::jnn[%,,s2CDQG HA M"DO"DN&+llA3&7J #r*)rDrOr!rArErrorrrxrrrrDr9rrHrPrrrIrNrrF)FN)F)NNFFNNNF)NFNN)rrrrrr rQrKrrr;rrrr<rrrmrrrnr)r{r6rrrs@r(r}r}&s% " SIjB :>IM#',)-)-%)#(+d3V #456+$Dd33;F3C4D/E*E%FG +  +!%+"#+"#+c]+!+dJN',)-)- d3V #456 $Dd33;F3C4D/E*E%FG  !%  "# "# &JN',)-)-w d3V #456w $Dd33;F3C4D/E*E%FG w !% w "#w "#w r7-++r*r}c[R"XUS9n[RRURSUS-S9 [RR URUS5 U$)N)rrrrYrZ)rrkrerfrc constant_)rqrrrrs r(rkrksS ^ LAGGOOAHH1-*=O>GGahh{+Q/ Hr*c[R"XU5n[RRUR5 U(a*[RR UR S5 U$)Nr)rrrexavier_uniform_rcrr) in_features out_featuresrrs r(rrsH +T2AGGAHH%  !&&#& Hr*c$[USS5Ul[USS5Ul[USS5Ul[USS5Ul[USS 5Ul[US S 5Ul[US S 5Ul[US S5Ul[USUR5Ul [USUR5Ul [USS5Ul [USS 5Ul [USS 5Ul [USS 5Ul[USS5Ul[USS5Ul[USS5Ul[USS5Ul[USS5Ul[USS5Ul[USS 5Ul[USS 5Ul[US S 5Ul[US!S 5Ul[US"S 5Ul[US#S 5Ul[US$UR5Ul[US%UR5Ul[US&S 5Ul[US'S 5Ul[US(S 5Ul[US)S 5Ul [US*S 5Ul!URB(aS+Ul [US,S5Ul"[US-S5Ul#[US.S5Ul$[US/S5Ul%[US0S5Ul&[US1S 5Ul'[US2S5Ul(g)3NrZrWiencoder_ffn_embed_dimirMencoder_attention_headsr7rFrrYrXdecoder_ffn_embed_dimrOdecoder_attention_headsrLrJattention_dropoutractivation_dropout activation_fnrelurg?r^r`rr\rUrrr{rArGdecoder_input_dimrrrMr]rETrJrNrrErrquant_noise_scalar))rPrZrWrrMrrrrYrXrrOrrLrJrrrrr^r`r\rUrrr{rArGrrrrMr]rErJrNrrErrrr!s r(rIrIs%d,@$GD$T+>DD!(/F!MD!$(8!%)%;%;=D!(/F)-)C)C"ED!$(8!DD%d,@#FD ?D4C0DL#*41J+/$1D $+D2L,-%/D!,3 0%-9D) '.De LD*1 .+7D'!$(8%@D%d,@%HD '.De LD%d,@&*&<&<>D$T+>%)%;%;=D&d,@%HD&t-BEJD '.De LD")$0H*/#1D&t-BEJD &*#")$0H$"OD")$0H$"OD$T+>BD$T+>BD!$(8!DD!(/F!MD#*41JB#OD !$(8"=D$+D2L,0%2D!$+D2L,0%2D!!$(8!DD!(/F!MD#*41JB#OD $T+>ED%d,@$GD4D1DLdr*)T)0rtypingrrrrrnumpyrtorch.nnrfairseqrfairseq.distributedr fairseq.modelsr r r r rfairseq.modulesrrrrrrrrr&fairseq.modules.checkpoint_activationsrfairseq.modules.quant_noiserrrrQrRr<r=rrwr}rkrrIrr*r(rs 33 )99OOO FI## Xf++f+R\\~R2Rj   8Evr*