i(SSKrSSKrSSKJs Jr SSKJr "SS\R 5r\R"5"SS\R 55r g)N)nnc2^\rSrSrSU4SjjrSrSrU=r$) EncoderLayerc>[TU]5 XlURUlURUl[ R "URU5Ul[ R "URU5Ul[ R "X5Ul [ R "X5Ul X l [ R"UR5Ul [ R"UR5UlSU-n[ R "X5Ul[ R "X15Ulg)N)super__init__ embed_dimkdimvdimrLineark_projv_projq_projout_proj num_heads LayerNormself_attn_layer_normfinal_layer_normfc1fc2)selfr r ffn_embed_dim __class__s `/var/www/html/ai-image-ml/venv/lib/python3.13/site-packages/executorch/exir/tests/transformer.pyr EncoderLayer.__init__s "NN NN ii 95 ii 95 ii 5  )7 "$&LL$@! " T^^ <  M 99Y699]6cUnU=n=pE[R"UUUURURURR UR R URR URR URRSSSSSSSS9upX!-nURU5nUn[R"URU55nURU5nX!-nURU5nU$)NFgT) q_proj_weight k_proj_weight v_proj_weightout_proj_weight out_proj_bias add_zero_attn dropout_puse_separate_proj_weightin_proj_weight in_proj_biasbias_kbias_v)Fmulti_head_attention_forwardr rrweightrrrbiasrrelurrr)rxresidualquerykeyvalue_s rforwardEncoderLayer.forward$s--    NN NN++,,++,,++,, MM00--,,%)% ( L  % %a ( FF488A;  HHQK L  ! !! $r) r rrrrr rrrrrr )r)__name__ __module__ __qualname____firstlineno__r r7__static_attributes__ __classcell__rs@rrrs7*rrc<^\rSrSrSrSU4SjjrSrSrSrU=r $) TransformerFav A simplified implementation of mt_model that does not have all those heavy dependencies but still be similar enough to the original model. Suitable to be put in exir end2end tests. E.g., we can use it to ease the testing of memory planning for dynamic shapes on REAL models. Some of the simplifications recorded here: 1. the original model will reset the embedding to a 0 vector for padding token. We skip that. 2. skip various configurations in the original model. E.g., original model has a config cfg.no_scale_embedding to control if the token embedding should be scaled or not. We just always scale the embedding. cT>[TU]5 XlSUl[R "URUR5Ul[R"UR5Ul [U5Vs/sHn[URS9PM snUl gs snf)N )r ) r r inp_vocab_size model_dimr Embeddingtoken_embed_tablemathsqrt embed_scalerangerencoder_layers)rrErFnum_encoder_layersr6rs rr Transformer.__init__Ws~ ,!#d.A.A4>>!R99T^^4> 2r?s@rrArAFs    J JrrA) rIr]torch.nn.functionalr functionalr,Modulerno_gradrArrrjsO 52995p3J"))3J3Jr