9i FSSKJr SSKrSSKJs Jr SSKJr SSKJr SSK J r /SQr "SS \R5r "S S \R5r"S S \R5r"SS\R5rSS\4SjjrSS\4SjjrSS\4SjjrSS\4Sjjrg)) OrderedDictN) LayerNorm)nn)DropPath)vit_base vit_large vit_large_336vit_hugec:\rSrSrSrS\R 4SjrSrg) QuickGELUz An activation function module. xc:U[R"SU-5-$)NgZd;?)torchsigmoidselfrs e/var/www/html/land-doc-ocr/venv/lib/python3.13/site-packages/modelscope/models/multi_modal/ofa/vit.pyforwardQuickGELU.forwards5==+++N) __name__ __module__ __qualname____firstlineno____doc__rTensorr__static_attributes__rrrr r s,,rr c^\rSrSrSrS S\S\S\R4U4SjjjrS\R4Sjr S\R4S jr S r U=r $) ResidualAttentionBlocka A residual attention block module. step 1. Calculate the self attention in input with layer normalization. step 2. Add input to the result of self attention's result as I. step 3. Calculate the mlp of input I with layer normalization. step 4. Add I to the result of mlp. d_modeln_head attn_maskc >[TU]5 [R"X5Ul[ U5Ul[R"[S[R"XS-54S[54S[R"US-U54/55Ul [ U5Ul X0l [U5Ulg)a Args: d_model (`int`): The embedding dimensions. n_head (`int`): The number of heads in self attention block. attn_mask (`Tensor`, **optional**, default to None): Attention mask using in self attention. drop_path_rate (`float`, **optional**, default to 0.0): Drop path rate. See more details about drop path from https://arxiv.org/pdf/1605.07648v4.pdf. c_fcgeluc_projN)super__init__rMultiheadAttentionattnrln_1 SequentialrLinearr mlpln_2r&r drop_path)rr$r%r&drop_path_rate __class__s rr-ResidualAttentionBlock.__init__)s ))': g& == 7aK89%299Wq[':;  g& "!.1rrcURb.URRURURS9OSUlUR XUSURS9S$)z A wrapper of self attention . N)dtypedeviceF) need_weightsr&r)r&tor:r;r/rs r attention ResidualAttentionBlock.attentionFsi ~~) NN  AGGAHH  =/3 yy !%4>>CCDF FrcXRURURU555-nXRURUR U555-nU$N)r5r>r0r3r4rs rrResidualAttentionBlock.forwardPsK t~~diil;< < txx ! 56 6r)r/r&r5r0r4r3N) rrrrrintrrr-r>rr __classcell__r7s@rr"r"s_,0 # 222"LL22:F5<<Frr"c ^\rSrSrSrS S\S\S\S\RS\4 U4Sjjjr S \R4S jr S r U=r $) TransformerVzg A transformer module using in `VisionTransformer`. Execute a sequential of `ResidualAttentionBlock`. widthlayersheadsr&r6c >[TU]5 XlX l[R "[ U5Vs/sHn[XXE5PM sn6Ulgs snf)a Args: width (`int`): The width of input image. layers (`int`): The number of `ResidualAttentionBlock` layers. heads (int): The number of self attention heads. attn_mask (`Tensor`, **optional**, default to None): Attention mask using in self attention. drop_path_rate (`float`, **optional**, default to 0.0): Drop path rate. See more details about drop path from https://arxiv.org/pdf/1605.07648v4.pdf. N) r,r-rKrLrr1ranger" resblocks)rrKrLrMr&r6_r7s rr-Transformer.__init__]sU&   6]) " #5 K")  ) sArc$URU5$rA)rPrs rrTransformer.forwardxs~~a  r)rLrPrKrC) rrrrrrErrfloatr-rr rFrGs@rrIrIVsc#' #      <<     6!!!rrIc p^\rSrSrSrS S\S\S\S\S\S\4 U4S jjjrS \R4S jr S r U=r $)VisionTransformer|ao Vision transformer module. step 1. Using conv2d to get the image embedding. step 2. If the resolution of input image doesn't equal to the initialized one do `bilinear` interpolate to get new patch position embedding. step 3. Add position embedding to image embedding to generate final image representation. step 4. Do `Transformer` to the image representation. input_resolution patch_sizerKrLrMr6c>>[TU]5 XlX l[R "SUUUSS9UlUS-nX0l[R"U[R"X-S-S-U5-5Ul [U5Ul [X4XVS9Ulg) a Args: input_resolution (`int`): The resolution of input image. patch_size (`int`): The resolution of each patch image. width (`int`): The dimension of each patch image. layers (`int`): The number of `ResidualAttentionBlock` in `Transformer`. heads (`int`): The number of heads in self attention block. drop_path_rate (`float`, **optional**, default to 0.0): Drop path rate. See more details about drop path from https://arxiv.org/pdf/1605.07648v4.pdf. F) in_channels out_channels kernel_sizestridebiasgrr6N)r,r-rYrZrConv2dconv1rK Parameterrrandnpositional_embeddingrln_prerI transformer) rrYrZrKrLrMr6scaler7s rr-VisionTransformer.__init__s(  0$YY"   t  $&LL  +a /! 3U:<2<%=!& & 5ArrcURSnURSUR-URSUR-pCURU5nURURSURSS5nUR SSS5nX R :waUR SSnUR UR-nURSXfS5R SSSS5n[R"XSU4SS9nUR SSSS5RX4-S5nXRUR5-nO*XR SSRUR5-nURU5nUR SSS5nURU5nUR SSS5nURupn URSS5RXX45nU$) Nrrrbr\bilinear)sizemode)shaperZrereshapepermuterYrhF interpolater=r:rirj transpose) rr resolutionheightrKold_pe patch_numnew_pebzseqhiddens rrVisionTransformer.forwardsWWR[  t6 9??9# JJqM IIaggaj!''!*  IIaA  .. ...qr2F--@I^^Ay$&((/1a(; ]]e_:?F^^Aq!Q/77KFIIagg&&A--ab144QWW==A KKN IIaA    Q  IIaA '' KK1  % %b& @r)rerYrirZrhrjrKrD) rrrrrrErUr-rrrr rFrGs@rrWrW|so"!$%A%A%A %A  %A  %A%A%ANrrWr6c"[SSSSSU5$)z An instance of base vision transformer model. Args: drop_path_rate (`float`, **optional**, default to 0.0): Drop path rate. See more details about drop path from https://arxiv.org/pdf/1605.07648v4.pdf. i rWrcs rrrs S"c1b. AArc"[SSSSSU5$)z An instance of large vision transformer model. Args: drop_path_rate (`float`, **optional**, default to 0.0): Drop path rate. See more details about drop path from https://arxiv.org/pdf/1605.07648v4.pdf. rrrrcs rr r  S"dBN CCrc"[SSSSSU5$)a An instance of large vision transformer model with 336 as input image width . Args: drop_path_rate (`float`, **optional**, default to 0.0): Drop path rate. See more details about drop path from https://arxiv.org/pdf/1605.07648v4.pdf. iPrrrrrrcs rr r rrc"[SSSSSU5$)z An instance of huge vision transformer model. Args: drop_path_rate (`float`, **optional**, default to 0.0): Drop path rate. See more details about drop path from https://arxiv.org/pdf/1605.07648v4.pdf. rrirrrcs rr r rrr) collectionsrrtorch.nn.functionalr functionalrvfairseq.modulesr utils.utilsr__all__Moduler r"rIrWrUrr r r rrrrs $ %! , ,4RYY4n#!"))#!LO Od BU B De D D% D DU Dr