9iSSKrSSKJr SSKrSSKJs Jr SSKJrJr "SS\R5r "SS\R5r "SS \R5r "S S \R5r "S S \R5r"SS\R5r"SS\R5r"SS\R5r"SS\5r\ \SSS.4\ \ \ \ SS04\\\ \R&\R(\R*\\\R,S.r\"\5rSr\"S5r\"S5r\"S5r\"S 5r\"S!5r\"S"5r\"S#5r \"S$5r!g)%N) OrderedDict)Tensornnc*\rSrSrSrS\S\4SjrSrg)NewGELUActivationz Implementation of the GELU activation function currently in Google BERT repo (identical to OpenAI GPT). Also see the Gaussian Error Linear Units paper: https://arxiv.org/abs/1606.08415 inputreturnc SU-S[R"[R"S[R- 5US[R "US5---5--$)N??@Hm?g@)paddletanhmathsqrtpipowselfr s o/var/www/html/land-doc-ocr/venv/lib/python3.13/site-packages/paddlex/inference/models/common/vlm/activations.pyforwardNewGELUActivation.forwards_  ++IIcDGGm,x&**UC*@@@B  N__name__ __module__ __qualname____firstlineno____doc__rr__static_attributes__rrrrrs  V    rrc\^\rSrSrSrS S\4U4SjjjrS\S\4SjrS\S\4Sjr S r U=r $) GELUActivation+a Original Implementation of the GELU activation function in Google BERT repo when initially created. For information: OpenAI GPT's GELU is slightly different (and gives slightly different results): 0.5 * x * (1 + torch.tanh(math.sqrt(2 / math.pi) * (x + 0.044715 * torch.pow(x, 3)))) This is now written in C in nn.functional Also see the Gaussian Error Linear Units paper: https://arxiv.org/abs/1606.08415 use_gelu_pythonc>[TU]5 U(aURUlg[R R UlgN)super__init__ _gelu_pythonactr functionalgelu)rr' __class__s rr+GELUActivation.__init__3s/  ((DH}}))DHrr r cnUS-S[R"U[R"S5- 5--$)Nr r r)rerfrrrs rr,GELUActivation._gelu_python:s,s{cFJJutyy~/E$FFGGrc$URU5$r)r-rs rrGELUActivation.forward=sxxrr6)F) rrr r!r"boolr+rr,rr# __classcell__r0s@rr%r%+sG***H&HVHVrr%c*\rSrSrSrS\S\4SjrSrg)FastGELUActivationAzu Applies GELU approximation that is slower than QuickGELU but more accurate. See: https://github.com/hendrycks/GELUs r r c^SU-S[R"US-SSU-U---5--$)Nr r g3E?r)rrrs rrFastGELUActivation.forwardFsG  ++el2cHuX:a[SUSUS35e[TU] 5 XlX lg)Nzmin should be < max (got min: z, max: )) ValueErrorr*r+rJrK)rrJrKr0s rr+ClippedGELUActivation.__init__gs8 9=cU'#aPQ Q rxr cl[R"[U5URUR5$r))rclipr/rJrK)rrPs rrClippedGELUActivation.forwardos!{{47DHHdhh77r)rKrJ) rrr r!r"floatr+rrr#r9r:s@rrHrHZs3 E88F88rrHc*\rSrSrSrS\S\4SjrSrg)SiLUActivationsa See Gaussian Error Linear Units (Hendrycks et al., https://arxiv.org/abs/1606.08415) where the SiLU (Sigmoid Linear Unit) was originally introduced and coined, and see Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learning (Elfwing et al., https://arxiv.org/abs/1702.03118) and Swish: a Self-Gated Activation Function (Ramachandran et al., https://arxiv.org/abs/1710.05941v1) where the SiLU was experimented with later. r r c.[R"U5$r))rDsilurs rrSiLUActivation.forward|vve}rrNrrrrrVrVssVrrVc*\rSrSrSrS\S\4SjrSrg)MishActivationz See Mish: A Self-Regularized Non-Monotonic Activation Function (Misra., https://arxiv.org/abs/1908.08681). Also visit the official repository for the paper: https://github.com/digantamisra98/Mish r r c.[R"U5$r))rDmishrs rrMishActivation.forwardr[rrNrrrrr]r]s Vrr]c*\rSrSrSrS\S\4SjrSrg)LinearActivationzS Applies the linear activation function, i.e. forwarding input directly to output. r r cU$r)rrs rrLinearActivation.forwards rrNrrrrrcrcsVrrcc(^\rSrSrU4SjrSrU=r$)ClassInstantiercl>[TU]U5n[U[5(aUOU04up4U"S0UD6$)Nr)r* __getitem__ isinstancetuple)rkeycontentclskwargsr0s rrkClassInstantier.__getitem__s7'%c*!+GU!;!;g'2 }V}rr)rrr r!rkr#r9r:s@rrhrhs rrhi )rJrKr'T)r/gelu_10 gelu_fastgelu_newgelu_pytorch_tanh gelu_pythonlinearr` quick_gelurelurelu6rErYswishrc U[;a [U$[SUS[[R5535e)Nz function z not found in ACT2FN mapping )ACT2FNKeyErrorlistkeys)activation_strings rget_activationrsBF"'(()**GV[[]H[G\ ]  rrxrvr/rurzrYr`ry)"r collectionsrrpaddle.nn.functionalrr.rDrLayerrr%r<rArHrVr]rcrhReLUReLU6SigmoidTanhACT2CLSrrrxrvr/rurzrYr` linear_actrrrrsc #   (RXX,    0"((08BHH82 RXX RXXrxxk %s2'>?#!*"%6$=> % GG XXzz   GG! $  ! ]+ * %f ; ' L ) ff H % r