?Ki SSKrSSKJr SSKJr SSKJr SSKJrJ r \ R"\ 5r \"5(aSSK r SrS"S\\S \S S \\S \S \44SjjrS"S\\S \S S \\S \S \44SjjrS"S\\S \S S \\S \S \44SjjrS#S\S S S \\S \S \44SjjrS#S\S S S \\S \S \44SjjrS#S\S S S \\S \S \44Sjjr\\\\\\S.rS$S\S\S\S\\S\\4 SjjrS#S\S\\4SjjrS#S\S\\4SjjrS#S\S\\4SjjrS#S\S\\4SjjrS#S\S\\4Sjjr S#S\S\\4S jjr!\\\\\ \!S.r"S#S\S\\4S!jjr#g)%Nwraps)Optional)PretrainedConfig)is_torch_availableloggingcD^^^SmSm[T5UUU4Sj5nU$)aD Decorator function to update the RoPE parameters in the forward pass, if the model is using a dynamic RoPE (i.e. a RoPE implementation that may recompute its frequencies in the forward pass). Args: rope_forward (Callable): The forward pass of the RoPE implementation. Returns: The decorated forward pass. c[R"U5S-n[URS5(aURRnOURR nX4:aR[US5(d%UR URX$S-S9uUlnURSURSS9 gURRU5Ul URSURSS9 g) zbLongrope uses long factor if sequence is larger than original pretraining length, short otherwise.r original_max_position_embeddings long_inv_freqseq_leninv_freqF persistentN) torchmaxhasattrconfigr max_position_embeddings rope_init_fnr register_bufferoriginal_inv_freqto)self position_idsdevicerr _s b/var/www/html/dynamic-report/venv/lib/python3.13/site-packages/transformers/modeling_rope_utils.pylongrope_frequency_update6dynamic_rope_update..longrope_frequency_update+s))L)A- 4;; B C C/3{{/[/[ ,/3{{/R/R ,  5411(,(9(9KKTU1U):)%"A  T-?-?E R&*%;%;%>%>v%FD "  T-C-CPU Vc[R"U5S-nX0R:a8URURX#S9uo@lUR SUSS9 X0lX0R:ahURUR:aMURRU5UlUR SURSS9 URUlggg)z dynamic RoPE layers should recompute `inv_freq` in the following situations: 1 - growing beyond the cached sequence length (allow scaling) 2 - the current sequence length is in the original scale (avoid losing precision with small sequences) rrrFrN) rrmax_seq_len_cachedrrattention_scalingroriginal_max_seq_lenrr)rrrrrs r dynamic_frequency_update5dynamic_rope_update..dynamic_frequency_update>s ))L)A- ,, ,/3/@/@f/@/f ,H,  X% H&- # .. .43J3JTMfMf3f&*%;%;%>%>v%FD "  T-C-CPU V&*&?&?D # 4g .r#c>SUR;aT"XURS9 O!URS:XaT"XURS9 T"XU5$)Ndynamic)rlongrope) rope_typer)rxrr(r! rope_forwards r wrapper$dynamic_rope_update..wrapperQsD  & $T I ^^z ) %d JD\22r#r)r/r0r(r!s` @@r dynamic_rope_updater2s/W&@& <33 Nr#rrz torch.devicerreturnz torch.Tensorc FURn[USS5n[USS5=(d URUR-n[ XT-5nSnSU[ R "SUS[ RS9RU[ RS9U- -- nX4$) a Computes the inverse frequencies according to the original RoPE implementation Args: config ([`~transformers.PretrainedConfig`]): The model configuration. This function assumes that the config will provide at least the following properties: * rope_theta (`float`): The base wavelength from which the inverse frequencies will be derived. * hidden_size (`int`): The numerator when deriving a head_dim, if not provided directly. * num_attention_heads (`int`): The denominator when deriving a head_dim, if not provided directly. Additionally, this function will make use of the following properties if they are found in the config: * head_dim (`int`, *optional*): The size of the key-value heads in the model. If None, this value will be derived as hidden_size // num_attention_heads. * partial_rotary_factor (`float`, *optional*): If less than 1.0, inverse frequencies will be returned for the first fraction of the head_dim. Defaults to 1.0. device (`torch.device`): The device to use for initialization of the inverse frequencies. seq_len (`int`, *optional*): The current sequence length. Unused for this type of RoPE. Returns: Tuple of (`torch.Tensor`, `float`), containing the inverse frequencies for the RoPE embeddings and the post-processing scaling factor applied to the computed cos/sin (unused in this type of RoPE). partial_rotary_factor?head_dimNrdtyperr:) rope_thetagetattr hidden_sizenum_attention_headsintrarangeint64rfloat) rrrbaser5r7dimattention_factorrs r _compute_default_rope_parametersrG\s>   D#F,CSIvz40dF4F4F&JdJd4dH h. /Cdu||AsAU[[ILLTZbgbmbmLnqttuvH  %%r#cJURSn[XU5upEXC-nXE4$)a Computes the inverse frequencies with linear scaling. Credits to the Reddit user /u/kaiokendev Args: config ([`~transformers.PretrainedConfig`]): The model configuration. This function assumes that the config will provide at least the following properties: * rope_theta (`float`): The base wavelength from which the inverse frequencies will be derived. * hidden_size (`int`): The numerator when deriving a head_dim, if not provided directly. * num_attention_heads (`int`): The denominator when deriving a head_dim, if not provided directly. Additionally, this function will make use of the following properties if they are found in the config: * head_dim (`int`, *optional*): The size of the key-value heads in the model. If None, this value will be derived as hidden_size // num_attention_heads. * partial_rotary_factor (`float`, *optional*): If less than 1.0, inverse frequencies will be returned for the first fraction of the head_dim. Defaults to 1.0. device (`torch.device`): The device to use for initialization of the inverse frequencies. seq_len (`int`, *optional*): The current sequence length. Unused for this type of RoPE. Returns: Tuple of (`torch.Tensor`, `float`), containing the inverse frequencies for the RoPE embeddings and the post-processing scaling factor applied to the computed cos/sin (unused in this type of RoPE). factor) rope_scalingrG)rrrrIrrFs r '_compute_linear_scaling_rope_parametersrKs:>  *F"B&RY!ZH  H  %%r#c xURn[USS5n[USURUR-5n[ XT-5nUR nUR SnSn UcUnOi[U[R5(a?[R"U[R"XrRURS95nO [X'5nX8U-U- US- - XfS- - --nSU[R"SUS[R S 9R#U[R$S 9U- -- n X4$) aP Computes the inverse frequencies with NTK scaling. Credits to the Reddit users /u/bloc97 and /u/emozilla Args: config ([`~transformers.PretrainedConfig`]): The model configuration. This function assumes that the config will provide at least the following properties: * rope_theta (`float`): The base wavelength from which the inverse frequencies will be derived. * hidden_size (`int`): The numerator when deriving a head_dim, if not provided directly. * num_attention_heads (`int`): The denominator when deriving a head_dim, if not provided directly. * max_position_embeddings (`int`): The default sequence length used to update the dynamic RoPE at inference time * rope_scaling (`dict[str, float]`): The standard RoPE scaling parameters, from which `factor` will be accessed. The value of `factor` is used to determine the new base frequency, along with the current sequence length (seq_len), the maximum positional embeddings (max_position_embeddings), and the computed dimensionality (dim) of the rotary embeddings. If seq_len <= max_position_embeddings, this factor has no effect. If seq_len <= max_position_embeddings, this factor effectively stretches the context window using an exponent derived from `dim`. Additionally, this function will make use of the following properties if they are found in the config: * head_dim (`int`, *optional*): The size of the key-value heads in the model. If None, this value will be derived as hidden_size // num_attention_heads. * partial_rotary_factor (`float`, *optional*): If less than 1.0, inverse frequencies will be returned for the first fraction of the head_dim. Defaults to 1.0. device (`torch.device`): The device to use for initialization of the inverse frequencies. seq_len (`int`, *optional*): The current sequence length, used to update the dynamic RoPE at inference time. If `None` or shorter than max_position_embeddings, this value will be overridden by max_position_embeddings. Returns: Tuple of (`torch.Tensor`, `float`), containing the inverse frequencies for the RoPE embeddings and the post-processing scaling factor applied to the computed cos/sin (unused in this type of RoPE). r5r6r7rIr:rrr8rr9r;)r<r=r>r?r@rrJ isinstancerTensormaximumtensorr:rrrArBrrC) rrrrDr5r7rErrIrFrs r _compute_dynamic_ntk_parametersrRs5T   D#F,CSIvz6+=+=A[A[+[\H h. /C$<<   *F) GU\\ * *--  LL0 gnn ]  g7 W$'>>6A:NTWab[bTcd dDdu||AsAU[[ILLTZbgbmbmLnqttuvH  %%r#c,^URn[USS5n[USURUR-5n[ XT-5nUR SnUR R S5nUR R S5n UR R S5n UR R S5=(d URn SS jn Uc1U (a"U (a[U "Xy5U "Xz5- 5nOU "U5nUR R S 5=(d S n UR R S 5=(d S nSmU4SjnSnU[R"SUS5RU[RS9U- -nSU- nSUU-- nUR R SS5nU"XXcU U5unnS U"UUUS-5RU[RS9- nUS U- -UU--nUU4$)a Computes the inverse frequencies with NTK scaling. Please refer to the [original paper](https://huggingface.co/papers/2309.00071) Args: config ([`~transformers.PretrainedConfig`]): The model configuration. This function assumes that the config will provide at least the following properties: * rope_theta (`float`): The base wavelength from which the inverse frequencies will be derived. * hidden_size (`int`): The numerator when deriving a head_dim, if not provided directly. * num_attention_heads (`int`): The denominator when deriving a head_dim, if not provided directly. * max_position_embeddings (`int`): The maximum length of the positional embeddings. * rope_scaling (`dict[str, float | int]`): The standard RoPE scaling parameters, from which the following keys will be accessed: * `attention_factor` (`float`, *optional*): The scaling factor to be applied to the computed cos/sin. If None, the value is inferred from `factor`, `mscale`, and `mscale_all_dim` as avaialble. * `beta_fast` (`float`, *optional*, defaults to 32): Parameter to set the boundary for extrapolation (only) in the linear ramp function. * `beta_slow` (`float`, *optional*, defaults to 1): Parameter to set the boundary for interpolation (only) in the linear ramp function. * `factor` (`float`, *optional*): The scaling factor applied when interpolating the position IDs to extend the possible context length. Additionally, if `attention_factor` is None, the log of this value is used to compute a value for `attention_factor`, possibly in conjunciton with `mscale` and `mscale_all_dim`, if provided. * `mscale` (`float`, *optional*): If `attention_factor` is None and both `mscale` and `mscale_all_dim` are provided, `mscale` acts scalar augmenting `log(factor)` when computing the numerator for the inferred value of `attention_factor`. If not provided, `attention_factor` will be calculated based on `factor` only. * `mscale_all_dim` (`float`, *optional*): If `attention_factor` is None and both `mscale` and `mscale_all_dim` are provided, `mscale_all_dim` acts scalar augmenting `log(factor)` when computing the denominator for the inferred value of `attention_factor`. If not provided, `attention_factor` will be calculated based on `factor` only. * `original_max_position_embeddings` (`int`, *optional*): The original max position embeddings used during pretraining. If not provided, the function falls back to `max_position_embeddings`. * `truncate` (`bool`, *optional*): Whether to truncate the correction range. Additionally, this function will make use of the following properties if they are found in the config: * head_dim (`int`, *optional*): The size of the key-value heads in the model. If None, this value will be derived as hidden_size // num_attention_heads. * partial_rotary_factor (`float`, *optional*, defaults to 1.0): If less than 1.0, inverse frequencies will be returned for the first fraction of the head_dim. device (`torch.device`): The device to use for initialization of the inverse frequencies. seq_len (`int`, *optional*): The current sequence length. Unused for this type of RoPE. Returns: Tuple of (`torch.Tensor`, `float`), containing the inverse frequencies for the RoPE embeddings and the post-processing scaling factor applied to the computed cos/sin. r5r6r7rIrFmscalemscale_all_dimr rcNUS::agSU-[R"U5-S-$)Nrr6g?)mathlog)scalerTs r get_mscale,_compute_yarn_parameters..get_mscale:s( A:V|dhhuo-33r# beta_fast beta_slowcU[R"X0S-[R-- 5-S[R"U5-- $)zPInverse dimension formula to find the dimension based on the number of rotationsr8)rWrXpi) num_rotationsrErDrs r find_correction_dim5_compute_yarn_parameters..find_correction_dimLs@dhh6!:Kdgg:UVWW\]`d`h`him`n\noor#c>T"XX45nT"XX45nU(a,[R"U5n[R"U5n[US5[ XrS- 54$)z.Find dimension range bounds based on rotationsrr)rWfloorceilrmin) low_rothigh_rotrErDrtruncatelowhighrbs r find_correction_range7_compute_yarn_parameters..find_correction_rangePsR!'N"8$P **S/C99T?D3{CAg...r#cX:XaUS- n[R"U[RS9U- X- - n[R"USS5nU$)NgMbP?r9rr)rrAfloat32clamp)rgrrE linear_func ramp_funcs r linear_ramp_factor4_compute_yarn_parameters..linear_ramp_factorYsH : 5LC||Cu}}=C R KK Q2 r#rr8r;rjT)r) r<r=r>r?r@rJgetrrCrrAr)rrrrDr5r7rErIrFrTrUr rZr\r^rmrt pos_freqsinv_freq_extrapolationinv_freq_interpolationrjrkrlinv_freq_extrapolation_factorrrbs @r _compute_yarn_parametersr{s$p   D#F,CSIvz6+=+=A[A[+[\H h. /C   *F**../AB  $ $X .F((,,-=>N BCevGeGe%4  n$Z%?*VBd%de )&1 ##'' 4:I##'' 49Ip/aa03363UX[[\I 9_ FY$67""&&z48H%iCGgiqrIC%&(:3cQh(O(R(RZ`hmhshs(R(t$t!!&C"CD #@ @ A  % %%r#cpURn[USS5n[USURUR-5n[ XT-5nUR SnUR SnUR R S5n UR R S5n [USS 5=n (aURU - n O URn U cMU S::aSn OD[R"S [R"U 5[R"U 5- -5n U(a*X+:a%[R"U[RUS 9n O$[R"U[RUS 9n [R"S US [RUS 9R!5U- n SXU --- nX4$)a9 Computes the inverse frequencies with LongRoPE scaling. Please refer to the [original implementation](https://github.com/microsoft/LongRoPE) Args: config ([`~transformers.PretrainedConfig`]): The model configuration. This function assumes that the config will provide at least the following properties: * rope_theta (`float`): The base wavelength from which the inverse frequencies will be derived. * hidden_size (`int`): The numerator when deriving a head_dim, if not provided directly. * num_attention_heads (`int`): The denominator when deriving a head_dim, if not provided directly. * max_position_embeddings (`int`): The maximum length of the positional embeddings. * original_max_position_embeddings (`int`, *optional*): The original max position embeddings used during pretraining. If not provided, defaults to `max_position_embeddings`. * rope_scaling (`dict[str, float]`): The standard RoPE scaling parameters, from which the following keys will be accessed: * `attention_factor` (`float`, *optional*): The scaling factor to be applied on the attention computation. If unspecified, it defaults to value recommended by the implementation, inferred from the value of `factor`. * `factor` (`float`, *optional*): The scaling factor to apply to the RoPE embeddings. If both `max_position_embeddings` and `original_max_position_embeddings` are provided, this value will be overridden s the ratio between those values. * `long_factor` (`float`, *optional*): The scale factor applied when computing the inverse frequencies if `seq_len` is provided and greater than `original_max_position_embeddings`. * `short_factor` (`float`, *optional*): The scale factor applied when computing the inverse frequencies if `seq_len` is None or less-than-or-equal-to `original_max_position_embeddings`. Additionally, this function will make use of the following properties if they are found in the config: * head_dim (`int`, *optional*): The size of the key-value heads in the model. If None, this value will be derived as hidden_size // num_attention_heads. * partial_rotary_factor (`float`, *optional*, defaults to 1.0): If less than 1.0, inverse frequencies will be returned for the first fraction of the head_dim. device (`torch.device`): The device to use for initialization of the inverse frequencies. seq_len (`int`, *optional*): The current sequence length. Returns: Tuple of (`torch.Tensor`, `float`), containing the inverse frequencies for the RoPE embeddings and the post-processing scaling factor applied to the computed cos/sin. r5r6r7 long_factor short_factorrIrFr NrrMrr8)r<r=r>r?r@rJrvrrWsqrtrXrrQrprArBrC)rrrrDr5r7rEr}r~rIrFr ext_factorsinv_freq_shapers r _compute_longrope_parametersrss^   D#F,CSIvz6+=+=A[A[+[\H h. /C%%m4K&&~6L  $ $X .F**../AB ,36;]_c+dd'd//2RR+1+I+I( S=" #yyTXXf-=Ii@j-j)jk 7=ll;emmFS ll))*LMO&8':$''kH$G[[!;X=NPXYN$.@EUEghM]*n= 1, got rJrvrrrrNrCrr)rrrJr-rrrIs r (_validate_linear_scaling_rope_parametersr8s&&L  l.>.>vt.LMI (+M ))+,M=Z ( #F ~Z66&3,QRXQYZ[;Gr#c.URnURSURSS55nSS1nS1n[UR55n[ X6XEUS9 USnUb[ U[ 5(aUS:a[RSU35 gg)Nr-rrIr rr6rr)rrrJr-rrrrIs r )_validate_dynamic_scaling_rope_parametersrDs&&L  l.>.>vt.LMI (+M78M ))+,M=]hi ( #F ~Z66&3,QRXQYZ[;Gr#c URnURSURSS55nSS1n1Skn[UR55n[ X6XEUS9 USnUb[ U[ 5(aUS:a[RSU35 URS5nUb3[ U[ 5(aUS :a[RS U35 URS 5n U b-[ U [ 5(d[RS U 35 URS 5n U b-[ U [ 5(d[RSU 35 U =(d SU =(d S:a[RSU SU S35 URRS5n U b5URU - n X:wa [RSUSU SUS35 gg[RS5 g)Nr-rrI>rTrjr\r^rUrFr rr6rrFrL`rope_scaling`'s attention_factor field must be a float greater than 0, got r\z6`rope_scaling`'s beta_fast field must be a float, got r^z6`rope_scaling`'s beta_slow field must be a float, got r]rzO`rope_scaling`'s beta_fast field must be greater than beta_slow, got beta_fast=z( (defaults to 32 if None) and beta_slow=z (defaults to 1 if None)r zHThe explicitly set RoPE scaling factor (config.rope_scaling['factor'] = z) does not match the ratio implicitly set by other parameters (implicit factor = post-yarn context length / pre-yarn context length = config.max_position_embeddings / config.rope_scaling['original_max_position_embeddings'] = z). Using the explicit factor (z) in YaRN. This may cause unexpected behaviour in model usage, please correct the 'max_position_embeddings' fields in the model config.a~config.rope_scaling['original_max_position_embeddings'], the pre-yarn context length, is unset. We will **assume** config.max_position_embeddings holds the pre-yarn context length. Some use cases may expect config.max_position_embeddings to hold the post-yarn context length (pre-yarn context length * factor) -- we recommend updating both fields for optimal downstream model usage.) rJrvrrrrNrCrrr warning_once) rrrJr-rrrrIrFr\r^r implicit_factors r _validate_yarn_parametersrRs&&L  l.>.>vt.LMI (+MM ))+,M=]hi ( #F ~Z66&3,QRXQYZ[#''(:;#Z8H%-P-PTdghThZ[kZl m   -IZ 5%A%AOPY{[\  -IZ 5%A%AOPY{[\RIN+]^g]hi66?[@X Z (.':':'>'>?a'b$'3 88;[[  $   Z[aZbcn###A&Ju u  %  _ r#cURnURSURSS55n1Skn1Skn[UR55n[ X6XEUS9 [ USS5n[ USUR UR-5n[X-5n URS 5n [U [5(d/[S U 55(a[RS U 35 [U 5U S -:wa'[RS U S -S[U 535 URS5n [U [5(d/[SU 55(a[RSU 35 [U 5U S -:wa'[RSU S -S[U 535 [US5(a[R!S5 gURS5n U c[RS5 O3[U ["5(aU S:a[RSU 35 URS5n U b5[U ["5(aU S:a[RSU 35 ggg)Nr-r>r-r}r~>rIrFr rr5r6r7r~c3N# UHn[U[[45v M g7fNrNr@rC.0r.s r 0_validate_longrope_parameters..s!1dWcRS*Qe 2M2MWc#%zC`rope_scaling`'s short_factor field must be a list of numbers, got r8z5`rope_scaling`'s short_factor field must have length z, got r}c3N# UHn[U[[45v M g7frrrs r rrs!0bVaQRAU|1L1LVarzB`rope_scaling`'s long_factor field must be a list of numbers, got z4`rope_scaling`'s long_factor field must have length r aYThis model has set a `original_max_position_embeddings` field, to be used together with `max_position_embeddings` to determine a scaling factor. Please set the `factor` field of `rope_scaling`with this ratio instead -- we recommend the use of this field over `original_max_position_embeddings`, as it is compatible with most model architectures.rIz1Missing required keys in `rope_scaling`: 'factor'rrFgr)rJrvrrrr=r>r?r@rNlistallrrlenrrrC)rrrJr-rrrr5r7rEr~r}rIrFs r _validate_longrope_parametersrsN&&L  l.>.>vt.LMI@MVM ))+,M=]hi#F,CSIvz6+=+=A[A[+[\H h. /C##N3L lD ) )c1dWc1d.d.d\]i\jkl <C1H$NsVWxjX^_bco_p^qrs""=1K k4 ( (S0bVa0b-b-b[\g[hij ;3!8#McUVhZW]^abm^n]opq v9:: A !!(+ > NNN OFE**fsl NNUV\U]^ _'++,>?  '.66:JS:Pbcsbtu;Q (r#c URnURSURSS55n1Skn[UR55n[ X5XAS9 USnUb[ U[ 5(aUS:a[RSU35 USnUS nUb[ U[ 5(d[RS U35 Ub[ U[ 5(d[RS U35 X::a[RS US U35 USn U b[ U [5(d[RSU 35 XR:a&[RSU SUR35 gg)Nr-r>rIr-rrr rrIr6rrrz<`rope_scaling`'s low_freq_factor field must be a float, got z=`rope_scaling`'s high_freq_factor field must be a float, got zc`rope_scaling`'s high_freq_factor field must be greater than low_freq_factor, got high_freq_factor=z and low_freq_factor=r zP`rope_scaling`'s original_max_position_embeddings field must be an integer, got zg`rope_scaling`'s original_max_position_embeddings field must be less than max_position_embeddings, got z and max_position_embeddings=) rJrvrrrrNrCrrr@r) rrrJr-rrrIrrr s r _validate_llama3_parametersrs&&L  l.>.>vt.LMIvM ))+,M=Z ( #F ~Z66&3,QRXQYZ["#45O#$67j%&H&HUVeUfghz2BE'J'JVWgVhij* q 5o5F H (44V'W$'/zBbdg7h7h ^/0 2 (+I+II u/00MfNlNlMm o Jr#c[USS5nUcgURSURSS55n[RU5nUbU"XS9 g[R SUS35 g) zG Validate the RoPE config arguments, given a `PretrainedConfig` object rJNr-rrrzTMissing validation function mapping in `ROPE_VALIDATION_FUNCTIONS` for 'rope_type'='')r=rvROPE_VALIDATION_FUNCTIONSrr)rrrJr- validation_fns r rope_config_validationrsu6>48L  l.>.>vy.QRI-11)rs$ 1.   H %;~*.'+!(& % &(& ^ $(&c](& >5 ! (&X*.'+!(& % &(& ^ $(&c](& >5 ! (&X*.'+!A& % &A& ^ $A&c]A& >5 ! A&JPTz& z&&4z&?G}z& >5 !z&|PTO& O&&4O&?G}O& >5 !O&fPT>, >,&4>,?G}>, >5 !>,J05. $,( $(!% llllC= l # l:[.>[XVY][ \5E \T\]`Ta \ \6F \U]^aUb \? &6? Xc]? D/*:/RU/d! (8! xPS}! L168 %-)  #3 (3- r#