9i#5SrSSKrSSKJr SSKJr SSKrSSKJr SSKJ r J r J r J r J r S/rS rS rS rS rS rS\S\\\\4S\S\S\4 SjrS\S\\S\R2S\4Sjr"SS\ 5rSS\ S\S\S\S\S\S3 -\lS \\S!\\S"\\S\S#\S$\S%\S\\\\4S\S\S\\S&\SS4S'jr\ "\S(9SS).S \\S!\\S"\\S*\\S\S#\S$\S%\S\\\\4S\S\S\\S&\4S+jj5rg),z%Implementation of the Muon optimizer.N)MutableMapping)Optional)Tensor)_disable_dynamo_if_unsupported _params_doc _to_scalar OptimizerParamsTMuongHz>guV @ggn@@gradns_coefficientsns_stepsepsreturnc |US:a [S5e[UR5S:wa [S5e[U5S:wa [S5eUupEnUR5nUR S5UR S5:a UR nUR UR5RUS 95 [U5H<nXwR -n [R"XXUS 9n [R"XzXtS 9nM> UR S5UR S5:a UR nU$) a Newton-Schulz iteration to compute the zeroth power / orthogonalization of G. We opt to use a quintic iteration whose coefficients are selected to maximize the slope at zero. For the purpose of minimizing steps, it turns out to be empirically effective to keep increasing the slope at zero even beyond the point where the iteration no longer converges all the way to one everywhere on the interval. This iteration therefore does not produce UV^T but rather something like US'V^T where S' is diagonal with S_{ii}' ~ Uniform(0.5, 1.5), which turns out not to hurt model performance at all relative to UV^T, where USV^T = G is the SVD. Implementation reference: https://github.com/KellerJordan/Muon/blob/master/muon.py with suggestions by @jxbz, @leloykun, and @YouJiacheng. dzBNumber of steps must be less than 100 for computational efficiencyz)Input tensor gradient must be a 2D matrixz0Coefficients must be a tuple of exactly 3 valuesrr)min)betaalpha)r) ValueErrorlenshapebfloat16sizeTdiv_normclamprangetorchaddmm) rrrrabc ortho_grad_ gram_matrix gram_updates Q/var/www/html/land-doc-ocr/venv/lib/python3.13/site-packages/torch/optim/_muon.py_zeropower_via_newtonschulzr. s3 P   4::!DEE ?q KLLGA!J yy|diil"\\ OOJOO%+++45 8_ <</ kk k [[*M  yy|diil"\\ lr adjust_lr_fn param_shapecUSSup4UbUS:Xa&[R"[SX4- 55nX-$US:Xa&S[R"[X455-nX-$SnX-$)z.Default learning rate adjustment used by Muon.Nroriginalrmatch_rms_adamwg?g?)mathsqrtmax)r0r1r2ABadjusted_ratios r- _adjust_lrr<Js| r?DA|z93q!%=1   * *tyyQ33   r/c^\rSrSrSSSS\\\4\\S4S\ S\ S \ S \ S \ S \ \ \ \ 4S \ S\ S\\SS4U4SjjjrS\S\\S\\S\\4Sjr\R,"5SSj5rSrU=r$)r YgMbP?g?gffffff?TNparamsr0 weight_decaymomentumnesterovrrrr1rc >[U[5(aUR5S:wa [S5eSU::d[SU35eSU::d[SU35eSU::d[SU35eU bU S;a[SU S 35eUUUUUUUU S .n [T U]X5 UR H;n U S H/n U RS :wdM[S U R535e M= g)NrzTensor lr must be 1-elementgz%Learning rate should be >= 0 but is: z momentum should be >= 0 but is: z$weight decay should be >= 0 but is: )r4r5zAdjust learning rate function z is not supported)r0r@rArBrrrr1r?rzIMuon only supports 2D parameters whereas we found a parameter with size: ) isinstancernumelrsuper__init__ param_groupsndimr)selfr?r0r@rArBrrrr1defaultsgroupp __class__s r-rG Muon.__init__Zs" b& ! !bhhjAo:; ;byDRDIJ Jh?zJK Kl"CL>RS S  # = ) 0>OP  (  . (   *&&E8_66Q;$cdedjdjdlcmn%'r/rLparams_with_gradgradsmuon_momentum_bufscUSHnURcM[R"U5(a [S5eURR(a [S5eUR U5 UR UR5 UR UnSU;a0[R"UR[RS9US'UR US5 M g)Nr?z(Muon does not support complex parametersz&Muon does not support sparse gradientsmomentum_buffer) memory_formatF) rr$ is_complex RuntimeError is_sparseappendstate zeros_likepreserve_format)rJrLrPrQrRrMrZs r- _init_groupMuon._init_groupsxAvv~"""#MNNvv"#KLL  # #A & LL JJqME -+0+;+;FF%*?*?,'(  % %e,=&> ?%!(r/c:SnUb%[R"5 U"5nSSS5 URHOnUSnUSnUSn/n/n/n URUUUU 5n [ UUU UUUUSUSUSUSUS U S 9 MQ U$!,(df  No=f) z$Performs a single optimization step.Nr0r@rArBrrrr1) r0r@rArBrrrr1 has_complex)r$ enable_gradrHr]muon) rJclosurelossrLr0r@rArPrQrRr`s r-step Muon.steps  ""$y%&&EtB 0LZ(H-/ "$E/1 ** " K  ")!z* %&7 8%Lz*">2' !'< C%$s B  B)N)__name__ __module__ __qualname____firstlineno__ DEFAULT_A DEFAULT_B DEFAULT_CEPSDEFAULT_NS_STEPSr floatbooltupleintrstrrGrlistrr]r$no_gradre__static_attributes__ __classcell__)rNs@r-r r Ys!7@)Y6W(&*-- - -  -  -ueU23---sm- --^v,F|  !L : ]]_%%r/u Implements Muon algorithm. .. math:: \begin{aligned} &\rule{110mm}{0.4pt} \\ &\textbf{input} : \gamma \text{ (lr)},\ \lambda \text{ (weight decay)},\ \mu \text{ (momentum)},\ \textit{nesterov}\in\{True,False\},\\ &\hspace{13mm}(a,b,c)\ \text{ (NS coefficients)},\ \varepsilon \text{ (epsilon)},\ k \text{ (NS steps)},\ \theta_0 \text{ (params)},\ f(\theta) \text{ (objective)} \\ &\textbf{initialize} : B_0 \leftarrow 0 \text{ (momentum buffer)} \\[-1.ex] &\rule{110mm}{0.4pt} \\ &\textbf{for}\ t=1\ \textbf{to}\ \ldots\ \textbf{do} \\[0.25ex] &\hspace{5mm} g_t \leftarrow \nabla_{\theta} f_t(\theta_{t-1}) \\[0.25ex] &\hspace{5mm} B_t \leftarrow \mu B_{t-1} + g_t \\[0.25ex] &\hspace{5mm} \widetilde{B}_t \leftarrow \begin{cases} g_t + \mu B_t, & \text{if nesterov}=True \\ B_t, & \text{if nesterov}=False \end{cases} \\[1.0ex] &\hspace{5mm} O_t \leftarrow \mathrm{NS}^{(a,b,c)}_{k}\!\big(\widetilde{B}_t;\ \varepsilon\big) \\[0.5ex] &\hspace{5mm} \theta_t \leftarrow \theta_{t-1} - \gamma\,\lambda\,\theta_{t-1} \quad\text{(decoupled weight decay)} \\[0.25ex] &\hspace{5mm} \gamma \leftarrow \mathrm{AdjustLR}\!\big(\gamma;\ \mathrm{shape}\!\big(\theta_t \big) \big) \\[0.25ex] &\hspace{5mm} \theta_t \leftarrow \theta_t - \gamma\, O_t \\ &\rule{110mm}{0.4pt} \\[-1.ex] &\mathbf{return}\ \theta_t \\[-1.ex] &\rule{110mm}{0.4pt}s \end{aligned} Here, :math:`\mathrm{NS}^{(a,b,c)}_{k}(\cdot;\varepsilon)` denotes :math:`k` iterations of the Newton–Schulz orthogonalization operator parameterized by coefficients :math:`(a,b,c)` with numerical stabilization :math:`\varepsilon`. The purpose for :math:`\mathrm{AdjustLR}\!\big(\gamma;\ \mathrm{shape}\!\big(\theta_t \big) \big)` is to make the orthogonalized update have a consistent :math:`RMS` across rectangular matrices. Keller's original implementation scales the update by :math:`\sqrt{\max\!\left(1, \frac{A}{B}\right)}`, where :math:`A` and :math:`B` are dimension of the matrix being optimized. Moonshot's implementation also focuses on matching :math:`RMS` of AdamW. The adjustment is computed as: :math:`\gamma \leftarrow {0.2}\gamma\,\sqrt{\max\!\left({A}, {B}\right)}` The method is adopted from `Muon is Scalable for LLM Training`_. Research results show that with this adjustment Muon can directly reuse the learning rate and weight decay tuned for AdamW. We provide two options for the learning rate adjustment: "original", which follows Keller's implementation, and "match_rms_adamw", which refers to Moonshot's implementation. This gives users the flexibility to choose between the two. If `adjust_lr_fn` is not specified, the default is "original". For further details regarding the algorithm we refer to `Muon: An optimizer for hidden layers in neural networks`_ and `Muon is Scalable for LLM Training`_. z Args: u. Note that Muon is an optimizer for 2D parameters of neural network hidden layers. Other parameters, such as bias, and embedding, should be optimized by a standard method such as AdamW. lr (float, Tensor, optional): learning rate (default: 1e-3). weight_decay (float, optional): weight decay (L2 penalty). (default: 0.1) momentum (float, optional): momentum factor (default: 0.95) nesterov (bool, optional): enables Nesterov momentum. Only applicable when momentum is non-zero ns_coefficients (tuple of three floats, optional): coefficients \(a,b,c\) for the Newton–Schulz orthogonalization polynomial (default: (z, zc)) eps (float, optional): term added to the denominator for numerical stability. (default: uY) ns_steps (int, optional): number of Newton–Schulz iteration steps. (default: a) adjust_lr_fn (str, optional): function to adjust learning rate. One of "original" and "match_rms_adamw". If not specified, we will default to use "original". (default: None) .. _Muon\: An optimizer for hidden layers in neural networks: https://kellerjordan.github.io/posts/muon/ .. _Muon is Scalable for LLM Training: https://arxiv.org/pdf/2502.16982 r?rQrRr@rArBr`c [U5nU (a [S5e[U5HupXnURS:wa [S5eX,nUR USU- 5 U(aUR X5OUn[ UXxU 5n[X:U R5nU RSX4-- 5 U RUU*S9 M g)Nz$Complex parameters are not supportedrz"Param gradient must be a 2D matrixr)r) r r enumeraterIlerp_lerpr.r<rmul_add_)r?rQrRr0r@rArBrrrr1r`iparamrbufupdate adjusted_lrs r-_single_tensor_muonrs BB?@@f%x 99>AB B # $H %-53)3,V_PST 5;;?  1r(() 6+ .&r/)single_tensor_fn)foreachrc \UbU(a [S5e[n U "UUUUUUUUU U U U S9 g)zfFunctional API that performs Muon algorithm computation. See :class:`~torch.optim.Muon` for details. Nz%Foreach is not supported for Muon yet) r0r@rArBrrrr1r`)rWr)r?rQrRrr0r@rArBrrrr1r`funcs r-rbrbCsI*wBCC D  !' ! r/) __doc__r6collections.abcrtypingrr$r optimizerrrr r r __all__rorlrmrnrprsrqrtr.ruSizer<r rvrrrrbrgr/r-rs, *  (     ' '#(u)<#='IL'SX' 'T   %c] 9>   s9sn5l  EFOKrR[Q\\^_h^ijaad`efXXhWi jmK `!/ L!/ <!/V !/  !/  !/!/!/5%./!/!/ !/3-!/!/ !/H 1DE # & L& <&V & d^ &  &&&&5%./&& &3-&&F&r/