i8&SSKrSSKJr SSKrSSKJr \R R SS5r\RS5 \RS5 \RS5 \RS 5 \RS \RRR/S 9 \RS 5 \RS 5 \RS5 \RS5 \RS5 \RS5 Sr Sr \RS5rSgS\S\S\S\\S\S\S\\S\S\S\S\S \S!\S"\S#\S$\S%\S&\4$S'jjr\ "S(5SgS\S\S\S\\S\S\S\\S\S\S\S\S \S!\S"\S#\S$\S%\S&\4$S)jj5rShS*\S+\S,\S-\S.\S/\\S0\\R*S&\4S1jjr\ "S25ShS*\S+\S,\S-\S.\S/\\S0\\R*S&\4S3jj5rS,\S&\\44S4jr\ "S55S,\S&\\44S6j5rS7\S8\S9\S:\S&\4 S;jr\ "S<5S7\S8\S9\S:\S&\4 S=j5rS7\S8\S9\S:\S>\S?\S/\\S@\\S0\\R*S&\4SAjr\ "SB5S7\S8\S9\S:\S>\S?\S/\\S@\\S0\\R*S&\4SCj5r\R"5SD5rSEr\ "SF5SG\SH\SI\SJ\4SKj5rS,\SL\SM\S&\4SNjr\ "SO5S,\SL\SM\S&\4SPj5rS*\SQ\SR\S,\SS\ST\SU\S/\\S0\R*4SVjr\ "SW5S*\SQ\SR\S,\SS\ST\SU\S/\\S0\R*S&\4SXj5r\ "SY5SZ\S[\S\\S]\S$\S^\S_\S0\R*S&\4S`j5rS,\SL\S&\4Sajr\ "Sb5S,\SL\S&\4Scj5rS*\SQ\S,\SS\S/\\S0\R*4 Sdjr \ "Se5S*\SQ\S,\SS\S/\\S0\R*S&\4Sfj5rg)iN)Optional)TensortorchaoFRAGMENTzrowwise_scaled_linear_sparse_cutlass_f8f8(Tensor input, Tensor input_scale, Tensor weight, Tensor weight_meta, Tensor weight_scale, Tensor? bias=None, ScalarType? out_dtype=None) -> TensorzLto_sparse_semi_structured_cutlass_sm9x_f8(Tensor weight) -> (Tensor, Tensor)z\swizzle_mm(Tensor mat1, Tensor mat2, bool mat1_is_swizzled, bool mat2_is_swizzled) -> Tensorzswizzle_scaled_mm(Tensor mat1, Tensor mat2, bool mat1_is_swizzled, bool mat2_is_swizzled, Tensor scale_a, Tensor scale_b, Tensor? bias=None, Tensor? scale_result=None, ScalarType? out_dtype=None) -> TensorzImx_fp8_bf16(Tensor a, Tensor b, Tensor a_scale, Tensor b_scale) -> Tensor)tagsa6qscaled_dot_product(Tensor query, Tensor key, Tensor value, Tensor? attn_mask=None, float dropout_p=0.0, bool is_causal=False, float? scale=None, float q_scale=1.0, int q_zp=0, float k_scale=1.0, int k_zp=0, float v_scale=1.0, int v_zp=0, float a_scale=1.0, int a_zp=0, float o_scale=1.0, int o_zp=0) -> Tensorzida8w4_linear_prepack_cpu(Tensor weight, Tensor scales, Tensor qzeros) -> (Tensor, Tensor, Tensor, Tensor)zda8w4_linear_cpu(Tensor input, Tensor input_scales, Tensor input_qzeros, Tensor weight, Tensor weight_scales, Tensor weight_qzeros, Tensor compensation, Tensor? bias, ScalarType output_dtype) -> Tensorz_scaled_embedding_bag(Tensor qweight, Tensor indices, Tensor offsets, Tensor weight_scale, float o_scale, int mode, bool include_last_offset, ScalarType output_dtype) -> TensorzKfloat8_linear_prepack_cpu(Tensor weight, Tensor scales) -> (Tensor, Tensor)zfloat8_linear_cpu(Tensor input, Tensor input_scales, Tensor weight, Tensor weight_scales, Tensor? bias, ScalarType output_dtype) -> Tensorc^U4SjnU$)NcP>[RRT5"U5$N)torchlibrary register_fakefuncnames J/var/www/html/ai-image-ml/venv/lib/python3.13/site-packages/torchao/ops.py decorator%register_custom_op..decorator4s}}**dV5d;;rrs` rregister_custom_opr3s< rc^U4SjnU$)NcN>[RRTSS9"U5$)Nr) mutates_args)r r custom_oprs rr*register_custom_op_impl..decorator;s$}}&&$r&B4HHrrrs` rregister_custom_op_implr:sI rc[RR[RR55nURS-UR -nU$)N )r cudaget_device_propertiescurrent_devicemajorminor) device_propscompute_capabilitys rcached_compute_capabilityr'AsE::33EJJ4M4M4OPL%++b0<3E3EE rquerykeyvalue attn_mask dropout_p is_causalscaleq_scaleq_zpk_scalek_zpv_scalev_zpa_scalea_zpo_scaleo_zpreturnc[RRRR UUUUUUUUUU U U U U UUU5$)a Quantized SDPA with quantized inputs and outputs. Arguments query: input query tensor, key: input key tensor, value: input value tensor, attn_mask: attention mask tensor, dropout_p: dropout probability, is_causal: causal flag, scale: scaling factor applied prior to softmax, q_scale: scale for query from linear quantization, q_zp: zero point for query from linear quantization, k_scale: scale for key from linear quantization, k_zp: zero point of key from linear quantization, v_scale: zero point for value from linear quantization, v_zp: zero point of value from linear quantization, a_scale: scale for attention from softmax quantization, a_zp: zero point for attention from softmax quantization, o_scale: scale for output from linear quantization, o_zp: zero point for output from linear quantization, Returns output of quantized SDPA )r opsrqscaled_dot_productdefaultr(r)r*r+r,r-r.r/r0r1r2r3r4r5r6r7r8s rr<r<Hs[T 99   0 0 8 8         # rztorchao::qscaled_dot_productcU$r rr>s r_r@s ( Lrinput input_scaleweight weight_meta weight_scalebias out_dtypec n[RRRR XX#XEU5$)a CUTLASS-based row-wise scaled F8F8 linear operator, for sparsified weight case. Args: input: quantized input tensor, in row-major layout. input_scale: scale factors for input tensor, has to be tensor of the same shape as the input tensor, minus the last dimension. weight: sparsified quantized weight matrix, in row-major layout. weight_meta: sparsify metadata for weight tensor. weight_scale: scale factors for weight tensor, one value per row of weight matrix (thus also tensor of the same shape as the weight tensor, minus the last dimension). bias: an optional vector of size equal to number of rows of weight tensor, or None. out_dtype: optional data type for output tensor. Returns: output: result tensor, in row-major layout. )r r;r)rowwise_scaled_linear_sparse_cutlass_f8f8r=)rArBrCrDrErFrGs rrIrIs0. 99   F F N N FY rz2torchao::rowwise_scaled_linear_sparse_cutlass_f8f8cUbUO URnURn[R"/URSSQURSP7XxS9$)Nrdtypedevice)rMrNr emptyshape) rArBrCrDrErFrGrMrNs rr@r@sN#.IK4E4EE \\F ;;;Sb);6<<?;5 XXrch[RRRR U5$)a CUTLASS-based conversion from sparsified input tensor to corresponding compressed tensor, along with corresponding metadata tensor. Args: weight: input tensor, in row-major layout. Returns: weight_compressed: compressed weight tensor, with sparsity eliminated, in row-major layout. weight_meta: metadata tensor, describing the sparsity structure of the input tensor, also in row-major layout. )r r;r)to_sparse_semi_structured_cutlass_sm9x_f8r=rCs rrRrRs% 99   F F N Nv VVrz2torchao::to_sparse_semi_structured_cutlass_sm9x_f8cURUSUSS-5URUS[USS-S5[RS94$)NrrM) new_emptymaxr charrSs rr@r@sV F1IN3Cq Q$;5::N rmat1mat2mat1_is_swizzledmat2_is_swizzledcj[RRRR XX#5$)zG Similar to torch.mm but Tensor inputs can be SwizzleTensor instances. )r r;r swizzle_mmr=r]r^r_r`s rrbrbs- 99   ' ' / / $ rztorchao::swizzle_mmcZURURSURS5$NrrUrZrPrcs rr@r@s% >>$**Q-A 77rscale_ascale_b scale_resultc x[RRRR UUUUUUUUU5 $)zH Similar to torch.mm but Tensor inputs can be SwizzleTensor instances. )r r;rswizzle_scaled_mmr= r]r^r_r`rgrhrFrirGs rrkrksB 99   . . 6 6     rztorchao::swizzle_scaled_mmc ZURURSURS5$rerfrls rr@r@s% >>$**Q-A 77rc[[S5(a [R[R4$[R4$)zGTODO: when e8m0 is hardened and major release lets remove uint8 supportfloat8_e8m0fnu)hasattrr uint8rorrr _get_dtypesrr%s3u&'' U1122 KK>rc^^[5n[R"TRU;U4Sj5 [R"TRU;U4Sj5 g)Nc">STR3$)Nz4A_scale tensor must be uint8 or float8_e8m0fnu, got rY)A_scalesr%_check_scale_dtypes..2Fw}}oVrc">STR3$)Nz4B_scale tensor must be uint8 or float8_e8m0fnu, got rY)B_scalesrrvrw6rxr)rrr _checkrM)rurzallowed_dtypess`` r_check_scale_dtypesr}-sB ]N LL 'V LL 'Vrztorchao::mx_fp8_bf16ABrurzc[R"URS5URS54[RURS9$)zMeta impl for mx_fp8_bf16rrUrL)r rOsizebfloat16rN)r~rrurzs rmeta_mx_fp8_bf16r:s4 ;;q 166!9-U^^AHH UUrscalesqzeroscj[RRRR XU5$)z Prepack weights for DA8W4 linear operator on CPU. Args: weight: weight tensor. scales: scales for weight tensor. qzeros: zero points for weight tensor. Returns: packed weight, scales, and zero points. )r r;rda8w4_linear_prepack_cpur=rCrrs rrr@s' 99   5 5 = =ff UUrz!torchao::da8w4_linear_prepack_cpuc2XU[R"54$r )r rrs rr@r@Qs 65<<> 11r input_scales input_qzeros weight_scales weight_qzeros compensationc x[RRRR UUUUUUUUU5 $)a DA8W4 linear operator on CPU. Args: input: input tensor. input_scales: scales for input tensor. input_qzeros: zero points for input tensor. weight: weight tensor. weight_scales: scales for weight tensor. weight_qzeros: zero points for weight tensor. compensation: compensation tensor for weight. bias: optional bias tensor. out_dtype: output data type. Returns: output tensor in out_dtype. )r r;rda8w4_linear_cpur=) rArrrCrrrrFrGs rrrVsB4 99   - - 5 5    rztorchao::da8w4_linear_cpuc UR5S:XdeURS5URS5-S-n UR"/URSSQU P7SU06$)NrrVrKrMdimrrZrP) rArrrCrrrrFrGNs rr@r@}s^ ::<1   AQ'!+A ?? AEKK, Aa Ay AArztorchao::_scaled_embedding_bagqweightindicesoffsetsw_scalesmodeinclude_last_offsetcpUS:XdeURSS- nURXRSUS9$)NTrrUrY)rPrZ) rrrrr7rrrG batch_sizes rr@r@sD $ && &q!A%J   Zq)9  KKrch[RRRR X5$)z Prepack weights for float8 linear operator on CPU. Args: weight: weight tensor. scales: scales for weight tensor. Returns: packed weight, packed scales )r r;rfloat8_linear_prepack_cpur=rCrs rrrs% 99   6 6 > >v NNrz"torchao::float8_linear_prepack_cpucX4$r rrs rr@r@s >rcr[RRRR UUUUUU5$)a float8 linear operator on CPU. Args: input: input tensor. input_scales: scales for input tensor. weight: weight tensor. weight_scales: scales for weight tensor. bias: optional bias tensor. out_dtype: output data type. Returns: output tensor in out_dtype. )r r;rfloat8_linear_cpur=)rArrCrrFrGs rrrs9( 99   . . 6 6    rztorchao::float8_linear_cpucUR5S;deUR5S:Xa#URS5URS5-OURS5nUR"/URSSQUP7SU06$)N)rVrrrrrKrMr)rArrCrrFrGrs rr@r@sp ::<6 !! !+1::<1+< AQ'&++a.A ?? AEKK, Aa Ay AAr)NgFN?rrrrrrrrr)NN)! functoolstypingrr rr Librarylibdefine_CTagneeds_fixed_stride_orderrr lru_cacher'floatboolintr<r@rMrIrRrbrkrrr}rrrrrrrrrs    mmIz2 C R b T  O ((,, / / 0  } o P w Q Q  #'!#< < < < <  <  < E?<< << << << < !<" #<$ %<~23 #'!#         E?     !" #$ %48"'+      6   $ 8HI"'+Y YY Y Y  Y 6  Y $Y YJY" W  W f WHI    f J  26JN )*8 88268JN8 8+8         6 6" $ 801 8  8  8 8 8  8  8 6  86" 8 $ 8  82 8  *+VV6VFVVV,V V V V V V"782f2f2f2292$ $$$  $  $  $$ 6 ${{$N/0 B  B B B  B  B  B B 6  B{{ B  B1 B 45 L  L  L L L  L  L L{{ L  L6 L O  O  O  O89ff:    6   {{ <01 B  B B  B B 6  B {{ B  B2 Br