i?(SrSSKrSSKJrJrJrJrJr SSKrSSK J r J r SSK J r SSKJrJr \R"r"SS\5rS \S \S \S \\\44S jrS\R"S\\S44SjrS+S \S\S \S\S\R,S\\S\\S\S \4SjjrS\R"S\\S\S\S\S\S \R"4SjrS \R"S \4S!jrS \R"S \4S"jrS \R"S \4S#jrS \R"S \4S$jrS%\\ \\ \ 4\\ 4S \4S&jrS'\\\ \\ \ 4\\ 4S \\ \ 44S(jr S)\\ \ 4S S4S*jr!g),z; Defines an nn module designed to be used during inference N)List NamedTupleOptionalTupleUnion) is_row_majorpad_tensor_for_matmul)FP8Granularity)is_MI300is_sm_at_least_89cD\rSrSr%SrSr\\S'Sr\\S'Sr \\S'Sr g) Float8MMConfiga{ Configuration for the scaled_mm in the forward and backward pass. Attributes: emulate (bool): Whether to emulate the matmuls in fp32. use_fast_accum (bool): Whether to use the fast-accumulation option for scaled_mm. pad_inner_dim (bool): Whether to pad the inner dimension of a and b with 0s. This is needed for matmuls not aligned to 16. Femulateuse_fast_accum pad_inner_dimN) __name__ __module__ __qualname____firstlineno____doc__rbool__annotations__rr__static_attributes__rW/var/www/html/ai-image-ml/venv/lib/python3.13/site-packages/torchao/float8/inference.pyrrs'GT ND M4rra_datab_datascaled_mm_configreturncUR(acURS5URS5:Xd+SURS5SURS535e[USS9n[USS9n[UR 55(dUR 5n[UR 55(a,UR 5R 5R 5nX4$)zPreprocess the inner fp8 data tensors for admmm Args: a_data: Input tensor A. b_data: Input tensor B. scaled_mm_config: Configuration for _scaled_mm. Returns: Preprocessed tensors A and B in the format for _scaled_mm. rz"Inner dims must match for mm, got z and )dims)rsizer rstride contiguoust)rrr s rpreprocess_datar))s%%{{1~Q/ 0Q0@fkkRSnEU V /'vA6&vA6   ( (""$FMMO$$&&(**, >r input_scale input_shape.cUR5S:XaURSS5$URS5nUR5S:aURSURS5nU$)z:Ensures input tensor is correctly formatted for _scaled_mmr#)numelreshape unsqueezedimshape)r*r+s rpreprocess_scaler4Csna""1a((''+K1!))"k.?.?.CD ra_scaleb_scale output_dtype output_scalebiasrc U[R:Xa!Ub[R"UUUUUUUS9nX-$[R"UUUUUUUUS9$)z This is the unwrapped version of addmm_float8, which does not take in Float8TrainingTensors as inputs. This is used to standardize the logic between subclassed and non subclassed versions of the linear module. )scale_ascale_b scale_result out_dtyper)r;r<r9r=r>r)torchfloat32 _scaled_mm) rr5rr6r7r8r9routputs r addmm_float8_unwrapped_inferencerCUso u}}$)9!!  %") }    !%  rscale data_shaper2startendstepc^^[RRnTRT:XaURR TX#XE5$[ UU4Sj[[T5555nU[U5:aT$XrnUS:XaURR TX#XE5$UbX8-OSn Ub XH-S- U-OSn US:a [S5eURR TX)U S5$)z Slice the scale tensor appropriately based on the data tensor slicing. This function calculates how the scale should be sliced when the data tensor is sliced along a given dimension, taking into account the block structure. c3N># UHnTUTRU-v M g7f)N)r3).0irErDs r -_slice_scale_for_dimension..s$XAWA 1 Q7AWs"%r#Nz;Slicing with step > 1 is not implemented for scale tensors.) r?opsatenr3sliceTensortuplerangelenNotImplementedError) rDrEr2rFrGrHrP block_sizesblock_size_for_dim scale_start scale_ends `` r_slice_scale_for_dimensionr[}s 99>>D {{j zz  C>>Xs:AWXXK c+ $)Qzz  C>>6;5Fe1D  % ).@ @  !8%M zz  )QGGrxc[US5(dS5e[UR5SUR5S- -URS4-:H$)rChecks if a quantized tensor is rowwise scaled Args: x: quantized tensor (should have `block_size` attribute) block_size.Expecting input to have `block_size` attribute)r#r#r-)hasattrrSr_r2r3r\s r_is_rowwise_scaledrcsO 1l # #U%UU #  $!%%'A+"6!''"+"G GGrc^[TS5(dS5e[U4Sj[TR555$)r^r_r`c3># UH=nTRUS:H=(d TRUTRU:Hv M? g7f)r-N)r_r3)rKrLr\s rrM(_is_tensorwise_scaled..s>HU1 Q2>aAGGAJ!>> sAA)raallrTndimrbs`r_is_tensorwise_scaledris@ 1l # #U%UU # HMaff  rc[US5(dS5eURn[U5S:=(a+ [R"USS5S:H=(a USS:H$)zChecks if a quantized tensor is scaled with a block size of 1x128 Args: x: quantized tensor (should have `block_size` attribute) r_r`r.Nr-r#)rar_rUmathprodr\bs r_is_1_128_scaledrpsZ 1l # #U%UU # A q6Q; B499QsV,1 BaeslBrc[US5(dS5eURn[U5S:H=(a USS:H=(a USS:H$)zChecks if a quantized tensor is scaled with a block size of 128x128 Args: x: quantized tensor (should have `block_size` attribute) r_r`r.rrkr#)rar_rUrns r_is_128_128_scaledrrsN 1l # #U%UU # A q6Q; 61Q43; 61Q43;6rgcSSKJn [U5S:H=(a' USU"SS/5:H=(a USU"SS/5:H$)Nr)PerBlockr.r#rk) torchao.quantization.granularityrurU)rsrus r!_granularity_is_a_1_128_w_128_128rwsF q6Q; V1Q48QH#55 V!A$(CQT:BV:VVr granularitycFSSKJnJn SnUcU"5U"54nU$[XU45(aX4nU$[U[[ 45(a[ U5S:Xa[USU5=(a [USU5n[USU5=(a [USU5n[U5nU(dU(dU(d[SUS35e[US[US55(d[SUS35e[ U5nU$[SUS35e) NrPerRow PerTensorr.r#zUnsupported granularity types: .zEDifferent granularities for activation and weight are not supported: z#Invalid granularity specification: ) rvr{r| isinstancerSlistrUrw ValueErrortype)rxr{r|processed_granularity is_per_tensor is_per_rowis_a_1_128_w_128_128s r_normalize_granularityrsH !!*ik :* ! ) KV!4 5 5!, :& ! % K% / /C 4D4I";q>9= * NIC  A7 J NF=  AM/C>{m1MN N+a.${1~*>??WXcWddef !&k 2 ! >{m1MNNr granularitiescSSKJnJn [USU5=(a [USU5n[USU5=(a [USU5n[ U5nU(dU(an[ R R5(dJ[ RR5(a[5(d[5(dS5egggU(a[5(dS5eg[SUS35e) a Validate that the hardware supports the requested granularities. Args: granularities: Tuple of (activation_granularity, weight_granularity) Raises: AssertionError: If hardware doesn't support the requested granularity ValueError: If invalid granularity type is provided rrzr#uUFloat8 dynamic quantization requires CUDA compute capability ≥8.9 or MI300+ or XPU.u[Float8 1x128 activation and 128x128 weight scaling requires CUDA compute capability ≥8.9.zInvalid granularities r}N) rvr{r|r~rwr?xpu is_availablecudar r r)rr{r|rrrs r_check_hardware_supportrs }Q/; a)AMM!,f5*a&;J=]K yy%%'' JJ # # % %*;*=*=  d  AK*=(  "" i "1-BCCr)NNF)"rrltypingrrrrrr?torchao.float8.float8_utilsrr torchao.float8.typesr torchao.utilsr r rRrr)intr4dtyperrCr[rcrirprrrrwrrrrrrsh  ;; K/   Z    % 66> 4%,,U38_0&*! % % % % % ++ % 6" % 6 %% %P0H <<0HS 0H 0H  0H 0H  0H \\0HfH%,,H4HU\\dC CC7%,,747 W  nn,- ^  W  W%!  ..0 1   " %! >> )*%!P&D78&D &Dr