i%* SrSSKrSSKrSSKrSSKrSSKrSSKJr SSKJ r J r SSK J r SSK JrJrJrJrJrJr SSK Jr SSKrSSKJr SSKJs Js Jr SSKrSSKJr SSKJrJ r J!r!J"r" SS K#J$r$ SS K%J&r& SS K'J(r( SS K)J*r*J+r+J,r,J-r-J.r. SS K/J0r0 SSK1J2r2 SSK3J4r4 SSK5J6r6 SSK7J8r8J9r9J:r:J;r;Jr>J?r?J@r@JArAJBrBJCrCJDrDJErEJFrFJGrG SSKHJIrIJJrJ SSKKJLrLJMrMJNrNJOrO SSKPJQrQJRrR SSKSJTrTJUrUJVrVJWrWJXrX SSKYJZrZJ[r[ SSK\J]r]J^r^ SSK_J`r` SSKaJbrbJcrcJdrdJere SSKfJgrgJhrh SSKJiri \R"\k5rl/SQrm\!\dR/0ro\!S0rpSrS\\\S 4S!S4S"jjrqS#rrSsS$jrsStS%jrtSS&.S'\RS(\\4S)\\4S*\\\R.R\v/\w44S+jjrxS,rySSS-.S.jrz\rS4S'\R.RS/\S*\\\R.R\v/\w4S0\\RR4S1jjr|S2\RS!\R4S3jr~S2\RS!\R4S4jrS2\RS!\R4S5jr\ "S6S7\55rSSS8.S9\\RS:\\R4S;jjr\J"\5S\R.RS/\S?\vS9\\RS:\\RS!\R.R4 S@jj5r\ "SASB\55rSCr\J"\5S\R.RS/\S?\vS!\R.R4SEjj5r\ "SFSG\55r\J"\5S\R.RS/\S?\vS!\R.R4SHjj5r\ "SISJ\55rSKr\J"\5S\R.RS/\S?\v4SLjj5rS2\RS!\R4SMjrS2\RS!\R4SNjrSO\TSP\TS!S4SQjr\ "SRSS\55rSTr\J"\5S\R.RS/\S!\R.R4SUjj5r\ "SVSW\55r\J"\5S\R.RS/\4SXjj5r\ "SYSZ\55rS[r\J"\5S\R.RS/\S?\vS!\R.R4S\jj5r\ "S]S^\55rS_r\J"\5S\R.RS/\S?\v4S`jj5rSa\DSb\RSc\GR4S!S4Sdjr\ "SeSf\55rSSS8.S9\\RS:\\R4Sgjjr\J"\5S\R.RS/\S?\vS9\\RS:\\RS!\R.R4 Shjj5r\ "SiSj\55r\rSk\\R.R\/\R.R4S!\w4SljrS>\R.RSm\vS/\4SnjrSm\vS/\4SojrS>\RSm\vS/\4SpjrS>\(S!\GRJ4Sqjr\GRNGRQ\~\/5 g)uap Quantization APIs Generally these APIs can be applied directly to any model with Linear modules to obtain quantized linear ops. The intended usage involves applying torch.compile to the model afterwards both because primitives were designed based on the fusions that come along with it and because that is how we access the intended quantized and mixed GEMM kernels N) OrderedDict) dataclassfield)partial)AnyCallableListOptionalTupleUnion) AOBaseConfig)AffineQuantizedTensor PlainLayoutTensorCoreTiledLayoutto_affine_quantized_intx)Layout) e4m3_dtype) Float8Linear)Float8MMConfigFP8Granularity_check_hardware_support!_granularity_is_a_1_128_w_128_128_normalize_granularity)(Float8StaticActivationFloat8WeightConfig)$LinearActivationWeightObservedTensor)AffineQuantizedObserverBase)KernelPreference)Float8PackingFormat Float8TensorInt4ChooseQParamsAlgorithmInt4PackingFormatInt4PlainInt32TensorInt4PreshuffledTensor Int4TensorInt4TilePackedTo4dTensor Int8TensorIntxChooseQParamsAlgorithmIntxOpaqueTensorIntxPackingFormatIntxUnpackedToInt8TensorQuantizeTensorToFloat8KwargsQuantizeTensorToInt8KwargsSparse2x4CUTLASSFloat8Tensor)_QUANTIZE_CONFIG_HANDLER register_quantize_module_handler)_fp8_mm_compat_linear_extra_repr_module_extra_repr_quantization_type)is_MI300is_sm_at_least_89) GranularityPerAxisPerGroupPerRow PerTensor)LinearActivationQuantizedTensorto_linear_activation_quantized)Int4WeightOnlyQuantizerInt8DynActInt4WeightQuantizer) intx_quantization_aware_training)_DTYPE_TO_QVALUE_BOUNDS MappingTypeZeroPointDomainquantize_affine) QuantizerTwoStepQuantizer)_get_per_token_block_size) swap_conv2d_1x1_to_linearrErFr>_get_subclass_inserter quantize_r@r?ModuleFqnToConfigF extra_args.returnc $U"XSS5(aUbURUS9 U"U/UQ76nU$[UR55nUH1upx[UUUUUS3UU5n XLdM U cM%[ XU 5 M3 UbURUS9 U$)a Recursively replaces each child module in `model` with the result of `replacement_fn(child)` if `filter_fn(child)` returns `True`. Args: model (torch.nn.Module): The model containing modules to be replaced. replacement_fn (Callable[[torch.nn.Module], torch.nn.Module]): The function to replace matching modules. filter_fn (Callable[[torch.nn.Module], bool]): The filter function to determine which modules to replace. cur_fqn (str, optional): The current fully qualified name of the module being processed. Defaults to "". device (device, optional): Device to move the model to before applying `filter_fn`. Defaults to None. extra_args (Tuple[Any, ...], optional): optional extra args to pass to `replacement_fn`. Returns: None Ndevice.)tolistnamed_children)_replace_with_custom_fn_if_matches_filtersetattr) modelreplacement_fn filter_fncur_fqnrQrLnamed_children_listnamechild new_childs ]/var/www/html/ai-image-ml/venv/lib/python3.13/site-packages/torchao/quantization/quant_api.pyrVrVs. %%   HHFH #u2z2 "5#7#7#9:.KDA)D6# I%)*?Y//   HHFH # cSSKJn [U[RR 5=(a [ US5=(a [UR[5(+=(a{ [UR[5(+=(aU [URU5(+=(a3 [U[RRR5(+$)Nr)_AffineFakeQuantizedTensorweight) 5torchao.quantization.qat.affine_fake_quantized_tensorrc isinstancetorchnnLinearhasattrrdrr<moduleslinearNonDynamicallyQuantizableLinear)modargsrcs r` _is_linearrps 3( S C " S3::'<= = S3::'FG G S3::'AB B  S 3 1 1 Q QR R rac l^^^^^TRSS5mTRSS5mUUUUU4SjnU$)a Returns a function which inserts the given subclass into all linear modules in the model. The inserted module will have its weight set to the result of `cls(mod.weight, **kwargs)`. If parametrization is enabled then this will be done using torch.nn.utils.parametrize instead of directly setting the attribute on the module. Args: cls (torch.Tensor): The class to insert as a child module. kwargs (Any): Any additional arguments for the constructor. constructorsubclass_constructormethod from_floatc>T(a[RRTR"UR40TD6SS9UlURR 5up[ R"US[TT5"U65 U$[RR[TT5"UR40TD6SS9UlU$)NF requires_gradrd) rgrh Parameterrurd__tensor_flatten__ parametrizeregister_parametrizationgetattr)lin_roclsrrenable_parametrizationrukwargss r`insert_subclass/_get_subclass_inserter..insert_subclasss !++szz4V4E,CJjj335GA  0 0XwsK8$?  ++Z(>v>#,CJ  ra)pop)rrrrrrrus``` @@r`rIrIs8**],BCKHl3J" racx^"SS[RR5mU4SjnUcSn[XUS9 g)za Changes all conv2d 1x1 modules to equivalent linear modules so that they can then be quantized. c.^\rSrSrU4SjrSrSrU=r$)2swap_conv2d_1x1_to_linear..PermuteSandwichc.>[TU]5 XlgN)super__init__rn)selfrn __class__s r`r;swap_conv2d_1x1_to_linear..PermuteSandwich.__init__s G  HracrURUSRSSSS55RSSSS5$)Nrr6)rnpermute)rros r`forward:swap_conv2d_1x1_to_linear..PermuteSandwich.forwards588DGOOAq!Q78@@Q1M Mra)rn)__name__ __module__ __qualname____firstlineno__rr__static_attributes__ __classcell__)rs@r`PermuteSandwichrs  N NrarcT>URS:Xde[RRURUR UR SLS9n[RRURRSS55UlUR UlT"U5$)Nr6r6)biasrO) kernel_sizergrhri in_channels out_channelsrryrdsqueeze)convr~rs r`replace_conv2d_1x15swap_conv2d_1x1_to_linear..replace_conv2d_1x1s6)))hhoo   d//tyyD7H XX'' (;(;B(CD 99s##raNcv[U[RR5=(a URS:H$)Nr)rfrgrhConv2dr)rnros r`+swap_conv2d_1x1_to_linear.. s,z ( ((oo'((rarZ)rgrhModulerV)rXrZrrs @r`rHrHs< N%((//N$( . YrarrXinput_observerweight_observerrZcp^^S[R4UU4Sjjn[UUUc [5 gU5 g)a6 Converts the weight of a linear module to a LinearActivationWeightObservedTensor. This function wraps the weight of the given linear module with a LinearActivationWeightObservedTensor, which enables observation of both input and weight tensors during forward passes. The wrapped weight is then re-wrapped as a nn.Parameter to maintain compatibility with PyTorch's module system. Example:: ``` import torch import torch.nn as nn from torchao.quantization import PerTensor from torchao.quantization.linear_observer_tensor import insert_observers_ from torchao.quantization.observer import ( AffineQuantizedMinMaxObserver, MappingType ) # Create observers input_observer = AffineQuantizedMinMaxObserver( MappingType.SYMMETRIC, torch.float8_e4m3fn, granularity_type=PerTensor(), eps=torch.finfo(torch.float32).eps, scale_dtype=torch.float, zero_point_dtype=torch.int, zero_point_domain=ZeroPointDomain.NONE, ) # Create a linear module linear_module = nn.Linear(10, 20) # Convert the linear module's weight to an observed tensor insert_observers_(linear_module, input_observer, weight_observer=None) # The linear_module can now be used as usual, with observers calculating statistics output = linear_module(torch.randn(10, 10)) # Get the scale and zero point of the input observer scale, zero_point = linear_module.weight.input_observer.calculate_qparams() ``` Args: model (nn.Module): The nn.Module to convert. input_observer (Optional[AffineQuantizedObserverBase]): Observer for input tensor. weight_observer (Optional[AffineQuantizedObserverBase]): Observer for weight tensor. filter_fn (Optional[Callable[[torch.nn.Module, str], bool]]): Filter function to select which modules to convert. If not provided, all linear modules will be converted. This function should take a module and its fully qualified name. Returns: nn.Linear: The modified linear module with its weight wrapped in a LinearActivationWeightObservedTensor. linear_modulec>[R"[R"URTTS9URR S9UlU$)N)rrrw)rhryrrurdrx)rrrs r`convert_to_linear_observer5insert_observers_..convert_to_linear_observerTsJ!|| 0 ; ;$$- /  (..<<  raN)rhrirVrp)rXrrrZrs `` r`insert_observers_rs;| "))  . "' .7racSURRSSURRSS[UR53$)Nznum_embeddings=rz, embedding_dim=r6z , weight=)rdshaper3rs r`_embedding_extra_reprrgsb T[[..q122B4;;CTCTUVCWBXXabtuyvAvAcBbC DDra)allow_requires_gradpropagate_biasc ^^^^UUUU4SjnU$)zHelper function to apply the constructor that quantizes the weight Tensor (with additional kwargs) to the weight of linear module c>T=(a URRnTS:XaURTS'[RR T"UR40TD6US9Ul[ R"[U5Ul U$)NTrrw) rdrxrrgrhrytypes MethodTyper1 extra_repr)r~rxrrrrrs r`r6_get_linear_subclass_inserter..insert_subclassrsv+H 0H0H T ! XXF6NXX''  -f -](  ))*rrrr)rB SYMMETRICrgrrrGrrrrrrrs r`_int8_symm_per_token_quantrsS((L::L CII # !!$ MM  rac.\rSrSr%Sr\R r\R\ S'\ "S5r \ \ S'\ Rr\ \ S'Sr\\R\ S'\ R$r\ \ S '\R*r\\ S '\R0r\\ S 'S r\\ S 'SrSrg)%Int8DynamicActivationIntxWeightConfigia\ Configuration for dynamically quantizing activations to torch.int8 and weights to torch.intx, with 1 <= x <= 8. More specifically, activations are dynamically quantized to 8-bits at a per-token granularity with scales/zeros. Weights are quantized with scales/zeros in a groupwise or channelwise manner using the number of bits specified by weight_dtype. args: `weight_dtype`: The dtype to use for weight quantization. Must be torch.intx, where 1 <= x <= 8. ` weight_granularity`: The granularity to use for weight quantization. Must be PerGroup or PerAxis(axis=0). `weight_mapping_type`: The type of mapping to use for the weight quantization. Must be one of MappingType.ASYMMETRIC or MappingType.SYMMETRIC. MappingType.SYMMETRIC requires ZeroPointDomain.NONE `weight_scale_dtype`: The dtype to use for the weight scale. `act_mapping_type`: The type of mapping to use for the activation quantization. Must be one of MappingType.ASYMMETRIC or MappingType.SYMMETRIC. `intx_packing_format`: The format to use for the packed weight tensor (version 2 only). - unpacked_to_int8: this format is the default and is intended for export applications like ExecuTorch. - opaque_torchao_auto: this format is optimized for CPU performance. `intx_choose_qparams_algorithm`: The algorithm to use for choosing the quantization parameters. `version`: version of the config to use, only subset of above args are valid based on version, see note for more details. Example: .. literalinclude:: ../../examples/inference/int8_dynamic_activation_intx_weight.py :language: python weight_dtype weight_granularityweight_mapping_typeNweight_scale_dtypeact_mapping_typeintx_packing_formatintx_choose_qparams_algorithmrversionc 2[RRS5 UR[ SS5Vs/sHn[ [SU35PM sn;dSUR35e[ UR[[45(dSUR35e[ UR[5(a8URRS:XdSURR35eUR[R[R[R4;dS UR35eUR [R[R4;dS UR 35egs snf) Nz:torchao.quantization.Int8DynamicActivationIntxWeightConfigr6 int $TT J4K\K\J] ^ T$11GX3FGG FtG^G^F_ ` G d--w 7 7**//14 *4+B+B+G+G*HI 4''  " "  ! !  1 1,   MNRNfNfMg h   $$  " "  ! !)   aaeavav`w x   #%UsFr)rrrr__doc__rgrrdtype__annotations__r9rr7rBrrrr rrr)UNPACKED_TO_INT8rr'AFFINErrr rrrrar`rrs2!& L%++*&.rl 2'2'<'<<04-4$/$:$:k:->-O-O*O"))"#=GS rar custom_scalecustom_zero_pointrrc FURnURnURnURnURn UR n UR n UR5S:XdSUR535e[U[5(a URn OW[U[5(a4URS:XdSUR35eURSn O[SU35eSU 4n URS:XdeU [ R":Xde[$R&[$R([$R*[$R,/nU [$R.:XdX;d SU 35eUb=UR0[2R4:XaUR7[2R85n[:R<"UU UUS U UUS 9nUbXR0:wa [?XU5 UnX;a[@RB"UUU S 9nSnUU4$) NrzFInt8DynamicActivationIntxWeightConfig only works for 2-d Tensor, got: r%axis must be 0 with PerAxis, but got rOz4weight_granularity must be PerGroup or PerAxis, got r6Unsupported packing format: int8_asym_per_token)ractivation_quantizationrrr)rr)"rrrrrrrdimrfr9 group_sizer8r rrrrBrr)OPAQUE_ATEN_KLEIDIAIOPAQUE_TORCHAO_AUTOOPAQUE_TORCHAO_KLEIDIAIOPAQUE_TORCHAO_LOWBITrrrgrrSrr*from_hp+_adjust_scale_dtype_in_intx_unpacked_tensorr(!from_intx_unpacked_to_int8_tensor)rdrrrrrrrrrrrr  block_sizeopaque_formats new_weightnew_biass r`4_int8_dynamic_activation_intx_weight_quantize_tensorr,IsF&&L22 4422.. 44$*$H$H! ::<1  PQWQ[Q[Q]P^_ $h//'22 & 0 0!&&!+ 34F4K4K3L M +\\"% BCUBV W  ZJ >>Q   {55 55 5..--11// N 0AAA  0< &&9%:;< 1$):)@)@EKK)O-00<)11( 5&C!+ J%*< *L3  2 H,%GG X;N  x rard)parameter_namerrrr-c [[X5URUUUS9upV[UU[R R USS95 UcSUl[R"[[URUS9U5Ul U$)NrFrworiginal_extra_reprr-) r,r}rrWrgrhryrrrr2r)rrr-rrr*r+s r`._int8_dynamic_activation_intx_weight_transformr1sP' !+ J  :U;  ((  & 1 1)  F Mrac\rSrSr%SrSr\\S'Sr\ \S'\ Rr \ \S'\ Rr\ \S'S r\\S 'S r\\S 'S rSrg)Int4WeightOnlyConfigia Configuration for int4 weight only quantization, only groupwise quantization is supported right now, and we support version 1 and version 2, that are implemented differently although with same support. In version 2, different target are mainly distinguished by `packing_format` arg, and in version 1, mainly by `layout`. Args: `group_size`: parameter for quantization, controls the granularity of quantization, smaller size is more fine grained, choices are [256, 128, 64, 32], used in both version 1 and 2 `int4_packing_format`: the packing format for int4 tensor, used in version 2 only `int4_choose_qparams_algorithm`: variants of choose qparams algorithm to use for int4, currently support TINYGEMM ("tinygemm") and HQQ ("hqq"), used in version 2 only `set_inductor_config`: if True, adjusts `torchinductor` settings to recommended values. used in both version 1 and 2 `version`: version of the config to use, default is 2 `int4_tile_packed_ntile`: ntile size for TILED_PACKED_TO_4D format, default is 8 for CUDA platform, 16 for ROCm platform Example: .. literalinclude:: ../../examples/inference/int4_weight_only.py :language: python r Tset_inductor_configint4_packing_formatint4_choose_qparams_algorithmint4_tile_packed_ntilerrcpURS;dS5e[RRS5 g)N)r8z-int4_tile_packed_ntile must be either 8 or 16z)torchao.quantization.Int4WeightOnlyConfig)r9rgrrrs r`r"Int4WeightOnlyConfig.__post_init__s5**g5 ; 5 $$%PQrarN)rrrrrr r rr5boolr!PLAINr6r TINYGEMMr7r9rrrrrar`r3r3sc*J $$->-D-D*D"++"#=#$C#GSRrar3cURnURnURnURnURSU-S:wa'[ R SURSU35 U$[[URS- 5Vs/sHnSPM snU/-5nURS:Xde[U5nU[R:XaU[R:Xd SUS35eU[R :Xa&["R$"UU[&R(S 9nU$U[R*:Xa[,R$"UU5nU$U[R.:Xa[0R$"UU5nU$U[R:Xa[2R$"UUUUS 9nU$[5S U35es snf) NrOrzZSkipping quantizing weight with int4 weight only quantization because the shape of weight z# is not compatible with group_size r6rzBInt4ChooseQParamsAlgorithm.HQQ is not supported by packing format zF, it's only supported by Int4PackingFormat.TILE_PACKED_TO_4D currentlyactivation_dtype)r7 ntile_sizez!Unsupported int4 packing format: )r r7r6r9rloggerinfotupler ndimrrTr HQQr!TILE_PACKED_TO_4D PRESHUFFLEDr#r%rgbfloat16r>r$ PLAIN_INT32r"r%r) rdrr r7r6r9rr(r*s r`!_int4_weight_only_quantize_tensorrMs ""J$*$H$H! 44#:: ||B*$) hioiuiuhvwZ[eZf g  5q#9:#9a#9:j\IJJ >>Q  j!J$(B(F(FF"&7&I&II PQdPefS T I /;;;*22  "^^   1 7 7 7''     1 = = =)11     1 C C C-55  *G-  <=PBRR\] &+ , - *3'J  :U; ((  & 1 1)  F Mrac(\rSrSr%SrSr\\S'Srg)'Float8DynamicActivationInt4WeightConfigi9aConfiguration for apply float8 dynamic per row quantization and int4 per group weight quantization to linear (only group_size 128 is supported right now since underlying kernel used only supports 128 and above and no benefits of making it bigger) Args: `int4_packing_format`: how the weight is packed, supported values are "preshuffled" and "plain" Example: .. literalinclude:: ../../examples/inference/float8_dynamic_activation_int4_weight.py :language: python preshuffledr6rN) rrrrrr6r!rrrrar`rZrZ9s .;*:rarZc [X5(dSUS3SUS3-5eURnUS;d SU35e[X5nSn[[ UR S- 5Vs/sHnSPM snU/-5nUS :Xa/[ R"URU[RS 9nO.[R"URU[RS 9n[UU[RRUS S 95 [R "[#[$UR&US 9U5UlU$s snf)NzMapplying float8 dynamic activation int4 weight quant requires module to have rPrQrR)r[plainzIonly preshuffled and plain int4_packing_format supported right now, got: r4r6r[rAFrwr/)rjr6r}rTr rGr#r%rdrg float8_e4m3fnr$rWrhryrrrr2r) rrr-r6rdr rr(r*s r`0_float8_dynamic_activation_int4_weight_transformr_Lsj 6 * * WXfWggqr &+ , - *!44 #  TTgShi   V ,FJ% a"89"8Qq"89ZLHIJm+*22 MM "00  '' MM "00   :U; ((  & 1 1)  F M;:s+ Ecn\rSrSr%SrSr\\\S'\ "5r \\ \S'Sr \ \S'Sr\\S 'S rS rg) Int8WeightOnlyConfigia Configuration for applying int8 weight-only symmetric per-channel quantization to linear layers. Args: group_size (version 1) - Controls the granularity of quantization. If None, applies per-channel quantization. Otherwise, applies per-group quantization with the specified group size. granularity (version 2) - Quantization granularity. PerRow() for per-channel quantization, PerTensor() for per-tensor quantization, PerGroup(group_size) for per-group quantization. set_inductor_config: bool = True - If True, adjusts `torchinductor` settings to recommended values for better performance with this quantization scheme. Example: .. literalinclude:: ../../examples/inference/int8_weight_only.py :language: python Nr  granularityTr5r6rc>[RRS5 URS:XanURb"SURSURS35e[ UR [[[45(dSUR 35egg)Nz)torchao.quantization.Int8WeightOnlyConfigrz1Only support version 2 with group_size=None, got z. Use granularity=PerGroup(z ) instead.z!A'>?:,NO -    -   ~~"K&:6>>:J$KK"''.get_weight_block_sizeis?U1557Q;%78%7!%78AGGBK=HI I8s A)rr_layoutzero_point_domainrrfrbr)rbact_quant_kwargs)rrzrr{rrDrErBrrCNONErgrrrrrirsrqrrr=r&rrbr%r,)rdrrzrr{ in_featuresrweight_zero_point_domainrrrrinput_quant_funcr(r*quantized_weightrtrs r`4_int8_dynamic_activation_int8_weight_quantize_tensorrVs ~~!22#66ll2& "  KK]^d^j^j]k2;-A M#,, #2#7#7  Jzz kk%--(,, ;; S  ;#8#88#K #> *62 -    -6  :*W /9.O.O   / ++~~"K&:6>>:J$KK"%-- *7+#44  rac UR(a([RRR 5 [ X5(dSUS3SUS3-5e[ [X5U5n[UU[RRUSS95 [R"[[UR US9U5UlU$)NzKapplying int8 dynamic activation int8 weight quant requires module to have rPrQrRFrwr/)r5rSrTrUrVrjrr}rWrgrhryrrrr2rrWs r`._int8_dynamic_activation_int8_weight_transformrs!!""EEG 6 * * UVdUeeop &+ , - *F'J  :U; ((  & 1 1)  F Mrac\rSrSr%SrSr\\R\ S'Sr \\R\ S'\ "5r \\ \\\\4\\4\ S'\R$r\\\ S'Sr\\ S 'S r\\ S 'S rS \4SjrSrg)$Int8StaticActivationInt8WeightConfigia Configuration for applying int8 static quantization to both activation and weight Args: act_quant_scale (torch.Tensor): The scale tensor for activation quantization. act_quant_zero_point (torch.Tensor): The zero_point tensor for activation quantization (asymmetric only). granularity (Optional[Union[Granularity, Tuple[Granularity, Granularity], List[Granularity]]] = PerRow()): The granularity for quantization. Can be either a single granularity (applied to both activations and weights) or a tuple / list of two granularities (first for activations, second for weights). If None, defaults to PerRow for both. Only PerTensor and PerRow are supported. act_mapping_type (MappingType): The mapping type for activation quantization. SYMMETRIC and ASYMMETRIC are supported. set_inductor_config (bool): if True, adjusts `torchinductor` settings to recommended values. version (int): the version of the config Nact_quant_scaleact_quant_zero_pointrbrTr5r6rc[RRS5 [R"UR 5up[ X5 g)Nz9torchao.quantization.Int8StaticActivationInt8WeightConfig)rgrrr&rrbrwr}s r`r2Int8StaticActivationInt8WeightConfig.__post_init__s? $$ G /9.O.O   / + #?GrarMcn[R"UR5up[UURS9$)zGet the activation quantization kwargs for static quantization. Returns: QuantizeTensorToInt8Kwargs with the configured granularity and mapping type. r)r&rrbr,rr}s r`get_act_quant_kwargs9Int8StaticActivationInt8WeightConfig.get_act_quant_kwargss; /9.O.O   / +*'..  rar)rrrrrrr rgTensorrrr:rbr r7r rTrBrrr5r=rr rr,rrrrar`rrs /3OXell+237(5<<07  k5k!9:D>Q G"6v~~6F GG &&L%%vxJ rac UR(a([RRR 5 [ X5(dS5e[ U[5(a [U5n[[X5U5n[UU[RRUSS95 [R "[#[$UR&US9U5UlU$)Nzsapplying float8 weight only quant requires module to have {parameter_name} attribute but {module} does not have oneFrwr/)r5rSrTrUrVrjrfr_unwrap_float8_linearrr}rWrgrhryrrrr2rrls r`_float8_weight_only_transformr?s!!""EEG 6 * * , * &,''&v.7'  +5A ((  & 1 1)  F Mrac*\rSrSr%Sr\r\R\ S'\r \R\ S'Sr \ \ \\\4\ S'\R"r\ \\ S'Sr\ \\ S'Sr\ \\ S 'Sr\ \\ S '\R2r\\ S 'S r\\ S 'Sr\\ S'SrSr g))Float8DynamicActivationFloat8WeightConfigiea~ Configuration for applying float8 dynamic symmetric quantization to both activations and weights of linear layers. Args: activation_dtype (torch.dtype): The target data type for activation quantization. Default is torch.float8_e4m3fn. weight_dtype (torch.dtype): The target data type for weight quantization. Default is torch.float8_e4m3fn. granularity (Optional[Union[FP8Granularity, List[FP8Granularity]]]): The granularity for quantization. Can be either a single granularity (applied to both activations and weights) or a tuple of two granularities (one for activations, one for weights). If None, defaults to PerTensor for both. Currently both quantizations need to be the same type. And only PerTensor and PerRow are supported. mm_config (Float8MMConfig): Configuration for the matrix multiplication. Default uses fast accumulation. activation_value_lb (Optional[float]): the lower bound for activation value for calculating scale activation_value_ub (Optional[float]): the upper bound for activation value for calculating scale kernel_preference (KernelPreference): kernel preference for ops like matmul, grouped matmul etc. by defalut (KernelPreference.AUTO) it will be chosen for user based on hardware or other information, this only needs to be set in weight set_inductor_config (bool): if True, adjusts `torchinductor` settings to recommended values. version (int): the version of the config, version 1 is deprecated, version 2 is using Float8Tensor (default) Example: .. literalinclude:: ../../examples/inference/float8_dynamic_activation_float8_weight.py :language: python rBrNrbpacking_format mm_configactivation_value_lbactivation_value_ubkernel_preferenceTr5rrc[RRS5 [UR5upX/UlSn[ UR5(aNUR [R[R4;dS5eURS:dS5eSn[RR5(aSnURc[US9Ul gg)Nz>torchao.quantization.Float8DynamicActivationFloat8WeightConfigT unimplementedrF)use_fast_accum)rgrrrrbrrrAUTOTORCHrxpu is_availablerr)rrrdefault_use_fast_accums r`r7Float8DynamicActivationFloat8WeightConfig.__post_init__s $$ L 6L   6 23G!% ,T-=-= > >)) %% &&.  <<1$ 5o 5$%* " 99 ! ! # #%* " >> !+;QRDN "ra)rbr)!rrrrrrrBrgrrrrbr r rr rr>rrrrfloatrrrrr5r=rr rrrrar`rres0%/ekk. *L%++*IMK%^0D DEFM4G4M4MNH01M*.Ix'.+/%/+/%/*:*?*?'? $$GSSrarc URnURnURnURnURnUR nUR nURn [U5 UupUR5S;a`[U [5(a[U [5(dS5eURSS-S:wdURSS-S:waU$O[U5(dU$URS:XdSUR35eU [R :Xa:[U ["5(a%UR$[&R(:XdS5e[+UU UUUS 9n U [R :Xa[,R."UUU UUU S 9n U $U [R0:Xa5[U ["5(dS 5e[2R."UUU U S 9n U $g) N)zH4D/5D tensor only supports per tensor activation and weight quantizationrr;r6rrfzBPerRow quantization only works for bfloat16 precision input weight) hp_value_lb hp_value_ubr)rrbrrrz8Sparse packing format only supports per-row quantization)rrbr)rBrrbrrrrrrrrfr;rr0rrr>r:rrgrKr+rr%SPARSE_CUTLASSr-)rdrrBrrbrrrrrrrrrs r`8_float8_dynamic_activation_float8_weight_quantize_tensorrs..&&L$$K  I 44 4400**NK(1<.zz|v0)<<  B B  V U V  <<?R 1 $ Q"(<(AM)B F # # >>Q G"6v~~6F GG ,222zF88||u~~- P -4''+ ,222'// %*/-    .== =,f55 F 58?? %*-     >rac n[RR5(a%[5(d[ 5(dS5eUR (a([ RRR5 [X5(dSUS3SUS3-5e[U[5(a [U5n[[X5U5n[!UU[R"R%USS95 [&R("[+[,UR.US9U5UlU$) NzPFloat8 dynamic activation quantization is only supported on CUDA>=8.9 and MI300+zKapplying float8 dynamic activation quant requires module to have parameter rPrQrRFrwr/)rgcudarr5r4r5rSrTrUrVrjrfrrrr}rWrhryrrrr2rrls r`2_float8_dynamic_activation_float8_weight_transformrs  zz   ""hjj ^ 0!!""EEG 6 * * UVdUeeop &+ , - *&,''&v.O'  +5A ((  & 1 1)  F Mraintx_unpacked_tensor hp_tensorrc [U[5(deURRU5Ul[UR up4[ UURURUR[RUUS9Ul g)aP Adjusts the scale_dtype on IntxUnpackedToInt8Tensor. Updating the scale dtype requires updating the qdata because qdata is calculated after the scale. This is used in IntxWeightOnlyConfig and Int8DynamicActivationIntxWeightConfig to make version=2 and version=1 numerically equivalent when the scale_dtype differs from the input dtype ) output_dtyperrN) rfr*scalerSrArrDr( zero_pointrgrqdata)rrrqminqmaxs r`r&r&s *,D E EE E!5!;!;!>!>{!K()=)J)JKJD!0''""''ZZ"rac \rSrSr%Sr\R r\R\ S'\ "S5r \ \ S'\ Rr\ \ S'Sr\\R\ S'\R&r\\ S '\R,r\\ S 'S r\\ S 'S rSrg)IntxWeightOnlyConfigi+a Configuration for quantizing weights to torch.intx, with 1 <= x <= 8. Weights are quantized with scales/zeros in a groupwise or channelwise manner using the number of bits specified by weight_dtype. args: `weight_dtype`: The dtype to use for weight quantization. Must be torch.intx, where 1 <= x <= 8. `granularity`: The granularity to use for weight quantization. Must be PerGroup or PerAxis(0). `mapping_type`: The type of mapping to use for the weight quantization. Must be one of MappingType.ASYMMETRIC or MappingType.SYMMETRIC. `scale_dtype`: The dtype to use for the weight scale. `intx_packing_format`: The format to use for the packed weight tensor (version 2 only). `intx_choose_qparams_algorithm`: The algorithm to use for choosing the quantization parameters. `version`: version of the config to use, only subset of above args are valid based on version, see note for more details. Example: .. literalinclude:: ../../examples/inference/intx_weight_only.py :language: python rrrbrNrrrrrc [RRS5 UR[ SS5Vs/sHn[ [SU35PM sn;dSUR35e[ UR[[45(dSUR35e[ UR[5(a8URRS:XdSURR35eUR[R[R[R4;dS UR35egs snf) Nz)torchao.quantization.IntxWeightOnlyConfigr6rr r z1granularity must be PerAxis or PerGroup, but got rrzvmapping_type must be MappingType.ASYMMETRIC, MappingType.SYMMETRIC, or MappingType.SYMMETRIC_NO_CLIPPING_ERR, but got )rgrrrr r}rfrbr8r9r rrBrrr rs r`r"IntxWeightOnlyConfig.__post_init__KsV $$%PQ  aQR $T 1WUc!I%> $TT J4K\K\J] ^ T$**Wh,?@@ ?@P@P?Q R @ d&& 0 0##((A- 78H8H8M8M7NO -   " "  ! !  1 1%   EFJFWFWEX Y   %UsEr)rrrrrrgrrrrr8rbr7rBrrrr r)rrr'rrrr rrrrar`rr+s(!& L%++*&qzK) + 5 5L+5)-K%++&-->-O-O*O"))"#=GS rarc URnURnURnURnURnUR n UR 5S:XaSn O3UR 5S:XaSn O[SUR 535e[U[5(a URn OW[U[5(a4URS:XdSUR35eURU n O[SU35eUR 5S:XaSU 4n OUR 5S:XdeSU SS4n URS:XdeUR[R :XazUb=UR"[$R&:XaUR)[$R*5n[,R."UU UUUUU S 9n UbXpR":wa [1XU5 U $[S U35e) NrrOrr6z>IntxWeightOnlyConfig only works for 2-d and 4-d Tensors, got: rrz-granularity must be PerGroup or PerAxis, got )rrrrr)rrbrrrrrrrfr9r r8r rrr)rrrgrrSrr*r%r&)rdrrrrrbrrrr input_dimr r(r*s r`!_intx_weight_only_quantize_tensorr`s&&L$$K&&L$$K 44$*$H$H! zz|q   LVZZ\N [  +x(( ++ K ) )1$ 3K4D4D3E F $\\), H VWW zz|q_ zz|q   Q* >>Q   !!%6%G%GG  (->-D-D -S 1 4 4UZZ @ -55   %%/*G   "{ll'B 7 K X78K7LMNNrac ,[X5(dSUS3SUS3-5e[[X5UUUS9n[UU[R R USS95 [R"[[URUS9U5Ul U$) Nz8applying intx weight only quant requires module to have rPrQrRrFrwr/) rjrr}rWrgrhryrrrr2r)rrr-rrr*s r`_intx_weight_only_transformrs 6 * * B>BRR\] &+ , - *3'!+ J   :U; ((  & 1 1)  F Mrac\rSrSr%Sr\"\S9r\\ \ \ 4\ S'\"\S9r \\ \ \ 4\ S'Sr\\ S'SrS rS rg ) riaConfiguration class for applying different quantization configs to modules or parameters based on their fully qualified names (FQNs). Args: `fqn_to_config`: typing.OrderedDict[str, Optional[AOBaseConfig]]: an ordered dictionary from (1). fully qualified name (fqn) of module or parameter (2). regex of fully qualified name (in python `re` module regex format), should start with prefix "re:" or (3). "_default" to the config that we want to apply to the module/param or None Config key ordered by precedence: * fully qualified parameter name, e.g. `language.layers.0.q_proj.weight` * fully qualified module name, e.g. `language.layers.0.q_proj` * regex for parameter names, must start with `re:`, e.g. `re:language\.layers\..+\.q_proj.weight`. The first regex that matches will be applied. * regex for module names, must start with `re:`, e.g. `re:language\.layers\..+\.q_proj`, whichever regex fully matches the module fqn first will be applied (order of keys for dictionary are kept consistent since we are using OrderedDict) * "_default", fallback if no match for all previous keys (Note, when using `_default`, the config is applied to all modules, to apply it to only a subset of modules, e.g. with some types, it's better to filter the modules that we don't want to quantize before hand and configure them to None, e.g. `{"re:.+norm.+": None, "_default": linear_config}`) "_default" is not supported when filter_fn is not specified. `module_fqn_to_config`: typing.OrderedDict[str, Optional[AOBaseConfig]]: To maintain BC with ModuleFqnToConfig, to be deprecated later `version`: int: Version of config to use. Note: - The order of patterns in the OrderedDict may matter as only the first matching pattern is applied - "_default" is ignored for parameter replacement. )default_factoryrmodule_fqn_to_configr6rcJ[RRS5 [UR5S:a>[UR 5S:a%URUR :wa [ S5e[UR 5S:a*[UR5S:XaUR Ul[UR5S:a*[UR 5S:XaURUlSUR;a[R"S5 gg)Nz torchao.quantization.FqnToConfigrzP`fqn_to_config` and `module_fqn_to_config` are both specified and are not equal!rzConfig Deprecation: _default is deprecated and will no longer be supported in a future release. Please see https://github.com/pytorch/ao/issues/3229 for more details.) rgrrlenrrrrgrhrs r`rFqnToConfig.__post_init__s $$%GH "" #a 'D--.2""d&?&??b  t(( )A -#d6H6H2IQ2N!%!:!:D  t!! "Q &3t/H/H+IQ+N(,(:(:D % ++ + MMy  ,raclSRS/SURR55QSP5$)N z FqnToConfig({c38# UHupSUSUS3v M g7f)z 'z': ,Nr).0keyvalues r` &FqnToConfig.__str__.. s(&@ #hugQ/&@sz}))joinrrrs r`__str__FqnToConfig.__str__sFyy &*&8&8&>&>&@      ra)rrN)rrrrrrrrOrderedDictTypestrr r rrrr rrrrrar`rrso@CH#CM?3(>#>?JO#J/#x /E*EFGS2  rarrcf[R"U5RRSS5SL$)zk Returns True if the handler function has a "parameter_name" kwarg in its type signature, False otherwise. r-N)inspect signature parametersget)rs r`"_handler_supports_fqn_quantizationrs/   W % 0 0 4 45Et LTX XXrafqncfSn/n[[UR555HDunupgU[U5;dM[ U5S:aUSU3OUnUR XVXx45 MF [U5H~upVpxXR ;dMSnUR Un U cURU5 M>[[U 5n [U 5(a U "X US9nMi[[U 5S35e U(d=XR ;a.UR Un U b[[U 5n U "X 5$U$UHupVpxUR Hn U RS5(dM[R"U S SU5(dM<SnUR U n U cMR[[U 5n [U 5(a U "X US9nM}[[U 5S35e M U(d|UR Hln U RS5(dM[R"U S SU5(dM<UR U n U b[[U 5n U "X 5s $Us $ U(d9UR RS S5n U b[[U 5n U "X 5$U$) aOThis function expects a module that either is specified in FqnToConfig or has a parameter that is specified in FqnToConfig. Args: module (torch.nn.Module): The module to be processed. fqn (str): The fully qualified name of the module containing the parameters. config (FqnToConfig): Configuration object containing regex patterns / fqn mapped to quantization configurations. Returns: torch.nn.Module: The modified module with quantized parameters. Raises: NotImplementedError: If the quantization configuration is not yet supported for parameter quantization. FrrRTNrNzs does not yet support parameter quantization! Please see https://github.com/pytorch/ao/issues/3252 for more detailsre:rr) enumeraterTnamed_parametersdirrappendrrr.rrNotImplementedError startswithre fullmatchr) rrrparameter_config_foundtop_level_paramsir-param parameter_fqncrpatterns r`rrs&#&/V5L5L5N0O&P" "N S[ (-0X\3%q()~   # #Q$M N 'Q488H3I/5 00 0%) "$$]3Ay $$Q'247;5g>>$V~NF-7)$WX4J$ "c-A-A&A   % =.tAw7G6% %M4D/5++G!!%((R\\'!"+}-U-U)-&((1=6tAw?G9'BB!(>!R1#Awi([\,4D "++G!!%((R\\'!"+s-K-K((1=6tAw?G"6--!M, "  $ $Z 6 =.tAw7G6% % MracXR;a"URS5(a SUS35egURH<nURS5(dM[R"USSU5(dM< g g)aMCheck if a given fqn matches the exact fqn or regex pattern specified in FqnToConfig. Args: fqn (str): The fully qualified name of the module. config (FqnToConfig): Configuration object containing regex patterns or raw FQNs for quantization. Returns: bool: True if the fqn is specified in FqnToConfig. False otherwise. rzError: Exact match but regex z specified.TrNF)rrrr)rr!maybe_module_or_param_fqn_patterns r`rr}s """>>%(( +C5 < (171E1E -0;;EBBr||1!"5sHH 2F racUR5H@up4U[U5;dM[U5S:aUSU3OUn[XR5(dM@ g g)aCheck if a given module contains top-level parameters that match the exact fqn or regex pattern specified in FqnToConfig. Args: module (nn.Module): The module to be checked. fqn (str): The fully qualified name of the module. config (FqnToConfig): Configuration object containing regex patterns or raw FQNs for quantization. Returns: bool: True if the module contains top-level parameters that match the fqn or regex pattern specified in FqnTo rrRTF)rrrr)rrrr]rrs r`rrsS..0  3v; /23x!|se1TFOM%m<< 1 rac[R"S5 [R"URUR 5nSSS5 UR WlURUlU$!,(df  N2=f)aE Unwrap a torchao Float8Linear by returning a nn.Linear with the same weights and bias. Torchao inference quantization techniques are generally only applicable to nn.Linear layers, so this helper is useful for unwrapping models trained with torchao float8 training, which replaces nn.Linear layers with Float8Linear layers. metaN)rgrQrhrir out_featuresrdr)r new_modules r`rrsW f YYv1163F3FG   JkkJO   s ,A// A=)rNr)Fr)rrloggingrrrg collectionsr dataclassesrr functoolsrtypingrrr r r r rrgtorch.nnrhtorch.nn.utils.parametrizerUr{rStorchao.core.configr torchao.dtypesrrrrtorchao.dtypes.utilsrtorchao.float8.configrtorchao.float8.float8_linearrtorchao.float8.inferencerrrrr(torchao.prototype.quantization.quant_apir=torchao.quantization.linear_activation_weight_observed_tensorrtorchao.quantization.observerr%torchao.quantization.quantize_.commonr(torchao.quantization.quantize_.workflowsrrr r!r"r#r$r%r&r'r(r)r*r+r,r-%torchao.quantization.transform_moduler.r/torchao.quantization.utilsr0r1r2r3 torchao.utilsr4r5rbr7r8r9r:r;"linear_activation_quantized_tensorr<r=linear_quant_modulesr>r?qatr@quant_primitivesrArBrCrDunifiedrErFrG getLoggerrrD__all__FLOATLAYOUT_TO_ZERO_POINT_DOMAINLAYOUT_TO_PRESERVE_ZEROSrVrprIrHrrr=rrrDevicerJrrrrrr,r1r3rMrXrZr_rarjrnrqrsrwryrrrrrrrrrrrr&rrrrrKrrrrrir serializationadd_safe_globalsrrar`r)s  #(>>1 00, (,5F$   1,   8 $ O112 5  ,. + sCx) + +\$DJCG N 99N89N9:N %((//3!7!=>? NbD ).e.CM+/ H 88??H H %((//3!7!=>?H  U[['' ( H V5<<ELL$ELLU\\,%,,5<<& @ L@  @ P,004 G 5<<( G   - G T""GH #+/04  HHOO 1  5<<(    -  XX__IB $R<$R $RN8Tv""67 #  HHOO    XX__ 8@ ;l; ;$""IJ # 1 HHOO1 31 1  XX__ 1K1h !<! !H6""67 #  HHOO  8@ $  ||  \\ .   #   4 -LL-L -L`>B""GH   HHOO 1  XX__ I@ / </  / d""FG  , HHOO, 0,H,^ T\T T4""89 # " HHOO" "" "  XX__ ":"J :S :S :SzE P""KL # # HHOO# 5# #M#L2|| 2 1 <1  1 p,004 ;O5<<( ;O   - ;O|""67 #+/04  HHOO   5<<(    -  XX__8D M ,M  M b Y uxx 5uxxF GY Y[ HHOO[ [ [|  6 II  0 , 299 $$#0ra