i nBSSKJr SSKJr SSKJrJrJrJrJ r J r J r J r SSK r SSKJs Jr SSKJs Js Jr SSKrSSKJrJr SSKJr SSKJr SSKJr SS K J!r! SS K"J#r# SS K$J%r% SS K&J'r'J(r(J)r) SS K*J+r+J,r,J-r-J.r.J/r/J0r0 Sr1"SS\5r2"SS\/5r3\2RhSSSSSS.S\R(RjS\RlRnRpS\2S\9S\ \:S\:S\ \\/\4S\ \\;\4S\R(Rj4S jjrg)0OpenVINOQuantizerJz Implementation of the Torch AO quantizer which annotates models with quantization annotations optimally for the inference via OpenVINO. int4_sym int4_asymr" int8_asym)moder?c XlUR[R;aU[R:Xa'[ R RRnSnOaU[R:Xa'[ R RRnSnO&Sn[RRRn[SX4S.UD6Ulg[RUn[#[R$"U540UD6nSUS'[R R&R(R*R,"S0UD6Ulg)a :param mode: Defines special quantization modes. - INT8_SYM: INT8 symmetric quantization for both activations and weights. - INT8_MIXED: INT8 asymmetric quantization for activations, symmetric for weights. - INT8_TRANSFORMER: Optimized INT8 quantization for transformer-based models Default value is INT8_SYM. - INT4_SYM: Symmetric INT4 Weights-Only Compression - INT4_ASYM: Asymmetric INT4 Weights-Only Compression :param kwargs: Arguments to pass to the NNCF MinMaxQuantization algorithm. N)preset model_type subset_sizer))r?r:WEIGHTS_ONLY_COMPRESSION_MODESr r/ quantizationstructsQuantizationPreset PERFORMANCEr0MIXEDnncf parameters ModelType TRANSFORMERr_algorCompressWeightsMode algorithmsweight_compression algorithmWeightCompression)selfr?kwargsrArBcompression_mode weight_compression_configurations r8__init__OpenVINOQuantizer.__init__Ws&  99-LL L'000%--@@LL! )444%--@@FF! !__66BB +8>DJ 1OO  0T(()9:00 ,  -] ;**55HHRRdd2DJr7returnc.[UR[RRR R R5(d9[S[UR5RSURS35eUR$)Nz5This method requires WeightCompression algo, but got  (mode=).) isinstancerOrKrFrQrRrSrT TypeErrortyper*r?rUs r8_require_wc_algo"OpenVINOQuantizer._require_wc_algos JJ    ( ( ; ; E E W W  DJJ'001 2G zzr7c[UR[5(d9[S[ UR5R SUR S35eUR$)Nz6This method requires MinMaxQuantization algo, but got r]r^)r_rOrr`rar*r?rbs r8_require_ptq_algo#OpenVINOQuantizer._require_ptq_algosY$**&899DJJ'001 2G zzr7cUR5nURURURURS.nU$)aw Returns a dictionary with all_layers, group_size, backup_mode and Quantization mode parameters used by the compress_pt2e weight compression algorithm. :return: A dictionary containing: 1. mode: Quantization mode. One of INT4 Sym, INT4 Asym, INT8 Sym, INT8 Asym. 2. group_size: group size to be used for group-wise compression. 3. all_layers: Indicates whether embeddings and last MatMul layers should be compressed to a primary precision. By default, the backup precision is assigned for the embeddings and last MatMul layers. 4. backup_mode: Defines a backup mode for mixed-precision weight compression. )r? group_size all_layers backup_mode)rcr?rirjrk)rUalgo%quantizer_initialized_algo_attributess r8get_weights_compression_config0OpenVINOQuantizer.get_weights_compression_configs@$$&II////++ 1 -54r7Nnamespatternstypes subgraphsvalidatec URR[R"U=(d /U=(d /U=(d /U=(d /US95 g)a Provides an option to specify portions of model to be excluded from compression. The ignored scope defines model sub-graphs that should be excluded from the quantization process. :param names: List of ignored node names. :param patterns: List of regular expressions that define patterns for names of ignored nodes. :param types: List of ignored operation types. :param subgraphs: List of ignored subgraphs. :param validate: If set to True, then a RuntimeError will be raised if any ignored scope does not match in the model graph. )rprqrrrsrtN)rOset_ignored_scoperK IgnoredScope)rUrprqrrrsrts r8rv#OpenVINOQuantizer.set_ignored_scopesC& $$   kr!Rkr#/r!   r7model nncf_graphcfUR5nURU5 URX5$N)rf_set_backend_entityfind_quantization_setuprUryrzrls r8get_nncf_quantization_setup-OpenVINOQuantizer.get_nncf_quantization_setups1%%'   '++E>>r7cfUR5nURU5 URX5$)a Collect weight compression parameters for the given FX model and NNCF graph. :param model: FX GraphModule to analyze for weight compression. :param nncf_graph: NNCFGraph representation of the model. :return: A tuple of: - all parameters eligible for weight compression, - ratio-defining parameters used to set primary/backup precisions, - parameters that are not compressible and remain in original precision. )rcset_backend_entity!get_weight_compression_parametersrs r8&get_nncf_weight_compression_parameters8OpenVINOQuantizer.get_nncf_weight_compression_parameterss1&$$& &55eHHr7graphnode_vs_torch_annotationc URX5tpVUHnUR(d.[R"S[ UR SS55 MBUR n[ RRX(R5n XIn URX5n URUS9n URXU 5 M U$)a  Annotates the model graph with weight-only quantization specs. Identifies compressible nodes in the NNCF graph and attaches the corresponding TorchAO quantization specifications to their weight edges for later transformation. :param model: The FX GraphModule to annotate. :param graph: The underlying FX graph. :param nncf_graph: The corresponding NNCF graph. :param node_vs_torch_annotation: A mapping of FX nodes to quantization annotations. :return: Updated mapping of FX nodes with weight compression annotations. zPSkipping weight compression for node '%s' because compression_config is missing. node_namez )wc_param) rcompression_configrdebuggetattrnode_with_weightnncf_fx node_utilsget_graph_node_by_namer_get_weight_edge+_get_torch_ao_qspec_from_nncf_config_for_wc_fill_torch_ao_annotation) rUryrrzr all_wc_params_rr target_node annotation edge_or_nodeqspecs r8_annotate_weight_compression.OpenVINOQuantizer._annotate_weight_compressions&!GG   &H..!!fH55{KP'88 !,,CC11K2>J00ILDDhDWE  * *< K& ('r7c ^^URX5mTRR5H:nURX#XT5upgUR U5nUR XhU5 M< TR R5GHHn URX9T5n TRU m[UU4SjU 55(aqU V s/sHn TRU PM n n [R"SU Vs/sH-oURR[UR54PM/ sn35e[R R#UTRR$5n UR'U TU5nU HJnX:XaM [)U5nTRUnURX#XT5upgUR XhU5 ML GMK U$s sn fs snf)ax Annotates the model graph with post-training quantization configurations. :param model: The FX GraphModule to annotate. :param graph: The underlying FX graph. :param nncf_graph: The corresponding NNCF graph. :param node_vs_torch_annotation: A mapping of FX nodes to quantization annotations. :return: Updated mapping of FX nodes with post-training quantization annotations. c3p># UH+nTRTRUR:gv M- g7fr|)qconfigquantization_points).0q_idquantization_setuproot_qps r8 IOpenVINOQuantizer._annotate_post_training_quantization..5s3)D#5#I#I$#O#W#WW)s36zBDifferent quantization configs are set to one unified scale group:)rrvalues _get_edge_or_node_and_annotation,_get_torch_ao_qspec_from_nncf_config_for_ptqrunified_scale_groups%_get_unified_scales_root_quantizer_idanyrK InternalErrorinsertion_point__dict__strrrrrtarget_node_name_get_edge_or_noder)rUryrrzrqprrr quantizer_idsroot_quantizer_idqidqpsroot_target_noderoot_edge_or_node quantizer_idrrs @@r8$_annotate_post_training_quantization6OpenVINOQuantizer._annotate_post_training_quantizations "==eP$88??AB'+'L'L2( $LAA"E   * *< KB0DDKKMM $ J J+=! )<<=NOG) LYKXC&::3?=((XQTUQT2++44c"**oFQTUVX '11HHw..??  !% 6 6 ':! !. 4./@A';;LI+/+P+Pr,( ..|JO!.3NH('5 Vs G:4G c[RRRU5nURn[ [ 5nUR[R;aURXX$5nOURXX$5nUR5H-upV[UR;deXeR['M/ U$r|)rnncf_graph_builderGraphConvertercreate_nncf_graphrrrr?r:rErritemsQUANT_ANNOTATION_KEYmeta)rUryrzrrnoders r8annotateOpenVINOQuantizer.annotateUs//>>PPQVW   . / ! 99)HH H'+'H'Hj( $(,'P'Pj( $!9 > > @ D'tyy8 88.8II* +!A r7rquantizer_setupcSnSnUHZnURURRnURU5nUbURU:dMLUnURnM\ UcSn[ R "U5eU$)a Identifies the earliest quantizer node ID based on the corresponding `nncf_node.node_id` in the given NNCFGraph. This is required by the `_get_obs_or_fq_map` function. Refer to: https://github.com/pytorch/ao/blob/main/torchao/quantization/pt2e/prepare.py :param nncf_graph: The NNCFGraph instance. :param quantizer_ids: The list of quantizer IDs to evaluate. :param quantizer_setup: The instance of SingleConfigQuantizerSetup. :return: The ID of the earliest quantizer node in terms of `nncf_node.node_id`. Nz Root quantizer ids can't be None)rrrget_node_by_namenode_idrKr) rzrrnncf_node_quantizer_idrrr nncf_nodemsgs r8r7OpenVINOQuantizer._get_unified_scales_root_quantizer_idks "& )L.BB o.. #334DEI&.$$'==$0!)2):):&*  $4C$$S) )  r7rc[RRXRR5nX4n[ R XBU5nXe4$)a Retrieves the edge or node and its corresponding QuantizationAnnotation based on the given graph, quantization point, and node-to-annotation mapping. :param graph: torch.fx.Graph instance. :param nncf_graph: NNCFGraph instance. :param qp: QuantizationPointBase instance. :param node_vs_torch_annotation: A dictionary mapping torch.fx.GraphNode objects to their respective QuantizationAnnotations. :return: A tuple containing the EdgeOrNode and its associated QuantizationAnnotation. )rrrrrr:r)rrzrrrrrs r8r2OpenVINOQuantizer._get_edge_or_node_and_annotationsN$((?? %%66 .: (::;JW ''r7rc URUR5n[X!5n[U5S:a-[R "SURSUSSUS35 UR USnX@4$)a Returns the FX node corresponding to the weight tensor input of a given operator node. Uses the NNCF graph to identify which input port of the target node holds the weight. If multiple weight ports are present, a warning is issued and only the first one is used. :param target_node: FX node representing a weighted operation (e.g., Linear, Conv). :param nncf_graph: NNCFGraph used to determine weight port indices. :return: Edge represented by a Tuple of (weight_node, target_node), where weight_node is the FX node supplying the weight. rCz"Quantization of the weighted node zK is not yet supported by the OpenVINOQuantizer. Only the weight on port ID rz> will be quantized. Quantizable weights are located on ports: .)rnamerlenrwarningall_input_nodes)rrzrweights_ports_ids weight_nodes r8r"OpenVINOQuantizer._get_weight_edges// 0@0@A 6yM  !A %   4[5E5E4F//@/C.DE>>O=PPQS  "112CA2FG ))r7cURnUR5(a[RX5$URcU$UR URnX@4$)a$ Returns the edge or node based on the given target node and quantization point. :param target_node: Target node instance. :param qp: QuantizationPointBase instance. :param graph: NNCFGraph instance. :return: The corresponding EdgeOrNode derived from the target node and quantization point. )ris_weight_quantization_pointr:r input_port_idr)rrrziprs r8r#OpenVINOQuantizer._get_edge_or_nodes`   * * , ,$55kN N   # **2+;+;<""r7rrannotation_to_updatec[U[RR5(aXlgXR US'g)ak Helper method to update the annotation_to_update based on the specified edge_or_node and qspec. :param edge_or_node: The target EdgeOrNode to be used for the update. :param qspec: An instance of QuantizationSpecBase representing the quantization specification to apply. :param annotation_to_update: The annotation to update based on the edge_or_node and qspec. rN)r_torchfxNode output_qspecinput_qspec_map)rrrs r8r+OpenVINOQuantizer._fill_torch_ao_annotations1 lEHHMM 2 205 -DI 0 0a Ar7rc j0nURRnXS'URRnU[RR [RR 4;aV[nU(dSOSnU(dSOSn[RnSnU(d[RO[Rn OU[nU(dSOSnU(dSOSn[RnSnU(d[RO[Rn [UUR"S 0UD6UUU US S 9$) z Returns a TorchAO QuantizationSpec based on NNCF weight compression parameter. :param wc_param: NNCF Weight compression parameters for the node. :return: A TorchAO QuantizationSpec. rirFdtypeobserver_or_fake_quant_ctr quant_min quant_maxqschemech_axis is_dynamicr))rr? is_asym_moderKrP INT4_ASYMINT4_SYMr rint8per_channel_symmetricper_channel_affinerr with_args) r extra_argsqmoderobserverrrr channel_axis torch_qschemes r8r=OpenVINOQuantizer._get_torch_ao_qspec_from_nncf_config_for_wcs&( ++00!):22??   $ $ . .  $ $ - -  *H".AI!-2IJJEL$++--* *H$0aI#/SIJJEL$++--   '/'9'9'GJ'G!   r7c SS0nUR5nURn[RnSnSnSnUR(aRUR [ RRRLa[RO[RnOQUR [ RRRLa[RO[RnU(an[n SnSn[RnSnUR [ RRRLa[RO[RnOgU[R[R4;a[O[n SnSnUR (a[RO[R"nSn[%UU R&"S 0UD6UUUUS S 9$) z Returns a TorchAO QuantizationSpec based on NNCF quantization point. :param qp: Quantization point from NNCF. :return: A TorchAO QuantizationSpec. epsgؗҜOpenVINOQuantizer._get_torch_ao_qspec_from_nncf_config_for_ptqs',U^ 335 **      <<<#7#7#J#J#T#TT++-- <<<#7#7#J#J#T#TT**,,  /HIIJJEL<<<#7#7#J#J#T#TT++-- !..0G0GHI".  II")"="=EJJ5;;EL'/'9'9'GJ'G!   r7cgr|r))rUrys r8rtOpenVINOQuantizer.validateas r7)rOr?)NNNNT)?r*r+r,r-r.r r4r5r2r3rEr/rYrKrFrQrRrSrTrcrrfrrrrnr r r boolrvrr GraphModulerrSingleConfigQuantizerSetuprrrGraphrrrrrr staticmethodintrQuantizationPointBaserrtuplerrrrrrrrtr6r)r7r8r:r:Js ##Z$$k##Z$$k &""2!:!:,,\    % % 8 8 B B T T #55S#X50&*(,%)AE  S " 49% S "  DtCy$s)';!<=>      :?XX))?7@?  % % @ @?Ixx##II  () () () +  I.'(xx##'(xx~~'( '( #.ehhmm=S.S"T '( UXX]]$:: ; '(R?(xx##?(xx~~?( ?( #.ehhmm=S.S"T ?( UXX]]$:: ; ?(Behh22uxx7K7K,!!Cy!&55PP! !!B(xx~~((  ( ( > >(#'uxx}}6L'L"M ( z11 2 ((0*XX]]** uxx}}ehhmm+ ,**4#XX]]#  ( ( > >##  ##.J J#J5J  JJ"1 -1 1 1 f?  ( ( > >? ? ? B ehh22 t r7r:i,TF)r?rDfast_bias_correction smooth_quant transform_fnextra_quantizer_optionscaptured_modelcalibration_datasetr?rDrrrrr[c U=(d 0nSU;a[SU35 US [SSU0UD6n [S5 SU;aSUS'[R"UU 4U[R "X5UUS.UD6n U $)a Quantizes a model using NNCF quantize_pt2e API. :param captured_model: The model to be quantized, represented as a torch.fx.GraphModule. :param calibration_dataset: A DataLoader containing calibration data for quantization. :param mode: Defines special quantization modes. - INT8_SYM: INT8 symmetric quantization for both activations and weights. - INT8_MIXED: INT8 asymmetric quantization for activations, symmetric for weights. - INT8_TRANSFORMER: Optimized INT8 quantization for transformer-based models Default value is INT8_SYM. :param subset_size: Size of a subset to calculate activations statistics used for quantization. :param fast_bias_correction: Setting this option to `False` enables a different bias correction method which is more accurate, in general, and takes more time but requires less memory. None disables the bias correction algorithm. :param smooth_quant: Setting this option to `True` enables the SmoothQuant algorithm. :param extra_quantizer_options: A dictionary containing additional configuration options for the OpenVINOQuantizer. :param kwargs: The keyword arguments for the nncf quantize_pt2e function. :return: The quantized model as a torch.fx.GraphModule. r?zBIgnoring "mode" from the quantizer_config. Using parameter mode = zPTQ: Quantize the model fold_quantizeF)rDrrrr))printr:r quantize_pt2erKDataset) rrr?rDrrrrrV quantizerquantized_models r8quantize_modelr%esB6; (( PQUPV W  $F +!GtG/FGI #$f$"'++  LL)<K1!  O r7)= collectionsrenumrtypingrrrrr r r r rKnncf.common.quantizationcommonrFnncf.experimental.torch.fx experimentalrrrtorch.fx0executorch.backends.openvino.quantizer.observersr rnncf.common.graph.graphrnncf.common.loggingr.nncf.quantization.algorithms.min_max.algorithmr6nncf.quantization.algorithms.weight_compression.configr nncf.quantization.quantize_modelrnncf.torch.model_graph_managerrtorchao.quantization.pt2errr#torchao.quantization.pt2e.quantizerrrrrrrrr r:r/rutilsdata DataLoaderrrrr%r)r7r8r:s:$PPP //,,.+ 1 t ,X X ~.66+/378<8HH((8))448  8  8 #4. 888SE3J/08&d38n58 XX8r7