Ki SSKJr SSKrSSKrSSKrSSKrSSKJr SSKJ r SSK r SSK r SSK J r JrJr SSK JrJrJr SSK Jr SSKJrJrJrJr SS KJr SS KJrJrJr SS K J!r! S r#S r$Sr%Sr&Sr'Sr(Sr)Sr*Sr+Sr,0r-\."\5Vs0sH$n\/"\0"\U5\15(dM\0"\U5U_M& snr2"SS\5r3"SS\5r4"SS\5r5"SS\5r6\R"Rn\ Rp"S5\R"Rr\ Rp"S5\R"Rt\ Rp"S 5\R"Rv\ Rp"S!5\R"Rx\ \R"Rz\\R"R|\0r?\R"Rr\ R"S\ RS"9\ R"S#\ RS"94\R"Rn\ R"S$\ RS"9\ R"S%\ RS"94\R"Rv\ R"S\ RS"9\ R"S&\ RS"94\R"Rt\ R"S'\ RS"9\ R"S(\ RS"94\R"R|\ R"S\S"9\ R"S)\S"94\R"Rz\ R"S*\S"9\ R"S+\S"940rE\R"Rn\ R"S,\ RS"9\ R"S%\ RS"94\R"Rt\ R"S-\ RS"9\ R"S(\ RS"940rF\R"Rr\ R"S\ RS"9\ R"S%\ RS"94\R"Rn\ R"S.\ RS"9\ R"S/\ RS"94\R"Rv\ R"S\ RS"9\ R"S(\ RS"94\R"Rt\ R"S0\ RS"9\ R"S1\ RS"94\R"R|\ R"S\S"9\ R"S+\S"94\R"Rz\ R"S2\S"9\ R"S3\S"940rGS4S5.S6jrHS`S7jrISaS8jrJS9rKSbScS:jjrLSbSdS;jjrMS`SeS<jjrNSfS=jrOSfS>jrPSgS?jrQShS@jrR"SASB5rS"SCSD5rT"SESF5rUSGrVSHrWSIrXSJrYSiSKjrZSLr[SjSMjr\SkSNjr]SlSOjr^SmSPjr_SnSQjr`SoSRjraSnSSjrbSoSTjrcSpSUjrdSqSVjreSrSWjrfSsSXjrgStSYjrhSuSZjriSuS[jrjSvS\jrkSvS]jrlSvS^jrmSvS_jrng!\"a Sr!GNf=fs snf)w) annotationsN)Enum)Path) float8_e4m3fnint4uint4) ModelProto TensorProtoexternal_data_helper)onnx_pb) make_graph make_model make_nodemake_tensor_value_info)ReferenceEvaluator)GraphOptimizationLevelInferenceSessionSessionOptions)to_array_extendedz onnx.quantizez0.1.0ai.onnxz com.microsoftQuantizeLinear_QuantizeLinear_InputDequantizeLinear_DequantizeLinear_Output _quantizedlc2\rSrSrSrSrSr\S5rSr g)QuantizationMode4rcUR$Nnameselfs f/var/www/html/dynamic-report/venv/lib/python3.13/site-packages/onnxruntime/quantization/quant_utils.py__str__QuantizationMode.__str__8 yycF[U$![a [5ef=fr!)rKeyError ValueError)modes r& from_stringQuantizationMode.from_string;s) #D) ) ,    N) __name__ __module__ __qualname____firstlineno__ IntegerOps QLinearOpsr' staticmethodr/__static_attributes__r2r*r&rr4s%JJr*rc2\rSrSrSrSrSr\S5rSr g)QuantizedValueTypeCrrcUR$r!r"r$s r&r'QuantizedValueType.__str__Gr)r*cF[U$![a [5ef=fr!)r<r,r-)vs r&r/QuantizedValueType.from_stringJs) %a( ( ,  r1r2N) r3r4r5r6Input Initializerr'r9r/r:r2r*r&r<r<Cs% EKr*r<cV\rSrSrSrSrSrSrSrSr Sr S r \ S 5r \S 5rS rg ) QuantTypeRrrcUR$r!r"r$s r&r'QuantType.__str__[r)r*cF[U$![a [5ef=fr!)rFr,r-)ts r&r/QuantType.from_string^s( Q<  ,  r1cU[R:Xa[R$U[R:Xa[R $U[R :Xa[R$U[R:Xa[R$U[R:Xa[R$U[R:Xa[R$U[R:Xa[R$[!SU<S35e)NzUnexpected value qtype=.)rFQInt8r INT8QUInt8UINT8QUInt16UINT16QInt16INT16 QFLOAT8E4M3FN FLOAT8E4M3FNQUInt4UINT4QInt4INT4r-r$s r& tensor_typeQuantType.tensor_typees 9?? "## # 9## #$$ $ 9$$ $%% % 9## #$$ $ 9** *++ + 9## #$$ $ 9?? "## #24(!<==r*r2N)r3r4r5r6rTrVr\rZrXr`r^r'r9r/propertyrbr:r2r*r&rFrFRsR E FM FG E F >>r*rFc2\rSrSrSrSrSr\S5rSr g) QuantFormatxrrcUR$r!r"r$s r&r'QuantFormat.__str__|r)r*cF[U$![a [5ef=fr!)rfr,r-)formats r&r/QuantFormat.from_strings) v& & ,  r1r2N) r3r4r5r6 QOperatorQDQr'r9r/r:r2r*r&rfrfxs%I Cr*rfint8uint8int16uint16dtypeiiiiiiii@ii@rIzero_point_indexcd/n[U5GHup4[R"[U5[R5(a&UR [R "U55 OB[U[R5(aUR U5 O[SUSU35eX0:XdMUSnUR[R:Xd UR[R:XdM[SUR35e [U5S:a [U5$US$)Nzarg z is not an array: r{zzero_point cannot be rr) enumeratenumpy issubdtypetypenumberappendarray isinstancendarray TypeErrorrtfloat32float16lentuple)r}argsnew_argsiarAs r& _check_typersH$   DGU\\ 2 2 OOEKKN + 5== ) ) OOA d1#%7s;< <  Aww%--'177emm+C"7y ABB "(ma/5?@Xa[@r*cU[;d SUS35eU[RR[RR[RR [RR 4;Ga/US:wa[SU<S35eUR[R:Xa[RnOHUR[R:Xa[RnO[SURS35e[[![#S/S/[$R&R)SU/S/5S 9[#S /S QS /5/S [+SUS5[+SUS5/[+S US5/55n[-U5n[/UR1SXS.5S5$[Un [3USSS9upUb [5X5OU n Ub [7X5OU n [R8"UR;[R5U- R=5U-5n[R>"XXS9 [/UR;U 55$)NUnexpected data type > requested. Only INT8, UINT8, INT16, and UINT16 are supported.rz2zero_point is expected to be null for float 8 not rSzUnexpected dtype Constant zero_point)valuer)XscalerYqurr)rrF) reduce_range symmetric)out) ONNX_TYPE_TO_NP_TYPE onnx_protor r]FLOAT8E4M3FNUZ FLOAT8E5M2FLOAT8E5M2FNUZNotImplementedErrorrtrrFLOATrFLOAT16r-rr ronnxhelper make_tensorrrrrunget_qmin_qmax_for_qTypemaxminasarrayastyperoundclip)qTypearrrrlowhigh onnx_type onnx_modelrefrtqminqmaxcliplowcliphigharr_fp32s r&quantize_nparrayrs! ( ( w&de ( ++--))--  ?%(Z[eZhhi&jk k 99 %#))I YY%-- '#++I0 1=> > "Bdkk>U>UVbdikmpqor>s.0LseT  *3 4@*7ItD(UD9:  !,3774s)CDQGHH%U+,URWX $'O#d.&*&63t?D==#**U]]";e"C!J!J!Lz!YZ 8h=8??5122r*c US:dUS:a[SUSU35e[R"U[R"SURS95n[R "U[R"SURS95nUb*[ X[R"XPRS9-5nU(aE[R "[R"U5[R"U55nU*nU7nX#::d SUSU35e[R"X- [RS9n[R"U[RS9[R"U[RS9- n[R"Xx- 5n U S:dS5eU [R"UR5R:aA[R"SURS9n [R"SURS9n X/$U(aZ[R"[R"X#-[R"S [RS9- 5URS9n O8[R"[R"X U - - 5URS9n U RUR5n X/$) aCalculate the scale s and zero point z for the quantization relation r = s(q-z), where r are the original values and q are the corresponding quantized values. r and z are calculated such that every value within [rmin,rmax] has an approximate representation within [qmin,qmax]. In addition, qmin <= z <= qmax is enforced. If the symmetric flag is set to True, the interval [rmin,rmax] is symmetrized to [-absmax, +absmax], where absmax = max(abs(rmin), abs(rmax)). :parameter rmin: minimum value of r :parameter rmax: maximum value of r :parameter qmin: minimum value representable by the target quantization data type :parameter qmax: maximum value representable by the target quantization data type :parameter symmetric: True if the floating-point range should be made symmetric. Defaults to False. :parameter min_real_range: Minimum floating-point range (i.e., rmax - rmin) to enforce. Defaults to None. :return: zero and scale [z, s] rBqmin and qmax must meet requirement: qmin <= 0 <= qmax while qmin:, qmmax:rszqmin=z > qmax=z scale issue?g@)r-rminimumrrtmaximumrrabsfloat64finfotinyrr) rminrmaxrrrmin_real_rangeabsmaxdrdqrrs r&compute_scale_zprs( ax4!8]^b]ccklpkqrss ==u{{1DJJ? @D ==u{{1DJJ? @D!4 nJJ OOPuyy $@ww <55htf55< T[ 6B T /%++d%--2X XB KK E A:$}$: u{{4::&+++ Ctzz2[[$**5      T[EKK5==,QQRZ^ZdZdJU[[u 1D%ETZZXJ TZZ(  r*cSnU[;aU[R:XaSSKJn Un[ S5Vs/sHn[ U5PM nn[R"UVs/sH?n[R"U5(aM [R"U5(aM=UPMA sn[RS9nO[SUS35eU[U'OU[R:XaSSKJn UnUc[SUS 35e[R"[U5n[R"SUS9n [R"X- URS9n X/$s snfs snf) aZCalculate the scale s for a float8 type (E4M3FN). The function assumes the coefficient distribution and the float 8 distribution are similar to two gaussian laws. :return: zero and scale [z, s] More details in notebook `quantization_fp8.ipynb `_. Nr)rrszQuantization to element_type=z not implemented.zUnexpected element_type rS)FLOAT8_DISTRIBUTIONSr r] ml_dtypesrrangefloatrrisnanisinfrr-rstdrt) element_typerzp_dtyperr all_valuesfvaluesstd_f8zerors r&compute_scale_zp_float8r,s"H// ;33 3 /$H,1#J7Jq%(JJ7[[&TJqekk!nU[[QR^JT\a\i\iF<\NJ[\] ]-3\* 11 1+ 2<.BCC YY+L9 :F ;;q )D KK CII 6E =#8TsE!EE!Ec[U[R5(d[S[ U5S35eUbUnO"[ U5(aUR 5OSnUbUnO"[ U5(aUR5OSn[R"XpRS9n[R"XRS9n[R"SURS9n U[R:Xa?U(a [S5e[R"U5n [X5up[XSS9$U[R [R"[R$[R&[R([R*4;aU[-XUS 9up[ U5(a[/XxXX$5upO[R"SU RS9n [XSS9$[1S US35e) a Returns the zero_point and scale for the given data. :param data: The data for which to compute quantization parameters. :param quant_type: The quantization data type. :param symmetric: whether symmetric quantization is used or not. :parameter reduce_range: True if the quantization range should be reduced. Defaults to False. :parameter min_real_range: Minimum floating-point range (i.e., rmax - rmin) to enforce. Defaults to None. :parameter rmin_override: The value of rmin to use if not None. Otherwise, uses min(data). :parameter rmax_override: The value of rmax to use if not None. Otherwise, uses max(data). :return: zero point and scale z%Weight must be given as an array not rSgrsrz1Unsupported option reduce_range=True for float 8.rr|rz Unexpected value for quant_type=)rrrrrrrrrrtr r] RuntimeErrorrrrrUrWr[rYrar_rrr-)data quant_typerrr rmin_override rmax_overriderrrrrrrs r&compute_data_quant_paramsrPs* dEMM * *?T |1MNN  YYtxxzC  YYtxxzC ;;t:: .D ;;t:: .D KK4:: .E[--- RS Siio3JD :qAA -ZQZ[  t99 0T c JQdjj9J:qAA 7 |1E FFr*c [UUUUUUU5upxU[R:Xa[XX5n [ U R [ R5R5S-S:H5(af[ R"U5n [SU R5SU R5SU R5SU R5S3 5eXxU 4$U[R[R[R[R [R"[R$4;a[XX5n XxU 4$['SUS35e)a  :param data: data to quantize :param qType: data type to quantize to. :param symmetric: whether symmetric quantization is used or not. :parameter reduce_range: True if the quantization range should be reduced. Defaults to False. :parameter min_real_range: Minimum floating-point range (i.e., rmax - rmin) to enforce. Defaults to None. :parameter rmin_override: The value of rmin to use if not None. Otherwise, uses min(data). :parameter rmax_override: The value of rmax to use if not None. Otherwise, uses max(data). :return: minimum, maximum, zero point, scale, and quantized weights To pack weights, we compute a linear transformation - when data `type == uint8` mode, from `[rmin, rmax]` -> :math:`[0, 2^{b-1}]` and - when data `type == int8`, from `[-m , m]` -> :math:`[-(2^{b-1}-1), 2^{b-1}-1]` where `m = max(abs(rmin), abs(rmax))` and add necessary intermediate nodes to transform quantized weight to full weight using the equation :math:`r = S(q-z)`, where - *r*: real original value - *q*: quantized value - *S*: scale - *z*: zero point rvz+One of the quantized value is NaN data in [z, z], quantized_data in [z].zUnexpected value for qType=rS)rr r]ranyviewrrpravelrrrrrUrWr[rYrar_r-) rrrrrrrrrquantized_datanp_datas r& quantize_datarsO82  J ((()%uI ##EKK06683>3F G GmmD)G=gkkm_Bw{{}o^&&4&8&8&:%;2n>P>P>R=SSUW .00  *%uI.00 25'; <4;a[3S US 35e[A[CUR-555n[RDRGUXR$US S9nU$[RDRIU5n[R"UUS9RUR$5n[RJRMUU5nU$)a Returns a quantized version of the given ONNX initializer. :param weight: The ONNX initializer to quantize. :param quant_type: The final quantized data type. :param zero_point: The zero-point value to use for quantization. :param scale: The scale value to use for quantization. :param axis: The quantization axis if quantizing per-channel. Defaults to None. :param quant_weight_name: The name of the quantized initializer. If not specified, the quantized name is generated. :return: The quantized ONNX initializer. NrzThe initializer of shape z! could not be created, expecting z, got z and shape=z raw=rSzQuantized weights for z. must be 8-bit before packing as 4-bit values.T)rawrs)'tensor_proto_to_arrayrrshapelistrtakerrrreshape concatenater#TENSOR_NAME_QUANT_SUFFIXrr r] data_typedimsextendflattencopytobytesraw_datarrstrrar_rtrrbytespack_bytes_to_4bitrrtensor_dtype_to_np_dtype numpy_helper from_array)weightrrraxisquant_weight_name weight_data q_weight_data channel_count channel_dimsquantized_channel_data_listr channel_data channel_scalechannel_zero_pointquantized_channel_data q_weight_nameq_weight_initializercheck packed_dataquant_np_dtypes r&quantize_onnx_initializerrs ((/K*.M |(5F5F5H%\ #))$/ K--.  &(#}%A&++A4L!HM!+ %5..0-& " ( . .u}}=S/T/\/\]i/j k&))*EL ):%6;;-PhOi@jMT%%222#//1)3&!!((5$1!(5(=(=(?(D(D(F(N(N(P%  (&&:;E{{k///5==?mF[F[F]3]"/ 0A0A/BBc$,,.s34F5==?3B;O:PP[\b\h\h[iS!56t<=Q@(  ((--t/?/?/E/EF F   tUm 3!7 Ftuv v.}/D/D/FGH  ${{66}jR]R]_jpt6u  ==jI m>JRRSYS^S^_ #00;;M=Y r*cU[RR:Xa [S5eSnU(a[R U5nO0U(aU[ ;a [ UnO[R U5nU(d[SUS35eUupEUS:dUS:a'[SUSUSURS US US U3 5eU$) z Return qmin and qmax, the minimum and maximum value representable by the given qType :parameter qType: onnx.onnx_pb.TensorProto.UINT8 or onnx.onnx_pb.TensorProto.UINT8 :return: qmin, qmax z;This function is not implemented for float 8 as not needed.Nrrrrrz, dtype=z, reduce_range=z , symmetric=z, qType=) rr r]rONNX_INT_TYPE_REDUCED_RANGEgetONNX_INT_TYPE_SYMMETRIC_RANGEONNX_INT_TYPE_RANGEr-rt)rrrqrangerrs r&rrs  &&333!"_`` F,007 u ==.u5$((/ 07uvwwJD ax4!86$x |?<.Y" 8E7 4  Mr*c"[XUS9up4XC- $)z Helper function to get the quantization range for a type. parameter qType: quantization type. return: quantization range. r)r)rrrrrs r&get_qrange_for_qTyper!:s ) RJD ;r*c@US:aX-OUnUS:=(a X!:nX24$)z Helper function that tries to return a normalized axis in the range [0, rank - 1]. :parameter axis: The axis to normalize. :parameter rank: The tensor rank (number of dimensions). :return (is_valid, axis_norm) rr2)r rank axis_normis_valids r&normalize_axisr&Ds- $ax TIA~2)"2H  r*c[U5nUS:Xa [5$US-S-n[U5nSnSnXAS- :a+XS-S-S-XS--X5'US- nUS- nXAS- :aM+XA:a XS-X5'U$)a. Copies a source array of 8-bit values into a destination bytearray of packed 4-bit values. Assumes that the source values are already in the appropriate int4 range. :parameter src_8bit: The 8-bit element values to pack. :return A bytearray with every two 8-bit src elements packed into a single byte. rrrHrwrJ)r bytearray)src_8bit num_elemsdst_sizedstsrc_idst_is r&rrPsH IA~{A !#H H CEE a-  *S0Q68?S;PQ      a-   _s* Jr*c(\rSrSrSr//S4SjrSrg)QuantizedInitializerinzB Represents a linearly quantized weight input from ONNX operators Nc pXlX lX0lX@lXPlX`lXplXlXlgr!) r# initializerrminsrmaxs zero_pointsscalesrrr ) r%r#r2r3r4r5r6rrr s r&__init__QuantizedInitializer.__init__ss4 &  &  , r*) r rr2r#rr4r3r6r5r3r4r5r6__doc__r7r:r2r*r&r0r0ns r*r0c*\rSrSrSrSSjrSrg)QuantizedValueizA Represents a linearly quantized value (input\output\intializer) Nc pXlX lX0lX@lXPlX`lXplXlXlgr!) original_nameq_name scale_namezp_name value_typer  node_type node_qtype scale_type) r%r#new_quantized_namer@zero_point_namequantized_value_typer rCrDrEs r&r7QuantizedValue.__init__s2"( $& . "$$r*) r rDrCr>r?r@rErBrA)NNNNr9r2r*r&r<r<s%r*r<c\rSrSrSrSrSrg)BiasToQuantizeiz# Represents a bias to be quantized c(XlX lX0lgr! bias_name input_name weight_name)r%rNrOrPs r&r7BiasToQuantize.__init__s"$&r*rMNr9r2r*r&rKrKs 'r*rKcURS:Xa[SURS35eURS:XaURnGO,URS:XaURnGOURS:Xa UR nOURS:Xa UR nOURS:Xa URnOURS :Xa URnOURS :Xa URnO}URS :Xa URnO`URS :Xa URnOCURS :Xa URnO&[SURSURS35eURU0$)z Convert attribute to kwarg format for use with onnx.helper.make_node. :parameter attribute: attribute in AttributeProto format. :return: attribute in {key: value} format. rz attribute z does not have type specified.rrHrIrJrKrLrx rz has unsupported type rS) rr-r#rrsrPgfloatsintsstringstensorsgraphs) attributers r&attribute_to_kwargr]s; ~~:inn%55STUU~~  1   1   1   1   1    1  1 !! 1 !! 2   :inn%55KINNK[[\]^^ NNE ""r*c~UVs/sHo"RU:XdMUPM nn[U5S:aUS$S$s snf)z Helper function to find item by name in a list. parameter item_name: name of the item. parameter item_list: list of items. return: item if found. None otherwise. rN)r#r) item_name item_listitemitemss r& find_by_namercs@( Bid99 +ATiE B5zA~58/4/ Cs::cXSn[[U55HnXU:XdM UnM U$)z; Helper function to return index of an item in a node list r{)rr) elem_name elem_listelem_idxrs r&get_elem_indexrhs2H 3y> " <9 $H# Or*cF[RRSX/U5$)z Helper function to create a Mul node. parameter inputs: list of input names. parameter output: output name. parameter name: name of the node. return: Mul node in NodeProto format. Mul)rrr)inputsoutputr#s r& get_mul_noderms ;; $ ??r*clURRURU-UR-5$)zh Helper function to generate a identifiable filepath by concatenating the given identifier as a suffix. )parentjoinpathstemsuffix)filename identifiers r&generate_identified_filenamerus+ ?? # #HMMJ$>$P QQr*cRSSKnSSKJn SSKnUR"UR S9 [ S5 [ U5 [ S5 [ U5 URXSS9 URS5 URS5 URS 5 UR5 g) Nr) thresholdz Histogram:zHistogram Edges:T)fillz Tensor valueCountszTensor value V.S. Counts) sysmatplotlib.pyplotpyplotrset_printoptionsmaxsizeprintstairsxlabelylabeltitleshow)hist hist_edgesrzpltrs r& apply_plotrs{# S[[1 , $K  *JJtdJ+JJ~JJxII()HHJr*c n ^^^^^SSKmSSKnSSKmSSKJs Js Jn SSKJs Js Jn SSK J mJ mJ m [R"SU35 "UUUUU4SjSTR5nTR!XS9n[#[$R&R)US5S 5nUR+U5 SSS5 TR-S5nUR/S 5n /n [1UR355Hn X n U R55n [7U R9S U5R;55[7U R9S U5R;55/n[=[?U55nU RAU 5nU RAU5nURCU 5 UREU U5 URGU U5 URIU 5nU RKU5 M URMU [OU 55 U HnU RQU5 M U RS5nURUU 5 URWU U5 URYU 5nU R[U5 U R]5n[#[$R&R)US 5S5nUR+U5 SSS5 [$R^R9SS5S;aURRaUS5nURc5n[eU5H\nURgU5n[R"URi55 [R"URk55 M^ [#[$R&R)US5S 5n[1UR355Hn X n U R55n [7U R9S U5R;55[7U R9S U5R;55/nU S-[=[?U55-nUR+U5 UR+S5 M SSS5 g!,(df  GN=f!,(df  GN=f!,(df  g=f)z6 Helper function to write calibration table to files. rN)CalibrationMethod TensorData TensorsDatazcalibration cache: c*>\rSrSrUUUUU4SjrSrg)*write_calibration_table..MyEncoderi#cj>[UTT45(aUR5$[UTR5(a'UR5[ UR 5SS.$[UT5(a"UR R[ U5S.$TRRX5$)Nz numpy.array)rrtCLS)rr) rto_dictrtolistrrt __class__r3 JSONEncoderdefault)r%objrrrjsonnps r&r2write_calibration_table..MyEncoder.default$s# K899{{}$#rzz** # s399~m\\#011"}}55CII##++D6 6r*r2N)r3r4r5r6rr:)rrrrrsr& MyEncoderr#s  7 7r*r)clszcalibration.jsonwihighestlowestzcalibration.flatbufferswbQUANTIZATION_DEBUG0)r1zcalibration.cache  )6r flatbuffersr5onnxruntime.quantization.CalTableFlatBuffers.KeyValue quantizationCalTableFlatBuffersKeyValue5onnxruntime.quantization.CalTableFlatBuffers.TrtTableTrtTable"onnxruntime.quantization.calibraterrrlogginginfordumpsopenospathjoinwriterBuildersortedkeysrrrrarr CreateString KeyValueStartKeyValueAddKeyKeyValueAddValue KeyValueEndrTrtTableStartDictVectorrPrependUOffsetTRelative EndVector TrtTableStartTrtTableAddDict TrtTableEndFinishOutputenvironGetRootAsTrtTable DictLengthrDictKeyValue)calibration_cachedirrrrr json_datafilerbuilderkey_value_listkeyrd_valuesrWrflat_key flat_value key_value main_dict cal_tablebufdict_lenrrrrrrs @@@@@r&write_calibration_tablers LLLL]] LL&'8&9:;77D$$7 , ># (,,y$/446 7 (,,x.335 6 CK '',))%0 w'2!!':6((1 i(#0& $$Wc..AB# '' 2$!!#I 7# Wi0$$W-I NN9 .. C bggll3 9:D AT 3 B zz~~*C0H<%%77Q? '')xA!q)I LL ) LL* +! bggll3 34c :d+0023C&+F~~'Hhll9d388:;hll8T2779:F#ICK 00E JJu  JJt 4 ; :g : 9L B A ; :s%"R7R6CR& R R#& R4cUS:HR[R5nUS:gR[R5nUR5nURU- nU(dgU[ U5-[ U5- nUS:dSUSUSU35eUR[R5nXqU-U*U--- nUS:*R5S:XdeU$)ajGiven a discrete distribution (may have not been normalized to 1), smooth it by replacing zeros with eps multiplied by a scaling factor and taking the corresponding amount off the non-zero values. Ref: http://web.engr.illinois.edu/~hanj/cs412/bk3/KL-divergence.pdf https://github.com//apache/incubator-mxnet/blob/master/python/mxnet/contrib/quantization.py rNrzn_zeros=z , n_nonzeros=z, eps1=)rrrsumsizer)pepsis_zeros is_nonzerosn_zeros n_nonzeroseps1rs r&smooth_distributionrosQu}}-H6//%--0KllnG'!J  w % "3 3D #:Q'- |74&QQ: 88EMM "D(Nte{2 22D AI??  !! ! Kr*c[R"UR5SS9n[SURR 55$)NF)load_external_datac3N# UHn[R"U5v M g7fr!)r uses_external_data).0 intializers r& *model_has_external_data..s!mUlz#66zBBUls#%)rloadas_posixrgraphr2) model_pathmodels r&model_has_external_datars9 IIj))+ FE mUZU`U`UlUlm mmr*c[5nUR5Ul[RUl0nS/US'[ UR5U4SS/0UD6ng)z Generate model that applies graph optimization (constant folding, etc.) parameter model_path: path to the original onnx model parameter opt_model_path: path to the optimized onnx model :return: optimized onnx model ConstantSharingdisabled_optimizers providersCPUExecutionProviderN)rroptimized_model_filepathrORT_ENABLE_BASICgraph_optimization_levelr)ropt_model_path sess_optionkwargs_s r&optimize_modelrsb!"K+9+B+B+DK(+A+R+RK( F%6$7F !,,. jH^G_jcijAr*cSS0nUR(a:URH*nURURUR05 M, [R R X5 g)z>Tag the model that it went through quantization pre-processingonnx.quant.pre_processonnxruntime.quantNmetadata_propsupdaterrrrset_model_props)rrprops r&add_pre_process_metadatar sS.0CDN ((D  ! !488TZZ"8 9)KK6r*cUR(a7URH'nURS:XdMURS:XdM' g g)zCCheck the model whether it went through quantization pre-processingrrTFrrr)rrs r&model_has_pre_process_metadatar s< ((Dxx33 FY8Y) r*cSS0nUR(a:URH*nURURUR05 M, [R R X5 g)N onnx.inferrr)rrrs r&add_infer_metadatarsS"$78N %%A  ! !155!''"2 3&KK6r*cUR(a7URH'nURS:XdMURS:XdM' g g)NrrTFr )rrs r&model_has_infer_metadatars; %%Auu $4G)G& r*cURVs/sH'oR(aURS:XdM%UPM) nn[U5S:wa [S5eUSRnU$s snf)Nrrz$Failed to find proper ai.onnx domainr) opset_importdomainrr-version)ropsetai_onnx_domain opset_versions r&get_opset_versionrsf).););m);<a?@@"1%--M  ns $A*A*c[U5nUn[USU5nUS:a;U[RR:Xa[ R "SUS35 SnOCUS:Xa[ R "SUS35 O"US:a[ R "SUS35 SnX2:wa*[RRX5n[U5nU$) Nrbz$The original model opset version is z, which does not support quantization to float 8. Please update the model to opset >= 19. Automatically update the model to opset 19. Please verify the quantized model.rze, which does not support node fusions. Please update the model to opset >= 11 for better performance.z, which does not support quantization. Please update the model to opset >= 11. Automatically update the model to opset 11. Please verify the quantized model. ) rgetattrrr r]rwarningversion_converterconvert_version&save_and_reload_model_with_shape_infer)r weight_typertarget_opset_versionweight_quant_types r&update_opset_versionr%s%e,M( ]KHr/43C3C3P3PP2=/B1 1 " " 2=/BM M  2=/B1 1 ",&&66uS7u= Lr*c[US5n[RR[ U5[ U55 [R "UR 55n[U5 UR5 U$)Nz -inferred) rurshape_inferenceinfer_shapes_pathrrrrunlink)rinferred_model_pathrs r&load_model_with_shape_inferr+s`6z;O**3z?C@SDEQUANT_OP_NAMErJrMODEL_SIZE_THRESHOLDrrrrrRr7rr<rFrfrUrtrWr[rYr]rar_rrrprorrrqrrrrrrrrrrrr!r&rr0r<rKr]rcrhrmrurrrrrr r rrrr%r+r!rr<r?rBrErHrK)ks0r&rcsd #   00>>&QQ-PP7      ,$2'!47 4Dq4Dq SZ[fhiSjloHp* Q'*4Dq  t   #>#>L $  V!4  %++g"6  %++g"6!!5;;x#8''  %  5;;q #DekkRU]b]h]hFi"j%++d%**"Eu{{SV^c^h^hGi!j!!EKK$F TYafamamHn#o  5;;vU[[#I5;;W\didodoKp"q  5;;q#> BV[@\"]%++b"=u{{1TX?Y!Z %++d%**"Eu{{SV^c^h^hGi!j  5;;vU[[#I5;;W\didodoKp"q!   5;;q #DekkRU]b]h]hFi"j%++c"DekkRT\a\f\fFg!h!!EKK$F TYafamamHn#o  5;;vU[[#I5;;W\didodoKp"q  5;;q#> AUZ@["\%++b"=u{{1TX?Y!Z )+ A 13h<~!P#'"&"&;G ;G);G;G ;G ! ;G  ;G ;G);G~hl:=7:=D$( L L )L L   L  L " L L ^@ <>%%8''"#J0@R$Yx2n k 77!H7+,2-3/$rs"Y3Z%Z3Y?>Y?