`i= SSKrSSKrSSKrSSKrSSKrSSKrSSKrSSKJ r SSK J r SSKJ r J r JrJr SSKJrJr SSKJr SSKJr SSKJr \R2"\5r\S S 4\S S 4\S S 4\S S 4\S S 4S.rSS\S\S\S\S\4 Sjjr S\S\ \\44Sjr!Sr"Sr#SSjr$Sr%Sr&\S:Xa\&"5 gg)N)Dict)deque) ModelProto TensorProto numpy_helper load_model) BertOnnxModelBertOptimizationOptions)BertOnnxModelTF)BertOnnxModelKeras) Gpt2OnnxModelpytorchTtf2onnxF keras2onnx)bertbert_tf bert_kerasgpt2gpt2_tfonnx_model_pathuse_gpuoptimized_model_path opt_levelreturncSSKnU(a+SUR5;a[RS5 U$UR 5nUS:XaUR R UlOEUS:XaUR RUlO#US:XdeUR RUlUc USSnS RXcU(aS OS 5nX%l U(dURXS /S 9nO'URX5nSUR5;de[RR!U5(a$[RR#U5(de[R%SRU55 U$)ai Use onnxruntime to optimize model. Args: onnx_model_path (str): the path of input onnx model. use_gpu (bool): whether the optimized model is targeted to run in GPU. optimized_model_path (str or None): the path of optimized model. opt_level (int): graph optimization level. Returns: optimized_model_path (str): the path of optimized model rNCUDAExecutionProviderz3There is no gpu for onnxruntime to do optimization.cz{}_o{}_{}.onnxgpucpuCPUExecutionProvider) providersz)Save optimized model by onnxruntime to {}) onnxruntimeget_available_providersloggererrorSessionOptionsGraphOptimizationLevelORT_ENABLE_BASICgraph_optimization_levelORT_ENABLE_EXTENDEDORT_ENABLE_ALLformatoptimized_model_filepathInferenceSession get_providersospathexistsisfiledebug)rrrrr% sess_options path_prefixsessions g/var/www/html/ai-image-ml/venv/lib/python3.13/site-packages/onnxruntime_tools/transformers/optimizer.pyoptimize_by_onnxruntimer</sR *+2U2U2WW JK--/LA~0;0R0R0c0c - a0;0R0R0f0f -B0;0R0R0a0a -#%cr* /66{X_uejk,@) ..YoXp.q..M&'*?*?*AAAA 77>>. / /BGGNNCW4X4XX X LL<CCDXYZ cN[USSS9n[USSS9nUR5$)z Get counter of fused operators in optimized model. Args: optimized_model_path (str): the path of onnx model. Returns: A dictionary with operator type as key, and count as value NTr/load_external_data ) num_heads hidden_size)rr get_fused_operator_statistics)rmodel optimizers r;get_fusion_statisticsrH_s0 +DT REersCI  2 2 44r=c [R"5nURSS[SS9 URSS[SS9 URSS[RS [ [ R55S S R[ R55-S 9 URS S[SSS9 URSS[SSS9 URSSSSS9 URSS9 URSSSSS9 URSS9 URSSSSS9 URSS9 URSSSS S9 URSS!9 URS"SSS#S9 URSS$9 URS%SSS&S9 URSS'9 URS(SSS)S9 URSS*9 URS+SSS,S9 URSS-9 URS.SSS/S9 URSS09 URS1SSS2S9 URSS39 URS4SSS5S9 URSS69 URS7SSS8S9 URSS99 URS:SSS;9 URSS<9 URS=SSS>S9 URSS?9 URS@SSSAS9 URSSB9 URSCS[/SDQSESFSG9 URSHSSSIS9 URSSJ9 UR5nU$)KNz--inputTzinput onnx model path)requiredtypehelpz--outputzoptimized onnx model pathz --model_typeFrz!Model type selected in the list: z, )rJrKdefaultchoicesrLz --num_headsrAzwnumber of attention heads. 12 for bert-base model and 16 for bert-large. For BERT, set it to 0 to detect automatically.)rJrKrMrLz --hidden_sizerBzwbert model hidden size. 768 for bert-base model and 1024 for bert-large. For BERT, set it to 0 to detect automatically.z --input_int32 store_truezKUse int32 (instead of int64) tensor as input to avoid unnecessary data cast)rJactionrL) input_int32z --float16zdIf your target device is V100 or T4 GPU, use this to convert float32 to float16 for best performance)float16z--disable_attentionzdisable Attention fusion)disable_attentionz--disable_skip_layer_normz%disable SkipLayerNormalization fusion)disable_skip_layer_normz--disable_embed_layer_normz&disable EmbedLayerNormalization fusion)disable_embed_layer_normz--disable_bias_skip_layer_normz2disable Add Bias and SkipLayerNormalization fusion)disable_bias_skip_layer_normz--disable_bias_geluz)disable Add Bias and Gelu/FastGelu fusion)disable_bias_geluz--disable_layer_normz!disable LayerNormalization fusion)disable_layer_normz--disable_geluzdisable Gelu fusion) disable_geluz--enable_gelu_approximationz+enable Gelu/BiasGelu to FastGelu conversion)enable_gelu_approximationz--use_mask_indexzBuse mask index instead of raw attention mask in attention operator)use_mask_indexz--no_attention_maskz1no attention mask. Only works for model_type=bert)no_attention_maskz --verbose)rJrPverbosez --use_gpuzuse GPU inference)rz--only_onnxruntimezoptimized by onnxruntime only)only_onnxruntimez --opt_level)rrrrrz;onnxruntime optimization level. 0 will disable onnxruntime.)rJrKrNrMrLz--use_external_data_formatzuse external data format)use_external_data_format) argparseArgumentParser add_argumentstrlowerlist MODEL_CLASSESkeysjoinint set_defaults parse_args)parserargss r;_parse_argumentsrons  $ $ &F  DsAXY  TB]^ !& YY & $]%7%7%9 :@499]M_M_MaCbb d   B    B  !&+jl E*  s u  & -lYst %0 3!&+DF 6 4!&+EG 7 8!&+QS U; -!&+HJ %0 .!&+@B 51 (5Tij U+ 5!&+JL %8 *!&+ac u- -!&+PR %0  eLI &  eLObc & ,u\Xwx /  !& - !Z \ 4!&+79 7    D Kr=c6[UR5nUR(aSUlUR(aSUlUR (aSUlUR(aSUl UR(aSUl UR(aSUl UR(aSUlUR (aSUlUR"(aUR%S5 UR&(aUR)5 U$)NFT)r model_typerY enable_gelurXenable_layer_normrSenable_attentionrTenable_skip_layer_normrUenable_embed_layer_normrVenable_bias_skip_layer_normrWenable_bias_gelurZr[use_raw_attention_maskr\disable_attention_mask)rnoptimization_optionss r;_get_optimization_optionsr|s24??C +0( 16. 05- ##6;3 $$7<4 ((;@8 05- %%9=6 33E: 335 r=c d[Uupn Sn US:a [XUS9n OU (a [USSS9n [U =(d USSS9n U R(aBXR:wa3[R SU SU RS U R S 35 Uc [U5nU"XU5n U(dU RU5 U (a:[R"U 5 [RS RU 55 S U RlS SKJn XRlU $)aOptimize Model by OnnxRuntime and/or offline fusion logic. The following optimizes model by OnnxRuntime only, and no offline fusion logic: optimize_model(input, opt_level=1, use_gpu=False, only_onnxruntime=True) If you want to optimize model by offline fusion logic. optimize_model(input, model_type, num_heads=12, hidden_size=768, optimization_options=your_options) Args: input (str): input model path. model_type (str): model type - like bert, bert_tf, bert_keras or gpt2. num_heads (int): number of attention heads. Default is 0 to allow detect the parameter from graph automatically (for model_type "bert" only). hidden_size (int): hidden size. Default is 0 to allow detect the parameter from graph automatically (for model_type "bert" only). optimization_options (OptimizationOptions or None): optimization options that can use to turn on/off some fusions. opt_level (int): onnxruntime graph optimization level (0, 1, 2 or 99). When the level > 0, onnxruntime will be used to optimize model first. use_gpu (bool): use gpu or not for onnxruntime. only_onnxruntime (bool): only use onnxruntime to optimize model, and no offline fusion logic is used. Returns: object of an optimizer class. Nr)rrFTr?z#Model producer not matched: Expect z, Got  z0. Please specify correct --model_type parameter.zRemove tempoary model: {}zonnxruntime.transformersr) __version__)rgr<r producer_namer'warningproducer_versionr optimizer3remover7r/rFr%r)inputrqrCrDr{rrr_optimizer_classproducerrun_onnxruntimetemp_model_pathrFrGonnxruntime_versions r;optimize_modelrs84A3L0_O1}1%T]^ 2%RST /%QU VE x+>+>>1(75CVCVBWWXY^YoYoXpqa b #6zB+>I /0 /" 077HI$>IOO!>':OO$ r=cfU(a[R"SSS9 g[R"SS9 g)NDEBUGz8[%(filename)s:%(lineno)s - %(funcName)20s()] %(message)s)levelfmtz%(funcName)20s: %(message)s)r) coloredlogsinstallr]s r; _setup_loggerrFs%'/ij =>r=c &[5n[UR5 [RR UR 5[RR UR5:Xa[RS5 [U5n[UR URURURURUUR UR"S9nUR$(aUR'5 UR((aUR+5 UR-URUR.5 UR15(a[R3S5 g[R3S5 g)NzYSpecified the same input and output path. Note that this may overwrite the original model)rr{rr_z#The model has been fully optimized.zThe model has been optimized.)rorr^r3r4realpathroutputr'rr|rrqrCrDrrr_rR convert_model_float32_to_float16rQchange_input_to_int32save_model_to_filer`is_fully_optimizedinfo)rnr{rGs r;mainrMs   D$,, ww #rww'7'7 'DDrt4T:tzz##~~#//)-4H'+||040E0EGI ||224 '')   d.K.KL##%% 9: 34r=__main__)FNr)rrrNrFF)'loggingronnxr3sysranumpynptypingr collectionsrrrrronnx_model_bertr r onnx_model_bert_tfr onnx_model_bert_kerasr onnx_model_gpt2r getLogger__name__r'rgrdboolrjr<rHror|rrrr=r;rs( BBB.4)   8 $It ,E2%|U; It ,y%0  -28<-/- S- %)- 25- (+- 58- ` 5 5S#X 5zz 6% (, $)>B?5B zFr=