9iE:SSKrSSKJr SSKJrJrJr SSKJrJ r SSK r SSK r SSK J r SSKJr SSKJr S\ \\\44S jr\"S S 55r\"S S 55r\"SS55r\"SS9"SS\\\55rSrS\S\ \\\S44Sjrg)N)deepcopy) dataclassfieldfields)ListUnion)CliArgumentParser)Config)DEFAULT_DATASET_NAMESPACEvaluesc[U[5(aURS5OUn0nU=(d /HBn[UR 55S:XaM"URS5upE[ U5X$'MD U$)N,r=) isinstancestrsplitlenstrip parse_value)r pairs_paramskvkeyvalues a/var/www/html/land-doc-ocr/venv/lib/python3.13/site-packages/modelscope/trainers/training_args.pyset_flatten_valuersj!+FC!8!8FLL fEGkrk rxxz?a  XXc] "5)  Nc\rSrSr%\"SSS0S9r\\S'\"SSS0S9r\\S'\"SSS 0S9r \\S '\"SSS 0S9r \\S '\"SSS 0S9r \\S'\"SSS0S9r \\S'\"\ SS0S9r\\S'\"\ SS0S9r\\S'\"SSS0S9r\\S'Srg) DatasetArgsNhelpzNThe dataset name used for training, can be an id in the datahub or a local dirdefaultmetadatatrain_dataset_namezOThe subset name used for evaluating, can be an id in the datahub or a local dirval_dataset_namez.The subset name used for training, can be Nonetrain_subset_namez0The subset name used for evaluating, can be Noneval_subset_namezThe split of train dataset train_splitzThe split of val dataset val_splitz'The dataset namespace used for trainingtrain_dataset_namespacez)The dataset namespace used for evaluatingval_dataset_namespaceaThe json file to parse all datasets from, used in a complex dataset scenario,the json format should be like: [ { "dataset": { # All args used in the MsDataset.load function "dataset_name": "xxx", ... }, # All columns used, mapping the column names in each dataset in same names. "column_mapping": { "text1": "sequence1", "text2": "sequence2", "label": "label", }, # float or str, float means to split the dataset into train/val, # or just str(train/val) "split": 0.8, } ] dataset_json_file)__name__ __module__ __qualname____firstlineno__rr%r__annotations__r&r'r(r)r*r r+r,r-__static_attributes__r.rrrrs3$  \   "  ]  c # D  s ! F  OS  0  K  .  Is $)) = $ S "') ? " 3 #    s rrc\rSrSr%\"SSSS.S9r\\S'\"SSS0S9r\\S '\"SSS 0S9r \\S '\"SS S S.S9r \\S'Sr g) ModelArgsjNzThe task code to be usedtaskr!cfg_noder"r!zA model id or model dirmodelzthe revision of modelmodel_revisionzJThe mode type, if load_model_config is False, user need to fill this fieldz model.type model_typer.) r/r0r1r2rr8rr3r;r<r=r4r.rrr6r6js.  D#  -  E3   +  NC  Y$  J rr6c\rSrSr%\"SSS0S9r\\S'\"SSS S .S9r\\S '\"S S SS .S9r \\S'\"SSSS .S9r \ \S'\"SSSS .S9r \ \S'\"SSSS .S9r \\S'\"S SSS .S9r\\S'\"SSSS .S9r\ \S'\"SS S!S .S9r\ \S"'\"S#S$S%S .S9r\\S&'\"S'S(S)S .S9r\\S*'\"S+S,S-S .S9r\\S.'\"S/S0S1S .S9r\\S2'\"S3S4S5S .S9r\\S6'\"S7S8S9\S:.S9r\\S;'\"S7S'\"S?S@SA/SBQSC.S9r\\SD'\"S SSE0S9r\\SF'\"S#SGSHSI.S9r\\SJ'\"S?SKSL/SBQSM.S9r\\SN'\"SOSPSQSI.S9r\\SR'\"S7SSSTSI.S9r \\SU'\"S?SVSW/SBQSM.S9r!\\SX'\"SOSYSZSI.S9r"\\S['\"SS\S]SI.S9r#\ \S^'\"S7S_S`SI.S9r$\\Sa'\"SbScSdSI.S9r%\\Se'\"S7SfSgSI.S9r&\\Sh'\"SOSiSjSI.S9r'\\Sk'\"SSlSmSI.S9r(\ \Sn'\"S7SoSpSI.S9r)\\Sq'\"S7SrSsSI.S9r*\\St'\"SuSvSwSI.S9r+\ \Sx'\"SySzS{SI.S9r,\\S|'\"SSlS}SI.S9r-\ \S~'\"S7SoSSI.S9r.\\S'\"S7SrSSI.S9r/\\S'\"SuSvSSI.S9r0\ \S'\"SySzSSI.S9r1\\S'Sr2g7) TrainArgs*r!zThe random seedr"seedz#train.dataloader.batch_size_per_gpuz:The `batch_size_per_gpu` argument for the train dataloader)r:r!per_device_train_batch_sizerz train.dataloader.workers_per_gpuz7The `workers_per_gpu` argument for the train dataloadertrain_data_workerFztrain.dataloader.shufflez/The `shuffle` argument for the train dataloader train_shuffleztrain.dataloader.drop_lastz1The `drop_last` argument for the train dataloadertrain_drop_lastz(evaluation.dataloader.batch_size_per_gpuz9The `batch_size_per_gpu` argument for the eval dataloaderper_device_eval_batch_sizez%evaluation.dataloader.workers_per_gpuz6The `workers_per_gpu` argument for the eval dataloadereval_data_workerzevaluation.dataloader.shufflez.The `shuffle` argument for the eval dataloader eval_shufflezevaluation.dataloader.drop_lastz0The `drop_last` argument for the eval dataloadereval_drop_lastztrain.max_epochszThe training epochs max_epochsz./train_targetztrain.work_dirz%The directory to save models and logswork_dirg-C6 ?ztrain.optimizer.lrz"The learning rate of the optimizerlrLinearLRztrain.lr_scheduler.typezThe lr_scheduler type in torch lr_schedulerAdamWztrain.optimizer.typez+The optimizer type in PyTorch, like `AdamW` optimizerNztrain.optimizerzThe optimizer params)r:r! cfg_setteroptimizer_paramsztrain.lr_schedulerzThe lr scheduler paramslr_scheduler_paramsby_epochz&train.lr_scheduler.options.lr_strategyzThe lr decay strategy)rWby_stepno)r:r!choices lr_strategyzThe local rank local_rankz+The interval of iter of logging informationztrain.logging.intervalr9logging_intervalz5Eval strategy, can be `by_epoch` or `by_step` or `no`zevaluation.period.eval_strategy)r!r:rZ eval_strategyz Eval intervalzevaluation.period.interval eval_intervalzThe metric name for evaluationzevaluation.metrics eval_metricsz>Checkpointing strategy, can be `by_epoch` or `by_step` or `no`z%train.checkpoint.period.save_strategy save_strategyz9The interval of epoch or iter of saving checkpoint periodz train.checkpoint.period.interval save_intervalz;Save the checkpoint(if it's the best) after the evaluation.ztrain.checkpoint.best.save_bestsave_best_checkpointz%The metric used to measure the model.z train.checkpoint.best.metric_keymetric_for_best_modelmaxzDThe rule to measure the model with the metric, can be `max` or `min`ztrain.checkpoint.best.rulemetric_rule_for_best_modelzBThe max number of checkpoints to keep, older ones will be deleted.z*train.checkpoint.period.max_checkpoint_nummax_checkpoint_numzGThe max number of best checkpoints to keep, worse ones will be deleted.z(train.checkpoint.best.max_checkpoint_nummax_checkpoint_num_bestz$Push to hub after each checkpointingz#train.checkpoint.period.push_to_hub push_to_hubz  L# #:;  d  K;  L#   h9  NC #-;  d #/<  s rr?F)initcR\rSrSr%\"SSS0S9r\\S'SrS S jr S S jr S r S r g) TrainingArgsiFr!zeUse the configuration of the model, default will only use the parameters in the CLI and the dataclassr"use_model_configc [UR55Ul[U5H7nURU;dM[ XRXR5 M9 0UlgN)listkeys manual_argsrnamesetattr _unknown_args)selfkwargsfs r__init__TrainingArgs.__init__sM .AvvfffVVn5 rNc,[U5nURU5up4UVs/sHnUS;dM SU;dMUPM nn0n[S[U5S5H(n[ XGS-5XdUR SS5'M* [ U5nU=RUR- slURRU5 [U5R5H(upU cM [X 5(dM[X U 5 M* U$s snf)zKConstruct a TrainingArg class by the parameters of CLI. Returns: Self )\ z --local-rank=rr_-)r parse_known_argsrangerrreplacevarsrrupdateritemshasattrr) r parser_argsparserargsunknownitem_unknowni args_dictrrs r parse_cliTrainingArgs.parse_clis  #4(// < $ $T<' ,;4,G W  q#g,*A4?A4OHQZ''R0 1+J  F... !!(+"9-335JC74#5#55)6  s DDDc h[5n[R"5nUc URn[ U5HnUR R S5nUR R S5=(d SnUbqURUR;dU(dN[U[5(aU/nUH.nURXv"[XR5505 M0 MM[XR5X4R'M URUR5 X#4$)zaConvert the TrainingArgs to the `Config` Returns: The Config, and extra parameters in dict. r:rTcU$r}r.xs r(TrainingArgs.to_config..sAr)r addictDictr{rr$getrrrrmerge_from_dictgetattrr)rignore_default_configcfgrrr:rT_nodes r to_configTrainingArgs.to_configs hKKM ($($9$9 !Azz~~j1H 5F+J#66T---5J!(C00$,:!)++"JwtVV/D$EFH"*6K%,D&&$9 &&! D../~rcT[U5nUHnURU:XdMUs $ gr})rr)rr_fieldsrs r get_metadataTrainingArgs.get_metadatas*,Avv}r)rrr}) r/r0r1r2rr{rvr3rrrrr4r.rrrzrzs;#  P  d  .8rrzchSSKJn /n/n[US5n[R"U5nUGHnUR"S0USD6R 5nUR nUSR5n UV s/sH oU ;dM U PM n n SSKJ n SSKJ n SSKJ n URR5Vs/sHoDSU ;dM UPM nn0nUH?n[US U5(aU "US R5UUS'M4US UUS'MA U "U5nUR!S U US 9R#US5nUS n[U[$5(a6US ;deUS:XaUR'U5 GMTUR'U5 GMh[U[(5(aSUs=:aS :de eUR+US9nUR'US5 UR'US5 GM SSS5 SSKJn U"U5U"U54$s sn fs snf!,(df  N.=f)a The filename format: [ { "dataset": { "dataset_name": "xxx", ... }, "column_mapping": { "text1": "sequence1", "text2": "sequence2", "label": "label", } "usage": 0.8, } ] r) MsDatasetrdatasetcolumn_mapping)Features)Value) ClassLabelr_cU$r}r.rs rr)build_dataset_from_file.."s!r)remove_columnsfeaturesusage)trainvalr) train_sizetestN)concatenate_datasetsr.) modelscoperopenjsonload to_hf_dataset column_namesrdatasetsrrrrrrdtypemaprename_columnsrappendrwtrain_test_splitr)filenamer train_seteval_setrds_jsondsr all_columns keep_columnscolumnrrrrr new_featuresrds_dictrs rbuild_dataset_from_filers8$%IH h ))A,Bnn5r)}5CCEG!..K./446L%0%06,4N[  * & +"++1133at|7K3 LadJ//).qtzz):L1&)*1L1&  $L1Lkk-%"'(6~b9I6J'K wKE%%% 0000G#$$W-OOG,!%//AMMAAMAA!22e2D  !120G N.  *,@,J JJE   s7A#H#: HH 0H#= H HD(H# H## H1rreturnc$SSSSSSSS.nX;aX$SU;dSU;a"URSS5RSS5$[R"SU5(a [U5$[R"SU5(a [ U5$U$) NTF)TruetrueFalsefalseNonenonenull"'rz^\d+$z4[+-]?(?=\d*[.eE])(?=\.?\d)\d*\.?\d*(?:[eE][+-]?\d+)?)rrematchrurw)r const_maps rrr6sI  }}S"%--c266 (E " "5z I  U| r)rcopyr dataclassesrrrtypingrrrr'modelscope.trainers.cli_argument_parserr modelscope.utils.configr modelscope.utils.constantr rrrr6r?rzrrwrvrr.rrrs 00 E*?eCcN3 L L  L ^     6 ^ ^  ^ B  I; 9IIX>KBsuS%t%;<r