9imrSrSSKrSSKrSSKrSSKrSSKrSSKJr SrSr Sr Sr S r S r S rS rS rg)zargparser configurationN) get_hostnamecURSS5nURSSSS9 URSSSS9 URS SS S9 URS [S S S9 URS[SSS9 URS[SSS9 URS[SSS9 URS[SSS9 URS[SSS9 URS[S SS9 URS [S S!S9 URS"[S#S$S9 URS%[S&S'S9 URS(SS)S9 URS*[S+S,S9 URS-SS.S9 URS/SS0S9 U$)1zModel argumentsmodelzmodel configurationz--transformer-xl store_truezuse transformer-xl for trainingactionhelpz--pretrained-bertzuse a pretrained bert-large-uncased model insteadof initializing from scratch. See --tokenizer-model-type to specify which pretrained BERT model to usez--encoder-decoderz0use the encoder-decoder architecture for blocklmz--attention-dropout皙?z)dropout probability for attention weightstypedefaultr z--num-attention-headsz"num of transformer attention headsz --hidden-sizeiztansformer hidden sizez--intermediate-sizeNzMtransformer embedding dimension for FFNset to 4*`--hidden-size` if it is Nonez --num-layersznum decoder layersz--layernorm-epsilongh㈵>zlayer norm epsilonz--hidden-dropoutz0dropout probability for hidden state transformerz--output-dropoutz%dropout probability for pooled outputz--max-position-embeddingsz,maximum number of position embeddings to usez --vocab-sizeiznvocab size to use for non-character-level tokenization. This value will only be used when creating a tokenizerz --deep-initzinitialize bert model similar to gpt2 model.scales initialization of projection layers by a factor of 1/sqrt(2N). Necessary to train bert models larger than BERT-Large.z--make-vocab-size-divisible-byzcPad the vocab size to be divisible by this value.This is added for computational efficieny reasons.z--cpu-optimizerzRun optimizer on CPUz--cpu_torch_adamz#Use Torch Adam as optimizer on CPU.add_argument_group add_argumentfloatintparsergroups d/var/www/html/land-doc-ocr/venv/lib/python3.13/site-packages/modelscope/models/nlp/mglm/arguments.pyadd_model_config_argsrs?  % %g/D EE  .0   ?A   8 :    1 3  c46NP  1 2  S"3GI   ! #    ? A    4 6  #  ; =      )* ( = >  ,5KM  24 McURSS5nURSSSS9 URSSSS9 URS SS S9 URS SS S9 URS SSS9 URS[SSS9 URS[SSS9 URS[SSS9 URS[SSS9 URS[SS9 U$)zMixed precision arguments.fp16zfp16 configurationsz--fp16rzRun model in fp16 moderz--fp32-embeddingzembedding in fp32z--fp32-layernormzlayer norm in fp32z--fp32-tokentypeszembedding token types in fp32z--fp32-allreducezall-reduce in fp32z --hysteresisz#hysteresis for dynamic loss scalingr z --loss-scaleNzsStatic loss scaling, positive power of 2 values can improve fp16 convergence. If None, dynamicloss scaling is used.z--loss-scale-windowz.Window over which to raise/lower dynamic scalez --min-scalerz)Minimum loss scale for dynamic loss scalez--attention-scale?r r)rrrrrs radd_fp16_config_argsr$tsF  % %f.C DE ,DF <6IK <6JL  ,. <6JL   2 4    !   = ?    8 :  *D MrcURSS5nURS[SSS9 URS[SS S9 URS [S S S9 URS [S SS9 URSSSS9 URS[S SS9 URSSSS9 URS[SSS9 URS[SSS9 URS[SSS9 URS [S!S"9 URS#[S$S%S9 URS&[S'S(S9 URS)[S*S+S9 URS,SS-S9 URS.SS/S9 URS0[SS1S9 URS2[S3/S4QS5S69 URS7[S8S"9 URS9[S:S;S9 URS<[S S=S9 URS>SS?S9 URS@[SSAS9 URSBSSC9 URSD[S SES9 URSF[SGSHS9 URSISSJS9 URSKSSLS9 URSM[SSNS9 URSOSSPS9 URSQSSRS9 URSSSSTS9 URSUSSVS9 URSWSSXS9 URSYSSZS9 URS[S\S]S\S^/S_9 URS`Sa/SbQScSd9 URSe[SSfS9 URSgSShS9 URSiSSjS9 URSk[SlS"9 URSm[SlS"9 URSn[SlS"9 URSo[S!S"9 URSp[SqS"9 URSr[SsS"9 URSt[S!S"9 URSu[S!S"9 URSvSSwS9 URSxSSyS9 URSzSS{S9 URS|SS}S9 URS~[S!S"9 URS[S!S"9 URSSSS9 U$)zTraining arguments.trainztraining configurationsz--experiment-namezgpt-345Mz.The experiment name for summary and checkpointr z --batch-sizezData Loader batch sizez--gradient-accumulation-stepsrz--weight-decayg{Gz?z.weight decay coefficient for L2 regularizationz--checkpoint-activationsrzLcheckpoint activation to allow for training with larger models and sequencesrz--checkpoint-num-layersz/chunk size (number of layers) for checkpointingz$--deepspeed-activation-checkpointingz,uses activation checkpointing from deepspeedz--epochsNz>Number of finetunning epochs. Zero results in evaluation only.z --clip-gradr"zgradient clippingz --train-itersrz:total number of iterations to train over all training runsz--label-smoothingr#z--log-intervaldzreport intervalz --summary-dirz"The directory to store the summaryz--seediz random seedz--reset-position-idsz/Reset position ids after end-of-document token.z--reset-attention-maskz7Reset self attention maske after end-of-document token.z--lr-decay-iterszUnumber of iterations to decay LR over, If None defaults to `--train-iters`*`--epochs`z--lr-decay-stylelinear)constantr+cosine exponentialzlearning rate decay functionr rchoicesr z--lr-decay-ratior z--lrg-C6?zinitial learning ratez--warmupzNpercentage of data to warmup on (.01 = 1% of all training iters). Default 0.01z--switch-linearz'Switch to linear decay for cosine decayz--savez(Output directory to save checkpoints to.z--new-save-directoryr z --save-epochznumber of epochs between savesz--save-intervaliz"number of iterations between savesz--no-save-optimzDo not save current optimizer.z --no-save-rngzDo not save current rng state.z--loadz2Path to a directory containing a model checkpoint.z--no-load-optimz.Do not load optimizer when loading checkpoint.z --no-load-rngz.Do not load rng state when loading checkpoint.z--no-load-lr-schedulerz1Do not load lr scheduler when loading checkpoint.z--no-deepspeed-loadz)Not use deepspeed when loading checkpointz --finetunezLoad model for finetuning. Do not load optimizer or rng state from checkpoint and set iteration to 0. Assumed when loading a release checkpoint.z--resume-dataloaderzResume the dataloader when resuming training. Does not apply to tfrecords dataloader, try resumingwith a different seed in this case.z--distributed-backendncclzBwhich backend to use for distributed training. One of [gloo, nccl]gloo)rr r0z --DDP-impltorch)localr4nonez4which DistributedDataParallel implementation to use.)rr0r z --local_rankz+local rank passed from distributed launcherz --block-lmz$whether use the BlockLM pre-trainingz --masked-lmz whether to use the mlm objectivez --bert-probg?z--gpt-infill-probz--gpt-min-ratioz--gap-sentence-probz--gap-sentence-ratiog333333?z--avg-block-lengthz--short-seq-probz--single-span-probz --task-maskz3Use different mask for generation and blank fillingz--no-shuffle-blockz-not shuffle the blocks when filling the blankz--no-block-positionz9Use (rough) absolute positions instead of block positionsz--sentinel-tokenz:Use sentinel (mask) tokens to replace 2d position encodingz--block-mask-probz--context-mask-ratioz--random-positionz>Use random start position to cover all the position embeddingsrrstrrrrs radd_training_argsr:sn  % %g/H IE   = ?  S!2JL '  % '    = ?  "+,  !  > @  . ;=   M O  E35HJ   I K  *D sC6GI   1 3  xc4mL  >@  !"  : ;   ? + -  )sC UF1HJ  ( )   68    7 9  -lC   - /    1 3   -/  -/   A C   =?  =?   @B  8: 56  ./ L "  * C E    : <   35  /1 }5#> *D (ucB ,5#F -E4H +#qA )sC +%E  BD  <>  HJ  IK *D -E3G  MO Mrc\URSS5nURS[SSS9 URS[SS S9 URS [S S S9 URS [SSS9 URS[SSS9 URS[SSS9 URS[SS9 U$)zEvaluation arguments. validationzvalidation configurationsz--eval-batch-sizeNzIData Loader batch size for evaluation datasets.Defaults to `--batch-size`r z --eval-itersr)z=number of iterations to run for evaluationvalidation/test forz--eval-intervalr!z5interval between running evaluation on validation setz --eval-epochrz2epoch between running evaluation on validation setz--eval-seq-lengthzMMaximum sequence length to process for evaluation. Defaults to `--seq-length`z--eval-max-preds-per-seqziMaximum number of predictions to use for evaluation. Defaults to math.ceil(`--eval-seq-length`*.15/10)*10z--overlapping-eval r#)rrrrs radd_evaluation_argsr>ss  % %l&A CE  % &        D F    A C   1 2  " 3 4 +#rB MrcURSS5nURS[SS9 URS[SS9 URS[S S9 URS [S S9 URS [S S9 URS[SS9 URS[S S9 URS[S S9 URSSS9 URS[SS9 U$)zText generate arguments.zText generationconfigurationsz --temperaturer"r#z--top_pr(z--top_krz--out-seq-lengthz --num-beamsrz--length-penaltyz--no-repeat-ngram-sizez--min-tgt-lengthz --select-topkrr1z--blank-maskratior rrs radd_text_generate_argsrBs  % %&79I JE UC@ yuc: ysA6 )SA }3: )sC /c1E )Q? |< *D MrcURSS5nURS[SSS9 URSSS S 9 URS SS 9 URS SSSS9 URSSSSS9 URSSSSS9 URS[SSS9 URS[SSS9 URSSSS9 URS S!S"S9 URS#SS$S9 URS%S&S'S9 URS(SS)S 9 URS*SS 9 URS+[SS,S9 URS-SS.S 9 URS/SS0S 9 URS1[S2S3S9 URS4[SS5S9 URS6[S7S8S9 URS9[S:/S;QSSS 9 URS?S[S@SA9 URSBSSCS 9 URSD[SESFS9 URSG[SHSIS9 URSJ[SSKS9 URSL[SMSN9 URSOSSPS 9 URSQ[SSRS9 URSS[SSTS9 URSU[SSVS9 URSWSSSXS9 URSY[SMSZS9 URS[[SSN9 URS\[SSN9 U$)]z Train/valid/test data arguments.datazdata configurationsz--model-parallel-sizerzsize of the model parallel.r z --shufflerzIShuffle data. Shuffling is deterministic based on seed and current epoch.rz--filter-englishr1z --train-data+Nz=Whitespace separated filenames or corpora names for training.)nargsrr z --valid-data*zFilename for validation data.z --test-datazFilename for testingz --data-dirz#The data path to all the data filesz--input-data-sizes-filez sizes.txtz,the filename containing all the shards sizesz--delim,z&delimiter used to parse csv data files)rr z --text-keysentencez(key to use to extract text from json/csvz--eval-text-keyzQS?9 URS@SSASB9 URSC[SS9 URSD[SS9 URSESSASB9 U$)FNfinetunezfinetune configurationsz--taskz Task name.)r r z--load-pretrainedzLoad pretrained model)r r rz --pool-token)startpadclsz-The token to pool the sequence representationrT)r r0r rz --cloze-evalrz"Evaluation dataset with cloze taskrz --multi-tokenz$Use multi token for cloze evaluationz--segment-lengthrz/The maximum segment length for cloze evaluationr z --loss-func) cross_entropyhinge generativemixrU)r r0rz--block-lm-ratior(r#z--adapetz.Use the decoupled cross entropy loss in AdaPETz --pattern-idz --fast-decodezRFast decode for multi-token cloze. Can only be used without checkpoint activation.z--few-supergluer1z --eval-validz!Whether evaluate on the valid setz--validation-metricz--unidirectionalz$Use the left to right language modelz--src-seq-lengthz--tgt-seq-lengthz --adam-beta1g?z --adam-beta2g+?z --adam-epsg:0yE>z --optimizeradam adafactorz--wsc-negativez --overwritez--no-validationz--continuous-promptzUse continuous prompt for PETz--num-prompt-tokensz --prompt-funclstm)r[mlpr6)rr0z--freeze-transformerF)r rz--tune-prefix-layersz--prefix-promptz --prompt-initr8rs radd_finetune_config_argsr]_s4  % %j2K LE xc =   $    ' <    13  35   > @   ? !  )sC  =? ~C; \   (>  02 ,3E  35 )TB )TB ~E3? ~E5A |%> C&+)>P ' = }\: (>  ,. ,3B 1HJ |UD -CF (sA> |UK Mrcz[R"SS9n[U5n[U5n[ U5n[ U5n[ U5n[U5n[U5n[R"U5nUR/S9nUR(dUR(d [S5 [R R#5Ul[%[&R("SS55Ul[%[&R("SS55Ul[/US 5(aUR0(a [3U5 O[&R("S 5(a[%[&R("S 55n[%[&R("S 55n[%[&R("S S55n[%[&R("S S55nX!lXS-U-UlXC-Ul[7UR8UR,5UlUR*S:Xa/[SR;UR,UR855 SUlUR>c"SUlUR*S:Xa [S5 UR@(dSUl!SUl"SUl#[/US5(aUR(aURHb[KURHSS9n[LRN"U5nSSS5 SW;a USUl(SU;a USUl)OSUl)SU;aWUSRUS05nURUSURV5Ul+URUSURX5Ul,U$!,(df  N=f)zParse all the args.zPyTorch BERT Model) description)argsz#WARNING: No training data specifiedRANK0 WORLD_SIZE1 deepspeed_mpiOMPI_COMM_WORLD_LOCAL_RANKOMPI_COMM_WORLD_LOCAL_SIZESLURM_JOB_NUM_NODES SLURM_NODEIDrz1using world size: {} and model-parallel size: {} FNTz > using dynamic loss scaling deepspeedzutf-8)encodingtrain_micro_batch_size_per_gpugradient_accumulation_stepsr optimizerparamslr weight_decay)-argparseArgumentParserrr$r:r>rBrOr]rjadd_config_arguments parse_args train_datadata_dirprintr4cuda is_availablerosgetenvrank world_sizehasattrrempi_define_env local_rankminmodel_parallel_sizeformatdynamic_loss_scale loss_scalerfp32_embeddingfp32_tokentypesfp32_layernormdeepspeed_configopenjsonload batch_sizermgetrprq) rr`r local_size num_nodesnodeidfileroptimizer_params_configs rget_argsrs $ $1E FF "6 *F !& )F v &F  (F #F +F 6 "F %f -F + +F 3F   "  %D ??4== 34 '')DIBIIfc*+DI"))L#67DOt_%%$*<*<t / 0 0#?@A #?@A  "7=> RYY~s34$'*4 #0"4#;#;T__MD yyA~ AHH OOT557 8$D "& 99> 1 2 99#$#t[^^(=(=(I $''' :d#yy ; +/? ?.02DO (,< >D ==?DJK qy"n **[q*1K&&(Iy)Ii.>?.>9n.>?@JTBJJv":BJJ| O OI +BJJ} JJ p  6"DOORZZ 5M =)2::m+D FG@sE4)__doc__rrr{rjrr4utilsrrr$r:r>rBrOr]rrrrrsV  Wt(VQh+\"jZM`K\Gr