9iUJ"SrSSKrSSKrSSKrSSKrSSKrSSKrSSKrSSK J r J r SSK J r SrSrSrSrSS jrS r"S S 5rS rSSjrSrSrSrSSjrSrSrSSjrSSjrSSjrSSjr SSjr!SSjr"Sr#g) z'Utilities for logging and serializationN)mpu print_rank_0)FP16_OptimizerrunscL[RRU[U5$N)ospathjoinSUMMARY_WRITER_DIR_NAME)namebases `/var/www/html/land-doc-ocr/venv/lib/python3.13/site-packages/modelscope/models/nlp/mglm/utils.py get_log_dirrs 77<<5t <<czS/n[R"USS9nURS5R5SnU$)Nz hostname -ITshellutf-8r) subprocess check_outputdecodesplit) hostname_cmdresult master_addrs r get_hostnamer"s=!?L  $ $\ >F--(..03K rc[RR5S:Xa[R"S/SS9n[ UR 55nXR:Xa/[R"S/SS9n[ UR 55n[RRU/5nO [RRS/5n[RRUS5 UR5nU$)Nrzshuf -n 1 -i 10000-65535Tr) torch distributedget_rankrrintstrip master_portcuda LongTensor broadcastitem)argsports rget_spare_portr+)s !!#q(&&(B'C-134::<  ## #**,F+G157Dtzz|$Dzz$$dV,zz$$aS) a( 99;D Krc U(aP[SSS9 [U5H7nSS[U5- -n[SRX4[ X55SS9 M9 Ub[ R RUS5n[US 5n[R"[U5USS 9 SSS5 UR(aURb[URS S 9n[R"U5nSSS5 [ R RUS 5n [U S 5n[R"WU5 SSS5 gggg!,(df  N=f!,(df  No=f!,(df  g=f)zPrint arguments.z arguments:Tflush.z {} {} {}Nz config.jsonw) sort_keysrencodingzconfig_gpt_large.json)printvarslenformatgetattrr r r openjsondump deepspeeddeepspeed_configload) r)verboselog_dirargdots json_fileoutputfiler>deepspeed_json_files rprint_and_save_argsrH:s* l$':C"s3x-(D ##Cwt/AB$ PGGLL-8 )S !V IId4j&D 9" >>d33?d++g>$#'99T? ?"$'',,w/F#H )3/6 *F30/ @>  ! !?>0/s$ E!E!,E2 E! E/2 Fc Sn[RR5nSnUn[U[5(a UR nUR HnUSHnUS- nURR5nURR5n URR5n USRXU[UR55- nUSRXU 5- nM M [USS9 g ) z+Print min, max, and norm of all parameters.rz6iteration, rank, index, model-parallel,min, max, norm paramsz{:7d}, {:4d}, {:4d}, {:2d}, z{:.6E}, {:.6E}, {:.6E} Tr-N)rr r! isinstancer optimizer param_groupsdataminmaxnormr8r"model_parallelr5) rM iterationindexrankstring optimizer_ param_groupparammin_max_rRs rprint_params_min_max_normr]Os E    % % 'D FFJ)^,,(( !..  *E QJE::>>#D::>>#D::??$D 4;;E,@,@(AC CF 077DI IF+/ &rcB\rSrSrSr"SS5rSrSrS SjrSr g ) TimersczGroup of timers.c:\rSrSrSrSrSrSrSrS Sjr Sr g ) Timers.TimerfzTimer.c`XlSUlSUl[R"5Ulg)NF)name_elapsed_started_time start_timeselfr s r__init__Timers.Timer.__init__is"JDM!DM"iikDOrcUR(aS5e[RR5 [R"5UlSUlg)zStart the timer.ztimer has already been startedTN)rhrr% synchronizerirjrls rstartTimers.Timer.startos9}} F&F F$ JJ " " $"iikDO DMrcUR(dS5e[RR5 U=R[ R "5UR - - slSUlg)zStop the timer.ztimer is not startedFN)rhrr%rprgrirjrqs rstopTimers.Timer.stopvsH== 8"8 8= JJ " " $ MMdiikDOO; >' " " Gd+ #rcB[RRUS5$)Nz!latest_checkpointed_iteration.txt)r r r )rs rget_checkpoint_tracker_filenamers 77<<(*M NNrcUUR5S.n[URUSS9n[U5 [R"X45 [ SR U55 g)N)rToptimizer_state_dictT)r successfully saved {}) state_dictrsaverrr5r8)r)rTrMzero_sdzero_checkpoint_names rsave_zero_checkpointrsW ) 4 4 6G/tyy)$O01 JJw- # * *+? @Arc Uc [U5nUR(aU(d [XX4US9 GO'[R"5S:XGa [ UR U5n [SR[RR5UU 55 SU0n UR(a URnUR5n U(a[0n UR5HupURX'M U R!5VVs0sHunnU U(dMUU_M n nnXS'UR"(d3U (d,UbUR5U S'UbUR5U S'UR$(d[&R("5U S '[*R&R-5U S '[R."5U S '[R0R/5U S '[R2"5R55U S '[7U 5 [R "X5 [SRU 55 U(a[RR95 [RR5S:Xa<[;UR 5n[=US5nUR?U5 SSS5 ggs snnf!,(df  g=f)Save a model checkpoint.N)tagrz"(-(;(;(=$%',zz'?'?'A#$+.+C+C,*,'( $O 4 JJr + +22?C D !!# !!#q(:499E "C (A GGCL) ()?B) (sK%KK%% K3c0nXS'UbUR5US'UR(d[R"5US'[RR 5US'[ R"5US'[ RR5US'[R"5R5US'URURXES 9 g) rrTNclient_lr_schedulerrrrrr) client_state)rrrrrrrrr%rrrrr)rTrrr)rrs rrrs B{O$0$;$;$= !   !'!2 YY002> % 3 3 5 $zz779 #&#;#;#=#H#H#J   $))S:rc:[U5n[RRU5(d[ SR U55 [RR U5(aP[RRU5n[RRU5up4[ S5 X4SS4$[ S5 USSS4$[USSS 9nUR5R5nUS :HnSSS5 UWWS4$!,(df  N=f) Nz-WARNING: could not find the metadata file {} z6Try to directly load the checkpoint from the directoryFTz< will not load any checkpoints and will start from randomrrrr3r) rr r isfilerr8isdirnormpathrr:readr#) load_pathrr load_dirrr metastringrs rget_checkpoint_iterationr/s6yA 77>>* + +DKK   77== # #77##I.DGGMM$/MH H J%- - !UE)) g 6!VVX^^%  ) 7 j'4 // 7 6s $D  Dcd[UR5upgpU (dgUR(aU(dURUUUR(+=(a U(+UR (+S9upUR (d%SU ;aUR U S5 [S5 U c&[R"5S:Xa [S5 U$GO![XgU5n [R"5S:Xa7[SR[RR5U 55 [R"U SS9n UR(a UR nUR U S S S 9upU (dU (a[S U S U 35 U(dXUR"(dGUR(d6U(d/UbUR U S5 UbUR U S5 UR"(dU(aSnOU SnU(dUR"(dUR&(d[(R*"U S5 [,R(R/U S5 [R0"U S5 [R2R1U S5 [R4"5R7U S5 [R"5S:Xa[SRU 55 U$![$a [SRU 55 GNDf=f![$a8 U SnGN:![$a! [SRU 55 SnGNdf=ff=f![$a [SRU 55 Nf=f)zLoad a model checkpoint.r)load_optimizer_statesload_lr_scheduler_statesrzLoad lr scheduler statezUnable to load checkpoint.z'global rank {} is loading checkpoint {}cpu) map_locationrF)strictz Missing keys z, unexpected keys rMrzUnable to load optimizer from checkpoint {}, exiting. Specify --no-load-optim or --finetune to prevent attempting to load the optimizer state.rT total_iterszbA metadata file exists but Unable to load iteration from checkpoint {}, starting from 0 iterationrrrrrzUnable to load random state from checkpoint {}, exiting. Specify --no-load-rng or --finetune to prevent attempting to load the random state.z successfully loaded {})rr?r=load_checkpoint no_load_optimno_load_lr_schedulerload_state_dictrrrr5rr8rr r!rfinetuneKeyError no_load_rngrsetstater set_state set_rng_stater%r set_states)rrMrr)rrrrrsuccessrr missing_keysunexpected_keysrTs rrrPs3'?tyy&I#H7  ~~l#33  &*&8&8"8"N=N)-)B)B%B 4D ((-Bb-H  ( (,A)B C 2 3  "))+q023J #.hWE  % % '1 , ;BB!!**,o? @ZZe < >>LLE(-(=(= xL)>)(% ?  ~-??PQ  t}}T5G5GP] 6(--bo>+ 00N1CD }}  ;I 4==1A1A 2 OOB12 3 II  > 2 3   #4 5 6 JJ $ $R(8%9 :  $ $ & 1 1"5I2J K !!#q( (//@A S 6$VO4 6 6  }-  EEKV'F)*   $ 2  0  2 2sO .LMBN$M?M NM&N<NNN$N/.N/cS[[U55;nUR5HupEU(a&URURnURnO%URnURURnU(a$SU;aUR 5R 5nURU5 M g)z Loads weights from src to dst via in place copy. src is a huggingface gpt2model, while dst is one of our models. dst2src=True loads parameters from our models into huggingface's. ^dst2src is still untested Conv1DweightN)rtyper _parametersrOt contiguouscopy_)srcdstdst2src conv_layernprOr?s r load_weightsr sSc^+J$$& ??1%**D66D66D??1%**D (a-668&&(D 4'rc[URURU5 [URURU5 gr)r c_fc dense_h_to_4hc_proj dense_4h_to_houroairs rload_mlprs.3,,g6S..8rc[URURU5 [URURU5 gr)r c_attnquery_key_valuer densers rload_attentionrs,S00':SYY0rc [URURU5 [URURU5 [ UR UR U5 [URURU5 gr) r ln_1input_layernormln_2post_attention_layernormrmlpr attentionattnrs rload_transformer_layerr sV3..8377A SWWcggw'3==#((G4rcURn[URURRU5 [URUR U5 [UR URU5 [URRURR5HupE[XEU5 M g)z Loads weights from `oai` to `our` via in place copy. `oai` is a huggingface gpt2model, while `our` is one of our models. dst2src=True loads parameters from our models into huggingface's. ^dst2src=True is still untested N) transformerr ln_ffinal_layernormwteword_embeddingswpeposition_embeddingsziplayershr )rrrtransformer_model our_layer oai_layers r move_weightsr/s"'')H)H"&&(;(;WE"&&(?(?I #COO$:$:COO##E* H  .q!t499;EU# ?  #((= !" 3 B (   " "  # #A &)  )  f./?@GGI JK :  q    $%56==? A B i!!*"6"="="?@A , $4!4 56r)TN)FF)NTFFF)F)$rr rrrir;numpyrr megatron_utilrrmegatron_util.fp16rr rrr+rHr]r_rrrrrrrrrr rrr r/rKrrrrOs.   +- ="4*(??D "!&" P , OB  ,1!&"'BJ;"0J"'"' bJ.9 1 5>&7r