9iO: SSKrSSKrSSKrSSKrSSKrSSKrSSKrSSKrSSKJ r J r J r J r SSK rSSKrSSKJr SSKJr SSKJr S\4SjrS\S\4SjrS rS/S \S \SS4S jjrS \SS4S jrS \SS4SjrS0S \S\ \SS4Sjjr S0S\ \\44Sjjr!Sr"Sr#Sr$Sr%Sr&S0Sjr'S0Sjr(Sr)Sr*Sr+\RX"5S5r-Sr.Sr/S0Sjr0S\RbRdS\4S jr3S!\ S"\Rh4S#\S$\S\Rh4S%jr5S&\6\S'\S(\S)\7\S\8\7\\Rh44 S*jr9S1S+\RbRtS,\RvS-\S\RbRt4S.jjr# UHnTRUT45S:gv M g7f)rN) connect_ex).0iprss r _is_free_port..#s#?3R1<<T +q03s#&)r gethostbyname_ex gethostnameappendrrall)ripsrs` @r _is_free_portr'sa  ! !&"4"4"6 7 ;CJJ{ v~~v'9'9 :a?3?? ; : :s 3B B!c 4[US5(aUR"S0UD6nU$[R"[R 5[R"S5:a[R"U40UD6nU$[ S[R S35 U$)Ncompilez 2.0.0.devzDCompiling model needs torch version > 2.0.0, your torch version is: z , origin model will be returned.)hasattrr)rparsetorch __version__print)modelcompile_optionss r compile_modelr2&sui   00 L u(( )W]];-G G e77 L  &&+&7&7%88X Z  Lrlauncherbackendc [R"SS9c[R"S5 US:Xa [U40UD6 gUS:Xa [ U40UD6 gUS:Xa [ U40UD6 g[ SU35e)NT) allow_nonespawnpytorchmpislurmzInvalid launcher type: )mpget_start_methodset_start_method_init_dist_pytorch_init_dist_mpi_init_dist_slurm ValueError)r3r4kwargss r init_distrC4ss d+3 G$97-f- U w)&) W +F+28*=>>rc [[RS5n[RR U5 [ R"SSU0UD6 g)N LOCAL_RANKr4r*)intosenvironr-cuda set_devicedistinit_process_groupr4rB local_ranks rr>r>As=RZZ -.J JJ*%6G6v6rc [[RS5n[RR U5 S[R;aS[RS'S[R;a [ S5e[RS[RS'[RS[RS '[R"S S U0UD6 g) NOMPI_COMM_WORLD_LOCAL_RANK MASTER_PORT29500 MASTER_ADDRz/The environment variable MASTER_ADDR is not setOMPI_COMM_WORLD_SIZE WORLD_SIZEOMPI_COMM_WORLD_RANKRANKr4r*) rFrGrHr-rIrJKeyErrorrKrLrMs rr?r?HsRZZ <=>J JJ*%BJJ&$+ =!BJJ&HII!zz*@ABJJ|$:;BJJv6G6v6rch[[RS5n[[RS5n[RSn[RR 5n[RR X%-5 [R"SUS35nUb[U5[RS'O]S[R;aOH[S5(aS [RS'O$[[55[RS'S [R;aU[RS '[U5[RS '[X%-5[RS '[U5[RS '[R"US9 g)a|Initialize slurm distributed training environment. If argument ``port`` is not specified, then the master port will be system environment variable ``MASTER_PORT``. If ``MASTER_PORT`` is not in system environment variable, then a default port ``29500`` will be used. Args: backend (str): Backend of torch.distributed. port (int, optional): Master port. Defaults to None. SLURM_PROCID SLURM_NTASKSSLURM_NODELISTzscontrol show hostname z | head -n1NrQiBJJ} %BJJ&$( =!"6{BJJ|"7#56BJJ|WBJJvG,rc[5(aZSSKJn Uc"U"5(aSSKJn UR 5n[ R"U5n[ R"U5nX44$SnSnX44$)zGet dist info of a specified group Args: group: The parallel group, default None, for the global group Returns: A tuple of the current rank and world_size of the group r)is_megatron_initialized)mpur ) is_distmodelscope.utils.megatron_utilsrh megatron_utilriget_data_parallel_grouprKget_rankget_world_size)grouprhrirank world_sizes r get_dist_infors|skyyK =466 )//1E}}U#((/     rcT[[RRSS55$)NrEr)rFrGrHgetr*rrget_local_rankrvs rzz~~lA. //rc[R"5(dg[R"5(dg[R"5$)Nr)rK is_availableis_initializedrnr*rrrnrns2          ==?rc[R"5(dg[R"5(dg[R"5$)Nr )rKrxryror*rrroros5              rc[R"5(dg[R"5(dg[R"5nUS:Xag[R"5 g)z^ Helper function to synchronize (barrier) among all processes when using distributed training Nr )rKrxryrobarrier)rrs r synchronizer}sI         $$&JQLLNrcd[R"5=(a [R"5$N)rKrxryr*rrrjrjs     84#6#6#88rcV[5(a[R"U5S:H$S$)NrT)rjrKrnrps r is_masterrs!(/ 4== 1 $;t;rc0^S[S[4U4SjjnU$)Nfuncr cJ>^[R"T5UU4Sj5nU$)Nc6>[T5(aT"U0UD6$gr)r)argsrBrrps rwrapper.master_only..decorate..wrappers#T,V,, r) functoolswraps)rrrps` rdecoratemaster_only..decorates%   -  -r)r)rprs` r master_onlyrsxH Orc[5(d[R"5$Sn[5(a[R"5n[R "5 [ US5nU$)zRMake sure each rank has the same temporary directory on the distributed mode. Nr)rjtempfilemkdtemprrKr| broadcast)tmpdirs r make_tmp_dirrsN 99!! F{{!!#LLN vq !F Mrc[R"5n[R"S/SS9nX!:Xa`[R"[ [ R "U55[RSS9n[R"URSS9n[R"5 [R"X15 X!:wa4[R"UR54S[RSS9n[R"5 [R"WU5 [ R"UR5R5R!55$)z Broadcasts the inputs to all ranks. Arguments: inputs : Any objects that can be serialized by pickle. src (int): Source rank. Returns: Each rank returns the same value as src. rrIdevicedtyper)rKrnr-tensor bytearraypickledumpsuint8shaper|rfullitemloadscpunumpytobytes)inputssrcrq shape_tensor inputs_tensors rrrs ==?D<<F3L { fll6* +5;;vO ||M$7$7G LLNNN<% { L$5$5$7#:#$).*02  LLNNN=#& << ))+113;;= >>rcUbqUS:ak[R"U5 [RRU5 [R"U5 [R R U5 g[SU35e)Nrz0Random seed should be positive, current seed is )randomseednpr- manual_seedrImanual_seed_allrA)rs rset_random_seedrsc DAI D t $ ""4(>tf EG Grc[R"5S:Xa[R"SS9$[RR$)z^ Return a process group based on gloo backend, containing all the ranks The result is cached. ncclgloor])rK get_backend new_grouprpWORLDr*rr_get_global_gloo_grouprs4  V#~~f--zzrc[R"U5nUS;de[R"US:XaSOS5n[R "U5n[ U5S:a:[RSR[5[ U5S- U55 [RRU5n[R"U5RUS9nU$)N)rrrrrIi@z;Rank {} trying to all-gather {:.2f} GB of data on device {}r)rKrr-rrrlenloggerwarningformatrn ByteStorage from_buffer ByteTensorto)datarpr4rbufferstoragers r_serialize_to_tensorrsu%G & && & \\7f#4%& AF \\$ F 6{W I F8:v;'*F 4 5++F3G   g & ) ) ) 8F Mrc [R"US9nUS:dS5e[R"UR 5/[R UR S9n[U5Vs/sH2n[R"S/[R UR S9PM4 nn[R"XSUS9 UVs/sHn[UR55PM nn[U5nX7:waG[R"Xs- 4[RUR S9n[R"X4SS9nXP4$s snfs snf)zj Returns: list[int]: size of the tensor, on each rank Tensor: padded tensor that has the max size rr zBcomm.gather/all_gather must be called from ranks within the group!rrdim)rKror-rnumelint64rrangezeros all_gatherrFrmaxrcat) rrprr local_size_ size_listsizemax_sizepaddings r_pad_to_largest_tensorr#s $$51JaLKLv||~.$)KK%+]]4J z""A  QCu{{6==A" OOI7.78idTYY[!iI89~H++x47$)KK%+]]4F,!4  ! 9s ,9D?#Ec H[5S:XaU/$Uc [5n[R"U5S:XaU/$[X5n[ X!5up2[ U5nUVs/sH2n[ R"U4[ RURS9PM4 nn[R"XbUS9 /n[X65HYupUR5R5R5SUn UR[ R""U 55 M[ U$s snf)a Run all_gather on arbitrary picklable data (not necessarily tensors). Args: data: any picklable object group: a torch process group. By default, will use a group which contains all ranks on gloo backend. Returns: list[data]: list of data gathered from each rank r Nrr)rorrKrrrr-emptyrrrziprrrr$rr) rrprrrr tensor_list data_listrrs rrrCs1v  }&( 5!Q&v !$ .F.v=I9~H A  XL FMMJ OOKu5II3 ##%--/6f-.4 s#9Dr0cf[SUR555S1- n[U5S:*$)Nc3L# UHn[UR5v M g7fr)rar)rps rr $is_on_same_device..is?,>qS]],>s"$rr )set parametersr)r0 device_sets ris_on_same_devicerhs1?E,<,<,>??5'IJ z?a r forward_fn. chunk_size chunk_dimc^^^ [U5S:d US35e[[R"T5R5nU[U5:wa[ SUS[U5S35eUS:aUSR TnUH4nUR TU:wdM[ SUSUR T35e USR TU-S:wa![ SUSR TS U35eUSR TU-m [ UU 4S jU55n[ U4S j[U655n[R"UTS 9$T"U6$) Nrz" has to be a tuple/list of tensorszforward_chunk_fn expects z arguments, but only z input tensors are givenz/All input tenors have to be of the same shape: z, found shape zThe dimension to be chunked z( has to be a multiple of the chunk size c3D># UHnURTTS9v M g7f)rN)chunk)r input_tensorr num_chunkss rr ,apply_chunking_to_forward..s(%/ -    zy  9 -s c3.># UH nT"U6v M g7frr*)rinput_tensors_chunkrs rr rs C'A# + ,'Asr) rinspect signaturerrArtuplerr-r) rrr input_tensorsnum_args_in_forward_chunk_fn tensor_shaperinput_tensors_chunks output_chunksrs ` ` @rapply_chunking_to_forwardrms }  K&'IJK $'*%00$2 #s='99'(D'EEZ[^_l[mZno  ! !A~$Q'--i8 )L!!), < El^T##/#5#5i#@"ACDD*   ! !) ,z 9Q >.}Q/?/E/Ei/P.QR"|%& &#1%++I6*D  %%/ -%/ /C'*,@'ACC yyI66 } %%rheadsn_heads head_sizealready_pruned_headscZ^[R"X5n[U5U- nUH mT[U4SjU55- mSUT'M" UR S5R 5R S5n[R"[U55UR5nX4$)Nc36># UHoT:aSOSv M g7f)r rNr*)rhheads rr 3find_pruneable_heads_and_indices..sM8L14x!Q.8Lsrrr ) r-onesrsumview contiguouseqarangerlong)rrrrmaskindexrs @r find_pruneable_heads_and_indicesr s ::g )D    EcM8LMMMT  99R= # # % ( ( +D#ll3t95d;@@BE <rlayerrrcURURR5nURRX!5R 5R 5nUR bZUS:Xa)UR R 5R 5nO+UR UR 5R 5n[URR55n[U5XR'[RRUSUSUR SLS9RURR5nSURl URRUR55 SURl UR bKSUR l UR RWR55 SUR l U$)Nr r)biasFT)rweightr index_selectdetachcloner listrrr-nnLinear requires_gradcopy_r)r rrWbnew_size new_layers rprune_linear_layerrsq HHU\\(( )E !!#-446<<>A zz !8 !!#))+A 5!((*002AELL%%'(HJHM Xa[uzz( R ++,&+I" 1<<>*%)I" zz', $Q\\^,'+ $ r)rr)r)=rrrGrrr r_rtypingrrrrrrr-torch.multiprocessingmultiprocessingr; packagingrrrKrarrFboolr'r2rCr>r?r@rsrvrnror}rjrrrrr lru_cacherrrrrModulerTensorrrrrr r LongTensorrr*rrr$s  22 "%@@@  ? ?c ? ?77$7 7C 7d 7$-c$-#$-$$-NsCx,0! 9<   ?BG     @"J UXX__  ,&ell*+,&,&,&  \\ ,&^Cy#&36!#h+0S5<<1G+H&#$ehhoo#..(-r