9iY/ SSKrSSKrSSKJrJr SSKJrJrJrJ r SSK r SSK J s J s Js Jr SSKJr SSKJrJr SSKJr SSKJr SSKJrJrJr SS KJr SS K J!r! SS K"J#r# SS K$J%r% SS K&J'r' /SQr(\%"\)5r*\"SS55r+"SS5r,S\ \\-S4S\.\S\-4Sjr/S\!S\0\\-\\144Sjr2S\+S\ \\-S4S\.\S\3\1\44Sjr4g)N) dataclassfield)AnyCallableOptionalUnion)get_default_numa_options)eventsmetrics) WorkerSpec)LocalElasticAgent)DefaultLogsSpecs LogsSpecsSignalException)ChildFailedError)RendezvousParameters)parse_rendezvous_endpoint) get_logger) NumaOptions) LaunchConfigelastic_launch launch_agentcL\rSrSr%Sr\\S'\\S'\\S'Sr\\ \S'Sr \ \S 'S r \ \S 'Sr \ \S 'S r\ \S'\"\S9r\\ \4\S'Sr\\S'Sr\\S'Sr\\S'Sr\ \S'Sr\\ \S'\"\S9r\\ \ 4\S'Sr\\ \S'Sr\ \S'Sr\\\S'SrS rg)!r%aw Creates a rendezvous config. Args: min_nodes: Minimum amount of nodes that the user function will be launched on. Elastic agent ensures that the user function start only when the min_nodes amount enters the rendezvous. max_nodes: Maximum amount of nodes that the user function will be launched on. nproc_per_node: On each node the elastic agent will launch this amount of workers that will execute user defined function. rdzv_backend: rdzv_backend to use in the rendezvous (zeus-adapter, etcd). rdzv_endpoint: The endpoint of the rdzv sync. storage. rdzv_configs: Key, value pair that specifies rendezvous specific configuration. rdzv_timeout: Legacy argument that specifies timeout for the rendezvous. It is going to be removed in future versions, see the note below. The default timeout is 900 seconds. run_id: The unique run id of the job (if not passed a unique one will be deduced from run environment - flow workflow id in flow - or auto generated). role: User defined role of the worker (defaults to "trainer"). max_restarts: The maximum amount of restarts that elastic agent will conduct on workers before failure. monitor_interval: The interval in seconds that is used by the elastic_agent as a period of monitoring workers. start_method: The method is used by the elastic agent to start the workers (spawn, fork, forkserver). metrics_cfg: configuration to initialize metrics. local_addr: address of the local node if any. If not set, a lookup on the local machine's FQDN will be performed. local_ranks_filter: ranks for which to show logs in console. If not set, show from all. event_log_handler: name of the event logging handler as registered in `elastic/events/handlers.py `_. .. note:: `rdzv_timeout` is a legacy argument that will be removed in future. Set the timeout via `rdzv_configs['timeout']` min_nodes max_nodesnproc_per_nodeN logs_specsrun_id default_rolerole rdzv_endpointetcd rdzv_backend)default_factory rdzv_configs rdzv_timeout max_restartsg?monitor_intervalspawn start_methodlog_line_prefix_template metrics_cfg local_addrnullevent_log_handler numa_optionscSnURS:waURURS'OSUR;aXRS'URc[5UlURc[ R R5(a][ R R5UR:Xa0[5Ul[RSUR5 gggg)Nir(timeoutzUsing default numa options = %r) r)r'rrr4torchcuda is_available device_countrr loggerinfo)selfdefault_timeouts ^/var/www/html/land-doc-ocr/venv/lib/python3.13/site-packages/torch/distributed/launcher/api.py __post_init__LaunchConfig.__post_init__cs    "+/+<+L$sCx.>L#L#!e!L#.2hsm2"'"=Kc3h= $J $#s#*.L(;'.NrBrc<\rSrSrSrS\S\\\S44Sjr Sr Sr g) rxa Launches an torchelastic agent on the container that invoked the entrypoint. 1. Pass the ``entrypoint`` arguments as non ``kwargs`` (e.g. no named parameters)/ ``entrypoint`` can be a function or a command. 2. The return value is a map of each worker's output mapped by their respective global rank. Usage :: def worker_fn(foo): # ... def main(): # entrypoint is a function. outputs = elastic_launch(LaunchConfig, worker_fn)(foo) # return rank 0's output return outputs[0] # entrypoint is a command and ``script.py`` is the python module. outputs = elastic_launch(LaunchConfig, "script.py")(args) outputs = elastic_launch(LaunchConfig, "python")("script.py") config entrypointNcXlX lgN_config _entrypoint)r=rQrRs r?__init__elastic_launch.__init__s  %rBcV[URUR[U55$rT)rrVrWlist)r=argss r?__call__elastic_launch.__call__sDLL$*:*:DJGGrBrU) rCrDrErFrGrrrrJrXr]rMrNrBr?rrxs04&&(C-.&HrBrrRr\returnc[U[5(a UR$[U[5(a)U[R :Xa[ SU5S5$U$g)aRetrieve entrypoint name with the rule: 1. If entrypoint is a function, use ``entrypoint.__qualname__``. 2. If entrypoint is a string, check its value: 2.1 if entrypoint equals to ``sys.executable`` (like "python"), use the first element from ``args`` which does not start with hifen letter (for example, "-u" will be skipped). 2.2 otherwise, use ``entrypoint`` value. 3. Otherwise, return empty string. c3:# UHoSS:wdM Uv M g7f)r-NrN).0args r? '_get_entrypoint_name..s>A# s  r) isinstancerrCrJsys executablenext)rRr\s r?_get_entrypoint_namerksR*h''""" J $ $  '>>C C rBrdzv_parameterscURS:wagURnUR5nU(d [S5e[ USS9up#US:Xa[SUS35eX#4$)Nstatic)NNzKEndpoint is missing in endpoint. Try to add --master-addr and --master-portr() default_portzport is missing in endpoint: z. Try to specify --master-port)backendendpointstrip ValueErrorr)rlrq master_addr master_ports r?_get_addr_and_portrvs~(*''H~~H  Y   9PRSKb+H:5S T    %%rBrQcUR(dD[[R"5R5n[ R SU5 X0l[X5n[ RSUURURURURURURURURUR UR"R$UR&UR(UR*S.5 [-S URURURURURUR.S.URD6n[1U5upg[3UR4URU[7U5[8R:"U5URUR UUUR.UR(UR*S9 n[=UUR"UR>UR@S9n Sn [BRD"[BRF"UR&55 U RI5n [JRL"U RO5UR(5 U RQ5(a[SUU RTS9eU RVU (aURXR[5 $$![Ra e[\a3 S n [JRL"U R_5UR(5 e[`a1 [JRL"U R_5UR(5 ef=f!U (aURXR[5 ff=f) Nz3config has no run_id, generated a random run_id: %saWStarting elastic_operator with launch configs: entrypoint : %(entrypoint)s min_nodes : %(min_nodes)s max_nodes : %(max_nodes)s nproc_per_node : %(nproc_per_node)s run_id : %(run_id)s rdzv_backend : %(rdzv_backend)s rdzv_endpoint : %(rdzv_endpoint)s rdzv_configs : %(rdzv_configs)s max_restarts : %(max_restarts)s monitor_interval : %(monitor_interval)s log_dir : %(log_dir)s metrics_cfg : %(metrics_cfg)s event_log_handler : %(event_log_handler)s numa_options : %(numa_options)s )rRrrrr r%r#r'r+r,log_dirr0r3r4)rprqr rrr1) r"local_world_sizerRr\ rdzv_handlerr+r,rtrur1r3r4)specrr.r/T)namefailuresFrN)1r rJuuiduuid4rHr;warningrkr<rrrr%r#r'r+r,r root_log_dirr0r3r4rr1rvr r"tuple rdzv_registryget_rendezvous_handlerr r.r/r initialize_metrics MetricsConfigrunr recordget_event_succeeded is_failedrr} return_valuesrzshutdownrget_event_failed Exception) rQrRr\r entrypoint_namerlrtrur{agent shutdown_rdzvresults r?rrs ==TZZ\%%&LfU *:rs (11 EE:5AX NEP>* = H  ONON ONd$H$HNhT)*26s),&)& 8C=(3- '(&&q) q)hT)*q) s)q) #s(^ q)rB