`i%tSSKJr SSKJr SSKJr SSKrSSKrSSKrSSKJr S SjrS Sjr S Sjr S r g) )absolute_import)print_function)divisionN)py3nvmlc U(d-S[RR5;a [S5eS[RS'US:Xag[R "[ 5n[R"5 [R"5nS/U-n Sn US:XaUnSn UcS/U-n O S/U-n SX'[U5Hn X(dM[R "U 5n [R""U 5nS R%U 5S R%UR&S - 5-SR%UR(S - 5-nUR+U5 M US:a [-US9nOS/U-n[U5Hn X(dMX(dM[R "U 5n [R""U 5nU(a5UR(S - U:a"UR/SR%U 55 MUR0S-UR(- U:aSX'MUR/SR%U 55 M [R2"5 [5U 5S:Xa-[R"S[5 URW5 g[5U 5U:a[7U 5V Vs/sHun nU(dMU PM nn nSR9[;SUS U555nUR+SR%[5U 555 UR/SR%U55 U[RS'U$U (aSR%[5U 5U5S-n[R"U[5 URU5 [7U 5V Vs/sHun nU(dMU PM nn nSR9[;SU555nUR+SR%[5U 555 UR/SR%U55 U[RS'[5U 5$g ! Sn[R"U[5 URU5 g=f![a" UHn SX'M GN! [S 5e=ff=fs snn fs snn f)a Checks for gpu availability and sets CUDA_VISIBLE_DEVICES as such. Note that this function does not do anything to 'reserve' gpus, it only limits what GPUS your program can see by altering the CUDA_VISIBLE_DEVICES variable. Other programs can still come along and snatch your gpu. This function is more about preventing **you** from stealing someone else's GPU. If more than 1 GPU is requested but not all were available, then it will set the CUDA_VISIBLE_DEVICES variable to see all the available GPUs. A warning is generated in this case. If one or more GPUs were requested and none were available, a Warning will be raised. Before raising it, the CUDA_VISIBLE_DEVICES will be set to a blank string. This means the calling function can ignore this warning and proceed if it chooses to only use the CPU, and it should still be protected against putting processes on a busy GPU. You can call this function with num_gpus=0 to blank out the CUDA_VISIBLE_DEVICES environment variable. Parameters ---------- num_gpus : int How many gpus your job needs (optional). Can set to -1 to take all remaining available GPUs. gpu_select : iterable A single int or an iterable of ints indicating gpu numbers to search through. If None, will search through all gpus. gpu_fraction : float The fractional of a gpu memory that must be free for the script to see the gpu as free. Defaults to 1. Useful if someone has grabbed a tiny amount of memory on a gpu but isn't using it. max_procs : int Maximum number of processes allowed on a GPU (as well as memory restriction). env_set_ok : bool If false, will complain if CUDA_VISIBLE_DEVICES is already set. gpu_min_memory: int The minimal allowed graphics card memory amount in MiB. Returns ------- success : int Number of gpus 'grabbed' Raises ------ RuntimeWarning If couldn't connect with NVIDIA drivers. If 1 or more gpus were requested and none were available. Will NOT raise a RuntimeWarning for mismatch in GPU availability if `num_gpus` is -1. ValueError If the gpu_select option was not understood (can fix by leaving this field blank, providing an int or an iterable of ints). CUDA_VISIBLE_DEVICESzlTrying to set CUDA_VISIBLE_DEVICES but it has been set already. Specify env_set_ok=True to avoid this error.rz_Couldn't connect to nvml drivers. Check they are installed correctly. Proceeding on cpu only...FTNz6}MB izTotal Mem: {:>6}MB) max_procsz*GPU {} does not have enough RAM. Skipping. z%GPU {} has processes on it. Skipping.z'Could not find enough GPUs for your job,c38# UHn[U5v M g7fNstr.0ss L/var/www/html/ai-image-ml/venv/lib/python3.13/site-packages/py3nvml/utils.py grab_gpus..s$O5NSVV5Nz{} Gpus found freezUsing {}zOnly {} GPUs found but {} z+requested. Allocating these and continuing.c38# UHn[U5v M g7frrrs rrrs$D^SVV^r)osenvironkeys ValueErrorlogging getLogger__name__rnvmlInitwarningswarnRuntimeWarningwarningnvmlDeviceGetCount TypeErrorrangenvmlDeviceGetHandleByIndexnvmlDeviceGetMemoryInfoformatusedtotaldebug get_free_gpusinfofree nvmlShutdownsum enumeratejoinlist)num_gpus gpu_select gpu_fractionr env_set_okgpu_min_memoryloggerstr_ numDevicesgpu_freewarn_about_fewer_gpus gpu_checkihandler0procs_okxavailable_gpususe_gpusrs r grab_gpusrH s3v 0BJJOO4EE"# # *,BJJ%&1}  x (F++-Jwz!H 2~ %FZ' Gj(  6$(I !: | 33A6..v6!!!$$++DIIy,ABC#**4::y+ABC  TA~ 958j(:  <A MM!^ , NN1 ,5h,?E,?DAq1a,?NExx$D^$D DEH LL-44S]C D KK ))(3 419BJJ- .x= #q4 dN+t" 6 6#A#'IL$ 6 "566  6xFFsB*Q7R>SS:S S0R R? R.. R;;R?cRU"U5nU$![Ra UnU$f=fr)rNVMLError_NotSupported)fhdefaultvs r try_get_inforOs8 aD H  ) )  Hs  &&c[R"[5n[R"5 [R"5nS/U-n[U5HKn[R"U5n[[RUS/5n[U5U::dMGSXE'MM [R"5 U$! Sn[ R "U[5 URU5 /s$=f! M=f)a  Checks the number of processes running on your GPUs. Parameters ---------- max_procs : int Maximum number of procs allowed to run on a gpu for it to be considered 'available' Returns ------- availabilities : list(bool) List of length N for an N-gpu system. The nth value will be true, if the nth gpu had at most max_procs processes running on it. Set to 0 to look for gpus with no procs on it. Note ---- If function can't query the driver will return an empty list rather than raise an Exception. ECouldn't connect to nvml drivers. Check they are installed correctly.F somethingTrrr rr!r"r#r$r%r&r(r)rO$nvmlDeviceGetComputeRunningProcesseslenr2)r r<r=r7r?rBrLprocss rr/r/s.  x (F))+HwxH 8_ 2215AWII1)], u: "HK  O)Z dN+t   sB=C4=2C14C9c[R"[5n[R"5 [R"5nS/U-n[U5HCn[R"U5n[[RUS/5n[U5X4'ME [R"5 U$! Sn[ R "U[5 URU5 /s$=f! M=f)aFGets the number of processes running on each gpu Returns ------- num_procs : list(int) Number of processes running on each gpu Note ---- If function can't query the driver will return an empty list rather than raise an Exception. Note ---- If function can't get the info from the gpu will return -1 in that gpu's place rQr rRrS)r<r=r7 gpu_procsrBrLrVs r get_num_procsrYs$  x (F))+HX I 8_ 2215AWII1)],5z   %Z dN+t   sB5C,52C),C1)Ngffffff?r FN)zN/A)r) __future__rrrrrr"rrHrOr/rYrr^s;&% IK15m!` .b'r]