6pjl6 SSKrSSKrSSKrSSKrSSKJr SSKJrJrJ r SSK J r J r J r SSKJr SSKJr \"5rSrSrS \4S jrS \4S jrS \4S jrS \S \4SjrS\S \4SjrSrSS\SS4SjrS\ \\\\ 4S \4SjrS S\ \\\\ 4S\ \S\ \S\ \ S \!4 Sjjr"S S\ \\4S\S\ \S\ \ S \!4 Sjjr#S!S\ \\\\ 4S\ \S\S\ \S\ \ S \!4 Sjjr$S\S \ 4Sjr%g)"N)Path)Errorcopy2copystat)BinaryIOOptionalUnion)urlparse) get_loggerc$[R"U5nURnURnUcUbgUR(d/O URnU(aUSS;aUR S5OU [ U5S:Xa USS;agg)zto decide if a func could receive dict inputs or not Args: func (class): the target function to be inspected Returns: bool: if func only has one arg ``input`` or ``inputs``, return True, else return False Fr)selfcls)inputinputsT)inspectgetfullargspecvarargsvarkwargspoplen)funcfull_args_specrrrs W/var/www/html/pdf-tiff/venv/lib/python3.13/site-packages/modelscope/utils/file_utils.pyfunc_receive_dict_inputsrs++D1N$$G  E O #((2n.A.ADT!W7DHHQKd 4yA~$q'%88 c[RR[R"5R SSS55nU$)z) default base dir: '~/.cache/modelscope' z.cache modelscopehub)ospath expanduserrhomejoinpath)default_cache_dirs r get_default_modelscope_cache_dirr')s9**499;+?+?,,'( rreturncz[RR[R"S[ 555$)z}Get modelscope cache dir, default location or setting with MODELSCOPE_CACHE Returns: str: the modelscope cache root. MODELSCOPE_CACHE)r!r"r#getenvr'rrget_modelscope_cache_dirr-2s0 77   $&F&HI KKrcR[RR[5S5$)zOGet model cache root path. Returns: str: the modelscope model cache root. modelsr!r"joinr-r,rrget_model_cache_rootr2=s 77<<02H ==rcR[RR[5S5$)zGet dataset raw file cache root path. if `MODELSCOPE_CACHE` is set, return `MODELSCOPE_CACHE/datasets`, else return `~/.cache/modelscope/hub/datasets` Returns: str: the modelscope dataset raw file cache root. datasetsr0r,rrget_dataset_cache_rootr5Fs 77<<02J ??r dataset_idcd[5nUcU$[RRXS-5$)zGet the dataset_id's path. dataset_cache_root/dataset_id. Args: dataset_id (str): The dataset id. Returns: str: The dataset_id's cache root path. /)r5r!r"r1)r6 dataset_roots rget_dataset_cache_dirr:Qs5*+L%-<(277<<3&4((rmodel_idcd[5nUcU$[RRXS-5$)zcache dir precedence: function parameter > environment > ~/.cache/modelscope/hub/model_id Args: model_id (str, optional): The model id. Returns: str: the model_id dir if model_id not None, otherwise cache root dir. r8)r2r!r"r1)r; root_paths rget_model_cache_dirr>`s4%&I (9#bggllc>/##rct[US5nUR5nSSS5 U$!,(df  W$=f)Nr)openread)r"ftexts r read_filerEos/ dCAvvx  K  Ks( 7Fc [R"U5nUb U"X5nO [5n[R"XS9 /n UGH\n X;aM [RR X 5n [RR X5n [RR U 5(a[R"U 5n U(a'[R"X5 [XU(+S9 M[RRU 5(d U(aM[RRU 5(a[U U UUUUS9 GMU"X5 GM[RRU 5(a[U U UUUUS9 GMTU"X5 GM_ [X5 U (a [U 5eU$![a*nU RURS5 SnAGMSnAf[ a(nU R#X[%U545 SnAGMSnAff=f![ a4n['USS5cU R#X[%U545 SnANSnAff=f)z7copy from py37 shutil. add the parameter dirs_exist_ok.N)exist_ok)follow_symlinks) dirs_exist_okrwinerror)r!listdirsetmakedirsr"r1islinkreadlinksymlinkrexistsisdir copytree_py37rextendrOSErrorappendstrgetattr)srcdstsymlinksignore copy_functionignore_dangling_symlinksrInames ignored_nameserrorsnamesrcnamedstnamelinktoerrwhys rrSrSvs JJsOE s*  KK, F  '',,s)'',,s)( 8ww~~g&&W-JJv/W8|L77>>&116N ww}}W--%##$")*7 9&g7w''!"/ 1g/O\0 Fm J ' MM#((1+ & & 8 MM7SX6 7 7 8 0 3 D ) 1 MM3SX. /0s[A&G7+G$2GG$2GG' H- H*G55 H*H%%H*- I+7*I&&I+file_path_or_objc[U[[45(a%[U5nUR5R$[U[ 5(a [ U5$[U[R5(aSUR5nURS[R5 UR5nURU5 U$[S5e)NrzCUnsupported type: must be string, Path, bytes, or io.BufferedIOBase) isinstancerWrstatst_sizebytesrioBufferedIOBasetellseekr!SEEK_END TypeError)rh file_pathcurrent_positionsizes r get_file_sizerws"S$K00)* ~~''' $e , ,#$$ $b&7&7 8 8+002a-$$&./  Q  rbuffer_size_mb tqdm_desc disable_tqdmc SSKJn [U5nUS:aSSnSn[U[[ 45(a.[U[ 5(aUO [ U5nUR nSUS3nUS-S-n[R"5n U"USS S S UUS 9n [U[[ 45(aSn [US 5n U RU5=n (aSU RU 5 U [U 5- n U R[U 55 U RU5=n (aMSS S S 5 U R5n X:wa[RSUSU SU35 U nGOz[U[ 5(aHU RU5 U R5n U R[U55 [U5nGO[U["R$5(aSn UR'S[(R*5 URU5=n (aSU RU 5 U [U 5- n U R[U 55 URU5=n (aMSU R5n UR'S[(R*5 X:wa[RSUSU SU35 U nOU R-5 [/S5eU R-5 UU US.$!,(df  GN=f)aaCompute SHA256 hash for a file path, bytes, or file-like object. Args: file_path_or_obj: File path, bytes, or file-like object. buffer_size_mb: Read buffer size in MB. Default 16MB. tqdm_desc: Progress bar description. disable_tqdm: Whether to disable progress bar. Returns: dict with keys: file_path_or_obj, file_hash, file_size. rtqdmi@Fz Large Filez[Validating Hash for ]TB)totalinitial unit_scale dynamic_ncolsunitdescdisablerbN4File size changed during hash computation: declared  bytes, actually hashed % bytes. File may have been modified: z5Input must be str, Path, bytes or a io.BufferedIOBaserh file_hash file_size) tqdm.autor}rwrjrWrrbhashlibsha256rArBupdater hexdigestloggerwarningrmrnrorqr!SEEK_SETclose ValueError)rhrxryrzr} declared_sizerbr" buffer_sizerprogress bytes_hashedrC byte_chunkrs r get_file_hashrs"!"23M))  &d 4 4'1 $((((#-12B-C 99D+D63  4'$.K I  H"S$K00 "D )Q !{ 33*3  ,J/ J0!"{ 33*3* '')  ( NN)?*B<.Q00@/AC D! $e , ,)*'') ,-.() $b&7&7 8 8 a-,11+>>j>   Z ( C O +L OOC O ,-11+>>j>'') a-  ( NN)?*B<.Q00@/AC D!   CE E NN- U* )s =A*K88 Lrtc ^^^^SSKnSSKnSSKJn [ T5m[ R RT5nUS-S-mURSS9mS/mUUUU4SjnURUSS 9n U R5 [R"5n Sn U"UUUSS SSS 9n TR5n U cO;U RU 5 U [U 5- n U R[U 55 MOU R!5 U R#5 TSbTSeX:wa[$R'S US U ST35 TU R)5U S.$)aCompute SHA256 with async I/O double-buffering for high-latency storage. Uses a producer-consumer pattern: a background thread reads file chunks into a bounded queue while the main thread computes the hash. This overlaps I/O latency with CPU computation. rNr|r)maxsizec4>[TS5nURT5nU(dOTRU5 M,SSS5 TRS5 g!,(df  N =f![anUTS'SnAN7SnAff=f!TRS5 f=f)Nrr)rArBput Exception)rCchunker chunk_queuert read_errors r _producer'_get_file_hash_async.._producerFs "i&!FF;/E OOE* ' OOD !'&  JqM  OOD !sK A(.AA( A%!A($B%A(( B2A<7B<BBBT)targetdaemonr)rrrrrr unit_divisorrrrr)queue threadingrr}rWr!r"getsizeQueueThreadstartrrgetrrr1rrrr)rtrxryrzrrr}rr reader_threadrrrrrrrs` @@@r_get_file_hash_asyncr.ssIIGGOOI.M 4'$.K++a+(KJ " "$$Id$CM IL  H ! = E " E #  NN!} m$%&>|nM,,5; 8 9 &((*! rasync_threshold_mbc[U[[45(a?[RR [U55nXRS-S-:a [ UUUUS9$[UUUUS9$)aCompute SHA256 hash with automatic optimization for large files. For file paths larger than async_threshold_mb, uses async double-buffered I/O to overlap disk reads with hash computation. For smaller files or non-path inputs (bytes, BinaryIO), falls back to synchronous hashing. Args: file_path_or_obj: File path, bytes, or file-like object. buffer_size_mb: Read buffer size in MB. Default 16MB for NAS optimization. async_threshold_mb: File size threshold for async mode. Default 32MB. tqdm_desc: Progress bar description. disable_tqdm: Whether to disable progress bar. Returns: dict with keys: file_path_or_obj, file_hash, file_size. r)rxryrz)rjrWrr!r"rrr)rhrxrryrzrs rcompute_file_hashr}sr0"S$K00GGOOC(8$9: T1D8 8' -#)   %!  rurl_or_filenamec[U5RS:H=(a$ [RR U5(+$)z- Check if a given string is a relative path. )r schemer!r"isabs)rs ris_relative_pathrs:  2 & M.0ggmmO.L*LMr) [Calculating]T)r rT)&rrrnr!pathlibrshutilrrrtypingrrr urllib.parser modelscope.utils.loggerr rrr'rWr-r2r5r:r>rErSrmintrwbooldictrrrrr,rrrs )),,!. 0K#K>c>@@ (c (c ( ## ## #! %+0 % FR E#tUH*D$E # (%'.#' ZCuh67ZSMZ}Z4. Z  Z~.#' LS$YLL}L4. L  Lb%' .#' (Cuh67(SM((} ( 4. (  (VMcMdMr