6pjKSrSSKrSSKJr SSKJr SSKJrJrJ r J r J r J r J r SSKJrJr SSKJrJr SSKJr SS KJrJrJr SS KJrJr SS KJr SS KJ r \ "5r!/S Qr"SS/r#\"SS9"SS55r$\"SS9"SS55r%\"SS9"SS55r&\"SS9"SS55r'S$S\ \ \(\4S\'4Sjjr)S\S\*4Sjr+S%S\S\(S \*4S!jjr,S"\S\(S\ \&4S#jr-g)&a Contains utilities to manage the ModelScope cache directory. :func:`scan_cache_dir` retains the legacy file-grain scan that is unique to this SDK; the modelscope_hub repo-grain :func:`scan_cache` and :func:`clear_cache` are re-exported here for forward compatibility. N) dataclass)Path)Dict FrozenSetListLiteralOptionalSetUnion) clear_cache scan_cache) CacheNotFoundCorruptedCacheException)ModelFileSystemCache)convert_readable_sizeformat_timesincetabulate)REPO_TYPE_DATASETREPO_TYPE_MODEL)get_modelscope_cache_dir) get_logger)CachedFileInfoCachedRevisionInfoCachedRepoInfoModelScopeCacheInfoscan_cache_dirr r z .DS_Storez ._____tempT)frozenc\rSrSr%Sr\\S'\\S'\\S'\\S'\\S'\ \S'\ \S '\ S \4S j5r \ S \4S j5r \ S \4S j5r Srg)r'aFrozen data structure holding information about a single cached file. Args: file_name (`str`): Name of the file. Example: `config.json`. file_path (`Path`): Path of the file in the `snapshots` directory. The file path is a symlink referring to a blob in the `blobs` folder. blob_path (`Path`): Path of the blob file. This is equivalent to `file_path.resolve()`. size_on_disk (`int`): Size of the blob file in bytes. blob_last_accessed (`float`): Timestamp of the last time the blob file has been accessed (from any revision). blob_last_modified (`float`): Timestamp of the last time the blob file has been modified/created. file_name file_pathfile_revision_hash blob_path size_on_diskblob_last_accessedblob_last_modifiedreturnc,[UR5$)z (property) Timestamp of the last time the blob file has been accessed (from any revision), returned as a human-readable string. Example: "2 weeks ago". )rr%selfs X/var/www/html/pdf-tiff/venv/lib/python3.13/site-packages/modelscope/hub/cache_manager.pyblob_last_accessed_str%CachedFileInfo.blob_last_accessed_strE  7 788c,[UR5$)z (property) Timestamp of the last time the blob file has been modified, returned as a human-readable string. Example: "2 weeks ago". )rr&r)s r+blob_last_modified_str%CachedFileInfo.blob_last_modified_strOr.r/c,[UR5$)zQ (property) Size of the blob file as a human-readable string. Example: "42.2K". rr$r)s r+size_on_disk_strCachedFileInfo.size_on_disk_strY%T%6%677r/N)__name__ __module__ __qualname____firstlineno____doc__str__annotations__rintfloatpropertyr,r1r5__static_attributes__r8r/r+rr's&NOO 9999998#88r/rc\rSrSr%Sr\\S'\\S'\\S'\ \ \S'\ \S'\ S\4S j5r \ S\4S j5r\ S\4S j5rS rg )rcaFrozen data structure holding information about a revision. Args: commit_hash (`str`): Hash of the revision (unique). Example: `"9338f7b671827df886678df2bdd7cc7b4f36dffd"`. snapshot_path (`Path`): Path to the revision directory in the `snapshots` folder. It contains the exact tree structure as the repo on the Hub. files: (`FrozenSet[CachedFileInfo]`): Set of [`~CachedFileInfo`] describing all files contained in the snapshot. size_on_disk (`int`): Sum of the blob file sizes that are symlink-ed by the revision. last_modified (`float`): Timestamp of the last time the revision has been created/modified. commit_hash snapshot_pathr$files last_modifiedr'c,[UR5$)z (property) Timestamp of the last time the revision has been modified, returned as a human-readable string. Example: "2 weeks ago". rrIr)s r+last_modified_str$CachedRevisionInfo.last_modified_str}  2 233r/c,[UR5$zV (property) Sum of the blob file sizes as a human-readable string. Example: "42.2K". r4r)s r+r5#CachedRevisionInfo.size_on_disk_strr7r/c,[UR5$)z3 (property) Total number of files in the revision. )lenrHr)s r+nb_filesCachedRevisionInfo.nb_filess 4::r/r8N)r9r:r;r<r=r>r?rr@rrrArBrLr5rTrCr8r/r+rrcs{" ^ $$ 43448#88#r/rc\rSrSr%Sr\\S'\\S'\\S'\\S'\\S'\ \ \S'\ \S '\ \S '\ S \4S j5r \ S \4S j5r\ S \4Sj5rSrg)ra>Frozen data structure holding information about a cached repository. Args: repo_id (`str`): Repo id of the repo on the Hub. Example: `"damo/bert-base-chinese"`. repo_type (`Literal["dataset", "model"]`): Type of the cached repo. repo_path (`Path`): Local path to the cached repo. size_on_disk (`int`): Sum of the blob file sizes in the cached repo. nb_files (`int`): Total number of blob files in the cached repo. revisions (`FrozenSet[CachedRevisionInfo]`): Set of [`~CachedRevisionInfo`] describing all revisions cached in the repo. last_accessed (`float`): Timestamp of the last time a blob file of the repo has been accessed. last_modified (`float`): Timestamp of the last time a blob file of the repo has been modified/created. repo_id repo_type repo_pathr$rT revisions last_accessedrIr'c,[UR5$)z (property) Last time a blob file of the repo has been accessed, returned as a human-readable string. Example: "2 weeks ago". )rr\r)s r+last_accessed_str CachedRepoInfo.last_accessed_strrNr/c,[UR5$)z (property) Last time a blob file of the repo has been modified, returned as a human-readable string. Example: "2 weeks ago". rKr)s r+rL CachedRepoInfo.last_modified_strrNr/c,[UR5$rPr4r)s r+r5CachedRepoInfo.size_on_disk_strr7r/r8N)r9r:r;r<r=r>r?rr@rrrArBr^rLr5rCr8r/r+rrs*LNOM+,, 434443448#88r/rcj\rSrSr%Sr\\S'\\\S'\ \ \S'\ S\ 4Sj5r S\ 4SjrS rg ) raFrozen data structure holding information about the entire cache-system. This data structure is returned by [`scan_cache_dir`] and is immutable. Args: size_on_disk (`int`): Sum of all valid repo sizes in the cache-system. repos (`FrozenSet[CachedRepoInfo]`): Set of [`~CachedRepoInfo`] describing all valid cached repos found on the cache-system while scanning. warnings (`List[CorruptedCacheException]`): List of [`~CorruptedCacheException`] that occurred while scanning the cache. Those exceptions are captured so that the scan can continue. Corrupted repos are skipped from the scan. r$reposwarningsr'c,[UR5$)zX (property) Sum of all valid repo sizes in the cache-system as a human-readable string. r4r)s r+r5$ModelScopeCacheInfo.size_on_disk_strs %T%6%677r/c S[S[S[[4Sjn/SQn[ UR SS9VVs/sH(n[ UR SS9H nU"X45PM M* nnn[UUS 9$s snnf) aGenerate a detailed table from the [`ModelScopeCacheInfo`] object. Returns a table with a row per repo and revision (thus multiple rows can appear for a single repo), with columns "repo_id", "repo_type", "revision", "size_on_disk", "nb_files", "last_modified", "local_path". Example: ```py >>> from modelscope.hub.cache_manager import scan_cache_dir >>> ms_cache_info = scan_cache_dir() ModelScopeCacheInfo(...) >>> print(ms_cache_info.export_as_table()) REPO ID REPO TYPE REVISION SIZE ON DISK NB FILES LAST_MODIFIED LOCAL PATH ---------------------- --------- ---------- ------------ -------- ------------- ------------------------------------------------------------- damo/bert-base-chinese model master 2.7M 5 1 week ago ~/.cache/modelscope/hub/models--damo--bert-base-chinese/... damo/structured-bert model master 8.8K 1 1 week ago ~/.cache/modelscope/hub/models--damo--structured-bert/... damo/t5-base model master 893.8M 4 7 months ago ~/.cache/modelscope/hub/models--damo--t5-base/... ``` Returns: `str`: The table as a string. reporevisionr'c URURURSRUR5UR UR UR[UR5/$)zHFormat a single repo and revision into a list of strings for tabulation.z{:>12}) rXrYrFformatr5rTr^rLr>rZ)rkrls r+format_repo_revisionAModelScopeCacheInfo.export_as_table..format_repo_revisions\ $$ 5 56 &&&&DNN#  r/)zREPO IDz REPO TYPEREVISIONz SIZE ON DISKzNB FILES LAST_ACCESSED LAST_MODIFIEDz LOCAL PATHcUR$N)rX)rks r+5ModelScopeCacheInfo.export_as_table..)sDLLr/)keycUR$ru)rF)rls r+rvrw+s X5I5Ir/)rowsheaders)rrrr>sortedrfr[r)r*rocolumn_headersrkrl table_datas r+export_as_table#ModelScopeCacheInfo.export_as_tables2 ~ += BFs)   tzz/HI I"$IK ! 0K 1I  "   s/A7r8N)r9r:r;r<r=r@r?rrrrrBr>r5rrCr8r/r+rrsK  ^ $$*++ 8#88< < r/r cache_dirr'cBUc [5n[U5R5R5nUR 5(d[ SUS3US9eUR 5(a[SUS35e[5n/nUS- nUS- nUR 5(aGUR5(a2[U[S9upVURU5 URU5 UR 5(aGUR5(a2[U[S9upxURU5 URU5 [U[SS 9upURU 5 URU 5 [[!U5[#S U55US 9$) aScan the entire ModelScope cache-system and return a [`ModelScopeCacheInfo`] structure. Use `scan_cache_dir` to programmatically scan your cache-system. The cache will be scanned repo by repo. If a repo is corrupted, a [`~CorruptedCacheException`] will be thrown internally but captured and returned in the [`~ModelScopeCacheInfo`] structure. Only valid repos get a proper report. ```py >>> from modelscope.hub.utils import scan_cache_dir >>> ms_cache_info = scan_cache_dir() ModelScopeCacheInfo( size_on_disk=3398085269, repos=frozenset({ CachedRepoInfo( repo_id='damo/t5-small', repo_type='model', repo_path=PosixPath(...), size_on_disk=970726914, nb_files=11, revisions=frozenset({ CachedRevisionInfo( commit_hash='master', size_on_disk=970726339, snapshot_path=PosixPath(...), files=frozenset({ CachedFileInfo( file_name='config.json', size_on_disk=1197 file_path=PosixPath(...), blob_path=PosixPath(...), ), CachedFileInfo(...), ... }), ), CachedRevisionInfo(...), ... }), ), CachedRepoInfo(...), ... }), warnings=[ CorruptedCacheException("Snapshots dir doesn't exist in cached repo: ..."), CorruptedCacheException(...), ... ], ) ``` Args: cache_dir (`str` or `Path`, `optional`): Cache directory to scan. Defaults to the default ModelScope cache directory. Raises: `CacheNotFound`: If the cache directory does not exist. `ValueError`: If the cache directory is a file, instead of a directory. Returns: a [`ModelScopeCacheInfo`] object. zCache directory not found: zQ. Please use `cache_dir` argument or set `MODELSCOPE_CACHE` environment variable.)rz1Scan cache expects a directory but found a file: modelsdatasetsrYT)rYinplacec38# UHoRv M g7fru)r$).0rks r+ !scan_cache_dir..s=ut**u)rfr$rg)rr expanduserresolveexistsris_file ValueErrorsetis_dir _scan_dirrupdateextendrr frozensetsum) rrfrg model_dir dataset_dir model_reposmodel_warnings dataset_reposdataset_warnings other_reposother_warningss r+rr4s~,. Y**,446I     ))5F G  ? {K\ ]  "%E.0HH$Ij(Ki..00&/ '2#  [!' 2 2 4 4*3 #4+6'  ]#()#,_d#<K LL OON# =u== r/dircUR5(dgUR5(dgUR5(agUR[;agg)z+Check if a directory is valid for scanning.FT)rr is_symlinknameFILES_TO_IGNORE)rs r+ _is_valid_dirrs@ ::<< ::<< ~~ xx?" r/rYrc[5n/nUR5HwnU(aURS;aM[U5(dM.UR5H5n[U5(dM[ XaS9nUbUR U5 M5M7 My X44$![ anURU5 SnAMaSnAff=f)zWScan a directory for cached repos and return a set of [`~CachedRepoInfo`] and warnings.)rrhubrN)riterdirrr_scan_cached_repoaddrappend) rrYrrfrg owner_dirname_dirinfoes r+rrs EEH[[] y~~)FF Y'' !))+H ** #(G#IIdO$ ,# ?+ #"" #s1B C%B<<CrZc ^UR5(d[SU35e[[U55nURnUR nUR 5RSS5nUS:Xag0m[5nUHn[RRXS5n URSS 5n [RRU 5(dM][U 5n U R!5TU 'UR#[%[RR'US5U U TU R(U TU R*TU R,S 95 M S n U(a,S U;a&UR/S 5S R/S5Sn U(a[1U4SjU55n OUR!5R,n [3U [5U5[7U4SjU55UU S9nT(aA[1STR9555n[1STR9555nO(UR!5nUR*nUR,n[;[=T5UUU[5U/5[7STR9555UUS9$![an[SU35eSnAff=f)z~Scan a single cache repo and return information about it. Any unexpected behavior will raise a [`~CorruptedCacheException`]. zRepo path is not a directory: ___.unknownNz"Failed to load cache information: rRevision)r r!r"r$r#r%r&masterz Revision:,rc3V># UHnTURRv M g7fru)r#st_mtimerfile blob_statss r+r$_scan_cached_repo..s(%D1B&0%?%H%H1B&)c3V># UHnTURRv M g7fru)r#st_sizers r+rrs'8%6T$DNN3;;%6r)rFrHr$rGrIc38# UHoRv M g7fru)st_atimerstats r+rr O;N4;Nrc38# UHoRv M g7fru)rrs r+rrrrc38# UHoRv M g7fru)rrs r+rr$sF2E$2Er)rTrXrZrYr[r$r\rI)rrrr> cached_filescached_model_revision get_model_idreplace ExceptionrospathjoingetrrrrrbasenamerrrsplitmaxrrrvaluesrrS)rZrYcacherrrXrcached_files_info cached_filer!r"r# revision_hashrevision_last_modifiedcached_revisionrepo_last_accessedrepo_last_modified repo_statsrs @r+rrs     %,YK 8: : P$S^4)) % ; ;$$&..uc: i  J$ GGLL,?@ (__Z<ww~~i(( O ) 0 9  ''**;v+>?##5' 2::##-i#8#A#A#-i#8#A#A  $,M / /177 DQGMMM!$%D1B%D"D"+!1!:!:)!)*8%688, O O:;L;L;N OO O:;L;L;N OO^^% '00'00 Z_-.F*2C2C2EFF((  A P%(J1#&NOOPsAJ22 K<K  Kru)F).r=r dataclassesrpathlibrtypingrrrrr r r modelscope_hub._cache_managerr r modelscope.hub.errorsrrmodelscope.hub.utils.cachingrmodelscope.hub.utils.utilsrrrmodelscope.utils.constantrrmodelscope.utils.file_utilsrmodelscope.utils.loggerrlogger__all__rrrrrr>rboolrrrr8r/r+rsS !GGGAH=DDH@.   - $888888v $111h $;8;8;8| $Y Y Y z15pE#t),-p9Lpf t  4C$.\\!$\)1.)A\r/