6pj7 S SK r S SKrS SKrS SKrS SKrS SKJrJr S SKJ r S SK
J r J r S SK
Jr S SKJr \" 5 r\R$ " \ S5 R' 5 R) 5 S:H r " S S
\5 r " S S \5 rg)
N)movermtree)Dict) FILE_HASH)MODELSCOPE_ENABLE_DEFAULT_HASH_VALIDATION) compute_hash)
get_loggerFalsetruec p \ rS rSrSrSrSrSr S\4S jr S r
S r S
r S r
S rS
rS rS rS rS rSrg)FileSystemCache z.mscz.mdlidz.mvcache_root_locationc [ R " 5 U l [ R " USS9 Xl U R
5 g)zBase file system cache interface.
Args:
cache_root_location (str): The root location to store files.
kwargs(dict): The keyword arguments.
Texist_okN) threadingRLock _cache_lockosmakedirsr
load_cache)selfr kwargss X/var/www/html/pdf-tiff/venv/lib/python3.13/site-packages/modelscope/hub/utils/caching.py__init__FileSystemCache.__init__ s2 %??,
'$7#6 c U R $ N)r r s r get_root_location!FileSystemCache.get_root_location1 s '''r c / U l [ R R U R [
R 5 n[ R R U5 ( as [ US5 n[ R " U5 nS S S 5 [ W[ 5 ( a X0l g [ R SU[ U5 R 5 g g ! , ( d f NV= f! [" aO n[ R SX5 [ R$ " UUS- 5 S nAg ! [& a S nAg f = fS nAff = f)Nrbz1Cache index %s has unexpected type %s, resetting.uk Failed to load cache index %s: %s. Resetting — already-downloaded files will be re-validated on next run.
.corrupted) cached_filesr pathjoinr r
KEY_FILE_NAMEexistsopenpickleload
isinstancelistloggerwarningtype__name__ ExceptionreplaceOSError)r cache_keys_file_pathfdataes r r FileSystemCache.load_cache4 s !ww||D,D,D,;,I,I K
77>>. / /
.5!;;q>D 6 dD))(,%NNK, T
++- 055
_(-JJ 3 3l BD D
sT % C) 1C#C) ,*C)
C&"C) )
E3D=
D))
D:3D=9D::D==Ec [ R R U R [ R
5 n[ R " SU R S9u p# [ R " US5 n[ R " [ U R 5 U5 UR 5 [ R " UR 5 5 SSS5 [ R " X15 g! , ( d f N%= f! [" ad [ R$ " U5 O! [& a Of = f[ R R) U5 ( a [ R* " U5 e f = f)z8Write .msc atomically. Caller must hold ``_cache_lock``.z.tmp)suffixdirwbN)r r) r* r r
r+ tempfilemkstempfdopenr. dumpr1 r( flushfsyncfilenor7 r6 closer8 r, unlink)r r9 fd
temp_filenamer: s r _save_cached_files_unlocked+FileSystemCache._save_cached_files_unlockedN s !ww||D,D,D,;,I,I K $ , ,t 7 7 9
2t$ D!2!2 3Q7 $ %
JJ} ; %$
ww~~m,, -( sI D 2A C0 D 0
C>:D
E/ D#"E/#
D0-E//D00?E/c p U R U R 5 SSS5 g! , ( d f g= f)z
Save cache metadata in order to verify that the cached content is consistent with the remote content.
Example of the cached content:
[{'Path': 'configuration.json', 'Revision': 'f01dxxx'}, {'Path': 'model.bin', 'Revision': '1159xxx'}, ...]
N)r rM r" s r save_cached_files!FileSystemCache.save_cached_filesc s%
, , .
s '
5c g)zCheck the key is in the cache, if exist, return the file, otherwise return None.
Args:
key(str): The cache key.
Raises:
None
N r keys r get_fileFileSystemCache.get_filem s
r c g)zPut file to the cache.
Args:
key (str): The cache key
location (str): Location of the file, we will move the file to cache.
Raises:
None
NrS )r rU locations r put_fileFileSystemCache.put_filex s
r c z XR ; a, U R R U5 U R 5 gg)z^Remove cache key in index, The file is removed manually
Args:
key (dict): The cache key.
N)r( removerP rT s r
remove_keyFileSystemCache.remove_key s5 ## # $ $S ) " " $ $r c 8 U R H
nX!:X d M
g g)NTF)r( )r rU
cache_files r r, FileSystemCache.exists s ++J , r c N [ U R 5 U R 5 g)zRemove all files and metadata from the cache
In the case of multiple cache locations, this clears only the last one,
which is assumed to be the read/write one.
N)r r r r" s r clear_cacheFileSystemCache.clear_cache s
t''( r c f [ R " UR 5 5 R 5 $ r! )hashlibsha256encode hexdigestrT s r hash_nameFileSystemCache.hash_name s ~~cjjl+5577r )r r r( N)r5
__module__ __qualname____firstlineno__r+ MODEL_META_FILE_NAMEMODEL_META_MODEL_IDMODEL_VERSION_FILE_NAMEstrr r# r rM rP rV rZ r^ r, rd rk __static_attributes__rS r r r
r
s] M! # "(4*/
%8r r
c ^ \ rS rSrSrSU 4S jjrS rS rS\4S jr S r
S r S
r S r
S rS
rS rS rS rSrU =r$ )ModelFileSystemCache zLocal cache file layout
cache_root/owner/model_name/individual cached files and cache index file '.mcs'
Save only one version for each file.
c l > Ub Uc= [ TU ] [ R R U5 5 U R 5 O[[ TU ] [ R R XU5 5 [ R U< SU< 30U l U R 5 U R 5 U l g)a Put file to the cache
Args:
cache_root(`str`): The modelscope local cache root(default: ~/.cache/modelscope/hub)
owner(`str`): The model owner.
name('str'): The name of the model
Returns:
Raises:
None
model_id = {owner}/{name}
N/) superr r r) r* load_model_metar
rq
model_metasave_model_metaload_model_versioncached_model_revision)r
cache_rootownername __class__s r r ModelFileSystemCache.__init__ s =DL G RWW\\* 5 6 " G RWW\\*T B C 33t5L DO
"%)%<%<%>"r c ~ [ R R U R [ R
5 n[ R R
U5 ( a^ [ US5 n[ R " U5 nS S S 5 [ W[ 5 ( a X0l g [ R SU5 [ R$ S0U l g ! , ( d f NX= f! [ aN n[ R SX5 [ R " XS- 5 S nANh! [" a S nANxf = fS nAff = f)Nr& z-Model meta %s has unexpected type, resetting.z,Failed to load model meta %s: %s. Resetting.r' unknown)r r) r* r r
rp r, r- r. r/ r0 dictr| r2 r3 r6 r7 r8 rq )r meta_file_pathr: r; r< s r r{ $ModelFileSystemCache.load_model_meta s d&>&>&5&J&JL
77>>. ) )
.$/1!;;q>D 0 dD))&*O N -/ +>> J 0/
M -2JJ~ /LMM
sT C$ *C#C$ %C$
C! C$ $
D<.D7D##
D4-D73D44D77D<c F [ R R U R [ R
5 n[ R R
U5 ( a4 [ US5 nUR 5 R 5 sS S S 5 $ g ! , ( d f g = f)Nr)
r r) r* r r
rr r, r- readstrip)r model_version_file_pathr: s r r~ 'ModelFileSystemCache.load_model_version sn "$'',, $ $o&M&M#O
77>>1 2 2-s3qvvx~~' 43 43s ) B
B
revision_infoc P [ R R U R [ R
5 n[
US5 n[ U[ 5 ( a" SUS < SUS < 3nUR U5 OUR U5 S S S 5 g ! , ( d f g = f)Nwz Revision:Revisionz ,CreatedAt: CreatedAt)
r r) r* r r
rr r- r0 r write)r r r r: version_info_strs r save_model_version'ModelFileSystemCache.save_model_version s~ "$'',, $ $o&M&M#O
)3
/1-..!*-}[/I$K ()
&
0
/
/s
A B
B%c < U R [ R $ r! )r| r
rq r" s r get_model_id!ModelFileSystemCache.get_model_id s B BCCr c [ R R U R [ R
5 n[
US5 n[ R " U R U5 S S S 5 g ! , ( d f g = f)NrA )
r r) r* r r
rp r- r. rE r| )r r r: s r r} $ModelFileSystemCache.save_model_meta sP d&>&>&5&J&JL
.$
'1 KK + (
'
's "A00
A>c
U R Hs nXS :X d M
[ R R U R US 5 n[ R R U5 ( a Us $ U R
U5 Mu g)zRetrieve the cache if there is file match the path.
Args:
file_path (str): The file path in the model.
Returns:
path: the full path of the file.
PathN)r( r r) r* r r, r^ )r file_path cached_filecached_file_paths r get_file_by_path%ModelFileSystemCache.get_file_by_path sn ,,K //#%77<<0H0H0;F0C$E 77>>"233++OOK0 - r c r U R H nXS :X d M
US R U5 ( d UR US 5 ( d MA [ R R U R
US 5 n[ R R
U5 ( a Us $ U R U5 M g)zRetrieve the cache if there is file match the path.
Args:
file_path (str): The file path in the model.
commit_id (str): The commit id of the file
Returns:
path: the full path of the file.
r r N)r(
startswithr r) r* r r, r^ )r r commit_idr r s r get_file_by_path_and_commit_id3ModelFileSystemCache.get_file_by_path_and_commit_id s ,,K //J'229==AUAUVablVmAnAn#%77<<0H0H0;F0C$E 77>>"233++OOK0 - r c & U R U5 nU R Hp nX2:X d M
[ R R U R
US 5 n[ R R
U5 ( a Us $ U R U5 g g)zCheck if exist cache file.
Args:
model_file_info (ModelFileInfo): The file information of the file.
Returns:
str: The file path.
r N)$_ModelFileSystemCache__get_cache_keyr( r r) r* r r, r^ )r model_file_info cache_keyr orig_paths r get_file_by_info%ModelFileSystemCache.get_file_by_info s| (( 9 ,,K' GGLL)A)A)4V)< > 77>>),,$$OOK0 - r c US US S.nU$ )Nr r )r r rS )r r r s r __get_cache_key$ModelFileSystemCache.__get_cache_key2 s! #F+'
3
r c U R U5 nSnUS n[ R R U R US 5 nU R
H nUS U:X d M US R
US 5 ( d US R
US 5 ( d MH U[ nUc MV [ R R U5 ( d M| [ ( a [ U5 nOUnXx:X a Sn O [ R SU S35 M U( a6 [ R R U5 ( a gU R U5 g)zCheck the file is cached or not. Note existence check will also cover digest check
Args:
model_file_info (CachedFileInfo): The cached file info
Returns:
bool: If exists and has the same hash, return True otherwise False
Fr r TzFile [z?] exists in cache but with a mismatched hash, will re-download.)r r r) r* r r( r r r, enable_default_hash_validationr r2 infor^ ) r r rU is_existsr cache_file_path
cached_key
expected_hashcache_file_sha256s r r, ModelFileSystemCache.exists9 s* ""?3 K '',,t'?'?'6v'>@ ++J&!Y. z*55c*oFF:11*Z2HII / :
, '2) 2) 65,8 ,I),9)$9$( $YK/n o! ,& ww~~o..#%r c 6 U R H nUS US :X d M U R U5 [ R R U R
US 5 n[ R R
U5 ( a [ R " U5 g g)zlWe in cache, remove it.
Args:
model_file_info (ModelFileInfo): The model file information from server.
r N)r( r^ r r) r* r r, r] )r r r r s r remove_if_exists%ModelFileSystemCache.remove_if_existsb sv ,,K6"of&== , GGLL)A)A)4V)< > 77>>),,IIi( -r c U R U5 U R U5 n[ R R U R
US 5 n[ R R
U5 n[ R R U5 ( d [ R " USS9 [ X$5 U R U R R U5 U R 5 SSS5 U$ ! , ( d f U$ = f)a+ Put model on model_file_location to cache, the model first download to /tmp, and move to cache.
Args:
model_file_info (str): The file description returned by get_model_files.
model_file_location (str): The location of the temporary file.
Returns:
str: The location of the cached file.
r Tr N)r r r r) r* r dirnamer, r r r r( appendrM )r r model_file_locationr cache_full_pathcache_file_dirs r rZ ModelFileSystemCache.put_fileq s
o.(( 9 '',,t'?'?'0'8: 9ww~~n-- KK 6
2
$ $Y / , , .
s ?,C55
D)r r| )NN)r5 rm rn ro __doc__r r{ r~ r r r r} r r r r r, r rZ rt
__classcell__)r s @r rv rv sX
?2K* ' 'D, (,,'R
r rv )rg r r. rB r shutilr r typingr modelscope.hub.constantsr r modelscope.hub.utils.utilsr modelscope.utils.loggerr r2 getenvr lowerr objectr
rv rS r r r s~
: 3 . II7AGGIOOQU[[ F8f F8Re ? e r