9i!2 S SK r S SKrS SKrS SKrS SKrS SKJrJr S SKJ r S SK
J r J r S SK
Jr S SKJr \" 5 r\R$ " \ S5 R' 5 R) 5 S:H r " S S
\5 r " S S \5 rg)
N)movermtree)Dict) FILE_HASH)MODELSCOPE_ENABLE_DEFAULT_HASH_VALIDATION) compute_hash)
get_loggerFalsetruec j \ rS rSrSrSrSrSr S\4S jr S r
S r S
r S r
S rS
rS rS rS rSrg)FileSystemCache z.mscz.mdlidz.mvcache_root_locationc [ R " 5 U l [ R " USS9 Xl U R
5 g)zBase file system cache interface.
Args:
cache_root_location (str): The root location to store files.
kwargs(dict): The keyword arguments.
Texist_okN) threadingRLock _cache_lockosmakedirsr
load_cache)selfr kwargss \/var/www/html/land-doc-ocr/venv/lib/python3.13/site-packages/modelscope/hub/utils/caching.py__init__FileSystemCache.__init__ s2 %??,
'$7#6 c U R $ N)r r s r get_root_location!FileSystemCache.get_root_location1 s '''r c N / U l [ R R U R [
R 5 n[ R R U5 ( a1 [ US5 n[ R " U5 U l S S S 5 g g ! , ( d f g = f)Nrb) cached_filesr pathjoinr r
KEY_FILE_NAMEexistsopenpickleload)r cache_keys_file_pathfs r r FileSystemCache.load_cache4 st !ww||D,D,D,;,I,I K
77>>. / /*D1Q$*KKN! 21 011s 0 B
B$c U R [ R R U R [
R 5 n[ R " SU R S9u p# [ R " US5 n[ R " U R U5 SSS5 [ X15 SSS5 g! , ( d f N#= f! [ ad [ R " U5 O! [ a Of = f[ R R# U5 ( a [ R$ " U5 e f = f! , ( d f g= f)z
Save cache metadata in order to verify that the cached content is consistent with the remote content.
Example of the cached content:
[{'Path': 'configuration.json', 'Revision': 'f01dxxx'}, {'Path': 'model.bin', 'Revision': '1159xxx'}, ...]
z.tmp)suffixdirwbN)r r r( r) r r
r* tempfilemkstempfdopenr- dumpr' r ExceptioncloseOSErrorr+ unlink)r r/ fd
temp_filenamer0 s r save_cached_files!FileSystemCache.save_cached_files< s
#%77<<0H0H0?0M0M$O ( 0 0 4#;#;!= B
YYr4(AKK 1 115 )]9
)(
HHRL 77>>-00IIm,
sZ AE (C?"B>!C>
C C
D=C10D=1
C>;D==C>>?D==E
Ec g)zCheck the key is in the cache, if exist, return the file, otherwise return None.
Args:
key(str): The cache key.
Raises:
None
N r keys r get_fileFileSystemCache.get_fileV s
r c g)zPut file to the cache.
Args:
key (str): The cache key
location (str): Location of the file, we will move the file to cache.
Raises:
None
NrC )r rE locations r put_fileFileSystemCache.put_filea s
r c z XR ; a, U R R U5 U R 5 gg)z^Remove cache key in index, The file is removed manually
Args:
key (dict): The cache key.
N)r' remover@ rD s r
remove_keyFileSystemCache.remove_keym s5 ## # $ $S ) " " $ $r c 8 U R H
nX!:X d M
g g)NTF)r' )r rE
cache_files r r+ FileSystemCache.existsw s ++J , r c N [ U R 5 U R 5 g)zRemove all files and metadata from the cache
In the case of multiple cache locations, this clears only the last one,
which is assumed to be the read/write one.
N)r r r r" s r clear_cacheFileSystemCache.clear_cache~ s
t''( r c f [ R " UR 5 5 R 5 $ r! )hashlibsha256encode hexdigestrD s r hash_nameFileSystemCache.hash_name s ~~cjjl+5577r )r r r' N)__name__
__module__ __qualname____firstlineno__r* MODEL_META_FILE_NAMEMODEL_META_MODEL_IDMODEL_VERSION_FILE_NAMEstrr r# r r@ rF rJ rN r+ rT r[ __static_attributes__rC r r r
r
sX M! # "(34
%8r r
c ^ \ rS rSrSrSU 4S jjrS rS rS\4S jr S r
S r S
r S r
S rS
rS rS rS rSrU =r$ )ModelFileSystemCache zLocal cache file layout
cache_root/owner/model_name/individual cached files and cache index file '.mcs'
Save only one version for each file.
c l > Ub Uc= [ TU ] [ R R U5 5 U R 5 O[[ TU ] [ R R XU5 5 [ R U< SU< 30U l U R 5 U R 5 U l g)a Put file to the cache
Args:
cache_root(`str`): The modelscope local cache root(default: ~/.cache/modelscope/hub)
owner(`str`): The model owner.
name('str'): The name of the model
Returns:
Raises:
None
model_id = {owner}/{name}
N/) superr r r( r) load_model_metar
rb
model_metasave_model_metaload_model_versioncached_model_revision)r
cache_rootownername __class__s r r ModelFileSystemCache.__init__ s =DL G RWW\\* 5 6 " G RWW\\*T B C 33t5L DO
"%)%<%<%>"r c n [ R R U R [ R
5 n[ R R
U5 ( a1 [ US5 n[ R " U5 U l
S S S 5 g [ R S0U l
g ! , ( d f g = f)Nr& unknown) r r( r) r r
ra r+ r, r- r. rm rb r meta_file_pathr0 s r rl $ModelFileSystemCache.load_model_meta sy d&>&>&5&J&JL
77>>. ) )nd+q"(++a. ,+ / B BI NDO ,+s ) B&&
B4c F [ R R U R [ R
5 n[ R R
U5 ( a4 [ US5 nUR 5 R 5 sS S S 5 $ g ! , ( d f g = f)Nr)
r r( r) r r
rc r+ r, readstrip)r model_version_file_pathr0 s r ro 'ModelFileSystemCache.load_model_version sn "$'',, $ $o&M&M#O
77>>1 2 2-s3qvvx~~' 43 43s ) B
B
revision_infoc P [ R R U R [ R
5 n[
US5 n[ U[ 5 ( a" SUS < SUS < 3nUR U5 OUR U5 S S S 5 g ! , ( d f g = f)Nwz Revision:Revisionz ,CreatedAt: CreatedAt)
r r( r) r r
rc r,
isinstancedictwrite)r r r r0 version_info_strs r save_model_version'ModelFileSystemCache.save_model_version s~ "$'',, $ $o&M&M#O
)3
/1-..!*-}[/I$K ()
&
0
/
/s
A B
B%c < U R [ R $ r! )rm r
rb r" s r get_model_id!ModelFileSystemCache.get_model_id s B BCCr c [ R R U R [ R
5 n[
US5 n[ R " U R U5 S S S 5 g ! , ( d f g = f)Nr5 )
r r( r) r r
ra r, r- r9 rm rx s r rn $ModelFileSystemCache.save_model_meta sP d&>&>&5&J&JL
.$
'1 KK + (
'
's "A00
A>c
U R Hs nXS :X d M
[ R R U R US 5 n[ R R U5 ( a Us $ U R
U5 Mu g)zRetrieve the cache if there is file match the path.
Args:
file_path (str): The file path in the model.
Returns:
path: the full path of the file.
PathN)r' r r( r) r r+ rN )r file_path cached_filecached_file_paths r get_file_by_path%ModelFileSystemCache.get_file_by_path sn ,,K //#%77<<0H0H0;F0C$E 77>>"233++OOK0 - r c r U R H nXS :X d M
US R U5 ( d UR US 5 ( d MA [ R R U R
US 5 n[ R R
U5 ( a Us $ U R U5 M g)zRetrieve the cache if there is file match the path.
Args:
file_path (str): The file path in the model.
commit_id (str): The commit id of the file
Returns:
path: the full path of the file.
r r N)r'
startswithr r( r) r r+ rN )r r commit_idr r s r get_file_by_path_and_commit_id3ModelFileSystemCache.get_file_by_path_and_commit_id s ,,K //J'229==AUAUVablVmAnAn#%77<<0H0H0;F0C$E 77>>"233++OOK0 - r c & U R U5 nU R Hp nX2:X d M
[ R R U R
US 5 n[ R R
U5 ( a Us $ U R U5 g g)zCheck if exist cache file.
Args:
model_file_info (ModelFileInfo): The file information of the file.
Returns:
str: The file path.
r N)$_ModelFileSystemCache__get_cache_keyr' r r( r) r r+ rN )r model_file_info cache_keyr orig_paths r get_file_by_info%ModelFileSystemCache.get_file_by_info s| (( 9 ,,K' GGLL)A)A)4V)< > 77>>),,$$OOK0 - r c US US S.nU$ )Nr r )r r rC )r r r s r __get_cache_key$ModelFileSystemCache.__get_cache_key s! #F+'
3
r c U R U5 nSnUS n[ R R U R US 5 nU R
H nUS U:X d M US R
US 5 ( d US R
US 5 ( d MH U[ nUc MV [ R R U5 ( d M| [ ( a [ U5 nOUnXx:X a Sn O [ R SU S35 M U( a6 [ R R U5 ( a gU R U5 g)zCheck the file is cached or not. Note existence check will also cover digest check
Args:
model_file_info (CachedFileInfo): The cached file info
Returns:
bool: If exists and has the same hash, return True otherwise False
Fr r TzFile [z?] exists in cache but with a mismatched hash, will re-download.)r r r( r) r r' r r r+ enable_default_hash_validationr loggerinforN ) r r rE is_existsr cache_file_path
cached_key
expected_hashcache_file_sha256s r r+ ModelFileSystemCache.exists s* ""?3 K '',,t'?'?'6v'>@ ++J&!Y. z*55c*oFF:11*Z2HII / :
, '2) 2) 65,8 ,I),9)$9$( $YK/n o! ,& ww~~o..#%r c 6 U R H nUS US :X d M U R U5 [ R R U R
US 5 n[ R R
U5 ( a [ R " U5 g g)zlWe in cache, remove it.
Args:
model_file_info (ModelFileInfo): The model file information from server.
r N)r' rN r r( r) r r+ rM )r r r r s r remove_if_exists%ModelFileSystemCache.remove_if_exists? sv ,,K6"of&== , GGLL)A)A)4V)< > 77>>),,IIi( -r c U R U5 U R U5 n[ R R U R
US 5 n[ R R
U5 n[ R R U5 ( d [ R " USS9 [ X$5 U R R U5 U R 5 U$ )a+ Put model on model_file_location to cache, the model first download to /tmp, and move to cache.
Args:
model_file_info (str): The file description returned by get_model_files.
model_file_location (str): The location of the temporary file.
Returns:
str: The location of the cached file.
r Tr )
r r r r( r) r dirnamer+ r r r' appendr@ )r r model_file_locationr cache_full_pathcache_file_dirs r rJ ModelFileSystemCache.put_fileN s
o.(( 9 '',, $ $ f 9ww~~n-- KK 6
2 + r )rp rm )NN)r] r^ r_ r` __doc__r rl ro r r r rn r r r r r+ r rJ re
__classcell__)rt s @r rg rg sX
?2O ' 'D, (,,'R
r rg )rW r r- r6 r shutilr r typingr modelscope.hub.constantsr r modelscope.hub.utils.utilsr modelscope.utils.loggerr r getenvr~ lowerr objectr
rg rC r r r s~
: 3 . II7AGGIOOQU[[ o8f o8dZ ? Z r