9i/ S r SSKrSSKrSSKrSSKrSSKJrJr SSKJ r SSK
J r J r J
r
SSKJr SSKJr \R$ " \ S 5 R' 5 R) 5 S
:H r " S S \5 r " S
S\5 rg)z
caching tools
N)movermtree)Dict) FILE_HASH'AISTUDIO_ENABLE_DEFAULT_HASH_VALIDATION CACHE_KEY ) compute_hash)logFalsetruec n \ rS rSrSrSrSrSrSr S\ 4S jr
S r S
r S r
S rS
rS rS rS rS rSrg)FileSystemCache infoz.mscz.mdlidz.mvcache_root_locationc Z [ R " USS9 Xl U R 5 g)zBase file system cache interface.
Args:
cache_root_location (str): The root location to store files.
kwargs(dict): The keyword arguments.
Texist_okN)osmakedirsr
load_cache)selfr s Z/var/www/html/land-doc-ocr/venv/lib/python3.13/site-packages/aistudio_sdk/utils/caching.py__init__FileSystemCache.__init__ s# '$7#6 c U R $ )location)r r s r get_root_location!FileSystemCache.get_root_location- s '''r c N / U l [ R R U R [
R 5 n[ R R U5 ( a1 [ US5 n[ R " U5 U l SSS5 gg! , ( d f g= f)loadrbN) cached_filesr pathjoinr r
KEY_FILE_NAMEexistsopenpickler% )r cache_keys_file_pathfs r r FileSystemCache.load_cache1 st !ww||D,D,D,;,I,I K
77>>. / /*D1Q$*KKN! 21 011s 0 B
B$c F [ R R U R [ R
5 n[ R " 5 u p#[ US5 n[ R " U R U5 SSS5 [ X15 g! , ( d f N= f)zSave cache metadata.wbN)
r r( r) r r r* tempfilemkstempr, r- dumpr' r )r r. fdfnr/ s r save_cached_files!FileSystemCache.save_cached_files: sm "ww||D,D,D,;,I,I K !!#
"d^q KK))1 - R& ^s "B
B c g)zCheck the key is in the cache, if exists, return the file, otherwise return None.
Args:
key(str): The cache key.
Raises:
None
N r keys r get_fileFileSystemCache.get_fileD s
r c g)zPut file to the cache.
Args:
key (str): The cache key
location (str): Location of the file, we will move the file to cache.
Raises:
None
Nr; )r r= r s r put_fileFileSystemCache.put_fileO s
r c z XR ; a, U R R U5 U R 5 gg)z^Remove cache key in index, The file is removed manually
Args:
key (dict): The cache key.
N)r' remover8 r< s r
remove_keyFileSystemCache.remove_key[ s5 ## # $ $S ) " " $ $r c 8 U R H
nX!:X d M
g g)checkTF)r' )r r=
cache_files r r+ FileSystemCache.existse s ++J , r c N [ U R 5 U R 5 g)zRemove all files and metadata from the cache
In the case of multiple cache locations, this clears only the last one,
which is assumed to be the read/write one.
N)r r r r! s r clear_cacheFileSystemCache.clear_cachem s
t''( r c f [ R " UR 5 5 R 5 $ )name)hashlibsha256encode hexdigestr< s r hash_nameFileSystemCache.hash_nameu s ~~cjjl+5577r )r r' N)__name__
__module__ __qualname____firstlineno____doc__r* MODEL_META_FILE_NAMEMODEL_META_MODEL_IDMODEL_VERSION_FILE_NAMEstrr r" r r8 r> rA rE r+ rL rT __static_attributes__r; r r r r s[ M! # (3'
%8r r c ^ \ rS rSrSrSU 4S jjrS rS rS\4S jr S r
S r S
r S r
S rS
rS rS rS rSrU =r$ )ModelFileSystemCachez zLocal cache file layout
cache_root/owner/model_name/individual cached files and cache index file '.mcs'
Save only one version for each file.
c l > Ub Uc= [ TU ] [ R R U5 5 U R 5 O[[ TU ] [ R R XU5 5 [ R U< SU< 30U l U R 5 U R 5 U l g)zPut file to the cache
Args:
cache_root(`str`): The aistudio local cache root(default: ~/.cache/aistudio/)
owner(`str`): The model owner.
name('str'): The name of the model
Returns:
Raises:
None
model_id = {owner}/{name}
N/) superr r r( r) load_model_metar r\
model_metasave_model_metaload_model_versioncached_model_revision)r
cache_rootownerrO __class__s r r ModelFileSystemCache.__init__ s =DL G RWW\\* 5 6 " G RWW\\*T B C 33t5L DO
"%)%<%<%>"r c n [ R R U R [ R
5 n[ R R
U5 ( a1 [ US5 n[ R " U5 U l
SSS5 g[ R S0U l
g! , ( d f g= f)zget metar& Nunknown) r r( r) r r r[ r+ r, r- r% rg r\ r meta_file_pathr/ s r rf $ModelFileSystemCache.load_model_meta sy d&>&>&5&J&JL
77>>. ) )nd+q"(++a. ,+ / B BI NDO ,+s ) B&&
B4c F [ R R U R [ R
5 n[ R R
U5 ( a4 [ US5 nUR 5 R 5 sSSS5 $ g! , ( d f g= f)zuse version inforN)
r r( r) r r r] r+ r, readstrip)r model_version_file_pathr/ s r ri 'ModelFileSystemCache.load_model_version sn "$'',, $ $o&M&M#O
77>>1 2 2-s3qvvx~~' 43 43s ) B
B
revision_infoc z [ R R U R [ R
5 n[
US5 n[ U[ 5 ( a7 SUS < SUR S5 =( d S< 3nUR U5 OUR U5 SSS5 g! , ( d f g= f)zsave file infowz Revision:Revisionz ,CreatedAt: CreatedAtrp N) r r( r) r r r] r,
isinstancedictgetwrite)r rz rx r/ version_info_strs r save_model_version'ModelFileSystemCache.save_model_version s "$'',, $ $o&M&M#O
)3
/1-..!*-}/@/@/M/ZQZ/Z$\ ()
&
0
/
/s
A B,,
B:c < U R [ R $ )r )rg r r\ r! s r get_model_id!ModelFileSystemCache.get_model_id s B BCCr c [ R R U R [ R
5 n[
US5 n[ R " U R U5 SSS5 g! , ( d f g= f)saver2 N)
r r( r) r r r[ r, r- r5 rg rq s r rh $ModelFileSystemCache.save_model_meta sP d&>&>&5&J&JL
.$
'1 KK + (
'
's "A00
A>c
U R Hs nXS :X d M
[ R R U R US 5 n[ R R U5 ( a Us $ U R
U5 Mu g)zRetrieve the cache if there is file match the path.
Args:
file_path (str): The file path in the model.
Returns:
path: the full path of the file.
PathN)r' r r( r) r r+ rE )r file_path cached_filecached_file_paths r get_file_by_path%ModelFileSystemCache.get_file_by_path sn ,,K //#%77<<0H0H0;F0C$E 77>>"233++OOK0 - r c r U R H nXS :X d M
US R U5 ( d UR US 5 ( d MA [ R R U R
US 5 n[ R R
U5 ( a Us $ U R U5 M g)zRetrieve the cache if there is file match the path.
Args:
file_path (str): The file path in the model.
commit_id (str): The commit id of the file
Returns:
path: the full path of the file.
r r} N)r'
startswithr r( r) r r+ rE )r r commit_idr r s r get_file_by_path_and_commit_id3ModelFileSystemCache.get_file_by_path_and_commit_id s ,,K //J'229==AUAUVablVmAnAn#%77<<0H0H0;F0C$E 77>>"233++OOK0 - r c & U R U5 nU R Hp nX2:X d M
[ R R U R
US 5 n[ R R
U5 ( a Us $ U R U5 g g)zCheck if exist cache file.
Args:
model_file_info (ModelFileInfo): The file information of the file.
Returns:
str: The file path.
r N)$_ModelFileSystemCache__get_cache_keyr' r r( r) r r+ rE )r model_file_info cache_keyr orig_paths r get_file_by_info%ModelFileSystemCache.get_file_by_info s| (( 9 ,,K' GGLL)A)A)4V)< > 77>>),,$$OOK0 - r c US US S.nU$ )Nr( sha)r r} r; )r r r s r __get_cache_key$ModelFileSystemCache.__get_cache_key s! #F+'.
r c U R U5 nSnUS n[ R R U R US 5 nU R
H nUS U:X d M US R
US 5 ( d US R
US 5 ( d MH U[ nUc MV [ R R U5 ( d M| [ ( a [ U5 nOUnXx:X a Sn O [ R " SU S35 M U( a6 [ R R U5 ( a gU R U5 g)zCheck the file is cached or not. Note existence check will also cover digest check
Args:
model_file_info (CachedFileInfo): The cached file info
Returns:
bool: If exists and has the same hash, return True otherwise False
Fr r( r} TzFile [z?] exists in cache but with a mismatched hash, will re-download.)r r r( r) r r' r r r+ enable_default_hash_validationr
r r rE ) r r r= is_existsr cache_file_path
cached_key
expected_hashcache_file_sha256s r r+ ModelFileSystemCache.exists s* ""?3 K '',,t'?'?'6v'>@ ++J&!Y. z*55c*oFF:11*Z2HII / :
, '2) 2) 65,8 ,I),9)$9$( $YK/n o! ,& ww~~o..#%r c 6 U R H nUS US :X d M U R U5 [ R R U R
US 5 n[ R R
U5 ( a [ R " U5 g g)zlWe in cache, remove it.
Args:
model_file_info (ModelFileInfo): The model file information from server.
r r( N)r' rE r r( r) r r+ rD )r r r r s r remove_if_exists%ModelFileSystemCache.remove_if_exists4 sv ,,K6"of&== , GGLL)A)A)4V)< > 77>>),,IIi( -r c U R U5 U R U5 n[ R R U R
US 5 n[ R R
U5 n[ R R U5 ( d [ R " USS9 [ X$5 U R R U5 U R 5 U$ )a+ Put model on model_file_location to cache, the model first download to /tmp, and move to cache.
Args:
model_file_info (str): The file description returned by get_model_files.
model_file_location (str): The location of the temporary file.
Returns:
str: The location of the cached file.
r Tr )
r r r r( r) r dirnamer+ r r r' appendr8 )r r model_file_locationr cache_full_pathcache_file_dirs r rA ModelFileSystemCache.put_fileC s
o.(( 9 '',, $ $ f 9ww~~n-- KK 6
2 + r )rj rg )NN)rV rW rX rY rZ r rf ri r r r rh r r r r r+ r rA r_
__classcell__)rm s @r ra ra z sX
?2O
'
'D,(,,'R
r ra )rZ rP r r- r3 shutilr r typingr aistudio_sdk.configr r r utilr
aistudio_sdkr getenvrw lowerr objectr ra r; r r r s
C C II5w?EEGMMOSYY a8f a8H_ ? _ r