9iGSSKrSSKrSSKrSSKrSSKrSSKrSSKJr SSKJ r J r SSK J r SSK J r SSK Jr SSKJrJrJrJrJrJrJrJrJrJrJr SSKJr SS KJr SS KJ r \"S 5r!/S Qr"\S r#/SQr$/SQr%"SS5r&\ "SS55r'\ "SS55r(\ "SS55r)\ "SS55r*S\+S\,4Sjr-\ "SS55r.\ "SS 55r/S!\,S\,4S"jr0\\/4r1g)#N)contextmanager) dataclassfield)datetimefnmatch)Path) AnyBinaryIOCallable GeneratorIterableIteratorListLiteralOptionalTypeVarUnion) DEFAULT_MODELSCOPE_DATA_ENDPOINT)convert_timestamp) get_file_hashT)z.gitz.git/*z*/.gitz **/.git/**z.cachez.cache/*z*/.cachez **/.cache/**)lfsnormal)+.7zz.aac.arrowz.audioz.bmp.bin.bz2z.flac.ftzz.gif.gz.h5z.jackz.jpegz.jpgz.pngz.jsonl.joblibz.lz4.msgpack.npy.npz.ot.parquet.pb.picklez.pcm.pklz.raw.rarz.sam.tar.tgz.wasmz.wavz.webmz.webp.zip.zstz.tiffz.mp3z.mp4z.ogg)rrrrz.ckptrr r!r"z.mlmodelz.modelr#r$r%z.onnxr&r'r(r)r*z.ptz.pthr+z .safetensorsr,z.tfliter-r.z.xzr/r0c\rSrSr\SSSS.S\\S\\\ \ \ 4S\\\ \ \ 4S\\ \/\ 4S\ \SS44 S jj5r \S \ S\ 4S j5rS rg) RepoUtilsuN)allow_patternsignore_patternskeyitemsr4r5r6returnc# ^# U(aUOSnU(aUOSn[U[5(aU/n[U[5(aU/nUb%UVs/sHn[RU5PM nnUb%UVs/sHn[RU5PM nnUcS[S[4SjnUnUHMnU"U5mUb[ U4SjU55(dM*Ub[ U4SjU55(aMIUv MO gs snfs snf7f)aFilter repo objects based on an allowlist and a denylist. Input must be a list of paths (`str` or `Path`) or a list of arbitrary objects. In the later case, `key` must be provided and specifies a function of one argument that is used to extract a path from each element in iterable. Patterns are Unix shell-style wildcards which are NOT regular expressions. See https://docs.python.org/3/library/fnmatch.html for more details. Args: items (`Iterable`): List of items to filter. allow_patterns (`str` or `List[str]`, *optional*): Patterns constituting the allowlist. If provided, item paths must match at least one pattern from the allowlist. ignore_patterns (`str` or `List[str]`, *optional*): Patterns constituting the denylist. If provided, item paths must not match any patterns from the denylist. key (`Callable[[T], str]`, *optional*): Single-argument function to extract a path from each item. If not provided, the `items` must already be `str` or `Path`. Returns: Filtered list of objects, as a generator. Raises: :class:`ValueError`: If `key` is not provided and items are not `str` or `Path`. Example usage with paths: ```python >>> # Filter only PDFs that are not hidden. >>> list(RepoUtils.filter_repo_objects( ... ["aaa.PDF", "bbb.jpg", ".ccc.pdf", ".ddd.png"], ... allow_patterns=["*.pdf"], ... ignore_patterns=[".*"], ... )) ["aaa.pdf"] ``` Nitemr8c[U[5(aU$[U[5(a [U5$[SUS35e)Nz9Please provide `key` argument in `filter_repo_objects`: `z` is not a string.) isinstancestrr ValueError)r:s [/var/www/html/land-doc-ocr/venv/lib/python3.13/site-packages/modelscope/utils/repo_utils.py _identity0RepoUtils.filter_repo_objects.._identitysFdC((KdD))t9$ OPTvUghc3<># UHn[TU5v M g7fNr.0rpaths r? 0RepoUtils.filter_repo_objects..s6>.<GD!$$nc3<># UHn[TU5v M g7frDrrEs r?rIrJs3?.=GD!$$orK)r<r=r2_add_wildcard_to_directoriesrany)r7r4r5r6pr@r:rHs @r?filter_repo_objectsRepoUtils.filter_repo_objectsws8b,:t-.<6>3>3>*s3?.=3?0?0?J/ sADD0 D9D A6Dpatternc"USS:XaUS-$U$)N/*)rRs r?rM&RepoUtils._add_wildcard_to_directoriess 2;# S= rBrW)__name__ __module__ __qualname____firstlineno__ staticmethodrrrrrr=r r rPrM__static_attributes__rWrBr?r2r2us;?;?,0 ^{^!tCy#~!67^"%S 3"78 ^ hsCx( ) ^ 1dD= ! ^^@ccrBr2cH\rSrSr%Sr\\S'\\S'\\S'\\S'SrSrg ) CommitInfoaData structure containing information about a newly created commit. Returned by any method that creates a commit on the Hub: [`create_commit`], [`upload_file`], [`upload_folder`], [`delete_file`], [`delete_folder`]. It inherits from `str` for backward compatibility but using methods specific to `str` is deprecated. Attributes: commit_url (`str`): Url where to find the commit. commit_message (`str`): The summary (first line) of the commit that has been created. commit_description (`str`): Description of the commit that has been created. Can be empty. oid (`str`): Commit hash id. Example: `"91c54ad1727ee830252e457677f467be0bfd8a57"`. commit_urlcommit_messagecommit_descriptionoidc`URURURURS.$)Nrbrcrdrerg)clss r?to_dictCommitInfo.to_dicts,..!00"%"8"877   rBrWN) rYrZr[r\__doc__r=__annotations__rir^rWrBr?r`r`s$*O H rBr`c\rSrSr%Sr\\\S'\\\S'\\\S'\\\S'\\\S'\\\S'\\\S '\\\S '\\\S '\\\S '\\\S '\ S\ SS4Sj5r S\ 4Sjr Sr g)DetailedCommitInfoiz8Detailed commit information from repository history API.idshort_idtitlemessage author_name authored_date author_emailcommitted_datecommitter_namecommitter_email created_atdatar8cU"URSS5URSS5URSS5URSS5URSS5[URSS55URS S5[URS S55URS S5URS S5[URS S55S9 $)z1Create DetailedCommitInfo from API response data.IdShortIdTitleMessage AuthorName AuthoredDateN AuthorEmail CommittedDate CommitterNameCommitterEmail CreatedAt rorprqrrrsrtrurvrwrxry)getr)rhrzs r?from_api_response$DetailedCommitInfo.from_api_responsesxxb!XXi,((7B'HHY+r2+DHH^T,JK-4,TXXot-LM88OR8 HH%5r:(+t)DE  rBc URURURURURUR UR URURURURS. $)zConvert to dictionary.rrselfs r?riDetailedCommitInfo.to_dict$sg'' ZZ||++!// --"11"11#33//  rBrWN)rYrZr[r\rkrr=rlr classmethoddictrrir^rWrBr?rnrnsB sm C= c]#H%%3-X&&SM!c]""" T .B    rBrnc\\rSrSr%Sr\\\\S'\\ \S'\ S\ SS4Sj5r Sr g ) CommitHistoryResponsei5z!Response from commit history API.commits total_countrzr8cURS05RS/5nU(dU"/SS9$UVs/sHn[RU5PM nnU"UURSS5S9$s snf)z4Create CommitHistoryResponse from API response data.DataCommitr)rr TotalCount)rrnr)rhrz commits_datacommitrs r?r'CommitHistoryResponse.from_api_response;sxx+//"=  ' &  0 0 8&  q1   sA-rWN)rYrZr[r\rkrrrnrlintrrrr^rWrBr?rr5sA+ d-. //# T .E  rBrc\rSrSr%Sr\\\S'Sr\\\S'Sr \\\S'Sr \\\S'Sr \\\S'\ r \\\S'S \4S jrS rg) RepoUrliQNurl namespace repo_namerepo_id repo_typeendpointr8c \SUSURSURSURS3 $)Nz RepoUrl('z ', endpoint='z', repo_type='z ', repo_id='z'))rrrrs r?__repr__RepoUrl.__repr__[s64& dmm_N4>>JZZfgkgsgsfttvwwrBrW)rYrZr[r\rrr=rlrrrrrrrr^rWrBr?rrQsfC##Ix}##Ix}#!GXc]!#Ix}#>Hhsm>x#xrBrrzr8cv[RS:aSS0O0n[R"[R 40UD6nU"5nUR S5 UR [[U55R55 UR S5 UR U5 UR5$)aw Computes the git-sha1 hash of the given bytes, using the same algorithm as git. This is equivalent to running `git hash-object`. See https://git-scm.com/docs/git-hash-object for more details. Note: this method is valid for regular files. For LFS files, the proper git hash is supposed to be computed on the pointer file content, not the actual file content. However, for simplicity, we directly compare the sha256 of the LFS file content when we want to compare LFS files. Args: data (`bytes`): The data to compute the git-hash for. Returns: `str`: the git-hash of `data` as an hexadecimal string. ) usedforsecurityFsblob ) sys version_info functoolspartialhashlibsha1updater=lenencode hexdigest)rz_kwargsrshas r?git_hashr_s$-0,<,<,F %(BG   W\\ 5W 5D &CJJxJJs3t9~$$&'JJuJJt ==?rBc\rSrSr%Sr\\S'\\S'\\S'\ SS\S\ 4S jj5r \ SS \S\ 4S jj5r \ SS \ S\ 4S jj5rSrg) UploadInfoi{a: Dataclass holding required information to determine whether a blob should be uploaded to the hub using the LFS protocol or the regular protocol Args: sha256 (`str`): SHA256 hash of the blob size (`int`): Size in bytes of the blob sample (`bytes`): First 512 bytes of the blob sha256sizesampleNrHfile_hash_infocU=(d [U5nUSnUSn[US5nURS5nSSS5 U"XCWS9$!,(df  N=f)N file_size file_hashrbrrr)ropenread)rhrHrrrfrs r? from_pathUploadInfo.from_paths['>=+>k*[) $ VVC[F#88 s A  Arzc\U=(d [U5nUSnU"[U5USSUS9$)Nrr)rrr)rr)rhrzrrs r? from_bytesUploadInfo.from_bytess4'>=+>[)D $t*SAArBfileobjcU=(d [U5nURS[R5 UR S5nURS[R5 U"USUSUS9$)Nrrrrr)rseekosSEEK_SETr)rhrrrs r? from_fileobjUploadInfo.from_fileobjsc-Gw1G Q $c" Q $!+. , rBrWrD)rYrZr[r\rkr=rlrbytesrrrrr rr^rWrBr?rr{s  K I M9S9$99BeBTBB 8TrBrcR\rSrSr%Sr\\S'\\\\ \ 4\S'\ "SSS9r \ \S'\ "\S9r\\S '\ "SSS S 9r\\\S '\ "SSS S 9r\\\S '\ "SSS S 9r\\\S'\ "SSSS 9r\\S'\ "SSSS 9r\\S'SSjr\S\\ 4Sj5rS\ 4Sjr\S\\4Sj5rSrg )CommitOperationAddizKData structure containing information about a file to be added to a commit. path_in_repopath_or_fileobjF)initrepr upload_info)default_factoryrN)rrdefault _upload_mode_should_ignore _remote_oid _is_uploaded _is_committedr8c[UR5Ul[UR[5(a[ UR5Ul[UR[ 5(az[ RR[ RRUR55n[ RRU5(d[SUS35eO:[UR[R[45(d [S5e[UR[R5(aEURR5 URR!S[ R"5 [UR[ 5(a0[(R+URUR,5Ulg[UR[5(a0[(R1URUR,5Ulg[(R3URUR,5Ulg![$[&4an[S5UeSnAff=f)z6Validates `path_or_fileobj` and compute `upload_info`.zProvided path: 'z(' is not a file on the local file systemzpath_or_fileobj must be either an instance of str, bytes or io.BufferedIOBase. If you passed a file-like object, make sure it is in binary mode.rzNpath_or_fileobj is a file-like object but does not implement seek() and tell()N)_validate_path_in_reporr<rr r=rrHnormpath expanduserisfiler>ioBufferedIOBasertellrSEEK_CUROSErrorAttributeErrorrrrrrr)rrexcs r? __post_init__ CommitOperationAdd.__post_init__s343D3DE d**D 1 1#&t';';#>/22 &&77_`3D0023D3De2LMM#$ $ d**B,=,= > > $$))+$$))!R[[9 d**C 0 0)33D4H4H484G4G ID  ,,e 4 4)44T5I5I595H5H JD  *66t7K7K7;7J7J LD ^,  d sAI##J3 I??Jc#R# [UR[5(d[UR[5(a$[ URS5nUv SSS5 g[UR[ 5(a#[ R"UR5v g[UR[ R5(aEURR5nURv URRUS5 gg!,(df  g=f7f)zq A context manager that yields a file-like object allowing to read the underlying data behind `path_or_fileobj`. rNr) r<rr=r rrrBytesIOrrr)rfileprev_poss r?as_fileCommitOperationAdd.as_files d**C 0 0J$$d5,5,d**D1T 21 ,,e 4 4**T112 2 ,,b.?.? @ @++002H&& &  % %h 2A 21sAD'DB:D' D$ D'cUR5n[R"UR55sSSS5 $!,(df  g=f)zC The base64-encoded content of `path_or_fileobj` Returns: `bytes` N)rbase64 b64encoderrrs r? b64contentCommitOperationAdd.b64contents- \\^t##DIIK0^^s $? A cURcgURS:XaURR$UR5n[ UR 55sSSS5 $!,(df  g=f)aReturn the OID of the local file. This OID is then compared to `self._remote_oid` to check if the file has changed compared to the remote one. If the file did not change, we won't upload it again to prevent empty commits. For LFS files, the OID corresponds to the SHA256 of the file content (used a LFS ref). For regular files, the OID corresponds to the SHA1 of the file content. Note: this is slightly different to git OID computation since the oid of an LFS file is usually the git-SHA1 of the pointer file content (not the actual file content). However, using the SHA256 is enough to detect changes and more convenient client-side. Nr)rrrrrrrs r? _local_oidCommitOperationAdd._local_oid sX    $   % '##** *4 , s A(( A6)rrr)r8N) rYrZr[r\rkr=rlrr rr rrrrrrr UploadModerboolrrrrrrrrpropertyrr^rWrBr?rrsU3eX566#U;K; 6ND6 */ *.L(:&. &+ &.NHTN. "'Et!LK#LEuEL$E UFM4F&LP3(+33 1E1-HSM--rBrrcURS5(aUSSnUS:XdUS:XdURS5(a[SUS35eURS5(aUS SnU$) NrU.z..z../z,Invalid `path_in_repo` in CommitOperation: ''z./) startswithr>)rs r?rr"ss###AB' sld2l6M6M 77:<. JL Lt$$#AB' rB)2rrrrrr contextlibr dataclassesrrrrpathlibr typingr r r r rrrrrrrmodelscope.hub.constantsrmodelscope.hub.utils.utilsrmodelscope.utils.file_utilsrrDEFAULT_IGNORE_PATTERNSrDATASET_LFS_SUFFIXMODEL_LFS_SUFFIXr2r`rnrrrr=rrrrCommitOperationrWrBr?rsZ %(====F85 CL _ % ,\ FggT ! !  ! H - -  - `     6  x x  x5S8 ++ +\ t-t- t-n   *+-rB