5pj"%SrSSKJr SSKrSSKrSSKrSSKrSSKrSSK r SSK r SSK r SSK r SSK r SSKJrJr SSKJr SSKJrJrJrJr SSKJr SS KJrJrJrJrJrJ r J!r!J"r"J#r#J$r$J%r%J&r&J'r'J(r(J)r)J*r*J+r+J,r,J-r-J.r.J/r/J0r0J1r1J2r2J3r3J4r4J5r5J6r6 SS K7J8r8J9r9J:r:J;r;Jr> SS K?J@r@ \(aSS KAJBrB SSKCJDrD SSKEJFrF \@"S5rG\H\-\I-\-\\I-rJSrK"SS5rLS*SjrMS+SjrNS,SjrOS-S.SjjrPS/SjrQS0S1SjjrR"SS5rS\SR\SR\SR\SR\SR\SR\SR\SR\SR\SR\SR\SR\SRS. r[S\\S'S2Sjr]"S S!5r^"S"S#5r_"S$S%5r`"S&S'5ra"S(S)5rbg)3uInternal file upload implementation. Strict behavioral parity with the old modelscope SDK upload pipeline: - LFS detection by suffix list + size threshold - Upload tracker for resumable uploads (.ms_upload_cache) - Adaptive batch sizing - Pipeline mode: ThreadPoolExecutor uploads + ordered batch commits - Per-file retry with exponential backoff - Per-commit retry - ReAct progressive retry fallback (parallel → serial → single-file) - Upload report ) annotationsN)ThreadPoolExecutor as_completed)Path)IO TYPE_CHECKINGAnyBinaryIO)tqdm)DATASET_LFS_SUFFIXDEFAULT_IGNORE_PATTERNSDEFAULT_MAX_WORKERSMODEL_LFS_SUFFIXUPLOAD_ADAPTIVE_BATCH_SIZE&UPLOAD_BATCH_CONSECUTIVE_FAILURE_LIMITUPLOAD_BLOB_MAX_RETRIESUPLOAD_BLOB_RETRY_BACKOFFUPLOAD_BLOB_RETRY_MAX_WAIT"UPLOAD_BLOB_TQDM_DISABLE_THRESHOLDUPLOAD_CACHE_FILEUPLOAD_COMMIT_BATCH_SIZEUPLOAD_COMMIT_MAX_RETRIESUPLOAD_COMMIT_MAX_TOTAL_WAITUPLOAD_FAILED_FILE_MAX_RETRIESUPLOAD_LEGACY_PROGRESS_FILEUPLOAD_LFS_ENFORCE_THRESHOLDUPLOAD_MAX_FILE_COUNTUPLOAD_MAX_FILE_COUNT_IN_DIRUPLOAD_MAX_FILE_SIZE#UPLOAD_NORMAL_FILE_SIZE_TOTAL_LIMIT!UPLOAD_REACT_BACKOFF_MAX_EXPONENTUPLOAD_REACT_ENABLEDUPLOAD_REACT_MAX_DELAYUPLOAD_REACT_ROUND2_BASE_DELAYUPLOAD_REACT_ROUND3_FILE_DELAYUPLOAD_USE_CACHEUPLOAD_VALIDATE_BLOB_BATCH_SIZE)FileIntegrityErrorHubErrorInvalidParameter NetworkError StorageError) compute_hash) get_logger) LegacyClient) OpenAPIClient) HubConfiguploadcN\rSrSrSrS SjrS S Sjjr\S Sj5rS Sjr Sr g)_CountedReadStreamZz?File wrapper that counts bytes read and updates a progress bar.cBXlX lX0lX@lSUlgNr)_file_expected_size_pbar _chunk_size _bytes_read)selffile_obj expected_sizepbar chunk_sizes R/var/www/html/pdf-tiff/venv/lib/python3.13/site-packages/modelscope_hub/_upload.py__init___CountedReadStream.__init__]s  + %cUS:a URO[XR5nURRU5nU(a;[ U5nU=R U- slUR RU5 U$r9)r=minr:readlenr>r<update)r?size read_sizechunkns rDrJ_CountedReadStream.readdsf(,qD$$c$@P@P6Q   * E A    !  JJ  a  rGcUR$N)r>r?s rD bytes_read_CountedReadStream.bytes_readmsrGcURUR:wa&[SURSURS35eg)NzUpload data incomplete: read z bytes, expected z2 bytes. File may have been modified during upload.)r>r;r)rTs rDverify_complete"_CountedReadStream.verify_completeqsR   t22 2$/0@0@/AB //01=>  3rG)r>r=r;r:r<N) r@r rAintrBr rCrZreturnNone))rMrZr[bytesr[rZr[r\) __name__ __module__ __qualname____firstlineno____doc__rErJpropertyrUrX__static_attributes__rGrDr6r6Zs(I  rGr6cU[:ag[U[[45(a#[U5RR 5OSnUS:Xa U[ ;$US:Xa U[;$U[:$)zIDetermine if a file should use LFS upload mode (suffix + size threshold).Tmodeldataset)r isinstancestrrsuffixlowerrr )pathrM repo_typeros rD_is_lfsrszsl ***4TC;*G*GT$Z   $ $ &RFG)))I+++ . ..rGc,[XU5(aS$S$)z)Return the commit/upload mode for a file.lfsnormal)rs)rqrMrrs rD _upload_moderwsD 225@@rGc`US::agUS::aU$US::a[S[SUS-55$g) z>Calculate optimal commit batch size based on total file count.rr di'@Pi)maxrI) total_filess rD_calculate_adaptive_batch_sizers?acf2s3 r 1233 rGc[U[5(a1U[R"U5R 5[ U5S.$[U[ [45(a@[U5nUR5Rn[US5n[ U5UUS.$UR5nU[R"U5R 5[ U5S.$)zCompute SHA256 hash and size for a file, bytes, or file-like object. Returns dict with 'file_hash', 'file_size', 'file_path_or_obj' keys. file_path_or_obj file_hash file_sizesha256) rmr^hashlibr hexdigestrKrnrstatst_sizer.rJ)rbuffer_size_mbrqrrdatas rD_compute_file_hashrs"E** 0 (89CCE-.  "S$K00$%IIK''  x0 #$4 5""   "D ^^D)335Y rGc>^U(dg[U4SjU55$)NFc3R># UHn[R"TU5v M g7frS)fnmatch).0patrqs rD $_matches_patterns..s>XcwtS))Xs$')any)rqpatternss` rD_matches_patternsrs  >X> >>rGc/nUHFnU(a[XA5(dMU(a[XB5(aM5URU5 MH U$)z;Filter file paths using fnmatch allow/ignore pattern lists.)rappend)itemsallow_patternsignore_patternsfiltereditems rD_filter_repo_objectsrsH H "3D"I"I  0GG   OrGcP\rSrSrSrSrSrSrSrSr Sr S r 1S kr \ SS j5rS rg )_ErrorCategorytransient_networktransient_server throttled auth_failed not_found file_invalid permanentunknown>rrrrcXR;$rS)_NON_RETRYABLE)clscategorys rD is_retryable_ErrorCategory.is_retryables1111rGrhN)rrnr[bool)rarbrcrdTRANSIENT_NETWORKTRANSIENT_SERVER THROTTLED AUTH_FAILED NOT_FOUND FILE_INVALID PERMANENTUNKNOWNr classmethodrrgrhrGrDrrsC+)IKI!LIGNN22rGr) E1001E1002E1003E1020E1021E1022E2020E3001E3002E3020E3021E3023E9001zdict[str, str]_CATEGORY_BY_ERROR_CODEc[U[5(aX[USS5nU(aU[;a [U$UR(a[ R $[ R$[U[5(a[ R$[U[R5(a[ R$[U[[45(a[ R$[U[[ 45(aa[#U5R%5nSU;dSU;a[ R$SU;dSU;a[ R$[ R$[ R $)uClassify an exception for retry strategy using the SDK error hierarchy. For :class:`HubError` instances, classification is driven by :attr:`~HubError.error_code` via a lookup table — no ``isinstance`` chains or string-matching heuristics needed for known error types. error_codeNz size changedz no such file permissionz access denied)rmr*getattrr retryablerrrFileNotFoundErrorr _builtinsPermissionErrorConnectionError TimeoutErrorrIOErrorOSErrorrnrp)errorcode error_strs rDclassify_errorrs%""ulD1 D33*40 0).~%%Vn>V>VV%*++***%2233***%/<899///%'7+,,J$$& Y &.I*E!.. . 9 $9(D!.. .///  ! !!rGc \rSrSrSrSrSrSrg) FileStatusiucfrhN)rarbrcrdUPLOADED COMMITTEDFAILEDrgrhrGrDrrsHI FrGrc\rSrSrSrSSjr\SSj5rSSjrSSjr SSjr SSjr SS jr SS jr SSS jjrSS jrSS jrSSjrSSjrSSjrSrg) UploadTrackerizIPersistent JSON cache at {folder}/.ms_upload_cache for resumable uploads.c[U5UlX l0Ul[R "5UlSUlUR5 gNF) r_path_repo_id_files threadingLock_lock_dirty_load)r? cache_pathrepo_ids rDrEUploadTracker.__init__s8*%  ') ^^%   rGcUSUSU3$)N|rh)rel_pathmtimerMs rD _make_keyUploadTracker._make_key"s1UG1TF++rGcURXU5nUR URRU5nSSS5 WbSU;agUUSUSS.$!,(df  N$=f)NhashrMrrrrgetr?rrrMkeyentrys rDget_hashUploadTracker.get_hash&sbnnXd3 ZZKKOOC(E =F%/ (vv  Zs A A'cURXU5nUR URRU05nUSUS'USUS'X`RU'SUlSSS5 g!,(df  g=f)NrrrrMT)rrrrr)r?rrrM hash_inforrs rDput_hashUploadTracker.put_hash2sfnnXd3 ZZKKOOC,E%k2E&M%k2E&M$KK DK ZZs AA** A8cURXU5nUR URRU5nSSS5 WSL=(a" URS5[R :H$!,(df  N;=fNstatus)rrrrrrrs rD is_committedUploadTracker.is_committed;s\nnXd3 ZZKKOOC(ED PUYYx%8JcURXU5nUR URRU5nSSS5 W(aURS5$S$!,(df  N(=frrrs rD get_statusUploadTracker.get_statusAsNnnXd3 ZZKKOOC(E&+uyy"55Zs A A+cURXU5nUR X@R;a'[RURUS'SUlSSS5 g!,(df  g=fNrT)rrrrrr)r?rrrMrs rD mark_uploadedUploadTracker.mark_uploadedGsMnnXd3 ZZkk!-7-@-@ C *" ZZs 7A A-cUR UHIup#nURX#U5nXPR;dM)[RURUS'MK SUlSSS5 g!,(df  g=fr )rrrrrr)r? file_keysrrrMrs rDmark_committed_batch"UploadTracker.mark_committed_batchNs^ ZZ)2%nnXd;++%1;1E1EDKK$X.*3DK ZZs(A-+A-- A;chURXU5nUR XPR;a9[RURUS'U(aX@RUS'O+S[R0nU(aXFS'X`RU'SUlSSS5 g!,(df  g=f)Nr error_typeT)rrrrrr)r?rrrMrrrs rD mark_failedUploadTracker.mark_failedVsnnXd3 ZZkk!-7->-> C *5?KK$\2)1:3D3D(E*4,'#( C DKZZs A;B## B1c nUR UR(d SSS5 g[URURR 5VVs0sHupU[ U5_M snnS.nSUlSSS5 URRRSSS9 [R"[URR5SS9upE[R"USSS 9n[R "WUSS 9 SSS5 [R""U[UR55 gs snnf!,(df  N=f!,(df  NP=f![$a [R&"U5 ef=f![(a n[*R-S U5 SnAgSnAff=f) N)versionrfilesFT)parentsexist_okz.tmp)dirrowzutf-8)encoding) ensure_asciiz!Failed to save upload tracker: %s)rr_TRACKER_VERSIONrrrdictrparentmkdirtempfilemkstemprnosfdopenjsondumpreplace BaseExceptionunlink Exceptionloggerwarning)r?kvrfdtmp_pathres rDsaveUploadTracker.savedsI ZZ;;Z,==151B1B1DE1D!T!W*1DED  DK C JJ   # #D4 # @#++DJJ4E4E0FvVLB YYr39QIIdAE:: 8S_5F Z:9!  (#  C NN> B B CseE-ED= . EAF E%3E 2E%=E E E"E%%"FF F4F//F4c URRSS9 UR  UR R5 SUlSSS5 g![a n[R SU5 SnAN\SnAff=f!,(df  g=f)NT) missing_okz!Failed to delete tracker file: %sF) rr+rr-r.rrclearr)r?r3s rDr8UploadTracker.clear{sp C JJ    .ZZ KK   DKZ C NN> B B C Zs"A"A? A<A77A<? B cNURR5(dUR5 g[UR5n[R "U5nSSS5 WRS5nUcURU5 gURSS5nU(a1XPR:wa"[RSUUR5 gURS05Ul [SURR555nUS:a*[R!S [#UR5U5 UR5 g!,(df  GN=f![R [4a n[RSU5 SnAgSnAff=f) Nz1Failed to load upload tracker, starting fresh: %srrrjzKTracker repo_id mismatch (cached: %s, current: %s), ignoring stale tracker.rc3n# UH+oRS5[R:XdM'Sv M- g7f)rr N)rrr)rr3s rDr&UploadTracker._load..s(i)=AxT^ThThAhaa)=s&5 5rz0Upload tracker loaded: %d entries, %d committed.)rexists_check_legacy_progressopenr'loadJSONDecodeErrorrr-r.r _migrate_v1rrsumvaluesinforK)r?rrr3r stored_repocommitted_counts rDrUploadTracker._loadsBzz  ""  ' ' )  djj!Qyy|" ((9% ?   T " hhy"- ;--7 NN]   hhw+ i););)=ii Q  KKBDKK   ##%9"!$$g.  NNNPQ R  s5E*EE* E'"E*'E**F$FF$c0nUR5H>up4[U[5(dMSU;dM$USURSS5S.X#'M@ X lSUlU(a [ RS[U55 gg)Nrrr)rrMTz2Migrated %d entries from legacy hash cache format.) rrmr rrrr-rErK)r?rmigratedrvalues rDrBUploadTracker._migrate_v1sx**,JC%&&;%+?!+.!IIk15! '    KKDH   rGcURR[- nUR5(a[R SU5 gg)Nz`Legacy upload progress file detected: %s. This file is no longer used. You may delete it safely.)rr!rr=r-r.)r? legacy_paths rDr>$UploadTracker._check_legacy_progresss=jj''*EE      NNr  rG)rrrrrN)r str | Pathrrnr[r\)rrnrfloatrMrZr[rn)rrnrrQrMrZr[z dict | None rrnrrQrMrZrr r[r\rrnrrQrMrZr[r)rrnrrQrMrZr[ str | NonerrnrrQrMrZr[r\)rzlist[tuple[str, float, int]]r[r\rj rrnrrQrMrZrrnr[r\r`)rr r[r\)rarbrcrdrerE staticmethodrrrrrr rrr4r8rrBr>rgrhrGrDrrs]S,,  Q 6 # C. !&F rGrcv\rSrSrSrSSjrSSjrSSjrSSjrSSjr SSjr SSS jjr SS jr SS jr S rg ) NullTrackeriz+No-op tracker for when caching is disabled.cgrSrhr?rrrMs rDrNullTracker.get_hashrGcgrSrh)r?rrrMrs rDrNullTracker.put_hash rGcgrrhr\s rDrNullTracker.is_committedsrGcgrSrhr\s rDrNullTracker.get_statusr^rGcgrSrhr\s rDr NullTracker.mark_uploadedrarGcgrSrh)r?rs rDr NullTracker.mark_committed_batchrarGcgrSrh)r?rrrMrs rDrNullTracker.mark_failedrarGcgrSrhrTs rDr4NullTracker.saverarGcgrSrhrTs rDr8NullTracker.clearrarGrhNrUrRrS)rlistr[r\rVrWr`)rarbrcrdrerrrrr rrr4r8rgrhrGrDrZrZs25      rGrZcr\rSrSrSrS Sjr\SSj5rSSjrSSjr SSjr SSjr SS jr SS jr S rg ) BatchTrackeriz4Thread-safe tracker for pre-assigned upload batches.c^X lUS:a US- U-S-OSUl[UR5Vs/sHn/PM snUl[UR5Vs/sHn/PM snUl/Ul[UR5H2nXB-n[ XR-U5nUR RXe- 5 M4 [UR5Vs/sHn[R"5PM snUl [R"5Ul gs snfs snfs snf)Nrr ) _batch_size _num_batchesrange_batch_results_batch_failures_batch_expectedrIrrEvent _batch_eventsrr)r?r~ batch_size_istartends rDrEBatchTracker.__init__s%CNQR?[1_;a?XY=B4CTCT=U0V=U=U0V?DTEVEV?W2X?W!2?W2X*,t(()ANEe(+6C  ' ' 4*QVVZVgVgPh4iPh1Y__5FPh4i^^% 1W2X 5js D ! D%D*cUR$rS)rurTs rD num_batchesBatchTracker.num_batchess   rGcXR-$rS)rt)r? file_indexs rD batch_indexBatchTracker.batch_indexs----rGcURU5nUR URURU5 UR U5(aUR UR 5 SSS5 g!,(df  g=frS)rrrwr_is_batch_completer{set)r?rresultidxs rDrecord_successBatchTracker.record_successsdz* ZZ    $ + +F 3&&s++""3'++-ZZs AA99 BcURU5nUR URURX#45 UR U5(aUR UR 5 SSS5 g!,(df  g=frS)rrrxrrr{r)r?rrrrs rDrecord_failureBatchTracker.record_failuresfz* ZZ   % , ,d] ;&&s++""3'++-ZZs AA:: BcURU5nUR URU==S-ss'URU5(aURUR 5 SSS5 g!,(df  g=fNr )rrryrr{r)r?rrs rDmark_file_skippedBatchTracker.mark_file_skipped saz* ZZ   % * %&&s++""3'++-ZZs A A22 BcURUR5 UR [URU5[UR U54sSSS5 $!,(df  g=frS)r{waitrrprwrx)r? batch_idxs rDwait_for_batchBatchTracker.wait_for_batchsU 9%**, ZZT((34T)))45ZZs 0A$$ A2c[URU5[URU5-nX RU:$rS)rKrwrxry)r?rcounts rDrBatchTracker._is_batch_completes@D'' 23c$:N:Ny:Y6ZZ,,Y777rG)r{ryrxrwrtrruN)r~rZr|rZr[r\r_)rrZr[rZ)rrZrr r[r\)rrZrtuplerr,r[r\)rrZr[r\)rrZr[ztuple[list[dict], list[tuple]])rrZr[r)rarbrcrdrerErfrrrrrrrrgrhrGrDrrrrs<> &!!....8rGrrc \rSrSrSrS!SS.S"SjjjrSSSSS S .S#S jjrS SSSSSS\S S S . S$SjjrS%Sjr \ S&Sj5r SSS S.S'Sjjr S SSSS.S(Sjjr S)SjrS\S.S*SjjrS+SjrS,SjrS-SjrS-SjrS.S/SjjrS0S1SjjrS0S2SjjrS rg)3 UploadManageri"zKInternal file upload implementation with production-grade retry and resume.N)create_repo_fnc4XlX lX0lX@lgrS)_client_config_openapi_create_repo_fn)r? legacy_clientconfigopenapi_clientrs rDrEUploadManager.__init__%s%  & -rGz Upload filemasterF)commit_messagecommit_descriptionrevisionr disable_tqdmc  Uc [S5e[U[[45(an[R R [R R[U555nU=(d [R RU5nOU(d [S5e[X85n U Sn U Sn [U[[[45(dU SnU=(d SUS3nURbURX5 [XLU5n U S :XaURUUU U UU S U3US 9nOSS S S S.nURUUU U USS9n[SUS3S S9 UR R#UUU/UUS9$)z%Upload a single file to a repository.Nz#Path or file object cannot be None!z#Arg `path_in_repo` cannot be empty!rrrzUpload z to ModelScope hubruz [Uploading] )rrrrrMrr tqdm_descrTFurl is_uploaded is_reusedis_blob_uploadedr path_in_repopath_or_fileobjr upload_moderzCommitting file to z ...)flushrrr operationsrr)r+rmrnrr%rqabspath expanduserbasenamerr^rrw _upload_blob_build_operationprintr create_commit)r?rrrrrrrrrrrrrr upload_res operations rD upload_fileUploadManager.upload_file5s  ""#HI I oT{ 3 3 ggoobgg.@.@_AU.VWO'L277+;+;O+LL&'LMM&G k* k* /Cu+=>>'(:;O'UW\NBT+U    +   4"||))!{) *  rGrj) rrrrrr max_workers use_cachersync_remote_repoc z^^^^ ^O^P^Q[R"5nT(d [S5eUc [S5eU c[n U=(d SnU c/n O$[U [5(aU /n O [ U 5n U [ - n UbU Vs/sH oU;dM UPM n nUbUOSTS3nU=(d Sn[[/nUVVs/sHnUSU34HoPM M nnnU U-n [RS5 TRUUTUU S 9nU (a+TRUUTSSS 9nUVVs1sHunnUiM nnnO [5nU(d[S US 35e[RS [U55 TRbTRTT5 [US S9n[ (a5[#[U55n[RSU[U55 O[$S:a[$O [U5n['U5R)5nU (aU[- n[+UTS9mQO [-5mQ[/[U5U5mO/n[5mP[1U5Hununn[2R4"U5nTQR7UUR8UR:5(a$TPR=U5 TOR?U5 MpUREUUU445 M 0n 0n!UHgununn[2R4"U5nTQRGUUR8UR:5n"U"b[IUU"ST5S:XaU"U4U!U'MgMi U!(aU!RK5V#Vs/sHun#nU#SU#SS.PM n$n#nTRMTTU$S9n%U%n [OSU%RK555n&[RS[U$5U&[U$5U&- 5 [TP5n'U'S:a[RSU'5 [RS[U5U'[U55 [RS[U5TORPU5 STSUUOU UUUUQ4Sjjjn(/n)/n*/n+TORPn,[SU S9n-UHCunn.Sn/UU!;a!U!USSn0U RUU05n/U/cSn/U-RWU(UU.U/5 ME Sn1[Y[[U,5S U,T S!9GHvn2U2U-n3[]U3U-[U55n4[_UP4S"j[[U3U4555(a[RS#U2S$-U,5 MaTORaU25un5n6U6(a7U+RcU65 U6H un7n8[ReS%U7SU85 M" TRgTQU55 TRiU5T5n9U9(d[ReS&U2S$-U,5 MUS'U2S$-S(U,S)3n:TRkTTU9U:US*9n;U)REU;5 U*RcU55 [RS+U2S$-U,[U555 TRmTQU55 Sn1GMy SSS5 TQR}5 U+(aH[~(a=TRU+TQTTUUU T S79un+n>n?U)RcU>5 U*RcU?5 OU+(aTRU+TQTTUUUU)U*T S89 n+TQR}5 Sn@U (aU+(dU(aURS(5OS9nATRTTUUUAS:9nBUB(ahTRWB5nCUS;[UB5S<3nDTRkTTUCUDUS*9nEU)REUE5 [UB5n@[RS=U@5 [R"5U- nF[U5nG[U+5nH[OS?U*55nI[OS@U*55nJ[OSAU*55nK[U*5nL[SB5 [SC5 [SD5 [SEUG35 [SFU'35 [SGUK35 [SHUI35 [SIUJ35 [SJUH35 [SKUL35 [SLW@35 [SMUFSNSO35 [SB5 U+(aPU+H4uunMnn8[ReSPUM[U85RU85 M6 WGWH- nN[UHSQUNSR35eU)(d&U'[U5:Xa[RSS5 gg[U)5S$:XaU)S$U)$s snfs snnfs snnf![@a"n[RCSUU5 SnAGNSnAff=f![@a GMf=fs snn#f![nGan[ReS,U2S$-U,U5 [qU5n<[rRuU<5(dRU5H!n=TQRwU=S-U=S.U=S/S0U<-S19 M# [ReS2U2S$-U,U<[U555 U1S$- n1ONU5Hn=U+REU=S-U=S34U45 M [RCS4U2S$-U,[U55U<5 U1S$- n1U1[x:a[{S5U1S6U35eSnAGMSnAff=f!,(df  GNk=f!TQR}5 f=f![na!n[ReS>U5 SnAGNoSnAff=f)VzEUpload a folder with resumable support, adaptive batching, and retry.z"The arg `repo_id` cannot be empty!Nz%The arg `folder_path` cannot be None!z Upload to z on ModelScope hubzUploading filesz*/zPreparing files to upload ...) folder_pathrrrrrz"No files to upload in the folder: z !zChecking %d files to upload ...c US$r9rh)xs rD-UploadManager.upload_folder..s!A$rG)rz&Adaptive batch size: %d (for %d files)r)rz'Cannot stat file %s, will re-upload: %srruroidrMrrrobjectsc3.# UH obMSv M g7frrh)rr0s rDr.UploadManager.upload_folder..sD$6q$6s zKPre-validated %d cached LFS hash(es): %d globally existing, %d need upload.z'%d file(s) already committed, skipping.zUup4[RSU5 T RUUT T T UTS9n[RSU5 TRX5 g![a2n[R SX65 TR XU5 SnAgSnAff=f)NzUploading: %s ...)rrrtracker pre_validatedrz Uploaded: %szUpload failed: %s - %s)r-debug_upload_single_filerr,rr) file_idx file_inforpath_in_repo_w file_path_wrr3 batch_trackerrrrrr?rs rD_upload_worker3UploadManager.upload_folder.._upload_worker0s*3 'N E 0.A11"#'#"/!-2 ^^<,,X> E 5~I,,X!DD EsAA B$(BBrTz[Committing batches])desctotaldisablec3,># UH oT;v M g7frSrh)rr~skipped_indicess rDrr]sW9VA/9Vz&Batch %d/%d fully committed, skipping.r z Failed: %s - %sz/Batch %d/%d: all files failed, skipping commit.z (batch /)rz"Batch %d/%d: committed %d file(s).zBatch %d/%d commit failed: %sfile_path_in_repo file_mtimefile_size_on_diskcommit_rzDBatch %d/%d: permanent failure (%s), %d file(s) will not be retried. file_pathzEBatch %d/%d: %d file(s) recovered to retry queue (error_category=%s).zUpload aborted: z/ consecutive batch commits failed. Last error: ) failed_filesrrrrrrrr) rrrrrrrr commit_infos all_resultsrrj)rrrrlocal_paths_in_repopath_in_repo_prefixz (sync: delete z orphan file(s))z,Sync: deleted %d orphan file(s) from remote.zSync delete commit failed: %sc3# UH5oRS5S:XdMURS5(dM1Sv M7 g7f)rrurr Nrrrs rDrrs2n+Q}9MQV9Vq[\[`[`al[mqq+s ?? ?c3T# UHoRS5(dMSv M g7f)rr Nrrs rDrrs UKq55AS;TKs( (c3R# UHoRS5S:XdMSv M g7f)rrvr Nrrs rDrrs VkUU=5IX5U11k' 'z<============================================================z Upload Reportz<------------------------------------------------------------z Total files : z Skipped (cached) : z Normal committed : z LFS existed : z LFS uploaded PUT : z Failed : z Committed : z Deleted (sync) : z Elapsed : .1fsz - %s: %s: %szL file(s) failed to upload. Please manually try again. Successfully uploaded z8 file(s) will be automatically skipped during the retry.z!All files were already committed.rS)rrZrrrr r[r\)Itimer+rrmrnrprrrr-rE_prepare_upload_folderrrKrsortedrrrrresolverrZrr enumerater%rrst_mtimeraddrrr.rrrwrD_validate_blobs_batchrCrrrsubmitr rvrIallrextendr_track_uploaded_batch_build_batch_operations_commit_with_retry_track_committed_batchr,rrrrr RuntimeErrorr4r#_retry_failed_files_react_retry_failed_simplestrip_compute_remote_orphans_build_delete_operationsrtyperar-)Rr?rrrrrrrrrrrrrr start_timep_internal_filesr_internal_ignore sorted_filesall_local_files_in_repor}all_local_paths_in_repocommit_batch_sizefolder_path_resolvedrfiles_to_uploadrrrstr3pre_validated_maplfs_hash_info_mapcachedrEr validatedreused skipped_countrrrtotal_failed_filesrexecutorrpv cached_hashconsecutive_failuresr batch_start batch_endresultsfailuresrerrrbatch_commit_message commit_inforr react_commits react_results deleted_countprefixorphans delete_opsdelete_commit_message delete_commitelapsedr~ failed_countlfs_reused_countlfs_uploaded_count normal_countrGpath_in_repo_f succeededrrrsR``` ` @@@rD upload_folderUploadManager.upload_folders $YY[ "#GH H  "#JK K  -K(/4  " O  - -./O"?3O22  %*9U/Qn=Tq/OU+9+EZX_W``rKs/D3D-.IJ'6N!bH 1A AN),<<  3422#%)+ 3  &*&A&A')## $ 'B' #6M&M5LTQq5L #&M #&)e #"%G }TV#WX X 5s<7HI    +  ) 4l?  & % >s@$'E8A,8O 4H4() WWY'''(92;; SS#''1!33H=T  " "H/@).L#M N9P 46DF8G 4H4() WWY' ))*;R[["**U%$-";/% ! !8>rl)(3& 9H$ \m\t\t\vw\vQXQUWXtK0${:KL\vGw227iah2iI ) DI$4$4$6DDF KK]G G v%  O, 1  KKA= Q J        N   % %    E E($& "$ *,#// p # <+:'Hi,0B#44&7&A!&D[&Q .22;?:!%BOONHiL,;()$!%+&/%( "I #,.?"?K #K2C$CSEV WIW{I9VWWW D%M' !(5(D(DY(O%GX*11(;)1ID#"LL)' %++K8#**73 @%M'L  33GWE/0,i"=^ LLN "6"6?C?]?]/#-!')@^ @ <  }    .   } - !%!:!:/#-#5!)')"; "    $60<\'',"F22#!$;$* 3G!::7C +9*:/#g,Wg(h%E$($;$; '"+#-'(#S -~tCy?Q?QSVW-?#l2I.!D+$% L 11 ?@"%l"3q"8|AJlJq VO.'NZ =% 6  xp%- ;%M'  $2!#4-::8DD%, ' 3 3$%&9$:$%lO$%&9$:/88/C !4!"&-#LL f )A + ( #G 1A50%, 2 9 9)*+>)?;(P()%&!"&-#NN g )A + #G ( 1A50/3YY"."23G2HI//0c!3#ZS-A=<^ LLNx!ELL!@!DDEs e;e;f0f;A$f  Af;g  k:(E)k(A-g?k(k:Al f8f33f8; g  g  k%C;k k( k%%k(( k72k::l  l:l55l:c[RS5 URRUUUSS9n/nUHnUR S5=(d UR S5=(d Sn U S:XaM=UR S5=(d2 UR S 5=(d UR S 5=(d S n U (dMUR U 5 M U(a.US -n UV s/sHoR U 5(dMU PM nn UV s/sH oU;dM U PM n n [RS [U5[U 55 U $s sn fs sn f)zCompute remote files that are not present locally (orphans). Only files under ``path_in_repo_prefix`` are considered. Returns a list of remote file paths to delete. z8Sync: fetching remote file list for orphan detection ...T)rrrr recursiveTyperblobtreerrqNamerjrz8Sync: %d remote file(s) in scope, %d orphan(s) detected.)r-rErlist_repo_filesrr startswithrK)r?rrrrrr raw_items remote_pathsr item_typerq scope_prefixrr:s rDr%UploadManager._compute_remote_orphans%s*  NOLL00 1 #% D(FDHHV,<FIF"88F#Qtxx'7Q488F;KQrD    % .4L'3R|!||L7QA|LR*Kl7J.J1lK F   L SKs+EE E!Ec BUVs/sH nSUSSSSSS.PM sn$s snf)z0Build delete operations for orphan remote files.deletervrrjactionrqrrMrcontentrrh) orphan_pathsrs rDr&UploadManager._build_delete_operationsQsD"  "#  "   s)rrrc Uc [5nSnSn [U[[R45n U (aF[R "U5n UR XRU R5n U bU nX(S'UcR[US9nU (aBU c[R "U5n URUU RU RU5 U cU (a[R "U5n USn USn [XU5nUS:XaSn[[5Hn[U[[R45(aB[RR![U55nUU :wa[#SU SUSU35eUR%UUU U UU=(d U [&:*S US 3US 9n O [?S[SUSU35Ue[U[[R45(aB[RR![U55nUU :wa[#SU SUSU35eSSSSS.nUUU (a U ROSU (a U ROURASS5USURASS5URASS5UUS. $![a Sn GN7f=f![a GNf=f![a GNf=f![([*[,4an[U[(5(aUR.(deUnU[S - :aU[1[2U-[45n[6R9S US -[UUU5 [:R<"U5 SnAGMZSnAGMaSnAff=f)Nr)rrrruz.File size changed since hash computation: was z, now z. File may have been modified: z [Uploading ])rrrrrMrrrrr z@Blob upload attempt %d/%d failed for %s: %s, retrying in %ds ...zBlob upload failed after z attempts for z: TFrrrrr) rrrrrrrrfile_hash_info)!rZrmrnr%PathLikerrrrrrrrwrvrrqgetsizer+rrr*rrrrIrrr-r.rsleepr-r)r?rrrrrrrr hash_info_d file_stat is_real_pathr&rrr last_errorattempt current_sizerr3rs rDr!UploadManager._upload_single_fileds ?!mG  !)c2;;-?@  !GGI.  ))*;=O=OQZQbQbc%"(K6? 23  ,iHK  ($&GGI$6 $$)!**!))#    GGI. %[1 $[1 "#4K % J !89%)!)c2;;-?@@')wws9~'F '94"2#''0k ~F@@Q?R!T# "&!2!2 '"+(&&&2&eiCe6e$/0A/B!"D&3"3 "J):P#/0G/HWhViiklvkwx!")c2;;%788!wws9~> 9,*(k ~>88I7JL #"$) J"3"09),,q7@)"3"3kooVacdFe%m4# U; */A5 I&)  w !  !   @!/<@)!!X..q{{!"J!81!<<"5w>6 ^#aK3-  4((= )sQAJ'AJ:K &BK' J76J7: KK KKN1BNNz [Uploading])rrrrc SSSSS.n U SLa%[RSUSS5 SU S'SU S'U $[U [5(aU n OYURR UUX4S./S 9n U R U5n U c%[RSUSS5 SU S'SU S'U $U n US -S -n[US SUUS 9n[U[[45(aN[US 5n[UXOU5nURRU UUS9 SSS5 WR5 O[U[5(aM[[R"U5XOU5nURRU UUS9 UR5 O7[XTX5nURRU UUS9 UR5 SSS5 XS'SU S'SU S'U $!,(df  N=f!,(df  N/=f)NFrTz'Blob %s already exists globally, reuse.rrrrB)runit unit_scalerrrb) upload_urlrrMrr)r-rErmrnrvalidate_blobsrr rr?r6 upload_blobrXr^ioBytesIO)r?rrrrrMrrrrrres_dror' maybe_urlrCrBrstreams rDrUploadManager._upload_blobs   %   D KKA6"1: N#'E- !%E+ L mS ) )+J 33#!'674I " f-I  EvbqzR'+m$%)k" "J#d*T1   $d ,,$%/4zJFLL,, VZ,[&&&(D%((+BJJt,# UH oT;v M g7frSrh)rrrs rDr3UploadManager._commit_with_retry.._sJ7I!I~7Ir<zRCommit total wait time would exceed %ds (already %.1fs elapsed), aborting retries.z4Commit attempt %d/%d failed: %s, retrying in %ds ...r zCommit failed after z attempts: )r monotonicrvrrr*rrnrrrr,rIrr-rr.r`rmr,)r?rrrrrrrzrdrrer3retryable_patternsrr>rs @rDr UploadManager._commit_with_retryFse(, ^^% [)G ||11#')#1% 2*N j( + + 1+k*VW]ggA {{ #AI*A;)O&J7IJJJ #\2     q'z2&Dnn&3G~ << h0  NNF!    JJt s/A22 C1<=B>>C1C C1%C,,C1cUS:XaSUSUSUSSSS.$[U[5(aUnOf[U[[45(a[U5R 5nO1UR 5nUR 5nURU5 SUSUSS[R"U5R5SS.$) NrucreaterrrjrVrvbase64) rmr^rnr read_bytestellrJseekr b64encodedecode)r?rrrrr content_bytesposs rDrUploadManager._build_operations % "$!+.#K0 /511 / Oc4[99 $_ 5 @ @ B %**, / 4 4 6 $$S)"$ !+.!++M:AAC$ rGc /nUHbnUSnUSnURS5=(d [USUSU5nURUSUUUUSS9nURU5 Md U$)Nrr]rrrrr)rrwrr) r?r1rrritem_drrrops rDr%UploadManager._build_batch_operationss F{+I/0I **]3|*++&8K &&#$78 )#'"=1 'B   b ! rGcnUHnURUSUSUS5 M! UR5 gNrrr)r r4r?rr1rs rDr#UploadManager._track_uploaded_batchs9 A  ! !!$7"8!L/1M`Ka b rGc URUVs/sHo3SUSUS4PM sn5 UR5 gs snfr)rr4rs rDr$UploadManager._track_committed_batchsM $$X_ `X_ST#$a oq9L7M NX_ `    as?c [U5R5R5nUR5(d[ SUS35e[ SUR S555n[U5[:a[ S[U5S[35e0nUH/n [U R5n URU S5S-X'M1 UR5H'upU [:dM[ SU S U S [35e Sn SnUHrn U R5RnU[ :a'[ S U S US - S - SS[ S - S - SS35eX- n [#[U 5X5(aMnX- nMt U[$:a[&R)SU[$5 UV s0sH+oR+U5R-5[U 5_M- nn [/[1UR355UUS9nU(aUR5S5S3OSnUVs/sH nUU-UU4PM nn[&R7S[U55 U$s sn fs snf)NzProvided path: 'z' is not a directoryc3R# UHoR5(dMUv M g7frS)is_file)rrqs rDr7UploadManager._prepare_upload_folder..sR,?D<<>44,?rz**/*zToo many files (z) in folder, max allowed: rr z) in directory z, max allowed per directory: zFile too large:  (rjrz MB), max allowed: z.0fz MBzjTotal normal (non-LFS) file size %d bytes exceeds soft limit %d bytes. Consider using LFS for large files.)rrrrjzPrepared %d files for upload.)rrris_dirr+rglobrKrrnr!rrrrrr rsr!r-r. relative_toas_posixrrpkeysrrE)r?rrrrrrfolder all_files dir_countsrqr!dir_pathr total_size normal_sizefsizerelpath_to_abspath filtered_keysr9relpathprepareds rDr$UploadManager._prepare_upload_foldersk"--/779}}"%5fX=Q#RS SRFKK,?RR y>1 1"%5c)n5EE_`u_v#wx x&( D%F!+!:Q!>J  *//1OH33&&ug_XJG22N1OQ 2  DIIK''E++&&tfBut|d/B3.GH$$84$?$$Fs#K3P  J3t9e77$  < < NNF3  ZccYbQU..v6??A3t9LYbc, #((* +)+ 3?L&&s+,A.BS`aS`Vg%'9''BCS`a 3S]Cdbs 52I#,I(c /n /n 0n /n [U5n /nU HnUuunnn[U5n[RU5(aUR U5 MCU R U5 [ U[ [R45(a[R"U5OSnURUU(a UROSU(a UROSUS9 [RSUUU5 M Un SS[!SUS-5SSS .S S SS ["S .S S SS[$S ./n['U5GHyunnU (d GOmUSn[R)SU[+U 55 /n/nUS(a[+U 5S:a[-USS9n0nU H/uunnnUR/UR0UUUUUUS9nUU4UU'M1 [3U5H-nUUunnUR55nUR U5 M/ SSS5 O['U 5Hun!uunnnUSS:agU!S:aaUS:XaUSS[9U![:5--OUSn"[9U"[<5n"[R)SU"U5 [>R@"U"5 UR1UUUUUUS9nUR U5 M U RCU5 [9US[!S[+U555n#[ES[+U5U#5Hn$UU$U$U#-n%URGUU%5 URIU%U5n&U&(dM8URKUUU&USUS3US9n'U R U'5 URMUU%5 [R)SU[+U%55 M /n)UGHnUuunnnU ROUS5S-U U'U US":aU R U5 [ U[ [R45(a[R"U5OSnURUU(a UROSU(a UROSS#S9 [RS$U5 M[U5n[RU5(aU)R U5 GMU R U5 [ U[ [R45(a[R"U5OSnURUU(a UROSU(a UROSUS9 [RS%UU5 GM [+U 5[+U)5- n*U*S:a"[R)S&UU*[+U)55 OU)(a[RQS'U5 U)n GM| X-n+U (a[RS([+U 55 U+X4$![a SnGNHf=f![6a!n UR UU4U 45 Sn A GMCSn A ff=f!,(df  GNo=f![6a9n [RSUUU 5 UR UU4U 45 Sn A GMVSn A ff=f![6an [RSUU 5 [U 5n[RU5(d.U%H!n(URU(SU(SU(SS U-S9 M# Sn A GMU%Hn(UR U(SU(S!4U 45 M Sn A GMSn A ff=f![a SnGN f=f![a SnGNGf=f))Nrrz&[ReAct] Permanent failure: %s (%s: %s)zRound 1 (parallel)Tr r~r)nameparallelworkersr|delayzRound 2 (serial+backoff)FrizRound 3 (single-file)rz#[ReAct] %s: retrying %d file(s) ...rrrrrrrrrz*[ReAct] Waiting %ds before retrying %s ...z[ReAct] %s: failed %s - %sr|rrrz![ReAct] %s: committed %d file(s).z[ReAct] %s commit failed: %srrrrrr4max_retries_exceededz#[ReAct] Max retries exceeded for %sz"[ReAct] Permanent failure: %s (%s)u@[ReAct] %s: made progress — %d file(s) resolved, %d remaining.z2[ReAct] %s: no progress, escalating to next round.z8[ReAct] %d file(s) still failing after all retry rounds.))rprrrrrmrnr%r^rrrrrr-rr}r%r&rrErKrr rrrr,rIr"r$rr`r rvrrrrrr.),r?rrrrrrrrrr all_successes retry_countspermanent_failuresr remainingitem_errpath_in_repo_r file_path_rr3rr# round_configs round_idxcfg round_nameround_successesround_failuresr+ future_map_errfuturerr3r~rr|r/batchrr5r new_retryableprogress all_failuress, rDr'UploadManager._retry_failed_files_reacts$& $& ') *,&  !H19 . )^[3%c*H**844  *"))(31;K#r{{I[1\1\-bfB##"#%BKK1"$BJJ!' $  <" #". - q+"23   3!7  0!7  / 0( 6NIsVJ KK5I  +-O*,N:3y>A#5'C NCx')J?H;5t!) 44*'$+&/$+)5"1"/=k-J 6*@I#/z":6@6H3 V%+]]_F+226: #;DC*AJ)@Tn[[\_L]]^,_%- '>' %++K8#**?;33GM E'!O0>"0 OAP"!e!NLL!:NAN"))NK+H!*LMMN.%" @'!O $2!#4-::8DD*9 ' 3 3$*+>$?$*<$8$*+>$?/88/C !4!"+:+: . 5 5-33F,G,2JJ{B,G)*)* %&!"+:!"s8!%D6;A2E;6 E8,E33E8; H+A&H&01H&&H+)rrrrrS) rr0rr2rzOpenAPIClient | Nonerr r[r\)rrnrrrnr PathOrFileObjrrnrrnrrTrrnrrZrrr[r )rrnrrrnrrPrrnrrTrrTrrnrlist[str] | Nonerrrz int | Nonerrrrrrr[zdict | list[dict] | None) rrnrrrnrrnrzset[str]rrnr[ list[str])rYrr[ list[dict])rrnrrnrrnrrrnrz"UploadTracker | NullTracker | Nonerr rrr[r )rrnrrrnrrnrMrZrrrrrrnrrZrr r[r )rrnrrrnrrr[zdict[str, str | None])rrnrrrnrrrrnrrnrzrZr[r ) rrnrrrr rrnrrr[r )r1rrrrnr[r)rUploadTracker | NullTrackerr1rr[r\)rkNN) rrPrrnrrrnrrrrr[zlist[tuple[str, str]])F)r list[tuple]rrrrnrrrnrrnrrnrrZrrr[z*tuple[list[tuple], list[dict], list[dict]])rrrrrrnrrrnrrnrrTrrnrrrrrrr[r)rarbrcrdrerErr'rErrXrrrr rrrrrrrrrrgrhrGrDrr"s(U 04 . # .# . .- .  .  ..,)-  "G G G ' G  G G 'G G G G  G d%))- +/,0"&*"!&_K_K_K _K  _K#_K'_K_K)_K*_K _K_K_K_K "!_KH *** * & * ! * *X    27;!"} } }  }  } 4} } }  } T#& !DD D  D  DDDDDD DR   <!45h5h 5h  5h  5h5h5h 5ht##'# #  #  # #J.,  ,  "!+/,0 ??? ? ) ? * ? ?Z#n9!n9-n9 n9  n9  n9n9n9n9 4n9|#V"!V"-V" V"  V"  V"'V"V"!V" V"V" V"V"rGr)rqrPrMrZrrrnr[r)rqrPrMrZrrrnr[rn)r~rZr[rZ)r)rz)str | Path | bytes | BinaryIO | IO[bytes]rrZr[r )rqrnrrr[r)NN)rrrrrrr[r)rr,r[rn)cre __future__rrbuiltinsrrrrrr'r%r#rrconcurrent.futuresrrpathlibrtypingrrr r tqdm.autor constantsr rrrrrrrrrrrrrrrrrrr r!r"r#r$r%r&r'r(errorsr)r*r+r,r-utils.file_utilsr. utils.loggerr/ _legacy_apir0rr1rr2r-rnr^rrr6rsrwrrrrrrrrrrrrr__annotations__rrrrZrrrrhrGrDrs #   ?33<+$)'! H d U"X-5 9 @ /A ? @?(,(,   $ &   22$ - -  , ,  , ,  - -  % %  ( (  , ,  ' '  ' '  % %  ( (  ( (  # #+""F ffR  H6868|}"}"rG