6pj@%SrSSKrSSKrSSKrSSKrSSKJr SSKJr SSK J r J r J r J r JrJrJrJrJrJr SSKJr SSKrSSKJrJrJrJrJrJrJrJrJ r J!r!J"r"J#r#J$r$J%r%J&r&Jr'J(r( SSKJ r) SS K+J,r, SS K-J.r. SS K/J0r0J1r1 SS K2J3r3J4r4 SS K5J6r6J7r7J8r8J9r9J:r: SSK;JJ?r? SSK@JArAJBrBJCrC SSKDJErE SSKFJGrG SSKHJIrI SSKJJKrK SSKLJMrN SSKLJOrOJPrPJQrQ SSKRJSrS SSKTJUrU SSKVJWrW SSKXJYrY SSKZJ[r[ SSK\J]r]J^r^ SSK_J`r` SSKaJbrb SSKcJdrd SS KeJfrfJgrgJhrh SS!KiJjrjJkrkJlrlJmrmJnrnJoroJprp SS"K5JqrrJsrt \d"5ru\S$rv\)b\)O\'rwS%\ 4S&jrxS'\yS(\S)\y4S*jrzSSS+SSS,.S-\yS.\\yS/\\{S0\|S1\\\|\y4S2\\ \vS)\N4S3jjr}0r~\ \\ \\P\Q44\S4'S-\yS.\yS5\yS6\|S)\\ \\P\Q44 S7jrSJS8S+SSSS9.S-\yS5\\yS6\|S2\|S.\\yS:\\yS1\\\|\y4S)\ \\P\Q44S;jjjrS+SSSS<.S-\yS=\\ \y\y4S2\|S.\\yS:\\yS1\\\|\y4S)\ \\P\Q44S>jjrSqSqSSS?.S@jrSKSAjr"SBSC5rSDrSErSFr"SGSH5r\GRSI5rg!\*a Sr)GNf=f!\*a SS#K5JrrrJtrt GNUf=f)LuModelScope dataset loading orchestration. This module provides :class:`DatasetsWrapperHF` and the :func:`load_dataset_with_ctx` context manager that monkey-patch the HuggingFace ``datasets`` library to work with the ModelScope Hub. Sub-modules: _compat – backward-compat shims for datasets>=4.0 script loading _module_factories – dataset module factory functions & data-file resolution N)fields)Path) AnyDictIterableListLiteralMappingOptionalSequenceTupleUnion) urlencode)DatasetDatasetBuilder DatasetDictDownloadConfigDownloadManager DownloadModeFeaturesIterableDatasetIterableDatasetDictSplitVerificationModeVersionconfig data_files LargeListr SplitDict)r)features)_FEATURE_TYPES) DataFilesDictEmptyDatasetError)DataFilesNotFoundErrorDatasetNotFoundError)CachedDatasetModuleFactory DatasetModule(HubDatasetModuleFactoryWithParquetExportPackagedDatasetModuleFactoryget_dataset_builder_class)_EXTENSION_TO_MODULE_PACKAGED_DATASETS_MODULES) file_utils)!_raise_if_offline_mode_is_enabled cached_pathrelative_to_absolute_path)is_small_dataset) tracked_str) hf_hub_url)OfflineModeIsEnabled) DatasetInfo)HfApiRepoFile RepoFolder) HfFileSystem)HubApi) get_endpoint)get_from_cache_ms)MS_DATASETS_CACHE)DEFAULT_DATASET_REVISIONREPO_TYPE_DATASET)has_attr_in_class)is_relative_path) get_logger)_HAS_SCRIPT_LOADING!HubDatasetModuleFactoryWithScript#LocalDatasetModuleFactoryWithScript)_resolve_pattern_download_repo_fileget_module_without_scriptget_module_with_script_compat_local_script_module_compat_hub_script_module _get_hub_api)HubDatasetModuleFactoryLocalDatasetModuleFactory)$HubDatasetModuleFactoryWithoutScript&LocalDatasetModuleFactoryWithoutScript)authorcardDatacitation createdAtdisabled description downloadsdownloadsAllTimegated lastModifiedlikespaperswithcode_idprivatesiblingsshatagsobjc L[U[5(aUVs/sHn[U5PM sn$SU;d[US[5(a/UR 5VVs0sHup!U[U5_M snn$[U5nUR S5nUS:Xa6UR S5nUR SS5n[[U5US9$[R "US5=(d [5R US5nUc-[SUS [[R"5535eU[:Xa&UR S5n[[U540UD6$[bDU[Ld[U[5(a&UR S5n[[U540UD6$[U5Vs1sHowR iM nnU"S 0UR 5V V s0sHupX;dM X_M sn n D6$s snfs snnfs snfs sn n f) aRegenerate the nested feature object from a deserialized dict. This is a ModelScope-patched version of ``features.generate_from_dict`` that handles backward compatibility for legacy ``Sequence`` types in datasets 4.0+ where ``Sequence`` is no longer a registered feature type. _typer featurelength)rdreNzFeature type 'z&' not found. Available feature types: ) isinstancelistgenerate_from_dict_msdictitemspopget SequenceHfr!globals ValueErrorkeysr _NativeList issubclassrname) ravaluekeyrcrdre class_typef field_nameskvs h/var/www/html/pdf-tiff/venv/lib/python3.13/site-packages/modelscope/msdatasets/utils/hf_datasets_util.pyrjrjs#t:=>#%e,#>>cZG d;;DGIIKPKjc*511KPP s)C GGG E '')$2&"7"@PP##E40NGIMM%4NJ>%0VW[\j\o\o\qWrVstuuY'')$.w7?3??J+$=JXcAdAd'')$09ASAA#)*#56#5a66#5K6  K#))+J+$!9I+J KK1?P*7JsH#HH5 H H url_or_filenamedownload_configreturnc[U5nURS5(aU[S5SnSH)nURU5(dMU[U5Sn O SU;aAURS5nX5S-SnUR S5nUS:XaUnSn O;USUnXgS-Sn O.UR SS 5n [ n[U 5S :aU S OSn [S XS .5n URU -nO3[U5(a#[ n[S XS .5n URU -n[XS 9n [U 5n U RU5 U $) zModelScope replacement for ``DownloadManager._download``. Rewrites relative paths and ``hf://`` URLs to ModelScope API endpoints. zhf://N)z datasets/zmodels/@/rfSDK)SourceRevisionFilePath)r) str startswithlenindexfindsplitr>r _base_pathrAr/r2 set_origin) selfr~rhf_path_prefixat_idxafter_at slash_idxrevision file_pathparts params_strouts r} _download_msrsT /*O!!'**!#g,-0/G!!'**!#g,-00 '>]]3'Fz{+H c*IB# #JY/$]^4 MM#q)E/H$'JNaI%X]^ //J6 / * *+%Xcd //J6 o GC c CNN?# JF)rtimeoutfiles_metadatatokenexpandrepo_idrrrrrc ZURUSUSUU[S9nUSU(aURS5SOSUSSSSS//SS. n/n UHRn [U [5(dMU R [ U RU RU RS95 MT XS'[S 0UD6$) z2ModelScope replacement for ``HfApi.dataset_info``.rF)r path_in_repor recursiverr repo_typerN) idr]rQr_rZrYrUrWr[r`rRrT) rfilenameblobIdsizer^rg) list_repo_treer?rrhr7appendrkrblob_idr HfDatasetInfo) rrrrrrrrepo_info_iter data_info data_siblings info_items r} _dataset_infors((#)N+2'--$Q' IM# i * *  '11$,," $*j  %9 %%r_repo_tree_cacherrcXSS4n[RU5nUcgU(aUS:waURS5OSn/nUHnURn U(aMU R US-5(dX:waM6U R US-5(aU [ U5S-SOSn OU n U(dSU ;aMtUR U5 M U$)z;Try to derive results from a cached recursive root listing.rTNrr)rrnstrippathrrr) rrrrroot_key root_cachedprefixresultsitem item_pathrel_paths r}_derive_from_recursive_cacher s3-H"&&x0K(49L\   $RTFGII '' 55):M6?6J6J6TW<6X6XyVq!12^`H HSH_ t NrT)rrrrrrc #"# U=(d [nU=(d SnXX4n [RU 5n U b U ShvN g[XX5n U bU [U 'U ShvN g[ 5n U R U[ S9n URS5upU RXU S9unn/nSnSnSnUU::aU RUUUUUUU US9nU(dOuUHRnUS S :XaS OSUSUSUSS.nUSS:Xa [S0UD6O [S0UD6nURU5 Uv MT [!U5U:aO US- nUU::aMU[U 'gGNN![a&n[RS US U35 SnAM:SnAff=f7f)z4ModelScope replacement for ``HfApi.list_repo_tree``.rNrr dataset_name namespaceendpointrii')rr root_pathr page_number page_sizerdataset_hub_idz Get dataset: z file list failed, message: Typetree directoryfilerSizeSha256)typerroidrrg)r>rrnrrLget_endpoint_for_readr?rget_dataset_id_and_typeget_dataset_files Exceptionloggererrorr7r8rr)rrrrrrrrnormalized_path cache_keycachedderivedapir_owner _dataset_namer_rrr max_pages dataset_filese file_info_d path_infors r}_list_repo_treer's33H")cOO?I  ! !) ,F *7oYG&-# .C((#4)6H$MM#.F33"x4INA24GKII  " 11!)#'#!-2 M (K'26':f'D &#F+#F+"8, I -6f,=,G8(i(ZMdZcMdD NN4 J) }  ) q A  "D#*Yo  6  LL= 1MaSQ R  sSrhrsetrrlrr) rrrrrrr paths_setr cached_itemsrmatchedrs r}_get_paths_inforss33H%E I$4#9#9#; Q<7 "ilh&> $0KLDII4J4LK N (C6 6I$<(( )N, F^TyyI/ED^ FF!L Gs$C;C7CCrrc [U/UQ7XS.UD6 U(av[U[5(a`[URS5(aURR c0URlSU3URR S'ggg)aCHfFileSystem.__init__ wrapper that injects ModelScope cookie auth. ModelScope's /resolve/ endpoint authenticates via `m_session_id` cookie rather than the `Authorization: Bearer` header used by HuggingFace Hub. This wrapper ensures the cookie is included in all subsequent HTTP requests made by the HfFileSystem instance. rheadersNz m_session_id=Cookie)_hf_fs_init_originalrhrhasattr_apir)rrrargskwargss r}_hf_fs_init_with_cookiersxOOxOO E3''tyy),, 0A0A0I "DII (5eW&= (#(urc FUS:XGa}SU;GavURU5nUR5nURU5nSnX`R;aJURUH7nUSU:XdMUR S5S:XdM%UR SS5n O US:Xa[ UR URURURURS 9n URR5n [R"XS S S 9n U RS :XacU R R S5n U (aA[#U 5n XS'URR U/5HnUSU:XdMXS' O ['X4SU0UD6$![$a Nf=f)a_Wrapper for HfFileSystem._open that fixes size=0 from ModelScope API. The ModelScope tree API may report Size=0 for files. When HfFileSystem caches this, AbstractBufferedFile treats the file as empty (0 bytes). This wrapper detects size=0 for files opened in read mode and resolves the actual size via a HEAD request before creating the file object. rbrNrurrrfr)rrfilenamerrT)rallow_redirectsrzContent-Lengthmode) resolve_path unresolve_parentdircachernr3rrrrrr_build_hf_headersrequestshead status_coderintr_hf_fs_open_original)rrrrresolved resolved_nameparent cached_sizeentryurlrrespcl actual_sizes r} _hf_fs_openrs t|f, ((.H$..0M\\-0FK&!]]62EV} 5%))F:Kv:U&+ii&; 3a $,,%..%22&00!]] ))557}}S4Y[\##s*))*:;B&)"g )4v%)]]%6%6vr%BE$V} =0;f %&C  @ @ @@   s+AF0FC2F=FF F F cz\rSrSrSr\R rSrSr Sr Sr Sr Sr \S 5r\S 5rS rS rg )_DryRunDownloadManageria Minimal download-manager stub for split discovery without I/O. Returns placeholder paths for all download/extract calls, allowing _split_generators() to execute and return SplitGenerator objects (which carry split names) without triggering actual network or disk I/O. c$URU5$N_mapr url_or_urlss r}download_DryRunDownloadManager.downloadyy%%rc$URU5$rrrs r}download_and_extract+_DryRunDownloadManager.download_and_extractrrc$URU5$rr)r path_or_pathss r}extract_DryRunDownloadManager.extractsyy''rc$URU5$rr)rrcustom_downloads r}download_custom&_DryRunDownloadManager.download_customrrc[/5$riter)rrs r} iter_archive#_DryRunDownloadManager.iter_archive Bxrc[/5$rr+)rrs r} iter_files!_DryRunDownloadManager.iter_filesr/rcgrrgrs r} manual_dir!_DryRunDownloadManager.manual_dirsrcg)NFrgr4s r} is_streaming#_DryRunDownloadManager.is_streamingsrc^[U[5(aUVs0sHo"TR_M sn$[U[[[ 45(a[ U5"U4SjU55$TR$s snf)Nc3<># UHnTRv M g7fr) _PLACEHOLDER).0rrs r} ._DryRunDownloadManager._map..sB6a 1 16s)rhrkr<rituplerr)rinput_r{s` r}r_DryRunDownloadManager._mapsh fd # #289&Qt(((&9 9 ftUC0 1 1<B6BB B   :sA;rgN)__name__ __module__ __qualname____firstlineno____doc__osdevnullr<rr r$r(r-r1propertyr5r8r__static_attributes__rgrr}rrs]::L&&(&!rrc4[US5(d [5$UR[55nUVs1sHn[ UR 5iM sn$s snf![ a,n[RSU35 [5sSnA$SnAff=f)aXDiscover available split names by dry-running _split_generators(). For script-based datasets that lack README split metadata, this calls the builder's _split_generators() with a stub download manager to extract split names without performing any actual downloads. Returns: A set of split name strings, or an empty set if discovery fails. _split_generatorsz(Failed to discover splits from builder: N) rrrMrrrurrdebug)builder_instance generatorssgrs r}_discover_splits_from_builderrRs~ #%8 9 9u %778N8PQ '12zBGG z222  ?sCDu s.A!AA!A!! B+!B BBc.UcgSSKJn U"U5nU(dg[5n[USS5nUb4UR(a#[URR 55nU(dI[USS5n[USS5n[ U[5(a[UR "55nU(d [U5nU(d[RS5 gX4- nU(a#[S[U5S [U535eg) aFail-fast check: raise ValueError before downloading if the requested split does not exist in the dataset metadata. Args: builder_instance: The DatasetBuilder instance with info/config. split: The user-requested split specification (may be None). Raises: ValueError: If any requested split name is not found among the available splits declared in the dataset metadata. Nr_extract_split_namesinforrz|Cannot determine available splits from dataset metadata; split validation skipped. Invalid splits will be caught downstream.zSplit z) not found in dataset. Available splits: ) -modelscope.msdatasets.utils._module_factoriesrUrgetattrsplitsrrrhrkrRrrNrqsorted) rOrrU split_names availablerVrrmissings r}_validate_split_existsr^s }R&u-K I #VT 2D DKK ((*+  )8T:V\48 j$ ' 'JOO-.I 12BC   R  %GVG_%&!!' !2 3 5  rcUcg[USS5nUb URcgSSKJn U"U5nU(dg[ URR 55nXE:agURR 5VVs0sHupgXd;dM Xg_M nnnU(dg[XRRS9Ulgs snnf)aAlign builder.info.splits with the actually requested split(s). When data_files have been filtered to a subset of splits (see _filter_data_files_by_split in _module_factories.py), the builder's info.splits metadata may still list all original splits from the README. download_and_prepare() calls verify_splits() which would then raise ExpectedMoreSplitsError. This helper prunes info.splits to only contain the splits that will actually be generated. NrVrrTr) rXrYrWrUrrrrlrr) rOrrVrUr[ existing_keysr{r|filtereds r}%_align_builder_splits_with_data_filesrcPs } #VT 2D |t{{*R&u-K  ((*+M#!%!2!2!4I!48H!4HI H;;3K3KLDK Js ? CCc/\rSrSr\SS\S\\S\\S\\\\\\ \\\\\444S\\\\ 4S\\S \\ S \\ S \\\ \4S \\\\4S \\S\S\\\\4S\\\\4S\S\\S\\S\S\\S\\\\\\44(Sjj5r\SS\S\\S\\S\\\\\\ \\\\\444S\\S \\ S \\ S \\\ \4S\\\\4S\\\\4S\\S\\S\\\\ 4S\4Sjj5r\SS\S\\\\4S \\ S \\\ \4S\\S\\S\\\\\\4S\\S\\S\4Sjj5rSrg)DatasetsWrapperHFitNrrudata_dirrr cache_dirr r download_modeverification_modekeep_in_memory save_infosrr streamingnum_procstorage_optionstrust_remote_codedataset_info_onlyrc & US:wa[R"S[5 Un US:wa[R"S[5 OSnUbU(d[SUS35e[ U[ R 5R5(a [S5eU(aUb [S5e[U=(d [R5n[U (dU =(d [RO[R5n U(a[RSUS 35 [ R""S!UUUUUUUUU U UUUSLUS .UD6nU(Ga&0n[%U[&5(a]UR)S 5(aG[*R,RU5(a#S S KJn U"U5nUVs0sHnU/_M nnU$Uc[R3SUS35 U$[5USS5nU(a[7US5(aUR95nOUVs/sHn[5USS5U4PM nnUHsunn['U5n[7US5(dM$UR:cM3[=UR:R?55V s/sHn ['U 5PM sn UU'Mu U(a%[ASURC555(a[5USS5=(d Sn!/n"[5USS5n#U#b4U#RD(a#[GU#RDR?55n"U"(d[IU5n$U$(a [GU$5n"U"(aU!U"0nU$U(dU!/0nU$[KUU5 U(aURMUS9$[OUU5 URQUUU UUS9 U bU O[SURTRV5n URYUU U S9n%UbJ[RZ"5 [R\"S[5 U%R_U5n%SSS5 U (aURa5 [c5n&[eU5(aNURgS5S:Xa9URiS5un'n(U&RkU[lS9n)U&RoU(U'U)S9 U%$s snfs snfs sn f!,(df  N=f![pa#n*[RS U*35 Sn*A*U%$Sn*A*ff=f)"N deprecated'use_auth_token' was deprecated in favor of 'token' in version 2.14.0 and will be removed in 3.0.0. You can remove this warning by passing 'token=' instead.zF'task' was deprecated in version 2.13.0 and will be removed in 3.0.0. zEmpty 'data_files': 'z3'. It should be either non-empty or None (default).zjYou are trying to load a dataset that was saved using `save_to_disk`. Please use `load_from_disk` instead.zLoading a streaming dataset in parallel with `num_proc` is not implemented. To parallelize streaming, you can wrap the dataset with a PyTorch DataLoader using `num_workers` > 1 instead.z3Use trust_remote_code=True. Will invoke codes from z9. Please make sure that you can trust the external codes.)rrurfrrgr rrhrrrnro_require_default_config_namer.pyr)get_dataset_config_namesz No builder instance created for z dataset.builder_configsrlrudefaultrc3.# UH o(+v M g7frrg)r=r|s r}r>1DatasetsWrapperHF.load_dataset..s"D2CQ552Cs config_namerV)r)rrhrirmrn)rri in_memoryignorerrrrz&Could not record download statistics: rg)9warningswarn FutureWarningrqrrDATASET_STATE_JSON_FILENAMEexistsNotImplementedErrorrREUSE_DATASET_IF_EXISTSr BASIC_CHECKS ALL_CHECKSrwarningreload_dataset_builderrhrendswithrHrdatasetsrvrrXrrlrrirrallvaluesrYrZrRr^as_streaming_datasetrcdownload_and_preparer1rV dataset_size as_datasetcatch_warnings simplefilterprepare_for_task _save_infosrLrAcountrrr?dataset_download_statisticsr)+rrurfrrrgr rrhrirjrkrruse_auth_tokentaskrlrmrnrorp config_kwargsrOret_dictrv subset_list_subset_tmp_builder_configs configs_iterctmp_config_nametmp_builder_configrr{rYrV discovereddsr _namespacerrrs+ r} load_datasetDatasetsWrapperHF.load_datasetvs: \ ) MM[  #E <  MMY  D  !*' |3fg  f88 9 @ @ B B78 8 -%34 4 %]&M(4(L(LN ,  >!1!>!>!1!  NNPQUPVWDD E-AA !+'+/)-  $ H$$$u)=)="''..QUBVBV=6t< 7BC{GGRK{C' ?vYOP$++;=NPT#U #/99#7#=#=#?LPd#ePd1WQ %BA$FPdL#e;G7O%7&)/&:O1<@@EWEbEbEnKOPbPmPmPrPrPtKu4vKu4SYKu4v1# #DKK$4$4$67F!>?O!PJ!!' !3 +V4HO" +R0HO/7 #88u8E E-.>F--+'/+ . -8N>N %%22?4  ( (/$)&  ((*%%h >((.+   ( ( * I.C%%$**S/Q*>,0JJsO) M44 ,=5?//]V`ks/t cD$f5w`+* I NNCA3G H H  Is7; S$SS  -SA(S# S # T-T  Tc U S:wa[R"S[5 U n [U=(d [R5nU b'U(aUR 5O [ 5nXlU b<U(aUR 5O [ 5nURRU 5 UbCU(aUR 5O [ 5nURc0UlXRS'[RUUUUUUUU U [U5US9 nURnURSU5nURSU5nURSU=(d URR 5nURSS5nUR"(aUR"R%U5OSnU[&;apUcmURR(S R*cIS US 3n[,Vs/sHn[,UU:XdMUPM nnU(a US US S 3- n[/U5e[1UUS9nUR2n[5US5(a[7UR8R;55n[7[<R>"UR@5RBR;55nUU-nURG5VVs0sHunnUU;dMUU_M nnnU"SUUUUUURHUUU U S. UDUD6nURKU5 U$s snf![.[D4a [75nNf=fs snnf)Nrrrsr) rrrhrfrrgrort_require_custom_configsrurfrr{rrz@Please specify the data files or data directory to load for the z dataset builder.z9 For example `data_files={"train": "path/to/data/train/*.z"}`r`__dataclass_fields__) rgrr{rfrhashrVr rrnrg)&r~rrrrcopyrrrnupdateredataset_module_factoryboolbuilder_kwargsrmbuilder_configs_parametersdefault_config_name dataset_infosrnr,rwrr+rqr*BUILDER_CONFIG_CLASSrrrrrinspect signature__init__ parameters TypeErrorrlr!_use_legacy_cache_dir_if_possible)rrurfrrgr rrhrrrrnrortrrdataset_modulerr{rrV error_msg extensionexample_extensions builder_cls _config_cls _valid_fields _init_paramsr{r|rOs r}r&DatasetsWrapperHF.load_dataset_builder&s, \ ) MM[  #E$]&M(4(L(LN   .22&4&6 $) !  & .22&4&6   + + 2 2? C   .22&4&6 ..624/7< + +G 4*AA +'!/)E$($7B  (66!%%j(;#'' jA $(( 4M88LLN &)).$? *88++// >B  . .:3E"==##-:6Z[_Z``qrI+?"+?i' 2d:+? ""Z[mno[pZqquvv Y' '/ 7 "66 ; 6 7 7 @ @ E E GHM %"%%k&:&:;FFKKM  *L8M!.!4!4!6!6A %1!6  ,7 , %#!$$+ ,  ,  ,  ::>J["* * %"u  %s+9L%L%?AL* M )M *MMdynamic_modules_pathc N U RSS5n U=(d [nUc [S+0U D6nURR SU 05 URR SU05 U(aUR c [ Ul[U=(d [R5nSUl SUl U[R:HUl [[SUR[ R"S5R%S555Sn U R'S5(dU S-n [ R(R+X 5nU[,;a[/UUUUUS9R15$UR'U 5(al[ R(R3U5(a1[4(a[7UUUUS 9R15$[9UUUUS 9$[;S [=U535e[ R(R3U5(a1[4(a[7UUUUS 9R15$[9UUUUS 9$[ R(R?U5(a[AUUUUS 9R15$[CU5(GaURES5S :XGa[G5 [I5RKUUURLS S9n[eUSUUS9n[ R(Rg[ R(RiU55nU URjVs/sHnURlPM sn;aSn[nRp(aOU(aH[s[tS5(a[uUUUS9R15$[uUUUS9R15$[4(aM[s[zS5(a[{UUUUUUS9R15$[{UUUUUUS9R15$[}UUUUUUS9$[s[~S5(a[UUUUUUS 9R15$[UUUUUUS!9R15$[;S [=U5S*35e![Nan[QU[R[TRVRX[TRVRZ45(a%[[SUS[]U5R^S35eS[aU5;a%SUS3n[cU(a USUS3-5eU5eS[aU5;a&SUS3nU(a USUS3-OUn[cUS-5eUeSnAff=fs snf![Na n[vRyU5 SnAGNSnAff=f![Nan[vRyS"US#U35 S$U0n[4(aUUS%'[U40UD6R15sSnA$![Na [QU[R5(a[[S&US'U35Se[QU[[b[45(aUSe[QU[:5(a4[;S [=U5S(US)[]U5R^S#U35SeUSef=fSnAff=f),NrurTcU$rrg)xs r}:DatasetsWrapperHF.dataset_module_factory..sQrrrfru)rfrrrh)rhrroz"Couldn't find a dataset script at )rfrrhrgY@)rrrrzCouldn't reach 'z' on the Hub ()404z Dataset 'z' doesn't exist on the Hubz at revision ''401zT. If the repo is private or gated, make sure to log in with `huggingface-cli login`.z README.md)rrrrF)rr) commit_hashr)rrrhrro)rrrhrro)rrfrrrh)rrfrrrhz>> Error loading z: rgrz1Couldn't reach the Hugging Face Hub for dataset 'z': z8 or any data file in the same directory. Couldn't find 'z"' on the Hugging Face Hub either: z( or any data file in the same directory.rg)Crmr>rrnrrgr=rrextract_compressed_file force_extractFORCE_REDOWNLOADforce_downloadrifilterreplacerHseprrrjoinr,r) get_moduleisfilerCrErJFileNotFoundErrorr0isdirrPrArr.r6 dataset_inforrrhr4r exceptionsConnectTimeoutConnectionErrorrrCrr%rGbasenamedirnamer^rrUSE_PARQUET_EXPORTr@r(rrrDrKrOr&r$r#)rrrrhrrfrrgrortrdownload_kwargs subset_namer combined_pathrrmsgdataset_readme_pathrsibling#can_load_config_from_parquet_exporte1_cached_factory_kwargss r}r(DatasetsWrapperHF.dataset_module_factorys5 +..vt< 77  ",??O''.. /DE''.. H/EF 88@(9O %$]&M(4(L(LN 26/(,%)6,:W:W)W& ;<<,2237 9::<>  ''%'H T4  - -/!% /+  jl  ]]8 $ $ww~~d##&&>&3-A*;  !jl # 3"/)=&7 (89RSW9X8YZWW^^M * *"":!"/)=&7  *,  /+%9"3   WW]]4 9!%+ -.8Z\  : d # # 31(<H '13! #(7#7#7 $!)-33 % $8$LD': !,$3% '# !gg..rww?R/ST 9E9N9N 9Ng))9N ;@7005X ,01Y[eff'O$(-54C(EFPZ\!R $L $,70?$ABLN+*,-NPZ[[#D $)10?.;5I2C $)jl+ A (3,;*71E.?  %*,'5!)(7&3-A*; ))MzZZC %-%-'1,;*7  %*,'@$/!)#-(7&3 !jl#D$45N}5]4^_789 9A! !0$//>>$//??..tfN47CSCSBTTUV#a& )$/IJ2>F .xj:;LO#a& )$/IJDLcnXJa$@@RU2#PP  5 H  ),"LLOO,d ' 0b=>'.99-E***I]./EF58 688B E '!"&:;;-OPTvUXY[X\]#$""#9#79J#LMM!d*!"&788/@AZ[hAi@jk..2V3UVZ[]V^VgVgUhhjkmjnp $ $ $&' 's3 V">$R"AV"9U0$V"4-U5"U5;;V"7V" V"!0V"V" U-CU((U--V"5 V?VV"VV"" Z$,Z-W;5Z$;B!ZZZ$rg)NNNNNNNNNNFNNrrrrFNNFF)NNNNNNNNNrrNNTN) NNNNNNNNTF) rCrDrErF staticmethodrr rr r rrrrrrrrrrrrrrkrrrrr"r'rrKrgrr}reretsW#"&KO-1#''+48<@DH)- 26,0# "&*."',1/mmsmm3-mU3 #*3c6>sm7D1E,E$F$FGH mc5j)*mC=m8$m".1m lC&7 89m$E*:C*?$@Am!mm5g./m dCi()!m&'m(3-)m*"$+m, -m.$D>/m2 {G%8/ 3mm^#"&KO#''+48<@26,0#*.,0%)-1#u u smu 3-u U3 #*3c6>sm7D1E,E$F$FGH u C=u 8$u ".1u  lC&7 89u 5g./u dCi()u "$u $D>u "c5j)*#u & 'u u n3748<@.2"&FJ#',0%) %e9e95g./e9".1e9 lC&7 89 e9 'sm e9 3- e9U4sM#ABCe9C=e9$D>e9 e9e9rrec/# [Rn[Rn[ [ S5(a[ R O[ Rn[Rn[Rn[Rn[Rn[Rn [ (a["ROSn [$R&n [(R*n [(R,n U [.LnU [0Ln[35[l[4[l[ [ S5(a[6[ lO[6[ l[8[l [:[l [<[l [>[l [@[l[ (a[B["l[D[$lU(dU q#[.[(lU(dU q$[0[(lURKSS5nSn[LRN"U0UD6nUnUv [PRS5 [TRVRS5 U(+nU(aU(d U [(lSq#U(d U [(lSq$U[lU[lU [$l[ [ S5(a U[ lO U[ lU[l U[l U[l U[l U [l[ (a U ["lggg![PRS5 [TRVRS5 U(+nU(aU(d U [(lSq#U(d U [(lSq$U[lU[lU [$l[ [ S5(a U[ lO U[ lU[l U[l U[l U[l U [l[ (a U ["lfff=f7f)zContext manager that monkey-patches ``datasets`` to use ModelScope. All monkey-patches are applied on entry and restored on exit (for non-streaming mode) or kept alive (for streaming mode, where lazy iteration needs the patches to remain active). _downloadNrlF),r HF_ENDPOINTr-get_from_cacherrr_download_singler6rrget_paths_inforresolve_patternrOrrCrDr generate_from_dictr9_openrrrr;r<rrrrrFrHrIrjr rrnrerrclearr:_dataset_id_type_cache)rrhf_endpoint_originget_from_cache_origin_download_origindataset_info_originlist_repo_tree_originget_paths_info_originresolve_pattern_origin get_module_without_script_originget_module_with_script_origingenerate_from_dict_originhf_fs_open_originhf_fs_init_originhf_fs_open_was_patchedhf_fs_init_was_patchedrl_streaming_dataset_returned dataset_resshould_restores r}load_dataset_with_ctxrs' ++&554;O[4Y4Y00  - -,,!00!00'77'K'V'V$8K8K)44QU" ( ; ;$**$--.+=.2II&F 1J ,,$0!+7(&E*E*E!1J6O(37M)4"7H !0(  !0 7  ;.I"' ]'44dEfE &/# %%++-88 )%6 "'+$)(9 %'+$!3F (=J %*CH ' 44,<)3C0!4E #8E #8E )?J &>^ 0 ;""?\1<#-   %%++-88 )%6 "'+$)(9 %'+$!3F (=J %*CH ' 44,<)3C0!4E #8E #8E )?J &>^ 0 ;""?\1<#- s HP>L50DP>5DP;;P>r)r)rG contextlibrrHr~ dataclassesrpathlibrtypingrrrrr r r r r r urllib.parserrrrrrrrrrrrrrrrrrror DatasetList ImportErrordatasets.featuresr datasets.features.featuresr!datasets.data_filesr"r#datasets.exceptionsr$r% datasets.loadr&r'r(r)r*datasets.packaged_modulesr+r,datasets.utilsr-datasets.utils.file_utilsr.r/r0datasets.utils.info_utilsr1datasets.utils.trackr2huggingface_hubr3huggingface_hub.errorsr4huggingface_hub.hf_apir5rr6r7r8huggingface_hub.hf_file_systemr9 modelscoper:modelscope.hub.utils.utilsr;)modelscope.msdatasets.utils.hf_file_utilsr<modelscope.utils.config_dsr=modelscope.utils.constantr>r?modelscope.utils.import_utilsr@modelscope.utils.file_utilsrAmodelscope.utils.loggerrB#modelscope.msdatasets.utils._compatrCrDrErWrFrGrHrIrJrKrLrMrOrNrPrExpandDatasetProperty_Trsrjrrfloatrrrr@__annotations__rrrr rrrrrRr^rcrecontextmanagerrrgrr}r+s  ```"99999 ,'5@L Y%OO6,&7?>>73G8Q;8.0M " 0)4k*  Ls LN$$n$QT$^## (,6:0& 0&sm 0& e_ 0&  0& E$)$ %0& T12 30&0&jDF$ud5:)=#>??@E   d5:-./0 >#'I* "#(,I* I*3-I*  I*  I*smI*}I* E$)$ %I*eHj()*I*b"#(,$G $G cC $G  $G sm $G}$G E$)$ %$G %*$ %&$GV37d >(AV)!)!X(1 hMHP9P9n U]U]W#Kr0000s$)I+.I:+I76I7:J  J