9iSSKrSSKrSSKJrJrJrJrJrJrJ r J r J r SSK r SSKJrJrJrJrJr SSKJr SSKJr SSKJr SSKJr SSKJrJrJrJ r SS K!J"r"J#r# SS K$J%r% SS K&J'r' SS K(J)r) SS K*J+r+ SSK,J-r- SSK.J/r/J0r0 SSK1J2r2 SSK3J4r4J5r5J6r6J7r7J8r8J9r9J:r:J;r;JJ?r?J@r@ SSKAJBrB \B"5rCS\4SjrD"SS5rEg)N) AnyCallableDictIterableListMappingOptionalSequenceUnion)Dataset DatasetDictFeaturesIterableDatasetIterableDatasetDict)_PACKAGED_DATASETS_MODULES)is_relative_path)DatasetRepository)DatasetContextConfig)LocalDataLoaderManagerLocalDataLoaderTypeRemoteDataLoaderManagerRemoteDataLoaderType)ExternalDatasetNativeIterableDataset)build_custom_dataset)DatasetDeleteManager)load_dataset_with_ctx)DatasetUploadManager)build_preprocessor)Config ConfigDict)MS_DATASETS_CACHE) DEFAULT_DATASET_NAMESPACEDEFAULT_DATASET_REVISIONREPO_TYPE_DATASET ConfigFieldsDatasetFormations DownloadModeHubsModeKeysTasks UploadMode)is_tf_availableis_torch_available) get_loggerreturncUc/nU$[U[5(aU/nU$[[U55[U5:a[ SU35eU$)Nz"List columns contains duplicates: ) isinstancestrlenset ValueError)paras `/var/www/html/land-doc-ocr/venv/lib/python3.13/site-packages/modelscope/msdatasets/ms_dataset.py format_listr9)s^ | K D#  v K SY#d) #=dVDEE Kc+d\rSrSr%SrSrSr\\S'SLS\ \ \ \ \ 4S\\4SjjrSrS rS r\S 5r\S 5r\SLS \ \ \\ 4S\S\ \S44Sjj5r\SLS\ \ \\ \ \ \4S\S\ \S44Sjj5r\\S\\ RBSSSS\"RF\$SSS\%"5SSS4S\ \\&4S\\S\\S\\S\\ S\\S\\S\\S\\ \\'\\(\\ \\'\444S\\"S\\S\\)S\\*S\\+S \\%S!\\S"\\*S#\\*S\ \S\ 44&S$jj5r,\\\SSS%\-R\4S&\S'\S\S\\S\\S(\\+S)\\+S*\\*S+\\-SS4S,jj5r/\\SS4S-\S.\S/\\S0\\S1\\SS4 S2jj5r0\\SSS4S-\S3\S/\\S0\\S1\\S4\*SS4S5jj5r1\\\4S&\S\S\\S\\S\4 S6jj5r2SMS7\ \\3\4S8\ \4\3\44S9\S:\5S;\*4 S<jjr6SNS=\+S>\*S8\ \4\3\44S7\ \\3\4S?\4S@\*SA\7\\84SB\ \\3\4SC\*4SDjjr9S\ 4SEjr:SF\7\\4S\ 4SGjr;SOS8\ \4\3\44S7\ \\3\4S;\*4SHjjr<SPS=\+S>\*S8\ \4\3\44S@\*SC\*SB\ \\3\4S7\ \\3\44SIjjr=SQS \%4SJjjr>SKr?g)R MsDataset3a ModelScope Dataset (aka, MsDataset) is backed by a huggingface Dataset to provide efficient data access and local storage managements. On top of that, MsDataset supports the data integration and interactions with multiple remote hubs, particularly, ModelScope's own Dataset-hub. MsDataset also abstracts away data-access details with other remote storage, including both general external web-hosted data and cloud storage such as OSS. N_dataset_context_config ds_instancetargetcXlUbUX RR;a<[S[URRR 55SU35eX lSUlg)Nz)"target" must be a column of the dataset(z , but got F)_hf_dsfeatures TypeErrorlistkeysr@ is_custom)selfr?r@s r8__init__MsDataset.__init__@se"  & 0D0D"D;DAUAUAZAZA\<];^^hiohpq  r:c#x# URH&nURbXRv M"Uv M( g7fN)rBr@)rHitems r8__iter__MsDataset.__iter__Ls/KKD{{&;;''  s8:c URU$rLrB)rHkeys r8 __getitem__MsDataset.__getitem__Ss{{3r:c,[UR5$rL)r4rBrHs r8__len__MsDataset.__len__Vs4;;r:cUR$rLrQrVs r8r?MsDataset.ds_instanceYs {{r:cn[UR[5(aURR$grL)r2rBr config_kwargsrVs r8r\MsDataset.config_kwargs]s& dkk? 3 3;;,, ,r:hf_dsr0c [R"S[5 [U[5(aU"X5$[U[ 5(aq[ UR55S:Xa)U"[[UR555U5$UR5VVs0sH up4X0"XB5_M snn$[U[5(aU"U5$[S[U535es snnf)zt @deprecated This method is deprecated and may be removed in future releases, please use `to_ms_dataset()` instead. z@from_hf_dataset is deprecated, please use to_ms_dataset instead.z2"hf_ds" must be a Dataset or DatasetDict, but got )warningswarnDeprecationWarningr2r r r4rFnextitervaluesitemsrrDtype)clsr^r@kvs r8from_hf_datasetMsDataset.from_hf_datasetds  N   eW % %u% % { + +5::< A%4U\\^ 45v>>27++-@-$!As1~%-@ @  / /u: DT%[MR  As'C3c H[U[5(aU"X5$[U[5(aq[UR 55S:Xa)U"[ [ UR555U5$UR5VVs0sH up4X0"XB5_M snn$[U[5(aU"U5$[U[5(aU"U5$[U[5(aU"U5$[U[5(aq[UR 55S:Xa)U"[ [ UR555U5$UR5VVs0sH up4X0"XB5_M snn$[S[U535es snnfs snnf)z&Convert input to `MsDataset` instance.r`z8"ds_instance" must be a Dataset or DatasetDict, but got )r2r r r4rFrdrerfrgrrrrrDrh)rir?r@rjrks r8 to_ms_datasetMsDataset.to_ms_dataset|sl k7 + +{+ +  [ 1 1;##%&!+4[%7%7%9 :;VDD2=2C2C2EF2E$!As1~%2EF F  _ 5 5{# #  %: ; ;{# #  _ 5 5{# #  %8 9 9;##%&!+4[%7%7%9 :;VDD2=2C2C2EF2E$!As1~%2EF FJ4P[K\J]^ GGs F)FFr` dataset_name namespaceversionhub subset_namesplitdata_dir data_files download_mode cache_dirrC use_streamingstream_batch_size custom_cfgtokendataset_info_onlytrust_remote_codec * U(aSSKJn U"5nURU5 [U =(d [R5n [ U=(d [ R 5nU[ R:Hn[U[5(d,[U[5(d[S[U535e[U[5(a0UcSn[R"X 05n[R!UUS9$["R$R'U5n["R$R)U5n[+U5(ajUR-S5S:XaUU(dNU(dGUR/S5nUSR15nUSR15nU(aU(dS eU(a[2R5S US 35 [7SUUUUUUUUUU U U U US .UD6nU[8;dH["R$R;U5(d$["R$R=U5(a|[?U5RA[BRD5n[R!UUS9n[U[5(a)UUl#U(aURH"SS U0UD6 SUl%U$U[ R:XaSSK&J n U"U4UUU U RNUS.UD6$U[ R :XGa)SSKJn U"5nURQUS-U-[RS9nURUUUUS9unn[U5[[VRXRN5:Xa6[[SUS-U-UUUUU U SU RNUUU UUS.UD6nUsSSS5 $[]U5n U RA[^R`5n[R!UUS9n[U[5(a3U RbUl#U(aURH"SS U0UD6 SUl%U$U[ Rd:Xa[fRh"S[j5 SSK6J7n! SSK8J9n" U[t:XaU"RvUl<U[z:XaU"R|Ul?U [:Xa/SSKAJBn# ["R$RU#SSS5n U UlDU!"U5n$U$R5 U$R$Se!,(df  g=f)aLoad a MsDataset from the ModelScope Hub, Hugging Face Hub, urls, or a local dataset. Args: dataset_name (str): Path or name of the dataset. The form of `namespace/dataset_name` is also supported. namespace(str, optional): Namespace of the dataset. It should not be None if you load a remote dataset from Hubs.modelscope, namespace (str, optional): Namespace of the dataset. It should not be None if you load a remote dataset from Hubs.modelscope, target (str, optional): Name of the column to output. version (str, optional): Version of the dataset script to load: subset_name (str, optional): Defining the subset_name of the dataset. data_dir (str, optional): Defining the data_dir of the dataset configuration. I data_files (str or Sequence or Mapping, optional): Path(s) to source data file(s). split (str, optional): Which split of the data to load. hub (Hubs or str, optional): When loading from a remote hub, where it is from. default Hubs.modelscope download_mode (DownloadMode or str, optional): How to treat existing datasets. default DownloadMode.REUSE_DATASET_IF_EXISTS cache_dir (str, Optional): User-define local cache directory. use_streaming (bool, Optional): If set to True, no need to download all data files. Instead, it streams the data progressively, and returns NativeIterableDataset or a dict of NativeIterableDataset. stream_batch_size (int, Optional): The batch size of the streaming data. custom_cfg (str, Optional): Model configuration, this can be used for custom datasets. see https://modelscope.cn/docs/Configuration%E8%AF%A6%E8%A7%A3 token (str, Optional): SDK token of ModelScope. dataset_info_only (bool, Optional): If set to True, only return the dataset config and info (dict). trust_remote_code (bool, Optional): If set to True, trust the remote code. Default to `False`. **config_kwargs (additional keyword arguments): Keyword arguments to be passed Returns: MsDataset (MsDataset): MsDataset object for a certain dataset. r)HubApiz.dataset_name must be `str` or `list`, but got Nr@)r@/r`zUThe dataset_name should be in the form of `namespace/dataset_name` or `dataset_name`.z3Use trust_remote_code=True. Will invoke codes from z9. Please make sure that you can trust the external codes.)rqrrrsrurvr@rtrwrxrycache_root_dirr{r|rr}T) load_dataset)namerv streamingryr)repo_id repo_type)rqrrendpoint)pathrrwrxrvrzrCdownload_configryrevisionr~rrrzMThe option `Hubs.virgo` is deprecated, will be removed in the future version.)VirgoDownloader)VirgoDatasetConfig) CACHE_HOMEvirgortdatasetszPlease adjust input args to specify a loading mode, we support following scenes: loading from local disk, huggingface hub and modelscope hub.)Gmodelscope.hub.apirloginr(REUSE_DATASET_IF_EXISTSr) modelscope huggingfacer2r3rErDrhr from_dictr<roosr expanduserexistsrcountrvstriploggerwarningrrisdirisfilerrrHF_DATA_LOADERr>to_custom_datasetrGrvalueget_endpoint_for_readr%get_dataset_id_and_typer'generalrrrMS_DATA_LOADERdataset_context_configrrarbrc-modelscope.msdatasets.data_loader.data_loaderrmodelscope.utils.constantrr#default_virgo_namespacerrr$default_dataset_versionrsr"modelscope.utils.config_dsrjoinrprocessdataset)%rqrrr@rsrtrurvrwrxryrzrCr{r|r}r~rrr\rapiis_huggingface_hub dataset_inst is_local_pathdataset_name_splitrr_apirdataset_id_on_hub dataset_type dataset_resremote_dataloader_managerrrrvirgo_downloaders% r8loadMsDataset.loads x  1(C IIe $]&M(4(L(LN 3)$//*!T%5%55,,,Zd6$6$@lAS@TU  lD ) )~!",,f-CDL**<*G Gww)),7 |4 L ) )l.@.@//"/8J!-!3!3C!8 *1-335I-a0668LLmm  NNEl^T44 5 "6"%#!'$'//""$ 5 5::!# !=!=1&((4 '66)8 %22<2OL, 227M 4 22@#-@1>@-1L*  D$$ $ -! '+11"3 ! ! !DOO # 18D11!C,6+2-H/3.J.J)#!/K/# + | < C(9(A(A(G(G$HH*)&_|;(!)#-#"+!)(,&3&9&9!(#"/*;*;)()-8&#))*-D*-,)8EE(77 9 (66  7 1 lI66;T;k;kL8!$66D'1D5BD15 .## DJJ  MM9:L N V D553E3]3]&0221C1[1[&.--AGGLLWe)35 8A&5./EF   $ $ &#++ +Q Qu))s T TT object_namelocal_file_path num_processes chunksizefilter_hidden_files upload_modec [R"S[5 U(d [S5e[ X#US9n [ U=(d [ R 5n[RRU5(aU RUUUS9 g[RRU5(aU RUUUUUUS9 g[US35e)zs @deprecated This method is deprecated and may be removed in future releases, please use git command line instead. zThe function `upload` is deprecated, please use git command or modelscope.hub.api.HubApi.upload_folder or modelscope.hub.api.HubApi.upload_file.zobject_name cannot be empty!rqrrrs)rrr)object_dir_namelocal_dir_pathrrrrz& is not a valid file path or directoryN) rarbrcr6rr, OVERWRITErrruploadr upload_dir) rrrqrrrsrrrr_upload_managers r8rMsDataset.uploadqs .   8:L  M ;< <.%GM!!D 0D0DE 77>>/ * *  " "' /' # )WW]]? + +  & & +.+#$7' ' )"##IJL Lr:dataset_work_dir dataset_idr auth_tokengit_pathc[R"S[5 [UUUUUS9nUR 5nU(a%[ R SRU55 g[ RSRU55 g)aClone meta-file of dataset from the ModelScope Hub. Args: dataset_work_dir (str): Current git working directory. dataset_id (str): Dataset id, in the form of your-namespace/your-dataset-name . revision (str, optional): revision of the model you want to clone from. Can be any of a branch, tag or commit hash auth_token (str, optional): token obtained when calling `HubApi.login()`. Usually you can safely ignore the parameter as the token is already saved when you login the first time, if None, we will use saved token. git_path (str, optional): The git command line path, if None, we use 'git' Returns: None zWThe function `clone_meta` is deprecated, please use git command line to clone the repo. repo_work_dirrrrrzAlready cloned repo to: {}zRepo dir already exists: {}N) rarbrcrclonerinfoformatr)rrrrr_repoclone_work_dirs r8 clone_metaMsDataset.clone_metast,  e  "*!!    KK4;;NK L NN-44^D Fr:commit_messageforcect[R"S[5 [USUUUS9nUR XUS9 g)aUpload meta-file of dataset to the ModelScope Hub. Please clone the meta-data from the ModelScope Hub first. Args: dataset_work_dir (str): Current working directory. commit_message (str): Commit message. revision(`Optional[str]`): revision of the model you want to clone from. Can be any of a branch, tag or commit hash auth_token(`Optional[str]`): token obtained when calling `HubApi.login()`. Usually you can safely ignore the parameter as the token is already saved when you log in the first time, if None, we will use saved token. git_path:(`Optional[str]`): The git command line path, if None, we use 'git' force (Optional[bool]): whether to use forced-push. Returns: None zuThe function `upload_meta` is deprecated, please use git command or CLI `modelscope upload owner_name/repo_name ...`.r)rbranchrN)rarbrcrpush)rrrrrrrs r8 upload_metaMsDataset.upload_metasH2   C   "*!   . Or:cj[XUS9nURUS9n[RSUS35 U$)a"Delete object of dataset. Please log in first and make sure you have permission to manage the dataset. Args: object_name (str): The object name of dataset to be deleted. Could be a name of file or directory. If it's directory, then ends with `/`. For example: your-data-name.zip, train/001/img_001.png, train/, ... dataset_name (str): Path or name of the dataset. namespace(str, optional): Namespace of the dataset. version (str, optional): Version of the dataset. Returns: res_msg (str): Response message. r)rzObject z successfully removed!)rdeleterr)rrqrrrs_delete_managerresp_msgs r8rMsDataset.deletesB&/%GM"))k)B gk]*@ABr:columns preprocessors task_name data_config to_tensorc [5(d [S5e[UR[5(aCUR SU05 UR URR 5 [XC5$UbURX!US9$URR5 URRSXS9 UR$)aCreate a torch.utils.data.Dataset from the MS Dataset. The torch.utils.data.Dataset can be passed to torch.utils.data.DataLoader. Args: preprocessors (Callable or List[Callable], default None): (list of) Preprocessor object used to process every sample of the dataset. The output type of processors is dict, and each (numeric) field of the dict will be used as a field of torch.utils.data.Dataset. columns (str or List[str], default None): Dataset column(s) to be loaded (numeric data only if `to_tensor` is True). If the preprocessor is None, the arg columns must have at least one column. If the `preprocessors` is not None, the output fields of processors will also be added. task_name (str, default None): task name, refer to :obj:`Tasks` for more details data_config (ConfigDict, default None): config dict for model object. Attributes of ConfigDict: `preprocessor` (Callable, List[Callable], optional): preprocessors to deal with dataset `type` (str): the type of task `split_config` (dict, optional): get the split config for ExternalDataset `test_mode` (bool, optional): is test mode or not to_tensor (bool, default None): whether convert the data types of dataset column(s) to torch.tensor or not. format_kwargs: A `dict` of arguments to be passed to the `torch.tensor`. Returns: :class:`torch.utils.data.Dataset` z>The function to_torch_dataset requires pytorch to be installed preprocessor)rrtorch)rhr format_kwargs) r. ImportErrorr2rBrupdater\r!_to_torch_dataset_with_processors reset_format set_format)rHrrrrrrs r8to_torch_datasetMsDataset.to_torch_datasetsB"##P  dkk? 3 3    > ?   t{{88 9' ? ?  $99):E E KK $ $ & KK " "g # L;; r: batch_sizeshuffle collate_fndrop_remaindercollate_fn_args label_colsprefetchc  [5(d [S5eUbURUUUUU UUS9$Uc[R S5 gUR R 5 UR RUUUUUUUU S9$)ahCreate a tf.data.Dataset from the MS Dataset. This tf.data.Dataset can be passed to tf methods like model.fit() or model.predict(). Args: batch_size (int): Number of samples in a single batch. shuffle(bool): Shuffle the dataset order. preprocessors (Callable or List[Callable], default None): (list of) Preprocessor object used to process every sample of the dataset. The output type of processors is dict, and each field of the dict will be used as a field of the tf.data. Dataset. If the `preprocessors` is None, the `collate_fn` shouldn't be None. columns (str or List[str], default None): Dataset column(s) to be loaded. If the preprocessor is None, the arg columns must have at least one column. If the `preprocessors` is not None, the output fields of processors will also be added. collate_fn(Callable, default None): A callable object used to collect lists of samples into a batch. If the `preprocessors` is None, the `collate_fn` shouldn't be None. drop_remainder(bool, default None): Drop the last incomplete batch when loading. collate_fn_args (Dict, optional): A `dict` of arguments to be passed to the`collate_fn`. label_cols (str or List[str], default None): Dataset column(s) to load as labels. prefetch (bool, default True): Prefetch data. Returns: :class:`tf.data.Dataset` z?The function to_tf_dataset requires Tensorflow to be installed.N)rrrrz?The `preprocessors` and the `collate_fn` should`t be both None.)rrrr)r-r_to_tf_dataset_with_processorsrerrorrBr to_tf_dataset) rHrrrrrrrrrs r8rMsDataset.to_tf_datasetOsH  Q   $66-!%7! !   LLQ    "{{((    )+!) r:cNURR5 UR$rL)rBrrVs r8 to_hf_datasetMsDataset.to_hf_datasets   "{{r:column_mappingclURR5 URRU5$)z Rename columns and return the underlying hf dataset directly TODO: support native MsDataset column rename. Args: column_mapping: the mapping of the original and new column names Returns: underlying hf dataset )rBrrename_columns)rHr s r8 remap_columnsMsDataset.remap_columnss)   "{{)).99r:c ^[U[5(aUOU/n[U5nURRR 5Vs/sH oUU;dM UPM nn/n/nU(Ga[ [UR55nUV s0sHo[R"X5_M n n UHRn U RU "U5R5V V s0sHupU [R"U 5_M sn n 5 MT Sn U R 5HOn U "X5(d,[RSU S35 URU 5 M>URU 5 MQ SSKm"U4SjSTR R"R$5nU"URUUXrU5$s snfs sn fs sn n f)Nc[R"UR[R5=(d/ [R"UR[R5$rL)np issubdtypedtypeintegerfloating)rs r8is_numpy_numberDMsDataset._to_torch_dataset_with_processors..is_numpy_numbers;}}U[["**=.KKB..r:zData of column z is non-numeric, will be removedrcJ>^\rSrSrS\4U4SjjrSrU4SjrSrSr U=r $)AMsDataset._to_torch_dataset_with_processors..MsMapDatasetirc>[[5R5 XlX lX`lX0lX@lXPlgrL) superr<rIrpreprocessor_listrretained_numeric_columnsretained_unumeric_columnsr)rHrrrrrr __class__s r8rIJMsDataset._to_torch_dataset_with_processors..MsMapDataset.__init__s7i ))+& ):&!*0H-1J.& r:c,[UR5$rL)r4rrVs r8rWIMsDataset._to_torch_dataset_with_processors..MsMapDataset.__len__s4<<((r:cL>UR(aTRU5$U$rL)r as_tensor)rHxrs r8type_converterPMsDataset._to_torch_dataset_with_processors..MsMapDataset.type_converters>> ??1--Hr:cURUnURVs0sH7o0R(aX0R;dM$X0R X#5_M9 nnUR HlnU"U5R 5HOup6UR(aX0R;aUR U5XC'M:X0R;dMKXdU'MQ Mn U$s snfrL)rrrrr%rrgr)rHindex item_dictrjresrrks r8rSMMsDataset._to_torch_dataset_with_processors..MsMapDataset.__getitem__s LL/ "\\)..9999A**9<88) %)$:$:L ,Y 7 = = ? $ !%B%B B%)%8%8%;CF"@"@@%&F !@%; s #CC)rrrrrr) __name__ __module__ __qualname____firstlineno__rrIrWr%rS__static_attributes__ __classcell__)rrs@r8 MsMapDatasetrs# ' ' )    r:r2)r2rEr9rBrCrFrdrerarrayrrgrrappendrutilsdatar )rHrrrrrRrrsamplerj sample_res processorrkrr2rs @r8r+MsDataset._to_torch_dataset_with_processorss .8 4.!.!M'4o g& ;;//446 6C.C6  $& $&! $t{{+,F:AB'QRXXfi00'JB. !!!*6!2!8!8!:[U5nTVs0sH)o"[R"TRUU5_M+ nnTH_nUR U"TRU5R 5VVs0sHup%U[R"U5_M snn5 Ma U(aU$[ [UR555$s snfs snnfrL) intrr3rBrrgtuplerErf) i return_dictrjr*rrkrretained_columnsrHs r8func6MsDataset._to_tf_dataset_with_processors..func sAA;KL;Kabhht{{1~a011;KCL 1  ,T[[^ < B B D Drxx{N D!2  cjjl+, ,Ms 0C9#C T)input_signaturec >TRTU/TR5Vs/sH(nTRRUR5PM* snS9n[ T5VVs0sH upX2U_M snn$s snfs snnf)N)inpTout)numpy_functionrfdtypesas_dtyper enumerate)r@valoutputrRrCr8tfs r8fetch_function@MsDataset._to_tf_dataset_with_processors..fetch_functions&&C *0022II&&syy12'F2;:1FG1FvqCN1FG G  Hs /A9 $A>)AUTOTUNE)num_parallel_callsc2>UR5VVs0sHupUT;dM X_M nnn[U5S:Xa"[[UR 555n[U5S:Xa"[[UR 555nX4$s snnf)Nr`)rgr4rdrerf) input_batchrRtensorlabelsrs r8split_features_and_labelsKMsDataset._to_tf_dataset_with_processors..split_features_and_labels+s(3'8'8':': cZ>O CK':{#q("&tK,>,>,@'A"BKv;!#!$v}}"78F"**s BBr`cF[[UR555$rL)rdrerf)r$s r8:MsDataset._to_tf_dataset_with_processors..9s$tAHHJ7G2Hr:)r)F)r2rEr9r5rBrCrF tensorflowr6r from_tensor_slicesraranger4int64rfunction TensorSpectensorflow.data.experimentalrRmapbatchr)rHrrrrrrrcols_to_retainrR tf_datasetrPrRrXrCrrBr8rOs` ` @@@@@r8r(MsDataset._to_tf_dataset_with_processorss.8 4.!.!M'4o !, g&c*w"678;;//446 6C:OC6  WW__77 IIc$++&bhh 79 #++DKK8H+IJ - -!T] bmmD"((&C%D E H F H :^^ $9  +$(ABJ \Q #(HIJ >#))*;J #,,X6J} s 5 GGc J[5(d [S5eU(dgSUlUcSU;aURS5nU[R :XaSOSnUR SU35nUcE[U[R5(a[URRS9O[SS9nUR[US 95 URnS U;aURS 5n[ R""U5nS U;aURS 5nUc/[US 5(aUR$n U (a ['X5n[)UR*[,5(aVUR[US 95 URUR*R.5 [1XaRS9UlgUb'URSS5n UR3X*S9UlgUR*R55 UR*R7SS9 g)aXConvert the input datasets to specific custom datasets by given model configuration and preprocessor. Args: custom_cfg (Config): The model configuration for custom datasets. preprocessor (Preprocessor, Optional): Preprocessor for data samples. mode (str, Optional): See modelscope.utils.constant.ModeKeys Returns: `MsDataset` z?The function to_custom_dataset requires pytorch to be installedNTmodetrainrMzdataset.)rh)rjtaskfieldr)r)cfgrr)rrr)r.rrGgetr*TRAINsafe_gethasattrr&modelr!rhrdictrlpopr+find_field_by_taskrrr2rBrr\rrrr) rHr}rrjkwargs ds_cfg_keydata_cfgr field_namepreprocessor_cfgrs r8rMsDataset.to_custom_datasetBs "##Q   <zz&)!% 6WE &&*'>?  AHL..B0B0zz'7'7'<'<=5?T5J $(OO V  6*I--i8 f G,J  GJ$G$G)66 12BO  dkk? 3 3 OODl; < OODKK55 6.9DK   # ;5I@@*AADK  KK $ $ & KK " " " 0r:)rBrGr@rL)NNNNT)NNNNNNT)NT)NTNN)NN)@r,r-r.r/__doc__rBr>r__annotations__r r rrrr r3rIrNrSrWpropertyr?r\ classmethodr rtrlrro staticmethodr#r$r)rr(rr"r rEr rrboolr>rr,rrrrrrrr!rrrrrr rrrr0rr:r8r<r<3s!F4818 *. #G_o$9%:; "#    '+$Wk?%JK #/4T;5F/G. %) #(+)>)8:M*N$O" .343D-E 6$= $!9"oo%)#"&KO0<#4'+(-+,'-x#,1,1+VQCI&VQC=VQ VQ# VQ d^ VQ c] VQ}VQ3-VQU3 #*3c6>sm7D1E,E$F$FGHVQ -VQC=VQ8$VQ  ~!VQ"$C=#VQ$V$%VQ&}'VQ($D>)VQ*$D>+VQ. t["77 8/VQVQp (A%=+/'(260:0D0DBLBL BLBL } BL c] BL $C= BL }BL"*$BL"*-BLJNBLBLH.F/3-1 $FS$F"$F%c]$F (}$F&c] $F7; $F$FL/G04.2"' $Pc$P$'$P&sm$P!) $P'sm $P  $P -1 $P$PL+D(@C "3-! FI4*.9="& 0sDI~&0XtH~560 0  0  0l:>)-##*.,0@@@XtH~56 @ sDI~& @  @@c3h@#tCy.)@@Dw :DcN :w :*. OKXtH~56OKsDI~&OK OKl $,0)-NNNXtH~56 N  N  N#tCy.)NsDI~&Nd(,#B&,BBr:r<)Frratypingrrrrrrr r r numpyrrr r rrrdatasets.packaged_modulesrdatasets.utils.file_utilsrmodelscope.hub.repositoryr4modelscope.msdatasets.context.dataset_context_configr5modelscope.msdatasets.data_loader.data_loader_managerrrrr!modelscope.msdatasets.dataset_clsrr9modelscope.msdatasets.dataset_cls.custom_datasets.builderr(modelscope.msdatasets.utils.delete_utilsr,modelscope.msdatasets.utils.hf_datasets_utilr(modelscope.msdatasets.utils.upload_utilsrmodelscope.preprocessorsrmodelscope.utils.configr r!rr"rr#r$r%r&r'r(r)r*r+r,modelscope.utils.import_utilsr-r.modelscope.utils.loggerr/rr9r<rr:r8rs %%%++@67FINI768DDD N. Q Q r: