9i BSSKrSSKJr SSKJrJr SSKrSSKJ r SSK J r SSK J r Jr SSKJr \"5rSrSS \S \\S \\4S jjrS \ S\S\S\S\S\S\4SjrS\4SjrS \\\4S\S\4SjrS \\\4S\4SjrSr\ 4S\S\S\S\ S\\4 Sjjrg)N) defaultdict)OptionalUnion)HubApi)DatasetContextConfig)DEFAULT_DATASET_REVISIONMetaDataFields) get_loggercUR5VVs0sH5upURS5(dURS5(dM3X_M7 snn$s snnf)Nmetafile)itemsget)dataset_structurekvs i/var/www/html/land-doc-ocr/venv/lib/python3.13/site-packages/modelscope/msdatasets/utils/dataset_utils.pyformat_dataset_structurersO&++- -DA EE&MMQUU6] -  s 2A Ar subset_namesplitcU(aX;d$U(d<[UR55S:a[SUSUR535eUnU(d:[[ UR555n[ R SU35 [X5nU(a$X$;a[SUSUR535eU(aX$U0nX44$)a  Args: dataset_structure (dict): Dataset Structure, like { "default":{ "train":{ "meta":"my_train.csv", "file":"pictures.zip" } }, "subsetA":{ "test":{ "meta":"mytest.csv", "file":"pictures.zip" } } } subset_name (str, optional): Defining the subset_name of the dataset. split (str, optional): Which split of the data to load. Returns: target_subset_name (str): Name of the chosen subset. target_dataset_structure (dict): Structure of the chosen split(s), like { "test":{ "meta":"mytest.csv", "file":"pictures.zip" } } z subset_name z not found. Available: z,No subset_name specified, defaulting to the zsplit )lenkeys ValueErrornextiterloggerinfor)rrrtarget_subset_nametarget_dataset_structures rget_target_dataset_structurer"sB  <$5$:$:$< = A;-'>?P?U?U?W>X Y  % !$'8'='='?"@A :;M:N O  8- / 6UG23K3P3P3R2S T   $)E+J#K  77hub_api max_limit is_recursive dataset_name namespaceversionreturnc /nURUUUUSUS9nUH.nURS5n U (dMURU 5 M0 U$)a List all objects for specific dataset. Args: hub_api (class HubApi): HubApi instance. max_limit (int): Max number of objects. is_recursive (bool): Whether to list objects recursively. dataset_name (str): Dataset name. namespace (str): Namespace. version (str): Dataset version. Returns: res (list): List of objects, i.e., ['train/images/001.png', 'train/images/002.png', 'val/images/001.png', ...] T)r'r(r%r& is_filter_dirrevisionKey)list_oss_dataset_objectsrappend) r$r%r&r'r(r)resobjectsitem object_keys rlist_dataset_objectsr5Rsb C..!! /GXXe_   :  Jr#cSnUR5H7up#[U[5(dMURS5(aM4Sn U$ U$)z To check whether input contains at least one directory. Args: file_map (dict): Structure of data files. e.g., {'train': 'train.zip', 'validation': 'val.zip'} Returns: True if input contains at least one directory, False otherwise. Fz.zipT)r isinstancestrendswith)file_mapr1rrs r contains_dirr;tsK C  a  ajj&8&8C  J ! Jr#c[U[5(aU/n[RR USR U55$)N_)r7r8ospathjoin)rr)s rget_subdir_hash_from_splitrAs3% 77<<% 11r#c~[U[5(aU/$[U[5(aU$S[U5S3e)z Unify the split to list-format. z/Expected format of split: str or list, but got .)r7r8listtype)rs rget_split_listrFs;%w E4  ?U }ANNr#c0nUR5H up4/X#'M UHYnUR5HBup4URURS5S-5(dM/X#RU5 MD M[ U$)a Get the map between dataset split and oss objects. Args: file_map (dict): Structure of data files. e.g., {'train': 'train', 'validation': 'val'}, both of train and val are dirs. objects (list): List of oss objects. e.g., ['train/001/1_123.png', 'train/001/1_124.png', 'val/003/3_38.png'] Returns: A map of split-objects. e.g., {'train': ['train/001/1_123.png', 'train/001/1_124.png'], 'validation':['val/003/3_38.png']} /)r startswithrstripr0)r:r2r1rrobj_keys rget_split_objects_maprLsu C !NN$DA!!!((3-#"566 g&% Jr#subset_split_intocontext_configr-c [[5n[[5n[[5n[[5n[5n URRn UR 5HnupU R SS5X'U RU R SS5XU5X['U R S5(aU SXk'U R S5X{'Mp /n UR 5GHupU(dMUR [R5(dM5X[n[R"X5nURR SS5n[R"USUS S 9nURURRR!S 5R#5n[%U5S :Xa)[&R)S US35 URS nOUS nUUR#5n XU 'GM U (d)[+U SSUUUS9n [-U5(a [/Xm5nXVXx4$)z Return: meta_map: Structure of meta files (.csv), the meta file name will be replaced by url, like { "test": "https://xxx/mytest.csv" } file_map: Structure of data files (.zip), like { "test": "pictures.zip" } customr r args delimiter,F\)iteratorrS escapecharz:FILErzNo column contains ":FILE" in rCT)r$r%r&r'r(r))rdictrdata_meta_configmeta_cache_dirrrget_dataset_file_url_originr ARGS_BIG_DATAfetch_meta_files_from_url config_kwargspdread_csvcolumnsr8containsto_listrrerrorr5r;rL)rMr'r(rNr-meta_mapr:args_mapcustom_type_mapmodelscope_apir[rrr2 args_dictmeta_csv_file_urlmeta_csv_file_path csv_delimitercsv_df target_cols rget_dataset_filesrps 4 H4 H4 H!$'OXN#44CCN(..0 !%(B!7(DD HHVR ,8E 88F  "6lHO((6* 1G$NN, 9~'C'CDD ( !'!A!A!"3 +88<<[#NM[["' !F  (:(:(C(C)!') :!# 45G4HJL#^^A. '] Z(002G%UO/-2 &"%   ! !,X?H x 88r#)NN)r> collectionsrtypingrrpandasr`modelscope.hub.apir4modelscope.msdatasets.context.dataset_context_configrmodelscope.utils.constantrr modelscope.utils.loggerr rrrYr8r"intboolrDr5r;rArFrLrpr#rr{s) #"%N. ?C8<58D58.6sm58(0 58p&S'*7:"%*.Dd"2eCI&6222 O%T *OtO81I E9E9$'E9!$E9';E9!) E9r#