3KibL SSKrSSKrSSKrSSKrSSKJr SSKJrJrJ r J r SSK J r SSK JrJrJrJr /SQr\ "S5r\ "SS S 9r\\\4r\\S 4r\ "S \\5r"S S\\5r"SS\\\ \5r"SS\\\S 45r"SS\\5r"SS\\5r"SS\5r "SS\\5r!\4S\\S\\"\#-S\S-S\$\!\4Sjjr%g) N)Sequence)castGenericIterableTypeVar) deprecated)default_generator GeneratorrandpermTensor)DatasetIterableDataset TensorDataset StackDataset ConcatDataset ChainDatasetSubset random_split_T_T_coT) covariant._T_stackc0\rSrSrSrS\4SjrSSjrSrg) r 'a}An abstract class representing a :class:`Dataset`. All datasets that represent a map from keys to data samples should subclass it. All subclasses should overwrite :meth:`__getitem__`, supporting fetching a data sample for a given key. Subclasses could also optionally overwrite :meth:`__len__`, which is expected to return the size of the dataset by many :class:`~torch.utils.data.Sampler` implementations and the default options of :class:`~torch.utils.data.DataLoader`. Subclasses could also optionally implement :meth:`__getitems__`, for speedup batched samples loading. This method accepts list of indices of samples of batch and returns list of samples. .. note:: :class:`~torch.utils.data.DataLoader` by default constructs an index sampler that yields integral indices. To make it work with a map-style dataset with non-integral indices/keys, a custom sampler must be provided. returnc[S5e)Nz3Subclasses of Dataset should implement __getitem__.)NotImplementedErrorselfindexs Z/var/www/html/dynamic-report/venv/lib/python3.13/site-packages/torch/utils/data/dataset.py __getitem__Dataset.__getitem__:s!"WXXc[X/5$N)rrothers r!__add__Dataset.__add__Asd]++r$N)r(zDataset[_T_co]rzConcatDataset[_T_co]) __name__ __module__ __qualname____firstlineno____doc__rr"r)__static_attributes__r+r$r!r r 's$YEY,r$r c,\rSrSrSrS\\4SjrSrg)rIaAn iterable Dataset. All datasets that represent an iterable of data samples should subclass it. Such form of datasets is particularly useful when data come from a stream. All subclasses should overwrite :meth:`__iter__`, which would return an iterator of samples in this dataset. When a subclass is used with :class:`~torch.utils.data.DataLoader`, each item in the dataset will be yielded from the :class:`~torch.utils.data.DataLoader` iterator. When :attr:`num_workers > 0`, each worker process will have a different copy of the dataset object, so it is often desired to configure each copy independently to avoid having duplicate data returned from the workers. :func:`~torch.utils.data.get_worker_info`, when called in a worker process, returns information about the worker. It can be used in either the dataset's :meth:`__iter__` method or the :class:`~torch.utils.data.DataLoader` 's :attr:`worker_init_fn` option to modify each copy's behavior. Example 1: splitting workload across all workers in :meth:`__iter__`:: >>> # xdoctest: +REQUIRES(env:TORCH_DOCTEST_DATALOADER) >>> # xdoctest: +SKIP("Fails on MacOS12") >>> class MyIterableDataset(torch.utils.data.IterableDataset): ... def __init__(self, start, end): ... super(MyIterableDataset).__init__() ... assert end > start, "this example only works with end >= start" ... self.start = start ... self.end = end ... ... def __iter__(self): ... worker_info = torch.utils.data.get_worker_info() ... if worker_info is None: # single-process data loading, return the full iterator ... iter_start = self.start ... iter_end = self.end ... else: # in a worker process ... # split workload ... per_worker = int(math.ceil((self.end - self.start) / float(worker_info.num_workers))) ... worker_id = worker_info.id ... iter_start = self.start + worker_id * per_worker ... iter_end = min(iter_start + per_worker, self.end) ... return iter(range(iter_start, iter_end)) ... >>> # should give same set of data as range(3, 7), i.e., [3, 4, 5, 6]. >>> ds = MyIterableDataset(start=3, end=7) >>> # Single-process loading >>> print(list(torch.utils.data.DataLoader(ds, num_workers=0))) [tensor([3]), tensor([4]), tensor([5]), tensor([6])] >>> # xdoctest: +REQUIRES(POSIX) >>> # Multi-process loading with two worker processes >>> # Worker 0 fetched [3, 4]. Worker 1 fetched [5, 6]. >>> # xdoctest: +IGNORE_WANT("non deterministic") >>> print(list(torch.utils.data.DataLoader(ds, num_workers=2))) [tensor([3]), tensor([5]), tensor([4]), tensor([6])] >>> # With even more workers >>> # xdoctest: +IGNORE_WANT("non deterministic") >>> print(list(torch.utils.data.DataLoader(ds, num_workers=12))) [tensor([3]), tensor([5]), tensor([4]), tensor([6])] Example 2: splitting workload across all workers using :attr:`worker_init_fn`:: >>> # xdoctest: +REQUIRES(env:TORCH_DOCTEST_DATALOADER) >>> class MyIterableDataset(torch.utils.data.IterableDataset): ... def __init__(self, start, end): ... super(MyIterableDataset).__init__() ... assert end > start, "this example only works with end >= start" ... self.start = start ... self.end = end ... ... def __iter__(self): ... return iter(range(self.start, self.end)) ... >>> # should give same set of data as range(3, 7), i.e., [3, 4, 5, 6]. >>> ds = MyIterableDataset(start=3, end=7) >>> # Single-process loading >>> print(list(torch.utils.data.DataLoader(ds, num_workers=0))) [3, 4, 5, 6] >>> >>> # Directly doing multi-process loading yields duplicate data >>> print(list(torch.utils.data.DataLoader(ds, num_workers=2))) [3, 3, 4, 4, 5, 5, 6, 6] >>> # Define a `worker_init_fn` that configures each dataset copy differently >>> def worker_init_fn(worker_id): ... worker_info = torch.utils.data.get_worker_info() ... dataset = worker_info.dataset # the dataset copy in this worker process ... overall_start = dataset.start ... overall_end = dataset.end ... # configure the dataset to only process the split workload ... per_worker = int(math.ceil((overall_end - overall_start) / float(worker_info.num_workers))) ... worker_id = worker_info.id ... dataset.start = overall_start + worker_id * per_worker ... dataset.end = min(dataset.start + per_worker, overall_end) ... >>> # Mult-process loading with the custom `worker_init_fn` >>> # Worker 0 fetched [3, 4]. Worker 1 fetched [5, 6]. >>> print(list(torch.utils.data.DataLoader(ds, num_workers=2, worker_init_fn=worker_init_fn))) [3, 5, 4, 6] >>> # With even more workers >>> print(list(torch.utils.data.DataLoader(ds, num_workers=12, worker_init_fn=worker_init_fn))) [3, 4, 5, 6] r(c[X/5$r&)rr's r!r)IterableDataset.__add__sTM**r$r+N) r,r-r.r/r0r rr)r1r+r$r!rrIsjX+WU^+r$rcT\rSrSr%Sr\\S4\S'S\SS4SjrSr S\ 4S jr S r g) rzDataset wrapping tensors. Each sample will be retrieved by indexing tensors along the first dimension. Args: *tensors (Tensor): tensors that have the same size of the first dimension. .tensorsrNc^^[U4SjT55(a [S5eTUlg)Nc3p># UH+nTSRS5URS5:gv M- g7f)rN)size).0tensorr8s r! )TensorDataset.__init__..s+J'wqzq!V[[^3's36zSize mismatch between tensors)allAssertionErrorr8)rr8s `r!__init__TensorDataset.__init__s' J'J J J !@A A r$cB^[U4SjUR55$)Nc3,># UH oTv M g7fr&r+)r<r=r s r!r>,TensorDataset.__getitem__..s>vE])tupler8rs `r!r"TensorDataset.__getitem__s>>>>r$c>URSRS5$Nr)r8r;rs r!__len__TensorDataset.__len__s||A##A&&r$)r8) r,r-r.r/r0rHr __annotations__rBr"intrMr1r+r$r!rrs<63; D ?''r$rcn\rSrSr%Sr\\-\S'S\\ S\\ SS4Sjr S r S \ 4S jr S\4S jrS rg)raMDataset as a stacking of multiple datasets. This class is useful to assemble different parts of complex input data, given as datasets. Example: >>> # xdoctest: +SKIP >>> images = ImageDataset() >>> texts = TextDataset() >>> tuple_stack = StackDataset(images, texts) >>> tuple_stack[0] == (images[0], texts[0]) >>> dict_stack = StackDataset(image=images, text=texts) >>> dict_stack[0] == {"image": images[0], "text": texts[0]} Args: *args (Dataset): Datasets for stacking returned as tuple. **kwargs (Dataset): Datasets for stacking returned as dict. datasetsargskwargsrNc^U(aRU(a [S5e[US5Tl[U4SjU55(a [S5eUTlgU(aY[ UR 55n[US5Tl[U4SjU55(a [S5eUTlg[S5e)NztSupported either ``tuple``- (via ``args``) or``dict``- (via ``kwargs``) like input/output, but both types are given.rc3T># UHnTR[U5:gv M g7fr&_lengthlenr<datasetrs r!r>(StackDataset.__init__..sDtG4<<3w</t%(zSize mismatch between datasetsc3T># UHnTR[U5:gv M g7fr&rXr[s r!r>r]sCsG4<<3w</sr^z%At least one dataset should be passed) ValueErrorrZrYanyrSlistvalues)rrTrUtmps` r!rBStackDataset.__init__s  ^tAw# UH oTv M g7fr&r+)r<r\r s r!r>+StackDataset.__getitem__..sA=U^=rG) isinstancerSdictitemsrH)rr kr\s ` r!r"StackDataset.__getitem__s\ dmmT * *8< 8K8K8MN8M*!Au~%8MN NA4==AAAOsA2indicesc [UR[5(aUVs/sHn0PM nnURR5HupE[ [ USS55(aeUR U5n[U5[U5:wa#[S[U5S[U535e[XcSS9H upxXxU'M M[XSS9H upXYX'M M U$UVs/sHn/PM n nURHn[ [ USS55(arUR U5n[U5[U5:wa#[S[U5S[U535e[XjSS9Hup{U RU5 M M[XSS9HupU RXY5 M M U V s/sHn [U 5PM n n U $s snfs snfs sn f)N __getitems__z0Nested dataset's output size mismatch. Expected z, got Tstrict) rirSrjrkcallablegetattrrprZr`zipappendrH)rrn_ dict_batchrlr\rkdatad_sampleidx list_batcht_samplesample tuple_batchs r!rpStackDataset.__getitems__s dmmT * *5<(=WWJ(="mm113 GG^TBCC#009E5zS\1()),WfSZLJ+.e*M&* +N*-W)N &-l *O4 /6!6g"g !6}}G>??,,W5u:W-$%%(\N&U F'*%D&INDOOD)'J&)T%JMCOOGL1&K%DN&N:uV}: &NA)>""7'Os G%4 G* G/cUR$r&)rYrLs r!rMStackDataset.__len__'s ||r$)rYrS)r,r-r.r/r0rHrjrOr rrBr"rbrprPrMr1r+r$r!rrsX$dlFgenFF4F(B #D#Jr$rc^\rSrSr%Sr\\\\S'\\ \S'\ S5r S\ \SS4U4Sjjr S\ 4S jrS r\\"S \S 9S 55rSrU=r$)ri+zDataset as a concatenation of multiple datasets. This class is useful to assemble different existing datasets. Args: datasets (sequence): List of datasets to be concatenated rScumulative_sizescb/Sp!UH%n[U5nURXB-5 X$- nM' U$rK)rZrv)sequencersels r!cumsumConcatDataset.cumsum7s711AAA HHQUO FAr$rNc0>[TU]5 [U5Ul[ UR5S:Xa [ S5eURH#n[ U[5(dM[ S5e URUR5Ul g)Nrz(datasets should not be an empty iterablez.ConcatDataset does not support IterableDataset) superrBrbrSrZrArirrr)rrSd __class__s r!rBConcatDataset.__init__@st X t}}  " !KL LA!_--$%UVV!% DMM :r$c URS$)NrrLs r!rMConcatDataset.__len__Js$$R((r$cUS:a)U*[U5:a [S5e[U5U-n[R"URU5nUS:XaUnOXRUS- - nUR UU$)Nrz8absolute value of index should not exceed dataset length)rZr`bisect bisect_rightrrS)rr{ dataset_idx sample_idxs r!r"ConcatDataset.__getitem__Ms 7tc$i Nd)c/C))$*?*?E ! J44[1_EEJ}}[)*55r$z>`cummulative_sizes` attribute is renamed to `cumulative_sizes`)categorycUR$r&rrLs r!cummulative_sizesConcatDataset.cummulative_sizes[s $$$r$)rrS)r,r-r.r/r0rbr rrOrP staticmethodrrrBrMr"propertyr FutureWarningrr1 __classcell__rs@r!rr+s75>""3i;'!2;t;)) 6H%  %r$rcR^\rSrSrSrS\\SS4U4SjjrSrS\ 4Sjr S r U=r $) ridaGDataset for chaining multiple :class:`IterableDataset` s. This class is useful to assemble different existing dataset streams. The chaining operation is done on-the-fly, so concatenating large-scale datasets with this class will be efficient. Args: datasets (iterable of IterableDataset): datasets to be chained together rSrNc.>[TU]5 Xlgr&)rrBrS)rrSrs r!rBChainDataset.__init__os  r$c## URH-n[U[5(d [S5eUShvN M/ gN 7f)N*ChainDataset only supports IterableDataset)rSrirrA)rrs r!__iter__ChainDataset.__iter__ss8Aa11$%QRRLL s5AA AcSnURH1n[U[5(d [S5eU[ U5- nM3 U$)Nrr)rSrirrArZ)rtotalrs r!rMChainDataset.__len__ysBAa11$%QRR SVOE r$)rS) r,r-r.r/r0rr rBrrPrMr1rrs@r!rrds6!'!2!t! r$rc\rSrSr%Sr\\\S'\\ \S'S\\S\\ SS4Sjr Sr S\ \ S\ \4S jr S\ 4S jrS rg) riz Subset of a dataset at specified indices. Args: dataset (Dataset): The whole Dataset indices (sequence): Indices in the whole set selected for subset r\rnrNcXlX lgr&r\rn)rr\rns r!rBSubset.__init__s   r$c[U[5(a,URUVs/sHo RUPM sn$URURU$s snfr&)rirbr\rn)rr{is r!r"Subset.__getitem__sP c4 <<# >#Qa# >? ?||DLL-..!?sAc([[URSS55(a8URRUVs/sHo RUPM sn5$UVs/sHo RURUPM sn$s snfs snf)Nrp)rsrtr\rprn)rrnr{s r!rpSubset.__getitems__su GDLL.$? @ @<<,,7-S7Cll3.?7-ST T?FGwLLc!23wG G.TGs B "%Bc,[UR5$r&)rZrnrLs r!rMSubset.__len__s4<<  r$r)r,r-r.r/r0r rrOrrPrBr"rbrprMr1r+r$r!rrsnU^ c]#4/ HDIH$u+H!!r$rr\lengths generatorrc l[R"[U5S5(a[U5S::a/n[U5HSupEUS:dUS:a[ SUS35e[R "[ U5U-5nURU5 MU [ U5[U5- n[U5HnU[ U5-nX8==S- ss'M Un[U5H&upIU S:XdM [R"SUS3SS9 M( [U5[ U5:wa [ S 5e[[U5US 9R5n [[[U5n[!["R$"U5US S 9V V s/sHup['X X- U 5PM sn n $s sn n f) a Randomly split a dataset into non-overlapping new datasets of given lengths. If a list of fractions that sum up to 1 is given, the lengths will be computed automatically as floor(frac * len(dataset)) for each fraction provided. After computing the lengths, if there are any remainders, 1 count will be distributed in round-robin fashion to the lengths until there are no remainders left. Optionally fix the generator for reproducible results, e.g.: Example: >>> # xdoctest: +SKIP >>> generator1 = torch.Generator().manual_seed(42) >>> generator2 = torch.Generator().manual_seed(42) >>> random_split(range(10), [3, 7], generator=generator1) >>> random_split(range(30), [0.3, 0.3, 0.4], generator=generator2) Args: dataset (Dataset): Dataset to be split lengths (sequence): lengths or fractions of splits to be produced generator (Generator): Generator used for the random permutation. rrzFraction at index z is not between 0 and 1zLength of split at index z- is 0. This might result in an empty dataset.) stacklevelzDSum of input lengths does not equal the length of the input dataset!)rTrq)mathisclosesum enumerater`floorrZrvrangewarningswarnr tolistrrrPru itertools accumulater) r\rrsubset_lengthsrfracn_items_in_split remainder idx_to_add_atlengthrnoffsets r!rrs< ||CL!$$W):$& )GAax4!8 #5aS8O!PQQ#zz#g,*=>   ! !"2 3 * L3~#66 y!AN 33M  )Q . )"!"7+IA{ /s3=> , 7|s7|# R  s7|y9@@BG8C='*G")"6"6w"?QUV VNF w&9:V  sF0)&rrrrcollections.abcrtypingrrrrtyping_extensionsrtorchr r r r __all__rrrjstr_T_dictrH_T_tuplerr rrrrrrrPfloatrbrr+r$r!rs?  $43(A@  T]4( sEz    :x 1,gen,Dn+genhuon+h'GE&#+./'.T78$Tn6%GEN6%r?<!WU^!H#4> R[> cEk ">4> &* >r$