DeMultiplexer

MapDataPipe for splitting a single datapipe into multiple

---

DemultiplexerMapDataPipe

 DemultiplexerMapDataPipe
                           (datapipe:torch.utils.data.datapipes.datapipe.M
                           apDataPipe, num_instances:int,
                           classifier_fn:Callable, drop_none:bool=False,
                           source_index:Union[Iterable,NoneType]=None)

Splits the input DataPipe into multiple child DataPipes, using the given classification function (functional name: demux). A list of the child DataPipes is returned from this operation.

For example we can have a function that splits a map of numbers into even and odd…

def odd_or_even(n):
    return n % 2
source_dp = dp.map.SequenceWrapper(range(5))
dp1, dp2 = source_dp.demux(num_instances=2, classifier_fn=odd_or_even)
list([dp1[i] for i in range(len(dp1))])

[0, 2, 4]

Note: The resulting datapipes from DemultiplexerMapDataPipe will be “re-indexed”.

list([dp2[i] for i in range(len(dp2))])

[1, 3]

It can also filter out any element that gets None from the classifier_fn

def odd_or_even_no_zero(n):
    return n % 2 if n != 0 else None

source_dp = dp.map.SequenceWrapper(range(5))
dp1, dp2 = source_dp.demux(num_instances=2, classifier_fn=odd_or_even_no_zero, drop_none=True)
list([dp1[i] for i in range(len(dp1))])

[2, 4]

list([dp2[i] for i in range(len(dp2))])

[1, 3]