Type Transforms

DataPipes for calling functions over map based __getitem__ calls

---

TypeTransformer

 TypeTransformer (*args, **kwds)

On __getitem__ functions in self.type_tfms get called over each element. Generally TypeTransformer as the name suggests is intended to convert elements from one type to another. reference documentation on how to combine this with InMemoryCacheHolder.

Similar to the fastrl.pipes.iter.transforms, TypeTransformer can be used to convert elements into different values…

add_one = lambda o:o+1
multiple_by_two = lambda o:o*2
pipe = TypeTransformer(range(10),[add_one,multiple_by_two])

results = [pipe[i] for i in range(10)] 
test_eq(results,[2, 4, 6, 8, 10, 12, 14, 16, 18, 20])

However if we want to convert types and keep them, we can combine TypeTransformer with InMemoryCacheHolder…

num2str = lambda o:str(o)
add_str_postfix = lambda o:o+'_postfix'
pipe = TypeTransformer(range(10),[num2str,add_str_postfix])
pipe = dp.map.InMemoryCacheHolder(pipe)

results = [pipe[i] for i in range(10)] 
test_eq(results,['0_postfix', '1_postfix', '2_postfix', '3_postfix', 
                 '4_postfix', '5_postfix', '6_postfix', '7_postfix', '8_postfix', '9_postfix'])