-
Notifications
You must be signed in to change notification settings - Fork 2
Description
Currently, Paralight logically splits a chain of iterator adaptors into two phases, represented by the ParallelSource and ParallelIterator traits. The flow of affine types in a chain of iterators is the following:
T -> .[into_]par_iter[_mut]() -> impl ParallelSource -> ... -> .with_thread_pool() -> impl ParallelIterator -> ... -> Output
One issue is that the combinators available in ParallelSource and ParallelIterator (via the ParallelSourceExt and ParallelIteratorExt extension traits) are currently disjoint, which may be confusing to users or overly restrictive.
For example, the following isn't allowed, because .copied() must be put after the .with_thread_pool() call:
// Not allowed
foo.par_iter().copied().with_thread_pool(...).for_each(...);
// Allowed
foo.par_iter().with_thread_pool(...).copied().for_each(...);Likewise, the following chain isn't allowed, because the .map() combinator isn't available on a ParallelSource context:
(
foo.par_iter().map(...),
bar.par_iter(),
).zip_eq().with_thread_pool(...).for_each(...);These patterns should be made possible for ergonomics, or potentially to allow expressing patterns that aren't possible today.
Note however that some patterns are not allowed for good reason: for example chaining .enumerate() after .filter() isn't allowed with Paralight1, because there is no way to efficiently compute correct post-filtering indices without sequentially processing all the items.
foo
.par_iter()
.with_thread_pool(...)
.filter(...)
.enumerate(...) // Not allowed.
.etc(...);Footnotes
-
The other chaining
.enumerate(...).filter(...)is allowed, but it produces a different result than.filter(...).enumerate(...)(as some indices will be skipped by the filtering). ↩