Thanks to visit codestin.com
Credit goes to Github.com

Skip to content

Make adaptor chaining more flexible and ergonomic #16

@gendx

Description

@gendx

Currently, Paralight logically splits a chain of iterator adaptors into two phases, represented by the ParallelSource and ParallelIterator traits. The flow of affine types in a chain of iterators is the following:

T -> .[into_]par_iter[_mut]() -> impl ParallelSource -> ... -> .with_thread_pool() -> impl ParallelIterator -> ... -> Output

One issue is that the combinators available in ParallelSource and ParallelIterator (via the ParallelSourceExt and ParallelIteratorExt extension traits) are currently disjoint, which may be confusing to users or overly restrictive.

For example, the following isn't allowed, because .copied() must be put after the .with_thread_pool() call:

// Not allowed
foo.par_iter().copied().with_thread_pool(...).for_each(...);
// Allowed
foo.par_iter().with_thread_pool(...).copied().for_each(...);

Likewise, the following chain isn't allowed, because the .map() combinator isn't available on a ParallelSource context:

(
    foo.par_iter().map(...),
    bar.par_iter(),
).zip_eq().with_thread_pool(...).for_each(...);

These patterns should be made possible for ergonomics, or potentially to allow expressing patterns that aren't possible today.


Note however that some patterns are not allowed for good reason: for example chaining .enumerate() after .filter() isn't allowed with Paralight1, because there is no way to efficiently compute correct post-filtering indices without sequentially processing all the items.

foo
    .par_iter()
    .with_thread_pool(...)
    .filter(...)
    .enumerate(...)  // Not allowed.
    .etc(...);

Footnotes

  1. The other chaining .enumerate(...).filter(...) is allowed, but it produces a different result than .filter(...).enumerate(...) (as some indices will be skipped by the filtering).

Metadata

Metadata

Assignees

No one assigned

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions