Thanks to visit codestin.com
Credit goes to github.com

Skip to content

issue-1182: foreachPar, foreachPar_, collectAllPar, reduceAllPar #1773

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 6 commits into from
Closed

issue-1182: foreachPar, foreachPar_, collectAllPar, reduceAllPar #1773

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
c577845
issue-1182: forachPar, forachPar_, collectAllPar, reduceAllPar
hanny24 Sep 22, 2019
e6865a4
issue-1182: forachPar, forachPar_, collectAllPar, reduceAllPar, forkAll
hanny24 Sep 22, 2019
3986e6b
Merge branch 'issue-1182-take2' of github.com:hanny24/scalaz-zio into…
hanny24 Sep 22, 2019
914bdb5
Merge remote-tracking branch 'origin/master' into issue-1182-take2
hanny24 Sep 22, 2019
35bd04f
PR comments
hanny24 Sep 22, 2019
7df8206
Merge branch 'master' into issue-1182-take2
LGLO Oct 6, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 28 additions & 0 deletions core-tests/shared/src/test/scala/zio/ZIOSpec.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -65,6 +65,34 @@ object ZIOSpec
assertM(ZIO.fail(42).raceAll(List(IO.succeed(24) <* live(ZIO.sleep(100.millis)))), equalTo(24))
}
),
suite("foreachPar")(
testM("runs effects in parallel") {
assertM(for {
p <- Promise.make[Nothing, Unit]
_ <- UIO.foreachPar(List(UIO.never, p.succeed(())))(a => a).fork
_ <- p.await
} yield true, isTrue)
},
testM("propagates error") {
val ints = List(1, 2, 3, 4, 5, 6)
val odds = ZIO.foreachPar(ints) { n =>
if (n % 2 != 0) ZIO.succeed(n) else ZIO.fail("not odd")
}
assertM(odds.flip, equalTo("not odd"))
},
testM("interrupts effects on first failure") {
for {
ref <- Ref.make(false)
promise <- Promise.make[Nothing, Unit]
actions = List(
ZIO.never,
ZIO.succeed(1),
ZIO.fail("C"),
promise.await *> ref.set(true)
)
e <- ZIO.foreachPar(actions)(a => a).flip
v <- ref.get
} yield assert(e, equalTo("C")) && assert(v, isFalse)
suite("option")(
testM("return success in Some") {
assertM(ZIO.succeed(11).option, equalTo[Option[Int]](Some(11)))
Expand Down
94 changes: 69 additions & 25 deletions core/shared/src/main/scala/zio/ZIO.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2022,29 +2022,78 @@ private[zio] trait ZIOFunctions extends Serializable {
*
* For a sequential version of this method, see `foreach`.
*/
final def foreachPar[R, E, A, B](as: Iterable[A])(fn: A => ZIO[R, E, B]): ZIO[R, E, List[B]] =
as.foldRight[ZIO[R, E, List[B]]](effectTotal(Nil)) { (a, io) =>
fn(a).zipWithPar(io)((b, bs) => b :: bs)
}
.refailWithTrace
final def foreachPar[R, E, A, B](as: Iterable[A])(fn: A => ZIO[R, E, B]): ZIO[R, E, List[B]] = {
def arbiter(
fibers: Iterable[Fiber[E, _]],
promise: Promise[E, List[B]],
buffer: Array[B],
todo: Ref[Int],
idx: Int
)(res: Exit[E, B]): ZIO[R, Nothing, Unit] =
res.foldM[R, Nothing, Unit](
e => promise.halt(e).unit *> Fiber.interruptAll(fibers),
a =>
todo.modify { t =>
buffer.update(idx, a)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's an effect which should be captured, making buffer a Ref[Array[B]] will help here.

if (t == 1) promise.succeed(buffer.toList).unit -> 0 else UIO.unit -> (t - 1)
}.flatten
)

(for {
size <- UIO.effectTotal(as.size)
todo <- Ref.make(size)
buffer <- UIO.effectTotal(new Array[Any](size).asInstanceOf[Array[B]])
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will this not cause a java.lang.ClassCastException ? A quick experiment from the scala repl:

scala> new Array[Any](100).asInstanceOf[Array[String]]
java.lang.ClassCastException: [Ljava.lang.Object; cannot be cast to [Ljava.lang.String;
  ... 36 elided

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the trick here is not to cast it to specific type, but only to a generic one.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I still don't get how this works at runtime though, because the array will need to get a concrete type at some stage right?

scala> def test[A](a:A):Array[A] = new Array[Any](1).asInstanceOf[Array[A]]
test: [A](a: A)Array[A]

scala> test(1)
java.lang.ClassCastException: [Ljava.lang.Object; cannot be cast to [I
  ... 28 elided

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Isn't is caused by specialization? There is even a test that uses foreachPar on a List[Int]: https://github.com/hanny24/scalaz-zio/blob/35bd04f28cedeb3c988a83af206c37d48d3b58ad/core-tests/shared/src/test/scala/zio/ZIOSpec.scala#L70

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah saw that test now. I have to admit that I don't understand how this works. It clearly does though :)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
buffer <- UIO.effectTotal(new Array[Any](size).asInstanceOf[Array[B]])
buffer <- UIO.effectTotal(Array.ofDim[B](size))

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, it's probably best to move the buffer into a Ref.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is gain of keeping mutable array in Ref? I think that reference will never change, so Ref#modify will never discover that value was changed.
Secondly - we know that each fiber operates only on it's index, so that are really independent operations.
What can go wrong?

Re: ofDim - I understand this gives better performance for primitive types but requires ClassTag[B] - do we want to change method signatures?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Perhaps answer for my first question is memory barrier.

promise <- Promise.make[E, List[B]]
c <- ZIO.uninterruptibleMask { restore =>
for {
as <- ZIO.traverse(as)(a => ZIO.interruptible(fn(a)).fork)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤔 Can we look for a solution that only traverses the structure once and forks one fiber per element?

ZIO.traverse(as.zipWithIndex){ case (a, i) => fn(a).flatMap(arbiter(...)).fork }

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For sure!

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, in #1937

_ <- ZIO.traverse_(as.zipWithIndex) {
case (f, idx) =>
f.await.flatMap(arbiter(as, promise, buffer, todo, idx)).fork
}
_ <- promise.succeed(Nil).when(as.isEmpty)
c <- restore(promise.await).onInterrupt(promise.interrupt *> Fiber.interruptAll(as))
} yield c
}
} yield c).refailWithTrace
}

/**
* Applies the function `f` to each element of the `Iterable[A]` and runs
* produced effects in parallel, discarding the results.
*
* For a sequential version of this method, see `foreach_`.
*/
final def foreachPar_[R, E, A](as: Iterable[A])(f: A => ZIO[R, E, _]): ZIO[R, E, Unit] =
ZIO
.succeed(as.iterator)
.flatMap { i =>
def loop(a: A): ZIO[R, E, Unit] =
if (i.hasNext) f(a).zipWithPar(loop(i.next))((_, _) => ())
else f(a).unit
if (i.hasNext) loop(i.next)
else ZIO.unit
}
.refailWithTrace
final def foreachPar_[R, E, A](as: Iterable[A])(f: A => ZIO[R, E, _]): ZIO[R, E, Unit] = {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This function is duplicating a lot from the foreachPar method, maybe this could be rewritten to leverage that function instead?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I know it's possible, I just wanted to avoid allocation of the extra array with results.

def arbiter(
fibers: Iterable[Fiber[E, _]],
promise: Promise[E, Unit],
todo: Ref[Int]
)(res: Exit[E, _]): ZIO[R, Nothing, Unit] =
res.foldM[R, Nothing, Unit](
e => promise.halt(e).unit *> Fiber.interruptAll(fibers),
_ =>
todo.modify { t =>
if (t == 1) promise.succeed(()).unit -> 0 else UIO.unit -> (t - 1)
}.flatten
)

(for {
size <- UIO.effectTotal(as.size)
todo <- Ref.make(size)
promise <- Promise.make[E, Unit]
c <- ZIO.uninterruptibleMask { restore =>
for {
as <- ZIO.traverse(as)(a => ZIO.interruptible(f(a)).fork)
_ <- ZIO.traverse_(as) { f =>
f.await.flatMap(arbiter(as, promise, todo)).fork
}
_ <- promise.succeed(()).when(as.isEmpty)
c <- restore(promise.await).onInterrupt(promise.interrupt *> Fiber.interruptAll(as))
} yield c
}
} yield c).refailWithTrace
}

/**
* Applies the function `f` to each element of the `Iterable[A]` in parallel,
Expand Down Expand Up @@ -2092,12 +2141,7 @@ private[zio] trait ZIOFunctions extends Serializable {
* composite fiber that produces a list of their results, in order.
*/
final def forkAll[R, E, A](as: Iterable[ZIO[R, E, A]]): ZIO[R, Nothing, Fiber[E, List[A]]] =
as.foldRight[ZIO[R, Nothing, Fiber[E, List[A]]]](succeed(Fiber.succeed[E, List[A]](List()))) { (aIO, asFiberIO) =>
asFiberIO.zip(aIO.fork).map {
case (asFiber, aFiber) =>
asFiber.zipWith(aFiber)((as, a) => a :: as)
}
}
foreachPar(as)(v => v).fork

/**
* Returns an effect that forks all of the specified values, and returns a
Expand Down Expand Up @@ -2258,7 +2302,7 @@ private[zio] trait ZIOFunctions extends Serializable {
final def mergeAllPar[R, E, A, B](
in: Iterable[ZIO[R, E, A]]
)(zero: B)(f: (B, A) => B): ZIO[R, E, B] =
in.foldLeft[ZIO[R, E, B]](succeed[B](zero))((acc, a) => acc.zipPar(a).map(f.tupled)).refailWithTrace
collectAllPar(in).map(_.foldLeft(zero)(f)).refailWithTrace

/**
* Returns an effect with the empty value.
Expand Down Expand Up @@ -2308,8 +2352,8 @@ private[zio] trait ZIOFunctions extends Serializable {
final def reduceAllPar[R, R1 <: R, E, A](a: ZIO[R, E, A], as: Iterable[ZIO[R1, E, A]])(
f: (A, A) => A
): ZIO[R1, E, A] =
as.foldLeft[ZIO[R1, E, A]](a) { (l, r) =>
l.zipPar(r).map(f.tupled)
a.zipPar(collectAllPar(as)).map {
case (head, tail) => tail.fold(head)(f)
}

/**
Expand Down