issue-1182: foreachPar, foreachPar_, collectAllPar, reduceAllPar #1773

hanny24 · 2019-09-22T08:52:58Z

Partially solves #1182, missing raceAll and forkAll.
Inspired by #1424

… issue-1182-take2

core-tests/shared/src/test/scala/zio/ZIOSpec.scala

freskog · 2019-09-22T10:10:29Z

core/shared/src/main/scala/zio/ZIO.scala

+    (for {
+      size    <- UIO.effectTotal(as.size)
+      todo    <- Ref.make(size)
+      buffer  <- UIO.effectTotal(new Array[Any](size).asInstanceOf[Array[B]])


Will this not cause a java.lang.ClassCastException ? A quick experiment from the scala repl:

scala> new Array[Any](100).asInstanceOf[Array[String]] java.lang.ClassCastException: [Ljava.lang.Object; cannot be cast to [Ljava.lang.String; ... 36 elided

I think the trick here is not to cast it to specific type, but only to a generic one.

I still don't get how this works at runtime though, because the array will need to get a concrete type at some stage right?

scala> def test[A](a:A):Array[A] = new Array[Any](1).asInstanceOf[Array[A]] test: [A](a: A)Array[A] scala> test(1) java.lang.ClassCastException: [Ljava.lang.Object; cannot be cast to [I ... 28 elided

Isn't is caused by specialization? There is even a test that uses foreachPar on a List[Int]: https://github.com/hanny24/scalaz-zio/blob/35bd04f28cedeb3c988a83af206c37d48d3b58ad/core-tests/shared/src/test/scala/zio/ZIOSpec.scala#L70

Yeah saw that test now. I have to admit that I don't understand how this works. It clearly does though :)

freskog · 2019-09-22T10:32:27Z

core/shared/src/main/scala/zio/ZIO.scala

+      c <- ZIO.uninterruptibleMask { restore =>
+            for {
+              as <- ZIO.traverse(as)(a => ZIO.interruptible(fn(a)).fork)
+              _ <- as.view.zipWithIndex.foldLeft[ZIO[R, E, _]](ZIO.unit) {


The call to .view here seems unnecessary, or am I missing something?

I did not want to materialize it. But if we change to ZIO.traverse we will need to materialize it anyway.

Rewritten in a way that uses ZIO.traverse_

freskog · 2019-09-22T10:39:25Z

core/shared/src/main/scala/zio/ZIO.scala

+              as <- ZIO.traverse(as)(a => ZIO.interruptible(fn(a)).fork)
+              _ <- as.view.zipWithIndex.foldLeft[ZIO[R, E, _]](ZIO.unit) {
+                    case (io, (f, idx)) =>
+                      io *> f.await.flatMap(arbiter(as, promise, buffer, todo, idx)).fork


An alternative to this is to combine this step with the ZIO.traverse(as)( ...) call above, so that one less fiber per element in as is created.

I don't think it actually that easy to combine it: we need to be able to interrupt all fibers from arbiter helper method.

freskog · 2019-09-22T10:48:44Z

core/shared/src/main/scala/zio/ZIO.scala

-        else ZIO.unit
-      }
-      .refailWithTrace
+  final def foreachPar_[R, E, A](as: Iterable[A])(f: A => ZIO[R, E, _]): ZIO[R, E, Unit] = {


This function is duplicating a lot from the foreachPar method, maybe this could be rewritten to leverage that function instead?

I know it's possible, I just wanted to avoid allocation of the extra array with results.

regiskuckaertz · 2019-09-22T15:45:04Z

core/shared/src/main/scala/zio/ZIO.scala

+    (for {
+      size    <- UIO.effectTotal(as.size)
+      todo    <- Ref.make(size)
+      buffer  <- UIO.effectTotal(new Array[Any](size).asInstanceOf[Array[B]])


Suggested change

buffer <- UIO.effectTotal(new Array[Any](size).asInstanceOf[Array[B]])

buffer <- UIO.effectTotal(Array.ofDim[B](size))

Also, it's probably best to move the buffer into a Ref.

What is gain of keeping mutable array in Ref? I think that reference will never change, so Ref#modify will never discover that value was changed.
Secondly - we know that each fiber operates only on it's index, so that are really independent operations.
What can go wrong?

Re: ofDim - I understand this gives better performance for primitive types but requires ClassTag[B] - do we want to change method signatures?

Perhaps answer for my first question is memory barrier.

regiskuckaertz · 2019-09-22T15:55:52Z

core/shared/src/main/scala/zio/ZIO.scala

+      promise <- Promise.make[E, List[B]]
+      c <- ZIO.uninterruptibleMask { restore =>
+            for {
+              as <- ZIO.traverse(as)(a => ZIO.interruptible(fn(a)).fork)


🤔 Can we look for a solution that only traverses the structure once and forks one fiber per element?

ZIO.traverse(as.zipWithIndex){ case (a, i) => fn(a).flatMap(arbiter(...)).fork }

Yes, in #1937

regiskuckaertz · 2019-09-22T15:56:59Z

core/shared/src/main/scala/zio/ZIO.scala

+        e => promise.halt(e).unit *> Fiber.interruptAll(fibers),
+        a =>
+          todo.modify { t =>
+            buffer.update(idx, a)


That's an effect which should be captured, making buffer a Ref[Array[B]] will help here.

regiskuckaertz · 2020-02-17T07:48:35Z

@hanny24 hi 👋 gonna close this as all these combinators have recently be refactor into stack-safe array-backed versions. Check out #2590

issue-1182: forachPar, forachPar_, collectAllPar, reduceAllPar

c577845

hanny24 force-pushed the issue-1182-take2 branch from 32333ad to c577845 Compare September 22, 2019 09:05

hanny24 added 2 commits September 22, 2019 11:19

issue-1182: forachPar, forachPar_, collectAllPar, reduceAllPar, forkAll

e6865a4

Merge branch 'issue-1182-take2' of github.com:hanny24/scalaz-zio into…

3986e6b

… issue-1182-take2

LGLO reviewed Sep 22, 2019

View reviewed changes

core-tests/shared/src/test/scala/zio/ZIOSpec.scala Outdated Show resolved Hide resolved

core-tests/shared/src/test/scala/zio/ZIOSpec.scala Outdated Show resolved Hide resolved

Merge remote-tracking branch 'origin/master' into issue-1182-take2

914bdb5

hanny24 force-pushed the issue-1182-take2 branch from 49bbb0c to 914bdb5 Compare September 22, 2019 09:51

freskog reviewed Sep 22, 2019

View reviewed changes

PR comments

35bd04f

LGLO changed the title ~~issue-1182: forachPar, forachPar_, collectAllPar, reduceAllPar~~ issue-1182: foreachPar, foreachPar_, collectAllPar, reduceAllPar Sep 22, 2019

regiskuckaertz reviewed Sep 22, 2019

View reviewed changes

Merge branch 'master' into issue-1182-take2

7df8206

This was referenced Oct 8, 2019

foreachPar tests #1930

Merged

Issue 1182: foreachPar, foreachPar_, collectAllPar, reduceAllPar #1937

Closed

regiskuckaertz closed this Feb 17, 2020

	buffer <- UIO.effectTotal(new Array[Any](size).asInstanceOf[Array[B]])
	buffer <- UIO.effectTotal(Array.ofDim[B](size))

Uh oh!

issue-1182: foreachPar, foreachPar_, collectAllPar, reduceAllPar #1773

issue-1182: foreachPar, foreachPar_, collectAllPar, reduceAllPar #1773

Uh oh!

Conversation

hanny24 commented Sep 22, 2019

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

regiskuckaertz commented Feb 17, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants