Cause filter fix #9254

eyalfa · 2024-10-22T05:56:52Z

found a couple of bugs in Cause.filter(..) while hacking on something completely different...
fix seemed quite easy, so here goes...

kyri-petrou · 2024-10-22T12:48:35Z

core/shared/src/main/scala/zio/Cause.scala

      def bothCase(context: Any, left: Cause[E], right: Cause[E]): Cause[E] =
-        if (p(left)) {
-          if (p(right)) Cause.Both(left, right)
+        if (left.nonEmpty) {


What's the rationale behind this change? Shouldn't we use the predicate when deciding if we're going to include left or right?

left and right are already filtered at this point

I think the fact that are already filtered is because of how bothCase and thenCase are used currently. I would be more comfortable keeping p(c) instead of assuming they're going to be filtered by the time we call these methods.

I actually think it's guaranteed by the fold semantics, furthermore applying p(c) won't be any cheaper thatn testing for emptiness.

My point is that while it's guaranteed by the fold semantics, what's not guaranteed is that the bothCase and thenCase methods might be used in a method other than fold. Since Folder.Filter is a public class (not sure why, but we can't ignore that it is), this test will fail using the implementation in this PR:

test("filter(false)") { val filter = Cause.Folder.Filter[Unit](_ => false) val cause = filter.bothCase((), f1, f2) assertTrue(cause.isEmpty) }

I'm also not sure why the Filter class exists as a class instead of a fold call or anonymous class, nevertheless seems like it's being public is simply a miss-sight.
anyway, as a Folder it's intended to be used by folds, I guess almost any other Folder implementation can be abused in similar ways to your example... what matters here is the semantics of filtering a Cause which are currently broken and have to be fixed, I think modifying the existing code is the path of least resistance, if you disagree we can always deprecate the exiting class and create a new one or simply replace it with a private/anonymous one.

I think for now let's go with using p(c) instead of nonEmpty in bothCase and thenCase. I'm planning on looking into the performance of error paths in the future (including Cause), so maybe add a TODO comment so that we don't forget it

core/shared/src/main/scala/zio/Cause.scala

kyri-petrou

By the way do we need to also modify the stacklessCase method to be aligned with failCase etc?

kyri-petrou · 2024-11-01T02:19:41Z

core/shared/src/main/scala/zio/Cause.scala

      def bothCase(context: Any, left: Cause[E], right: Cause[E]): Cause[E] =
-        if (p(left)) {
-          if (p(right)) Cause.Both(left, right)
+        if (left.nonEmpty) {


I think for now let's go with using p(c) instead of nonEmpty in bothCase and thenCase. I'm planning on looking into the performance of error paths in the future (including Cause), so maybe add a TODO comment so that we don't forget it

eyalfa · 2024-11-04T20:24:33Z

@kyri-petrou
I think replacing nonEmpty with p(c) is a beaking change due to multiple reasons:

the original code never applied p over a Then or a Both, doing so may introduce regressions.
it's quite possible these applications of p may fail with runtime match errors. (I suspect they are implemented using pattern matching).

kyri-petrou · 2024-11-05T05:44:24Z

the original code never applied p over a Then or a Both, doing so may introduce regressions.

@eyalfa I'm not sure I understand your comment. This is the current code for thenCase which uses p. Am I misunderstanding something?

      def thenCase(context: Any, left: Cause[E], right: Cause[E]): Cause[E] =
        if (p(left)) {
          if (p(right)) Cause.Then(left, right)
          else left
        } else if (p(right)) right
        else Cause.empty

eyalfaZS · 2024-11-06T10:26:34Z

the original code never applied p over a Then or a Both, doing so may introduce regressions.

@eyalfa I'm not sure I understand your comment. This is the current code for thenCase which uses p. Am I misunderstanding something?
      def thenCase(context: Any, left: Cause[E], right: Cause[E]): Cause[E] =
        if (p(left)) {
          if (p(right)) Cause.Then(left, right)
          else left
        } else if (p(right)) right
        else Cause.empty

it never applies p on the Then instance itself, it only considers it for composition.

kyri-petrou · 2024-11-07T04:30:49Z

it never applies p on the Then instance itself, it only considers it for composition.

@eyalfa I'm not suggesting we should apply p on the Then instance itself, just p(left) and p(right) instead of left.nonEmpty and right.nonEmpty in this PR

eyalfaZS · 2024-11-07T11:58:30Z

the original code never applied p over a Then or a Both, doing so may introduce regressions.

@eyalfa I'm not sure I understand your comment. This is the current code for thenCase which uses p. Am I misunderstanding something?
      def thenCase(context: Any, left: Cause[E], right: Cause[E]): Cause[E] =
        if (p(left)) {
          if (p(right)) Cause.Then(left, right)
          else left
        } else if (p(right)) right
        else Cause.empty

it would in the case of a composite Then (when one of the side is also a Then)

jdegoes · 2024-11-08T22:09:01Z

@eyalfa Would love to get this fix in. Looks like @kyri-petrou suggested a minor tweak, and then looks good to merge!

eyalfaZS · 2024-11-09T07:09:06Z

@jdegoes , @kyri-petrou ,
sorry for being less responsive the past few days, I've taken a new position and last few days were packed.

I think Kyrie's suggestion changes the semantics of the filter in such a way that it may break existing code.
original implementation applied p only on the 'leafs', never on the Both and Then cases (Stackless neither), Kyrie's suggestion does that, i.e. when we have something like Both(c0, Both(c1, c2)) p would be applied on the nested Both as well which is likely to fail if p is implemented as a partial function.

looking at the original code it's quite clear that Both, Then and Stackless are treated as containers/collections of causes and are never filtered on their own. I think a better approach is to replaced filtered out causes with a Cause.Empty and pattern match when composing - which is very similar to what the current code is doing (with a bug 😎 ).

another issue I see with this operator is losing annotations, seems like there are two 'levels' of implementing a Fold and for some reason Filter chose the one losing annotations.

CLAassistant · 2024-11-09T09:06:41Z

All committers have signed the CLA.

eyalfa · 2024-11-09T18:52:40Z

@kyri-petrou I think this is the best way to go, if you're still concerned about anyone using this class directly we can always deprecate it and replace it with a new one or just use the fold override taking lambdas

kyri-petrou · 2024-11-11T06:03:28Z

I think Kyrie's suggestion changes the semantics of the filter in such a way that it may break existing code.
original implementation applied p only on the 'leafs', never on the Both and Then cases (Stackless neither)

@eyalfa This is not true. In fact, what's changing the semantics of filter is the new implementation. I managed to create a reproducer that showcases a change in behaviour with your implementation:

object App {
  val c1   = Cause.fail("foo")
  val c2   = Cause.fail("bar")
  val c3   = Cause.fail("baz")
  val c23  = Cause.Both(c2, c3)
  val c123 = Cause.Both(c1, c23)

  def main(args: Array[String]): Unit = {
    val c = c123.filter { c =>
      println(s"applying filter on: ${c.failures}")
      !c.isInstanceOf[Cause.Both[?]]
    }

    println(s"\nfiltered cause: $c")
  }
}

This code yields the following outputs:

series/2.x:

applying filter on: List(bar)
applying filter on: List(baz)
applying filter on: List(foo)
applying filter on: List(bar, baz)

filtered cause: Fail(foo,Stack trace for thread "zio-fiber-":
)

PR (current bothCase implementation)

applying filter on: List(foo)
applying filter on: List(bar)
applying filter on: List(baz)

filtered cause: Both(Fail(foo,Stack trace for thread "zio-fiber-":
),Both(Fail(bar,Stack trace for thread "zio-fiber-":
),Fail(baz,Stack trace for thread "zio-fiber-":
)))

PR (/w my suggestion)

applying filter on: List(foo)
applying filter on: List(bar)
applying filter on: List(baz)
applying filter on: List(bar)
applying filter on: List(baz)
applying filter on: List(foo)
applying filter on: List(bar, baz)

filtered cause: Fail(foo,Stack trace for thread "zio-fiber-":
)

As you can see, both the current code (series/2.x) and my suggestion apply the filter on Both, but that's not the case with your proposed changes.

Having said that, the downside of my suggestion is that it ends up evaluating p(c) on same Cause multiple times, which is suboptimal. There are ways to improve performance in this case, but since I find it extremely difficult to picture Cause#filter being used in CPU hotpaths, I think we can ignore the performance penalty for now. In the future I plan to visit Cause methods and optimize them as much as possible

eyalfa · 2024-11-11T06:51:39Z

@kyri-petrou ,
I actually understood that the original impl also applies p to Both and Then in the nested case, unfortunately I wrote my comment before this understanding hitting me 😎 .

I think at this point it's a matter of deciding what's the correct behavior, I suspect it's applying p on the Both case only when one is actually constructed, but tbh I'm not 100% sure.

another thing is, I wouldn't expect filter to apply the predicate more than once per Cause, original impl has this property. but it includes a bug for the 'single' case.

I think the logic for Both should be something like:
if both sides are filtered out return Empty
if exactly one side is filtered out, return the other
otherwise, construct a Both, apply the predicate on the new instance, in case of false return Empty otherwise return the new instance.

sketched the implementation and got these printouts when running your sample:

applying filter on: List(foo)
applying filter on: List(bar)
applying filter on: List(baz)
applying filter on: List(bar, baz)

filtered cause: Fail(foo,Stack trace for thread "zio-fiber-":
)

I think this approach yields the correct results while maintaining the 'apply once' property.

eyalfaZS · 2024-11-22T05:51:17Z

@kyri-petrou ☝️

kyri-petrou · 2024-11-26T07:44:42Z

core/shared/src/main/scala/zio/Cause.scala

-          else left
-        } else if (p(right)) right
-        else Cause.empty
+        (left, right) match {


Let's avoid the intermediate tuple creation here and in the methods below and use simple if (left eq Cause.Empty) { ... } else if (right eq Cause.Empty) { ... } else { ... } statements

eyalfa added 4 commits October 22, 2024 08:23

cause_filter_fix: introuce a failing test

15be6fe

cause_filter_fix: couple more tests

773668f

cause_filter_fix: fix

00d07d8

cause_filter_fix: fmt

6c03eb7

kyri-petrou reviewed Oct 22, 2024

View reviewed changes

eyalfa commented Oct 22, 2024

View reviewed changes

core/shared/src/main/scala/zio/Cause.scala Show resolved Hide resolved

eyalfa commented Oct 22, 2024

View reviewed changes

core/shared/src/main/scala/zio/Cause.scala Show resolved Hide resolved

eyalfa commented Oct 22, 2024

View reviewed changes

core/shared/src/main/scala/zio/Cause.scala Show resolved Hide resolved

eyalfa requested a review from kyri-petrou October 23, 2024 11:50

kyri-petrou reviewed Nov 1, 2024

View reviewed changes

eyalfa force-pushed the cause_filter_fix branch from 58a0a04 to 41c17c6 Compare November 9, 2024 18:33

cause_filter_fix: check for specific Empty when composing..

9e2c66c

eyalfa force-pushed the cause_filter_fix branch from 41c17c6 to 9e2c66c Compare November 9, 2024 18:50

cause_filter_fix: traversal test, guaranteed apply once

8db869b

kyri-petrou reviewed Nov 26, 2024

View reviewed changes

cause_filter_fix: address review comments

9627161

kyri-petrou approved these changes Nov 28, 2024

View reviewed changes

kyri-petrou merged commit 252e23e into zio:series/2.x Nov 28, 2024
18 checks passed

Uh oh!

Cause filter fix #9254

Cause filter fix #9254

Uh oh!

Conversation

eyalfa commented Oct 22, 2024

Uh oh!

kyri-petrou Oct 22, 2024

Choose a reason for hiding this comment

Uh oh!

eyalfa Oct 22, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kyri-petrou Oct 29, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

eyalfa Oct 29, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kyri-petrou Oct 29, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

eyalfa Oct 29, 2024

Choose a reason for hiding this comment

Uh oh!

kyri-petrou Nov 1, 2024

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kyri-petrou left a comment

Choose a reason for hiding this comment

Uh oh!

kyri-petrou Nov 1, 2024

Choose a reason for hiding this comment

Uh oh!

eyalfa commented Nov 4, 2024

Uh oh!

kyri-petrou commented Nov 5, 2024

Uh oh!

eyalfaZS commented Nov 6, 2024

Uh oh!

kyri-petrou commented Nov 7, 2024

Uh oh!

eyalfaZS commented Nov 7, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jdegoes commented Nov 8, 2024

Uh oh!

eyalfaZS commented Nov 9, 2024

Uh oh!

CLAassistant commented Nov 9, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

eyalfa commented Nov 9, 2024

Uh oh!

kyri-petrou commented Nov 11, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

eyalfa commented Nov 11, 2024

Uh oh!

eyalfaZS commented Nov 22, 2024

Uh oh!

kyri-petrou Nov 26, 2024

Choose a reason for hiding this comment

Uh oh!

eyalfaZS Nov 26, 2024

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

eyalfa Oct 22, 2024 •

edited

Loading

kyri-petrou Oct 29, 2024 •

edited

Loading

eyalfa Oct 29, 2024 •

edited

Loading

kyri-petrou Oct 29, 2024 •

edited

Loading

eyalfaZS commented Nov 7, 2024 •

edited

Loading

CLAassistant commented Nov 9, 2024 •

edited

Loading

kyri-petrou commented Nov 11, 2024 •

edited

Loading