ZIO Test: Support Managed Resources Per Suite #1664

adamgfraser · 2019-09-13T03:47:24Z

Implements a new ManagedRunnableSpec, which takes a Managed as a parameter in addition to a Spec and provides the managed resource once to the entire suite.

Resolves #1632.

iravid · 2019-09-13T12:47:23Z

This can definitely work, but are we sure we want to go down the route of requiring a different "base class" for this sort of usecase?

Looks pretty different compared to how the rest of the library is built around plain values :-)

adamgfraser · 2019-09-13T14:04:47Z

@iravid I hear you. I think we are limited by our current encoding right now because individual tests are effects but suites aren't. So if I want to perform effects across an entire suite I have to modify the logic of the executor that is executing the tests, instead of just being able to operate on the tests themselves.

I think this is definitely an indicator that we may not have the optimal encoding yet. I think the solution would be to make suites themselves effectual in addition to tests, but I get the sense that @jdegoes is not bought into this. So I am conscious that you need a solution to this problem and I don't want to hold you up when I don't have a clear path forward on a better approach.

jdegoes · 2019-09-15T14:38:35Z

@adamgfraser @iravid I am not a fan of new specs.

I think effectful suites is preferable, because then we can accomplish per-suite provision with TestAspect.

Already, we can do per-test provision with TestAspect, using something like:

suiteM("my suite") { ... } @@ TestAspect.provided(env)

The goal should be to enhance aspects until they can do this for the whole suite, rather than just per test, i.e. achieve cross-test sharing within a suite; and this requires effectful suites, which is something we have already discussed and seems to be broadly useful.

What do you two think?

adamgfraser · 2019-09-15T15:36:16Z

@jdegoes @iravid I completely agree with this approach. Let me proceed on that basis.

iravid · 2019-09-15T18:25:53Z

@adamgfraser @jdegoes Yes, effectful suites definitely look like the right way to go. Thanks guys 🙏🏻

adamgfraser · 2019-09-17T02:27:35Z

@jdegoes I just pushed a commit to make suites effectual. I think everything went relatively smoothly, but a couple of things I'm thinking about:

Do we need more type parameters on ZSpec now? We now have the possibility of "suite level" resources and failures and "test level" resource and failures. Right now I have these fixed to be the same in the definition of ZSpec but not sure that really works (we introduce dependencies and error types unnecessarily and if something changes the type at one level and not the other it may not be a ZSpec so things like test aspects won't work). That would really be a lot of type parameters for test aspects. Would be nice to say that the suite type is the supertype / subtype of the test type based on variance, but not sure how to do that with the test type of Spec being parameterized on T and not ZIO.
I'm still struggling a little to do managed resources and not have the release action trigger too early before test evaluation. Not sure if I just don't have the combinators right or if it has to do with the way we have effectual tests that we transform and then later execute them so the release action has triggered by the time we actually execute the specs.

Appreciate any thoughts you have as always.

jdegoes · 2019-09-17T10:23:03Z

test/shared/src/main/scala/zio/test/Spec.scala

-  sealed trait SpecCase[+L, +T, +A] { self =>
-    final def map[B](f: A => B): SpecCase[L, T, B] = self match {
-      case SuiteCase(label, specs, exec) => SuiteCase(label, specs.map(f), exec)
+  sealed trait SpecCase[-R, +E, +L, +T, +A] { self =>


Ah, I see you propagated R and E into SpecCase. A lot of type parameters but inference should be extremely good due to lower order nature + variance annotations.

What the design really wants to be is:

sealed trait SpecCase[F[_], +L, +T, +A]

So we can have:

final case class Effect[F[_], +L, +T, +A](fa: F[A]) extends SpecCase[F, L, T, A]

where in a common case, F[A] = ZIO[R, E, A]. And in another case, F[A] = A.

The problem is ZIO eschews higher-kinded types for teachability / inference, so doing so here would be rather inconsistent with the rest of ZIO, and is almost certain to diminish inference. Although I suppose we could test it.

Yes I was thinking the same thing but if we managed to do functional effects and streaming without higher order types it seems like we should be able to do make it work for testing. 😃

True, we should keep trying. 😄

jdegoes · 2019-09-17T10:40:11Z

@adamgfraser

I'm still struggling a little to do managed resources and not have the release action trigger too early before test evaluation.

The tests are effectful, which means they need to be flattened into the overall suite effect before provision (otherwise out-of-ordering problems will occur). I think this would be simpler with a separate EffectCase constructor, because then T can become much simpler (no effect, just result). Doesn't solve the problem of "pure specs" but maybe that's a non-problem—i.e. maybe it's OK to require an effect just to traverse a spec.

adamgfraser · 2019-09-17T10:43:30Z

@jdegoes Yes, I think the next step is for me to rework this with the EffectCase constructor. I agree about the pure specs. It would be nice but in this version we're already doing everything with the specs being effectual.

jdegoes · 2019-09-17T11:09:29Z

Yes, I think the next step is for me to rework this with the EffectCase constructor. I agree about the pure specs. It would be nice but in this version we're already doing everything with the specs being effectual.

I think this makes sense... and this is the one thing to sacrifice without much user-level pain (users won't really notice this given the API; maybe power users).

adamgfraser · 2019-09-17T21:44:20Z

@jdegoes I refactored the effectual suites along the lines we discussed and now have a working prototype of providing a managed resource once to an entire suite. It is still a little rough around the edges but we are getting closer.

A few things I am thinking about:

1. Handling suite level errors - Errors in suite level effects are pretty dangerous because unlike the prior iteration, if a suite fails we don't even have a label for the suite. In the previous version when we executed a Spec if there was a failure in the effect I took the label and the failure and constructed a new test node with a runtime exception. Now we can't really do anything with such a failure except log it since we are polymorphic in L so can't even construct a label for a node to put it in. I am inclined to think we should stay polymorphic in the error type but say all suite level errors have to be handled before the TestExecutor executes the tests.
2. Type signature for ZSpec - I tried your suggestion of pushing the effect type up so that you only have the effect in one place, but ran into some challenges in making that work with test aspects. In that setup we don't even know if we have a suite or a test until we run an effect, so it isn't clear how we would implement for example test level timeouts since by the time we "know" we have a test it has already run. We do need to do something though. Maybe we create a super polymorphic version of ZSpec and then a type alias that only has effects at the test level? I'm open to other ideas.
3. Use of error versus value channels - At this point do we want to push test failures into the value channel so we more clearly distinguish them from any errors that can occur with suite level effects. Answer may depend on where we come out on the above but I am leaning towards thinking that would be cleaner.

Other thoughts?

adamgfraser · 2019-09-18T18:18:56Z

@jdegoes The other thing I realized here is I don't think we can currently provide an environment through test aspects because the test aspect can enlarge the types but can't narrow them.

adamgfraser · 2019-09-18T19:10:15Z

@jdegoes This is ready for review.

adamgfraser · 2019-09-20T03:44:02Z

One idea on this. What if we said that all tests were effectual? Then we could have:

final case class SuiteCase[-R, +E, +L, +T, +A](label: L, specs: ZIO[R, E, Vector[A]], exec: Option[ExecutionStrategy]) extends SpecCase[R, E, L, T, A]
final case class TestCase[-R, +E, L, T](label: L, test: ZIO[R, E, T]) extends SpecCase[R, E, L, T, Nothing]

Advantages: (1) We unify the suite and test level environment and error types, (2) we maintain the guarantee that all test nodes in ZSpec are effectual, which I think is important for implementing test aspects, (3) we guarantee that while specs can be created effectually there is always some label for them, which gives us the ability to recover from errors by turning suites with failed effects into test nodes with values indicating failure.

I know there is a ton of other stuff going on right now so no rush in responding. Just something to think about.

jdegoes · 2019-09-21T21:49:02Z

@adamgfraser

The other thing I realized here is I don't think we can currently provide an environment through test aspects because the test aspect can enlarge the types but can't narrow them.

This is true, although combinators can narrow them, e.g. @@ myCominator(all >>> other >>> aspects), so it's not a huge limitation. We could also (probably) have a narrowing type of aspect, but it could not compose with the more common widening aspect, so combinator may be better (I'm not sure).

Will address other things soon, sorry for delay, been super busy with hackathon!

adamgfraser · 2019-09-22T05:45:33Z

@jdegoes Yes, I don't think it is a huge limitation. It just means that some things like providing resources will need to be done through methods on Spec versus test aspects.

No worries at all. I know that the hackathon and ZIO 1.0 release is top priority right now. There will be plenty of time for more work on ZIO Test after that.

jdegoes · 2019-09-26T09:13:30Z

@adamgfraser

Advantages: (1) We unify the suite and test level environment and error types, (2) we maintain the guarantee that all test nodes in ZSpec are effectual, which I think is important for implementing test aspects, (3) we guarantee that while specs can be created effectually there is always some label for them, which gives us the ability to recover from errors by turning suites with failed effects into test nodes with values indicating failure.

I think this makes a lot of sense. It's a simpler representation than the one that factors out the effect, and more uniform. Less flexile but the ergonomics will be higher. I think it's the right set of tradeoffs.

adamgfraser · 2019-09-26T10:01:37Z

@jdegoes I'm on it!

jdegoes · 2019-10-02T23:07:18Z

test-sbt/shared/src/main/scala/zio/test/sbt/ZTestEvent.scala

+  ): UIO[Seq[ZTestEvent]] =
+    executedSpec.mapLabel(_.toString).fold[UIO[Seq[ZTestEvent]]] {
+      case Spec.SuiteCase(_, results, _) =>
+        results.flatMap(UIO.collectAll(_).map(_.flatten))


collectAll makes a choice of sequentiality—intentional?

My thought was that since we are already dealing with an executedSpec here the potentially compute intensive work of running the test has already been done so we don't need parallelism here.

jdegoes · 2019-10-02T23:19:01Z

test/shared/src/main/scala/zio/test/DefaultTestReporter.scala

+            hasFailures   = failures.exists(identity)
+            status        = if (hasFailures) Failed else Passed
+            renderedLabel = if (hasFailures) renderFailureLabel(label, depth) else renderSuccessLabel(label, depth)
+            rest          <- UIO.foreach(specs)(loop(_, depth + tabSize)).map(_.flatten)


Fixed to sequential here

Similar thought to the above that since we've already run the tests we don't need parallelism in rendering the results.

jdegoes · 2019-10-02T23:19:08Z

test/shared/src/main/scala/zio/test/DefaultTestReporter.scala

-          rendered(Suite, label, status, depth, renderedLabel) +: executedSpecs.flatMap(loop(_, depth + tabSize))
+          for {
+            specs <- executedSpecs
+            failures <- UIO.foreach(specs)(_.exists {


Fixed to sequential here

Same as above.

jdegoes · 2019-10-02T23:20:39Z

test/shared/src/main/scala/zio/test/Spec.scala

-  final def exists(f: SpecCase[L, T, Unit] => Boolean): Boolean =
-    fold[Boolean] {
-      case c @ SuiteCase(_, specs, _) => specs.exists(identity) || f(c.map(_ => ()))
+  final def exists[R1 <: R, E1 >: E](f: SpecCase[R, E, L, T, Any] => ZIO[R1, E1, Boolean]): ZIO[R1, E1, Boolean] =


This maybe should be called existsM?

Yes we can definitely do that.

jdegoes · 2019-10-02T23:23:19Z

test/shared/src/main/scala/zio/test/Spec.scala

-      case c @ SuiteCase(_, specs, _) => specs.forall(identity) && f(c.map(_ => ()))
+  final def forall[R1 <: R, E1 >: E](f: SpecCase[R, E, L, T, Any] => ZIO[R1, E1, Boolean]): ZIO[R1, E1, Boolean] =
+    fold[ZIO[R1, E1, Boolean]] {
+      case c @ SuiteCase(_, specs, _) => specs.flatMap(ZIO.collectAll(_).map(_.forall(identity))).zipWith(f(c))(_ && _)


Sequential specialization. Possibly look at the strategy in the node?

We could. The only time this method is used in the code base today is on executed specs. This goes a little to some of the other points about the value of these methods on specs that haven't been executed now that the suites are effectual.

jdegoes · 2019-10-02T23:25:54Z

test/shared/src/main/scala/zio/test/Spec.scala

-    case TestCase(_, _)          => 1
+  final def provideManaged[E1 >: E](managed: Managed[E1, R]): Spec[Any, E1, L, T] =
+    transform[Any, E1, L, T] {
+      case SuiteCase(label, specs, exec) => SuiteCase(label, specs.provideManaged(managed), exec)


It seems like transform is awfully low-level for this, do we have anything that just allows function application to specs and test?

I agree. How about something like:

final def mapTests[R1, E1, L1 >: L, T1]( suiteCase: ZIO[R, E, Vector[Spec[R1, E1, L1, T1]]] => ZIO[R1, E1, Vector[Spec[R1, E1, L1, T1]]], testCase: ZIO[R, E, T] => ZIO[R1, E1, T1] ): Spec[R1, E1, L1, T1] = caseValue match { case SuiteCase(label, specs, exec) => Spec.suite(label, suiteCase(specs.map(_.map(_.mapTests(suiteCase, testCase)))), exec) case TestCase(label, test) => Spec.test(label, testCase(test)) }

Then we could do:

final def provideManaged[E1 >: E](managed: Managed[E1, R]): Spec[Any, E1, L, T] = mapTests[Any, E1, L, T](_.provideManaged(managed), _.provideManaged(managed))

jdegoes · 2019-10-02T23:26:52Z

test/shared/src/main/scala/zio/test/Spec.scala

+   * act of creating the environment is expensive and should only be performed
+   * once.
+   */
+  final def provideManagedSuite[E1 >: E](managed: Managed[E1, R]): Spec[Any, E1, L, T] = {


How about provideManagedShared if this is global sharing? provideManagedSuite suggests operation at the level of a suite, maybe each suite (i.e. each nested suite gets its own resource).

Yes! Changed!

jdegoes · 2019-10-02T23:27:20Z

test/shared/src/main/scala/zio/test/Spec.scala

+   * once.
+   */
+  final def provideManagedSuite[E1 >: E](managed: Managed[E1, R]): Spec[Any, E1, L, T] = {
+    def loop(r: R)(spec: Spec[R, E, L, T]): ZIO[Any, E, Spec[Any, E, L, T]] =


I feel like this loop suggests the need for another traversal scheme.

I agree. I've been struggling a bit to find the right encoding. I think what we want is a function that takes a ZIO[R, E, T] in the test case to a ZIO[R1, E1, T], where there are no constrains on R1 and E1 being subtypes / supertypes of R and E. And then I think for the suite case we basically need a universally quantified function that takes a ZIO[R, E, X] to a ZIO[R1, E1, X], so the suite case can change the effect but should delegate changing the child specs to recursively calling the traversal. Otherwise we run into issues with the environment and error types getting widened to the union of R and R1 and E and E1 instead of just getting R1 and E.

jdegoes · 2019-10-02T23:28:08Z

test/shared/src/main/scala/zio/test/Spec.scala

+  /**
+   * Computes the size of the spec, i.e. the number of tests in the spec.
+   */
+  final def size: ZIO[R, E, Int] =


I wonder just how useful all these become in an effectful world. Basically computing the size here could, in theory, allocate and de-allocate resources required only to run the tests!

I totally agree. I wonder if we move these to some type of syntax class for executed Specs or require implicit evidence that the spec has been executed. The only variants of these fold type methods we use are exists and forall in checking if all executed specs passed.

jdegoes · 2019-10-02T23:29:51Z

test/shared/src/main/scala/zio/test/TestAspect.scala

+     * `ZTest` and the new encoding. We can remove them once we refactor test
+     * aspects to use the new encoding
+     */
+    private def to[R, E, S](


Will this be part of this pull request, or come along later?

I was planning on doing it as part of this pull request, I just wanted to make sure we were aligned before I went through and changed all the test aspects. The one thing that may change this is if we are going to do a release today it could be nice to get this in so that projects that need the shared resource functionality can start using ZIO Test, even if we need to do a little cleanup on the backend afterwards.

I will have this done shortly.

jdegoes · 2019-10-02T23:38:23Z

test/shared/src/main/scala/zio/test/package.scala

   */
  final def fail[E](cause: Cause[E]): ZTest[Any, E, Nothing] =
-    ZIO.fail(TestFailure.Runtime(cause))
+    ZIO.halt(cause)


Nice simplification! 👍

jdegoes

Just a few minor questions / comments. Overall looks great!

We lose some power (on the consumer side) with effectful suites, but we knew that was going to happen. Obviously we gain power on the other side, not just managed for suites and so forth, but also dynamic tests / suites, something we can have combinators for eventually.

adamgfraser · 2019-10-03T14:49:24Z

@jdegoes I updated the name of the method to provideManagedShared and updated all the test aspects to the new encoding. My strong inclination at this point is that we should get this in for this release. We have some potential users who really need this feature and we could benefit from having more time to get feedback and iterate on the effectual suites. I think we can definitely further simplify but can also pick that up in a subsequent PR if we don't get to it now.

jdegoes · 2019-10-03T23:11:07Z

@adamgfraser Agreed, done! Thanks for all your hard work on this. It gets us that much closer to ZIO Test 1.0!

* implement ManagedRunnableSpec * add documentation * make suites effectual * linting change * initial work on new encoding * rework Spec combinators * add test * format * clean up * make all specs effectual * rename provideManagedSuite to provideManagedShared * move test aspects to new encoding * format * convert unchecked exceptions to test failures * ignore flaky and timing out supervision test * merge upstream changes * fix import

adamgfraser added 2 commits September 12, 2019 23:38

implement ManagedRunnableSpec

b8b143b

add documentation

5be49a3

adamgfraser requested review from iravid and jdegoes September 13, 2019 03:58

adamgfraser added 2 commits September 16, 2019 17:49

make suites effectual

e653b52

linting change

103ee26

adamgfraser mentioned this pull request Sep 17, 2019

ZIO Test: Add effectful suites #1693

Closed

jdegoes reviewed Sep 17, 2019

View reviewed changes

adamgfraser added 4 commits September 17, 2019 14:20

initial work on new encoding

ac9ce98

rework Spec combinators

eb18369

add test

f53d204

format

83fc29c

clean up

a05371f

jdegoes reviewed Oct 2, 2019

View reviewed changes

jdegoes requested changes Oct 2, 2019

View reviewed changes

adamgfraser added 9 commits October 3, 2019 05:21

merge upstream changes

67b3f0b

rename provideManagedSuite to provideManagedShared

6ea86f6

move test aspects to new encoding

435ac23

format

3742e3c

convert unchecked exceptions to test failures

5946299

ignore flaky and timing out supervision test

3aebe4c

merge upstream changes

436f8ed

merge upstream changes

e7d6cf3

fix import

575f2a3

jdegoes approved these changes Oct 3, 2019

View reviewed changes

jdegoes merged commit f41be68 into zio:master Oct 3, 2019

adamgfraser deleted the managed branch October 4, 2019 02:23

This was referenced Oct 4, 2019

#1790 - A suite annotated with .only("some-label"), will ignore all non matching labels #1838

Merged

ZIO Test: Add Support for Test Annotations #1899

Closed

adamgfraser mentioned this pull request Oct 22, 2019

ZIO Test: Fix Bug in TestAspect#nonFlaky #2013

Merged

Uh oh!

ZIO Test: Support Managed Resources Per Suite #1664

ZIO Test: Support Managed Resources Per Suite #1664

Uh oh!

Conversation

adamgfraser commented Sep 13, 2019

Uh oh!

iravid commented Sep 13, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

adamgfraser commented Sep 13, 2019

Uh oh!

jdegoes commented Sep 15, 2019

Uh oh!

adamgfraser commented Sep 15, 2019

Uh oh!

iravid commented Sep 15, 2019

Uh oh!

adamgfraser commented Sep 17, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jdegoes commented Sep 17, 2019

Uh oh!

adamgfraser commented Sep 17, 2019

Uh oh!

jdegoes commented Sep 17, 2019

Uh oh!

adamgfraser commented Sep 17, 2019

Uh oh!

adamgfraser commented Sep 18, 2019

Uh oh!

adamgfraser commented Sep 18, 2019

Uh oh!

adamgfraser commented Sep 20, 2019

Uh oh!

jdegoes commented Sep 21, 2019

Uh oh!

adamgfraser commented Sep 22, 2019

Uh oh!

jdegoes commented Sep 26, 2019

Uh oh!

adamgfraser commented Sep 26, 2019

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

iravid commented Sep 13, 2019 •

edited

Loading

adamgfraser commented Sep 17, 2019 •

edited

Loading