[test-sbt] More uniformity. #10120

dubinsky · 2025-08-19T21:04:01Z

TL;DR: make sbt.testing.Framework implementation:

cleaner and easier to understand;
600+ lines shorter;
behave more uniformly across the back-ends;
testable on all the back-ends.

Currently, code implementing sbt.testing.Framework for JVM, Scala.js and Scala Native suffers from severe code triplication: code doing the same thing is present in three places. This pull request removes this gratuitous redundancy.

An unfortunate - and inevitable - consequence of code duplication is code drift: different copies of the code do things differently for no good reason.

Backends sbt.testing.Framework runs on do have inherent differences: tests running on Scala.js and Scala Native are running outside of the JVM, and thus their outcomes need to be communicated to the process managing the test run; Scala.js is single-threaded by nature, which affects the way tests have to be run. Code differences not warranted by the inherent differences between the backends seem gratuitous, and this pull request removes some of them.

One area where implementations of the sbt.testing.Framework diverged from one another is the machinery to test them: only JVM implementation is currently testable. With this pull request, sbt.testing.Framework tests run on all the backends.

Another divergence is in the storage of test summaries: JVM uses AtomicReference[Vector], Scala.js uses Buffer, Scala Native uses ConcurrentLinkedQueue; this pull request unifies summary storage for all backends: AtomicReference[Vector] works for all of them.

Another divergence is the way test summary is rendered: counts of tests executed, ignored, or failed are included in the summary on JVM but not on Scala.js or Scala Native; setting test renderer to IntelliJ in the test arguments is honored only on JVM. This pull request makes test summary uniform across the backends.

Another divergence is: currently, signal handlers are installed only on JVM. This pull request installs them on all backends.

Another divergence affects a recent enhancement, which turned out to be effective only on JVM; this pull request applies it to all backends (and tests that it works ;)).

@kyri-petrou, your remark

I gave it a go at this (basically trying to unify the JVM / JS / Native implementations so that we can reuse the existing event test suites that exist for the JVM, but it doesn't seem possible.
... writing a full test suite from scratch (which frankly it's going to be extremely time consuming)

is on target: this was not trivial ;)

@kyri-petrou

Currently, code implementing `sbt.testing.Framework` for JVM, Scala.js and Scala Native suffers from severe code triplication: code doing the same thing is present in three places. This pull request removes this gratuitous redundancy. An unfortunate - and inevitable - consequence of code duplication is code drift: different copies of the code do things differently for no good reason. Backends `sbt.testing.Framework` runs on _do_ have inherent differences: tests running on Scala.js and Scala Native are not running on JVM, and thus their outcomes need to be communicated to the process managing the test run; Scala.js is single-threaded by nature, which affects the way tests have to be run. Code differences not warranted by the inherent differences between the backends seem gratuitous, and this pull request removes some of them. One area where implementations of the `sbt.testing.Framework` diverged from one another is the machinery to test them: only JVM implementation is currently testable. With this pull request, `sbt.testing.Framework` tests run on all the backends. Another divergence is in the storage of test summaries: JVM uses `AtomicReference[Vector]`, Scala.js uses `Buffer`, Scala Native uses `ConcurrentLinkedQueue`; this pull request unifies summary storage for all backends: `AtomicReference[Vector]` works for all of them. Another divergence is the way test summary is rendered: counts of tests executed, ignored, or failed are included in the summary on JVM but not on Scala.js or Scala Native; setting test renderer to IntelliJ in the test arguments is honored only on JVM. This pull request makes test summary uniform across the backends. Another divergence is: currently, signal handlers are installed only on JVM. This pull request installs them on all backends. Another divergence affects a recent [enhancement](zio#9756), which turned out to be effective only on JVM; this pull request applies it to all backends (and tests that it works ;)). @kyri-petrou, your [remark](zio#9629 (comment)) > I gave it a go at this (basically trying to unify the JVM / JS / Native implementations so that we can reuse the existing event test suites that exist for the JVM, but it doesn't seem possible. > ... writing a full test suite from scratch (which frankly it's going to be extremely time consuming) is on target: this was not trivial ;)

dubinsky · 2025-08-24T20:53:53Z

@kyri-petrou what do I need to do for this to get merged? Thanks!

kyri-petrou · 2025-08-25T14:57:27Z

what do I need to do for this to get merged? Thanks!

@dubinsky I'm trying to get through the PR but it's taking some time. I'll leave comments when I'm finished

dubinsky · 2025-08-25T15:03:05Z

what do I need to do for this to get merged? Thanks!

@dubinsky I'm trying to get through the PR but it's taking some time. I'll leave comments when I'm finished

Thank you!!

kyri-petrou

Overall looks good to me, just a few minor-ish comments.

The one main comment I do have though is, can you please make sure that these changes don't break the IntelliJ ZIO plugin?

kyri-petrou · 2025-08-25T14:54:36Z

test-sbt/shared/src/main/scala/zio/test/sbt/Colors.scala

+  lazy val red: String => String    = colored(Console.RED)
+  lazy val green: String => String  = colored(Console.GREEN)
+  lazy val cyan: String => String   = colored(Console.CYAN)
+  lazy val blue: String => String   = colored(Console.BLUE)
+  lazy val yellow: String => String = colored(Console.YELLOW)


I don't think we need these to be lazy, right?

They always were for some (or no) reason; changing this now.

done in 174ab24

kyri-petrou · 2025-08-25T15:04:52Z

test-sbt/shared/src/main/scala/zio/test/sbt/TestRunner.scala

+
+  protected def sharedRuntimeSupported: Boolean
+
+  private val summaries: AtomicReference[Vector[Summary]] = new AtomicReference(Vector.empty)


Very minor, but ConcurrentLinkedQueue is supported for all Scala flavours (JVM, JS and Native) and should be a lot more memory-friendly than using a Vector and appending items to it.

Alternatively, you could use AtomicReference[List[Summary]], prepend the summaries and the reverse the order in the end

I followed the approach currently used on JVM :)
Changing to ConcurrentLinkedQueue now.
Thank you!

done in 174ab24

kyri-petrou · 2025-08-25T15:09:39Z

test/shared/src/main/scala/zio/test/ZIOSpecAbstract.scala

-    testEventHandler: ZTestEventHandler,
-    console: Console


I would keep the old method with the old signature, overload it, deprecate it and use the new method (where you need to use it). The reason for this is there is a discrepancy between the zio-test-sbt and zio-test versions for some reason in a codebase it'll start throwing errors

Thanks! Doing that.

done in 174ab24

kyri-petrou · 2025-08-25T15:15:05Z

test-sbt/jvm/src/test/scala/zio/test/sbt/ZTestFrameworkSbtSpec.scala

-import scala.collection.mutable.ArrayBuffer
-import scala.util.Try
-
-object ZTestFrameworkSbtSpec {


Out of curiosity, why do you want to throw away this code? I'm not against it but I thought perhaps it had a reason for being there

I think it was there since the beginning, when there was no way to test using sbt.testing.Framework at all, even on JVM. When this became possible, some tests were moved (or copied) to real automated testing. As the behavior of test-sbt changed, the test left behind in this class eventually started failing, since they were checking the actual framework output, not the events emitted, and were disabled one-by-one. I moved whatever was salvageable and deleted the rest, which did not do anything anyway. The reason I think this should be removed is: it creates confusion about where new tests should be added; for instance, when working on #9756, I added tests in this class - and that was the wrong place to add them ;)

kyri-petrou · 2025-08-25T15:24:26Z

test-sbt/shared/src/main/scala/zio/test/sbt/TestRunner.scala

+  final override def done(): String = {
+    // If tests are forked, this will only be relevant in the forked
+    // JVM, and will not be set in the original JVM.
+    sharedRuntimes.get.foreach(_.unsafe.shutdown()(zio.Unsafe))


Should we use getAndSet(Vector.empty) instead here? Or it doesn't matter?

Also potentially we can use ConcurrentLinkedQueue as per my previous comment as well to ensure that we're closing each one exactly once

Calling Runner.done() more than once is not a supported scenario. I am switching to ConcurrentLinkedQueue as you suggested :)

done in 174ab24

kyri-petrou · 2025-08-25T15:30:03Z

test-sbt/jvm/src/main/scala/zio/test/sbt/ZTestFramework.scala

-import sbt.testing._
-
 /**
+ * TODO update/remove outdated/misplaced doc comment.


Is this still TODO?

I think so.

This comment was outdated and/or misplaced before my pull request; it's ok for it to stay that way a bit longer I think ;)

turned TODO into a note

kyri-petrou · 2025-08-25T15:30:28Z

test-sbt/shared/src/main/scala/zio/test/sbt/TestTask.scala

+  // TODO SBT loggers should be hooked in...
+  final private[sbt] def run(eventHandler: EventHandler)(implicit trace: zio.Trace): ZIO[Any, Throwable, Unit] = {


Does this TODO still needs to be resolved

Yes. Currently, test-sbt ignores the sbt loggers supplied via Task.execute() and just prints the progress of test execution to the Console. My pull request does not change that - but ideally, it should change.

It does not need to be resolved for this pull request though.

turned TODO into a note

kyri-petrou · 2025-08-25T15:30:46Z

test-sbt/shared/src/test/scala/zio/test/sbt/ExecuteSpecs.scala

+      .mapError((error: Throwable) => ::(error, Nil))
+
+    for {
+      // TODO make this test console silent


Does this needs to be resolved?

Currently, test-sbt is not testable on Scala.js nor Scala Native; with my pull request, it is. During testing I noticed that progress of execution of tests within framework instances used by tests is visible on non-JVM back-ends - but not on JVM. It would be nice to make this output more uniform across the back-ends - but it is not needed for this pull request.

turned TODO into a note

kyri-petrou · 2025-08-25T15:32:54Z

test-sbt/shared/src/test/scala/zio/test/sbt/ExecuteSpecs.scala

+    specs: Seq[ZIOSpecAbstract],
+    args: Array[String],
+    selectors: Array[Selector] = Array(new SuiteSelector)
+  ): ZIO[Any, ::[Throwable], (Seq[String], Seq[Event])] = {


Since we're initializing a mutable object inside, we need to be referentially transparent

Suggested change

): ZIO[Any, ::[Throwable], (Seq[String], Seq[Event])] = {

): ZIO[Any, ::[Throwable], (Seq[String], Seq[Event])] = ZIO.suspendSucceed {

Thanks! Fixing it.

done in 174ab24

dubinsky · 2025-08-25T17:32:34Z

Overall looks good to me, just a few minor-ish comments.

Thank you!

The one main comment I do have though is, can you please make sure that these changes don't break the IntelliJ ZIO plugin?

AFAIK, coupling between the IntelliJ ZIO plugin and zio.test.sbt classes is lose enough to not break.

Same seems to be true about the ZIO Test runner support for the ZIO IntelliJ plugin, which anyway is archived and was not touched for two years...

Address review comments.

hearnadam · 2025-08-20T20:40:41Z

test-sbt/js/src/main/scala/zio/test/sbt/ZTestTask.scala

+      runtime
+    ) {
+  // Note: on Scala.js this method is never called - next one is.
+  override def execute(eventHandler: EventHandler, loggers: Array[Logger]): Array[Task] = ???


Can you throw an error with a clear message stating the comment, something like:

new IllegalStateException("ZTestTask.execute unexpectedly invoked on Scala.js...

done in eadcaa9

hearnadam · 2025-08-27T18:20:27Z

test-sbt/js/src/main/scala/zio/test/sbt/ZTestTask.scala

+
+  override def execute(eventHandler: EventHandler, loggers: Array[Logger], continuation: Array[Task] => Unit): Unit =
+    unsafeAPI
+      .fork(run(eventHandler)(zio.Trace.empty))(zio.Trace.empty, zio.Unsafe)


Can you import these to avoid the prefix? same with exit?

done in eadcaa9 throughout the code; Runtime, System and Console remain zio.-prefixed to disambiguate them from the ones from Java/Scala.

hearnadam · 2025-08-28T01:15:51Z

test-sbt/jvm-native/src/main/scala/zio/test/sbt/ZTestTask.scala

+    try {
+      resOutter = unsafeAPI.runToFuture(run(eventHandler)(zio.Trace.empty))(zio.Trace.empty, zio.Unsafe)
+      Await.result(resOutter, Duration.Inf)
+    } catch {


try/finally?

I don't think so: resOutter.cancel() needs to be invoked only when the try block throws...

Address more review comments.

kyri-petrou

I think it looks good. I'm not too familiar with the sbt integration part but I'm really in favour of unifying the implementations for JVM/JS/Native as it's been the source of several bugs on the JS / Native side

@hearnadam are you okay with merging this one in as well?

kyri-petrou · 2025-09-04T04:10:51Z

Many thanks @dubinsky! I know this was not a trivial task - really appreciate this :)

dubinsky · 2025-09-04T04:12:42Z

Many thanks @dubinsky! I know this was not a trivial task - really appreciate this :)

Glad I could be of help :)

Thank you @kyri-petrou and @hearnadam!

dubinsky mentioned this pull request Aug 19, 2025

Out of my hands dubinsky/scalajs-gradle#97

Closed

dubinsky force-pushed the zio-test-more-uniform-reporting branch 6 times, most recently from 61d736b to 246ce39 Compare August 20, 2025 04:52

dubinsky marked this pull request as draft August 20, 2025 05:29

dubinsky force-pushed the zio-test-more-uniform-reporting branch 6 times, most recently from e0332e7 to a908303 Compare August 20, 2025 23:32

dubinsky mentioned this pull request Aug 15, 2025

ZIO Test events and summary dubinsky/scalajs-gradle#106

Closed

2 tasks

dubinsky marked this pull request as ready for review August 20, 2025 23:46

dubinsky force-pushed the zio-test-more-uniform-reporting branch 8 times, most recently from e4458d5 to 5464462 Compare August 24, 2025 17:26

dubinsky force-pushed the zio-test-more-uniform-reporting branch from 5464462 to af719de Compare August 24, 2025 18:25

kyri-petrou reviewed Aug 25, 2025

View reviewed changes

[test-sbt] More uniformity.

174ab24

Address review comments.

dubinsky force-pushed the zio-test-more-uniform-reporting branch from 90bfe81 to 174ab24 Compare August 25, 2025 23:01

hearnadam reviewed Aug 28, 2025

View reviewed changes

dubinsky added 2 commits August 27, 2025 21:57

[test-sbt] More uniformity.

eadcaa9

Address more review comments.

[test-sbt] More uniformity.

663d732

Address more review comments.

kyri-petrou approved these changes Aug 30, 2025

View reviewed changes

kyri-petrou merged commit 07efbcb into zio:series/2.x Sep 4, 2025
19 checks passed


		protected def sharedRuntimeSupported: Boolean

		private val summaries: AtomicReference[Vector[Summary]] = new AtomicReference(Vector.empty)

		// TODO SBT loggers should be hooked in...
		final private[sbt] def run(eventHandler: EventHandler)(implicit trace: zio.Trace): ZIO[Any, Throwable, Unit] = {

	): ZIO[Any, ::[Throwable], (Seq[String], Seq[Event])] = {
	): ZIO[Any, ::[Throwable], (Seq[String], Seq[Event])] = ZIO.suspendSucceed {

Uh oh!

[test-sbt] More uniformity. #10120

[test-sbt] More uniformity. #10120

Uh oh!

Conversation

dubinsky commented Aug 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dubinsky commented Aug 24, 2025

Uh oh!

kyri-petrou commented Aug 25, 2025

Uh oh!

dubinsky commented Aug 25, 2025

Uh oh!

kyri-petrou left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dubinsky Aug 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dubinsky Aug 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dubinsky Aug 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dubinsky Aug 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

dubinsky commented Aug 19, 2025 •

edited

Loading

dubinsky Aug 25, 2025 •

edited

Loading

dubinsky Aug 25, 2025 •

edited

Loading

dubinsky Aug 25, 2025 •

edited

Loading

dubinsky Aug 25, 2025 •

edited

Loading

dubinsky Aug 25, 2025 •

edited

Loading

dubinsky Aug 28, 2025 •

edited

Loading