.map(_.map(_.mapTest(...)))

😆

Do we have to expose this map? It would be nice to make this as tiny a surface area as possible, which means annotate and get.

If we want executing a spec to give you the annotation map we need a way to retrieve the annotation map. We could make it package private, though that would mean any user implementing their own test executor would not be able to access the annotation map.

I like the idea that executing the spec gives you back the annotation map (at each node) always. 👍

It may be empty in some configurations, that's fine.

By the way, this structure is beginning to get complex enough that we should consider a custom type:

sealed trait TestResult[+E, +S] {
  def annotations: TestAnnotationMap
}
object TestResult {
  final case class Failed[E](failure: TestFailure[E], annotations: TestAnnotationMap) extends TestResult[E, S]
  final case class Succeeded[A](success: TestSuccess[S], annotations: TestAnnotationMap) extends TestResult[E, S]
}

Then we can simplify to:

type ExecutedSpec[+L, +E, +S] = 
  Spec[Any, Nothing, L, TestResult[E, S]]

(I think we're already using TestResult name for something...)

Yes. After getting this merged my next priority is to simplify some of these types.

Is this actually helpful, because there is just one annotation map here?

It is definitely questionable. For the vast majority of users they are just going to use the TestEnvironment and maybe provideSome to enrich that with additional functionality. But if a power user wants to provide their own environment type this would allow them to include test annotations functionality in that type.

Ditto for this — is it helpful?

See above.

I think we are missing some type of interface something like this:

type TestAnnotationRenderer[A] = 
  TestAnnotation[A] => RenderedResult

Then we can push knowledge of which aspects need rendering to a higher level, and get it out of here, so the user can pass their own rendering for custom test aspects (of course, we will supply the ones for the built in test annotations like timing, etc.).

Agreed, I was thinking about whether test annotations should contain their own logic for rendering them but was worried that was coupling data with presentation too much. This is a better way to do that. Let me work on adding. Though I think we may need to modify it slightly to be something like

type TestAnnotationRenderer[A] =
  (TestAnnotation[A], ExecutedSpec[Any, Any, Any]) => UIO[String]

Conceptually I think we want the renderer to be able to introspect the test annotations from all the tests in a spec (e.g. to sort the rendered test times from longest to shortest or to report at the level of suites instead of individual tests).

We should think how to pull this logic out of here and push it higher.

Yes. I think once we have TestAnnotationRenderer, TestReporter can take a List[TestAnnotationRenderer] as input, and the DefaultTestReporter in particular can have a set of default renderers in addition to whichever ones are passed to it. Then the DefaultTestReporter can traverse the executed spec, report all annotations it has renderers for, and either ignore or report an error for any others.

Don't we need (in the most general possible case), one TestAnnotationMap per test? Of course, we can compose all TestAnnotation inside a suite into one, for the suite itself, and so forth; but it seems to me to have the most flexibility, we will need one per test.

(This, btw, explains my confusion about the interface for TestAnnotations, specifically the utility about the make* methods.)

Yes, I think we want one TestAnnotationMap per test. Right now I am doing this because the TestExecutor provides a separate copy of the TestEnvironment to each test. Does that make sense or would you approach it differently?

Ah, I didn't see that. Where is that code?

The TestExecutor uses Spec#provideManaged here and provideManaged (as opposed to provideManagedShared) provides a separate copy of the resources to each test here.

So the only problem I see is that if the user chooses to share an environment across tests (e.g. to share database connection costs), then suddenly the annotations will not be per test, but rather, "global" per the sharing.

It would be nice if we could guarantee the same annotations regardless of environmental sharing; and also, if the annotations for suites were computed through semigroup composition of the annotations of their contents (it's a stronger guarantee).

What do you think?

I tried that initially and ran into some issues. For a lot of annotations we don't actually want to combine them using the semigroup for the test annotation. For example, for the Timing annotation it isn't that interesting to just add up the duration for all the tests, we want to combine them using more of a "map merge" semigroup with mappings from labels to durations. The more serious issue is that we want to write information to test annotations through test aspects but test aspects don't have access to test labels. So if we have each test write its duration to a shared annotation map we don't have any keys to associate with the durations in a sensible way to then aggregate.

Maybe another way we could address is by taking the TestAnnotations service out of the test environment and having the TestExecutor provide it separately to all the tests so it would be impossible or at least very hard for a user to provide a shared annotation map.

I do think the semigroup solution is the most natural one so would love to do that if possible but not sure how to overcome the issues there.

I'll write up some ideas here soon!

If we use a fiber ref, then during the evaluation of tests, the executor can create a fresh map, and replace the contents of the fiber ref with that fresh map, and then after the test, capture the map, and in the end, generate one map per test, and one map per suite. The suite maps would be generated by using the annotation append functionality, which merges collisions, which means that suite maps would always be an aggregation of their children.

Then the renderer can look at the map and decide what to render, per node, based on the desired rendering test annotations (timing, failure count, success count, etc.).