ZIO Test: Support Stateful Property Based Testing #4497
Conversation
|
Thanks for this @adamgfraser it seems like a great start! I've been having a ton of fun trying this out today, here are some things I've learned: I've tried (and been somewhat successful) with writing stateful end to end tests for a zio-grpc backend using your example. I've run into a couple of things that I think are worth discussing It would be extremely useful if we could somehow split the command/state generation into multiple phases; one that generates the next command based on the current state, and one that computes the next state based on the result of some effect needed to run the command. I'll try to provide a more elaborate example in code below, but the use case for this is e.g testing a REST API. When we The API I'm testing actually allows you to provide an optional ID when creating resources. Fortunately the API now I'm testing supports this, which means I can use Even though if it's probably not directly related to this PR per-se, I'm seeing much lower performance than I expected. I generate two commands, each consisting of a case class or 5 or so fields that are generated (some are nested generators, such as list of values). Not doing anything at all in the import zio._
import zio.test._
import zio.random._
import zio.test.Assertion._
object E2ETest extends DefaultRunnableSpec {
val spec = suite("end to end tests")(testM("End to end tests") {
// This is a bit contrived to make the example simpler.
case class CreateRequest(id: String, name: String)
case class UpdateRequest(id: String, name: String)
case class GetRequest(id: String)
case class GetResponse(id: String, name: String)
case class Resource(id: String, name: String)
sealed trait Command
case class Create(r: CreateRequest) extends Command
case class Update(r: UpdateRequest) extends Command
case class Get(r: Resource) extends Command
def genCreate(ids: Set[String]) =
Gen
.zipN(
Gen.anyUUID.map(_.toString()).filterNot(ids.contains(_)),
Gen.anyString
)((id, name) => CreateRequest(id, name))
.map(Create(_))
def genUpdate(ids: Set[String]) =
Gen
.zipN(
Gen.fromIterable(ids),
Gen.anyString
)((id, name) => UpdateRequest(id, name))
.map(Update(_))
def genGet(state: Iterable[Resource]) =
Gen.fromIterable(state).map(Get(_))
def matchesState(expected: Resource) =
hasField("id", (r: GetResponse) => r.id, not(isEmptyString)) &&
hasField(
"name",
(r: GetResponse) => r.name,
equalTo(expected.name)
)
val genCommands: Gen[Random with Sized, List[Command]] =
unfoldGen(Map(): Map[String, Resource]) {
state =>
state.isEmpty match {
case true =>
genCreate(state.keySet)
.map(cmd =>
(
state + (cmd.r.id -> Resource(cmd.r.id, cmd.r.name)),
cmd: Command
)
)
case false =>
Gen.weighted(
genCreate(state.keySet)
.map(cmd =>
(state + (cmd.r.id -> Resource(cmd.r.id, cmd.r.name)), cmd)
)
-> 1,
genUpdate(state.keySet).map(cmd =>
(
state + (cmd.r.id -> state(cmd.r.id)
.copy(name = cmd.r.name)),
cmd
)
) -> 1,
genGet(state.values).map(cmd =>
(
state,
cmd
)
) -> 2
)
}
}
checkM(genCommands) {
commands =>
{
ZIO.foldLeft(commands)(assert(ZIO.unit)(anything))({
case (acc, cmd) => {
if (acc.isFailure) ZIO.succeed(acc)
else
cmd match {
case Create(r) =>
API.create(r).unit *> ZIO.succeed(acc)
case Update(r) =>
API.update(r) *> ZIO.succeed(acc)
case Get(r) =>
(for {
actual <- API.get(
GetRequest(r.id)
)
} yield assert(actual)(matchesState(p)) && acc)
}
}
})
}
}
})
}PS: I'm sure there's a better way to write the assertions in my |
|
After some discussions with @adamgfraser on #zio-users this is what I ended up with, which perfectly solves my previous issues! sealed trait Command[S, R <: Has[_], A] {
def run: ZIO[R, Throwable, A]
def nextState(s: S, a: A): S
val verify: Assertion[A]
def exec(s: S): ZIO[R, Nothing, (S, TestResult)] =
for {
res <- run.orDie
} yield (nextState(s, res), assert(res)(verify))
}
val genResults = unfoldGen(Map(): State) { state =>
genCommand(state).mapM(cmd => cmd.exec(state))
}
checkM(genResults) { results =>
results.dropWhile(_.isSuccess).headOption match {
case Some(x) => ZIO.succeed(x)
case None => ZIO.succeed(assertCompletes)
}
}Amazing work. This is an incredibly powerful tool for testing and has already helped me catch two non-obvious bugs in the zio-grpc API I'm testing (one which was that PostgreSQL will fail if you happen to have a I'm super happy with this, but I think this construct is powerful enough that it might be worth to consider packaging it in an a way that's even easier for devs new to property based testing to get started with, either through thorough documentation or perhaps even a more out-of-the-box solution resembling my snippet above. Anyway, @adamgfraser, thanks for all the help and your fantastic work on this! 🙇 |
For stateful property based testing we need to be able to generate values, for example commands to a system under test, that may depend on the prior generated values.
To support this, this PR implements a new
unfoldGenconstructor with the following signature:If we conceptualize the state transition function
fas generating a tree of all possible states from the initial state, then this represents a generator of all possible traversals of the tree up to a given depth.With this we can implement stateful property based testing like so: