Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Conversation

@iravid
Copy link
Member

@iravid iravid commented May 22, 2020

Fixes #3659.

We were already pulling the leftover into the carry, so concatenating it to the chunk was redundant.

I was going to write a smarter generator to catch this, but ran out of time.

@iravid iravid force-pushed the feature/fix-splitlines branch from 06d667c to 34a291a Compare May 22, 2020 20:48
@iravid
Copy link
Member Author

iravid commented May 22, 2020

@simpadjo Could you review this?

@simpadjo
Copy link
Contributor

simpadjo commented May 23, 2020

@iravid LGTM!
I tried to write a property-based test for it. That's what I came up with so far:


testM("pbt"){
          val breakGen: Gen[Random, String] = Gen.elements("'\r\n", "\n", "", "", "")
          val stringWithBreaks: Gen[Random with Sized, String] = Gen.string(Gen.printableChar).zipWith(breakGen)(_ ++ _)
          val g: Gen[Random with Sized, Chunk[String]] = Gen.listOf(stringWithBreaks).map(Chunk.fromIterable(_))
          val gen = Gen.listOf(g).map(Chunk.fromIterable(_))
          checkM(gen) { lines =>
            ZStream.fromChunks(lines).flattenChunks.transduce(ZTransducer.splitLines).runCollect.map(res => {
              println(s"${res.length}  " + res)
              val input = lines.map(_.mkString).mkString
              val expected = Source.fromString(input).getLines().toList
              assert(res)(equalTo(expected))
            })
          }
        }

But it doesn't catch this particular issue. Maybe you could improve the generator.

Although it does catch something. Could be added anyway.

.fromChunks(Chunk("aa", "bb"), Chunk("\nbbc\n", "ddb", "bd"), Chunk("abc", "\n"), Chunk("abc"))
.transduce(ZTransducer.splitLines)
.runCollect
)(equalTo(List("aabb", "bbc", "ddbbdabc", "abc")))
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
)(equalTo(List("aabb", "bbc", "ddbbdabc", "abc")))
)(equalTo(List("aabb", "theguardian", "ddbbdabc", "abc")))

🀣

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🀣

@iravid
Copy link
Member Author

iravid commented May 23, 2020

@simpadjo I will follow up with a better generator. If you want to pick that up, the idea is to:

  • Generate a list of strings - keep that around as target
  • Randomly intersperse "\n", "\r\n" or ""
  • .mkString on the list
  • Generate a random list of indices on which to split the string
  • Use .grouped(n) with a random chunk size to generate the chunks
  • Create a stream from the resulting list of chunks, transduce(splitLines).runCollect and compare that to target

@iravid iravid merged commit ab08958 into zio:master May 23, 2020
@simpadjo
Copy link
Contributor

@iravid not planning to work on it right now. Btw, logic of chunking is related to this ticket #3550

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ZTransducer.splitLines fails with OutOfMemoryError

3 participants