inflate and gunzip transducers #3825

LGLO · 2020-06-16T21:25:37Z

Resolves #3879

Ready for review

docs
same and better exceptions in inflate and ungzip
property test for random input compressed with Java, decompressed by this code
tests for gzipped headers with extra, file name, file comment, with/without CRC16 of header

deflate and gzip are out of scope of this PR, decompression only

simpadjo · 2020-06-17T17:35:43Z

Would be great to have property-based tests:

doSomething(bytes) == doSomethingUsingJavaLib(bytes)
undoSomethingUsingJavaLib(doSomething(bytes)) == bytes

LGLO · 2020-06-17T17:41:25Z

There is only decompression implemented in this PR, so I think I could add decompress(compressWithJava(input)) == input property tests only, it's equivalent to
decompressWithJava(compressWithJava(input)) == decompress(compressWithJava(input)).

iravid

Amazing work @LGLO. I left some minor comments and agree on the necessity of the property check. Big big thanks for working on this!

iravid · 2020-06-20T13:53:48Z

streams/jvm/src/main/scala/zio/stream/compression/package.scala

+        }
+        .map {
+          case (buffer, inflater) => {
+            case None => ZIO.succeed(Chunk.empty)


When a transducer receives a None, it should treat it as an end-of-stream marker, so at this point, it should flush everything it has (so it should do inflater.inflate) and prepare for a new set of input chunks (inflater.reset).

pullOutput, which I'll rename to pullAllOutput does it, when after reading some part of data it is .finished(). What is more if there are multiple delfated parts inside one stream, it would work.
However it wouldn't work if not whole deflated part was transduced (Inflater wouldn't be .finished()). Thanks for catching this! I'll add test for this case and fix!

@iravid actually I don't know if I can test it at ZStream level, should I write the test on level of calling push?

iravid · 2020-06-20T14:08:26Z

streams/jvm/src/main/scala/zio/stream/compression/Gunzipper.scala

+
+import zio._
+
+private[compression] class Gunzipper private (var state: Gunzipper.State) {


I suggest making the functionality in this file mutable and ZIO-less. It'd be much faster this way and you're only doing synchronous effects and error handling, so you can just replace that with exceptions.

iravid · 2020-06-20T14:11:00Z

streams/jvm/src/main/scala/zio/stream/compression/Gunzipper.scala

+
+  private val fixedHeaderLength = 10
+
+  sealed trait State {


I love how you organized the steps here. I'm not knowledgeable about the GZIP encoding, so I won't review everything here, but it looks great from the maintenance angle.

Hopefully ZIO Codec will make such parsers easier to write 💪🏻

iravid · 2020-06-21T08:32:02Z

Yeah that’s fine. There’s an abstraction in the transducer tests that lets you manually drive the transducer.

…

On 21 Jun 2020, 10:56 +0300, Lech Głowiak ***@***.***>, wrote: @LGLO commented on this pull request. In streams/jvm/src/main/scala/zio/stream/compression/package.scala: > + * HTTP 'deflate' content-encoding should use nowrap = false. See . + * */ + def inflate( + bufferSize: Int = 64 * 1024, + noWrap: Boolean = false + ): ZTransducer[Any, CompressionException, Byte, Byte] = { + def makeInflater( + bufferSize: Int + ): ZManaged[Any, Nothing, Option[zio.Chunk[Byte]] => ZIO[Any, CompressionException, Chunk[Byte]]] = + ZManaged + .make(ZIO.effectTotal((new Array[Byte](bufferSize), new Inflater(noWrap)))) { + case (_, inflater) => ZIO.effectTotal(inflater.end()) + } + .map { + case (buffer, inflater) => { + case None => ZIO.succeed(Chunk.empty) @iravid actually I don't know if I can test it at ZStream level, should I write the test on level of calling push? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

iravid

@LGLO Looks excellent. This is good to merge by me if you don't have any planned follow-ups. I left some minor comments but they're not critical.

iravid · 2020-06-26T14:50:30Z

streams/jvm/src/main/scala/zio/stream/compression/Gunzipper.scala

+
+  def close(): Unit = state.close()
+
+  def onChunk(c: Chunk[Byte]): ZIO[Any, CompressionException, Chunk[Byte]] =


In terms of code organization, I'd be fine with this just being : Chunk[Byte] and wrapping the exceptions into the CompressionException in compression.gunzip.

iravid · 2020-06-26T14:53:26Z

streams/jvm/src/main/scala/zio/stream/compression/package.scala

+        .map {
+          case (buffer, inflater) => {
+            case None =>
+              ZIO.succeed {


Technically this should be ZIO.effectTotal (ZIO.succeed is lazy and just forwards to effectTotal but it's nicer for maintainability to use effectTotal).

iravid · 2020-06-26T14:55:42Z

streams/jvm/src/main/scala/zio/stream/compression/Gunzipper.scala

+
+  private class ParseHeaderStep(acc: Array[Byte], crc32: CRC32) extends State {
+
+    //TODO: If whole input is shorther than fixed header, not output is produced and no error is singaled. Is it ok?


Sounds like we should throw an exception if the input does not contain a valid header. Is that possible to do?

Should I change:

case None => ZIO.succeed { gunzipper.reset() Chunk.empty }

to

case None => ZIO.succeed { if(gunzipper.isFinished()){ gunzipper.reset() Chunk.empty } else { <fail because stream wasn't complete gzipped> } } ``` ?

I would prefer the error and not the empty chunk :) Is there an "auto decompress" path that decompresses if the header exists, and leaves the bytes intact if not?

If we would like to have such path, then invalid header would have to fallback to clear text stream. Because there is no criterion to distinguish between invalid header and no header.

@LGLO To your question: Yes, you should add that else branch, but you should use ZIO.effect instead.

Thanks!
I didn't left this work here. Hopefully I could spend some time today in the evening and next week.

Thank you @LGLO!

iravid · 2020-06-26T15:00:02Z

Oh, one last request: can we move the inflate and gunzip transducer constructors to a platform-specific trait mixed into ZTransducer?

LGLO · 2020-06-27T19:09:38Z

I'll move this to trait and mix in to platform 👍
I'd like to add test for valid and invalid headers with various properties

… bug in pulling all from inflater

LGLO · 2020-07-07T18:59:44Z

Docs are added. Tests against more involved headers are added (and some bugs fixed as well). Removed WIP.

iravid · 2020-07-07T20:56:26Z

@LGLO Fantastic work and coverage. Thank you!!

LGLO requested a review from iravid as a code owner June 16, 2020 21:25

LGLO added help wanted Extra attention is needed stream ZIO Stream WIP Work in progress labels Jun 16, 2020

LGLO force-pushed the comprezzion branch from f76726a to 75fecf5 Compare June 17, 2020 06:18

iravid reviewed Jun 20, 2020

View reviewed changes

LGLO force-pushed the comprezzion branch 3 times, most recently from b733b8b to 6bfc164 Compare June 22, 2020 21:34

iravid previously approved these changes Jun 26, 2020

View reviewed changes

LGLO dismissed iravid’s stale review via c4ab1e5 July 5, 2020 20:57

LGLO force-pushed the comprezzion branch from 1e411dc to c4ab1e5 Compare July 5, 2020 20:57

LGLO added 8 commits July 6, 2020 22:35

WIP of un-deflate and un-gzip transducers

515672b

Fixing JDK8 build

1496072

Moved to JVM-only sources

208abd1

Less ZIO for performance gains. Naming, cleaning.

9602216

Property tests

da745d8

Moved to platform specific mix-ins

c6f12b4

Fail if input completes when output is not completed

2ab5cca

Fixed: wrong gzip header CRC16 computation, wrong gzip extra parsing,…

009a146

… bug in pulling all from inflater

LGLO force-pushed the comprezzion branch from c4ab1e5 to 009a146 Compare July 6, 2020 20:35

Docs

c36e71f

LGLO requested a review from iravid July 7, 2020 18:58

LGLO changed the title ~~WIP of un-deflate and un-gzip transducers~~ inflate and gunzip transducers Jul 7, 2020

??? is unallowd in mdoc

18aa6a6

iravid approved these changes Jul 7, 2020

View reviewed changes

LGLO merged commit f0c31bc into master Jul 8, 2020

LGLO deleted the comprezzion branch July 8, 2020 19:49

ghostbuster91 mentioned this pull request Aug 24, 2020

Add support for compression: Gzip and deflate monix/monix#1244

Merged


		import zio._

		private[compression] class Gunzipper private (var state: Gunzipper.State) {


		def close(): Unit = state.close()

		def onChunk(c: Chunk[Byte]): ZIO[Any, CompressionException, Chunk[Byte]] =


		private class ParseHeaderStep(acc: Array[Byte], crc32: CRC32) extends State {

		//TODO: If whole input is shorther than fixed header, not output is produced and no error is singaled. Is it ok?

Uh oh!

inflate and gunzip transducers #3825

inflate and gunzip transducers #3825

Uh oh!

Conversation

LGLO commented Jun 16, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

simpadjo commented Jun 17, 2020

Uh oh!

LGLO commented Jun 17, 2020

Uh oh!

iravid left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

iravid commented Jun 21, 2020 via email

Uh oh!

iravid left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

iravid commented Jun 26, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

LGLO commented Jun 27, 2020

Uh oh!

LGLO commented Jul 7, 2020

Uh oh!

iravid commented Jul 7, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

LGLO commented Jun 16, 2020 •

edited

Loading

iravid commented Jun 26, 2020 •

edited

Loading