I think this not good:

• Unlike mapZIO processing is done chunk by chunk. When f fails for an
• element in the middle of a Chunk, the other elements of the chunk are lost.

If processing fails in the middle of a chunk, the stream should first emit the elements where processing succeeded and then fail the stream.

the act of doing this breaks chunking structure though

the act of doing this breaks chunking structure though

No, not if you implement it right.

I've written an implementation that will retain chunking while also emitting every element that was processed:
https://github.com/mberndt123/zio/tree/mapziochunked

I'd open a merge request to @erikvanoosten's repo, but for whatever reason, I can't create a PR targeting his ZIO fork.

I've written an implementation that will retain chunking while also emitting every element that was processed: https://github.com/mberndt123/zio/tree/mapziochunked

I'd open a merge request to @erikvanoosten's repo, but for whatever reason, I can't create a PR targeting his ZIO fork.

Thanks @mberndt123 ! I have cherry picked your implementation and updated the scaladoc and unit test. I was playing around with foldLeft to get the same result, but your solution is simpler.

@regiskuckaertz Thanks for your approval. Can you merge as well please?

For the ZIO contributor that is reviewing this. @mberndt123 proposed another solution:

    ZPipeline.chunks.mapStream { (chunk: Chunk[In]) =>
      val builder = ChunkBuilder.make[Out](chunk.size)
      def writeNonEmpty = {
        val out = builder.result()
        ZStream.when(out.nonEmpty)(ZStream.fromChunk(out))
      }
      ZStream.unwrap {
        chunk
          .mapZIODiscard(f(_).map(builder += _))
          .foldCause(
            cause => writeNonEmpty ++ ZStream.failCause(cause),
            _ => writeNonEmpty
          )
      }

Please let me know if that has preference and I'll swap it in.

@kyri-petrou I feel like I'm not in a position to merge anymore but I feel like @erikvanoosten has exercised all due diligence and this has my vote, for what it's worth.

@paulpdaniels, a thumbs-down emoji without further explanation is a bit disappointing. My implementation is significantly terser and relies on higher level combinators (i. e. mapStream), both of which are significant maintainability advantages. If you're going to argue performance, then I'd like to see some evidence. Using low-level ZChannel primitives doesn't magically make code faster.

I've done some cursory benchmarks with this thing in StreamBenchmarks.scala:

  @Benchmark
  def zioMapZIOChunked: Unit = {
    val stream = ZStream
      .fromChunks(zioChunks: _*)
      .mapZIOChunked(i => ZIO.succeed((i * i).toLong))
    
    val result = stream.runDrain

    unsafeRun(result)
  }

// run with `benchmarks/Jmh/run zio.StreamBenchmarks.zioMapZIOChunked`

I didn't see any meaningful (i. e. beyond noise) performance difference between the different implementations.

@mberndt123

then I'd like to see some evidence

Running your above example with the two implementations (current vs proposed) side-by-side

// Using  17.0.12-amzn
// benchmarks/jmh:run .*StreamBenchmarks.zioPipelineMapChunked
chunkCount=10000 chunkSize=5000: current impl faster ~15% 
chunkCount=5000 chunkSize=5000: current impl faster ~12%
chunkCount=5000 chunkSize=50: current impl faster ~450%
chunkCount=5000 chunkSize=5: current impl faster ~1000% (10x)
chunkCount=5000 chunkSize=1: current impl faster ~1300% (13x)

🪄

I haven't run a memory profile on these, but I would expect it would lean toward the current implementation as well.

and relies on higher level combinators

This is a mistake for low-level operators, for instance mapStream is meant as an interop operator for user-land code that has a Stream but wants it to work in a pipeline without having to unpack it into a ZChannel. You can very quickly check these operators implementations to see the amount of extra work the operator would have to do if they were included.

both of which are significant maintainability advantages

The current implementation is written in line with the rest of the library, consistency is just as important in maintainability.

This does open up some questions in my mind about why the performance is so bad in some of the cases above (perhaps they are too naively reliant on other operators), certainly some fruitful avenues to explore on how to optimize those!

Edit

There is a case where the proposed is faster than the current:

// Seems consistently faster, though for some reason current implementation generates a lot variance with this one.
chunkCount=5000 chunkSize=0: proposed impl faster ~2%

@kyri-petrou this PR is ready for merging!

@kyri-petrou

Can we add a @ see [[mapZIOChunked]] to the scaladoc of mapZIO as well?

We could introduce some more documentation around these chunk-breakers on https://zio.dev as well. This zio-kafka page could be used as a start (see the page source for hidden sections).

We could introduce some more documentation around these chunk-breakers on https://zio.dev/ as well

Yeah I agree. I'll create an issue for it