Problem Description

I really need to await an async's function result inside a stream chunk callback, and all the workarounds I can think of are unsatisfactory.

Proposal

I'd like to either:

async def responseheaders(self, flow: http.HTTPFlow) -> None:
  async def process_chunk(chunk: bytes) -> bytes:
      return await my_transform(chunk)
  
  flow.response.stream = process_chunk

or fit streaming more directly into the hooks system, as streaming callbacks are an odd man out right now. I'm sure there's reasons for that though, so this may not be feasible

async def responsechunk(self, flow: http.HTTPFlow) -> list[bytes] | bytes:
  return await my_transform(chunk)

An alternative would be to let me return additional data from the response callback even in the streaming case. Right now I don't see a way to do it, as in mitmproxy/proxy/layers/http/__init__.py:send_response the only send not gated behind if not already_streamed flag is the trailer headers.

Alternatives

1. Do the work in a thread, block the mitmproxy callback on a thread future
   • I already have a loop (the one mitmproxy runs on) and would like to avoid the overhead and complexity
   • This will serialize the whole mitmproxy processing while the future resolves
2. Don't use streaming
   • no-go due to SSE, plus I need to stream the responses for other reasons
3. ...?