@vigith @kohlisid - I successfully reduced image size by 60% (from ~971MB to ~385MB) by fixing the broken multi-stage builds in all 17 Dockerfiles. The optimization removes build tools from final images while maintaining full functionality.
However, build times remain similar across both versions. Looking for suggestions on how to optimize build speed while keeping these size improvements.
The key change was switching from FROM builder AS udf to FROM python:3.10-slim-bullseye AS udf to properly separate build and runtime stages.

The jobs are failing with the following error, please run poetry lock

Using virtualenv: /home/runner/.cache/pypoetry/virtualenvs/pynumaflow-Fn3toN8H-py3.9
Installing dependencies from lock file

pyproject.toml changed significantly since poetry.lock was last generated. Run `poetry lock` to fix the lock file.
Error: Process completed with exit code 1.

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 94.26%. Comparing base (42f9fbd) to head (05a5649).
Report is 1 commits behind head on main.

@@           Coverage Diff           @@
##             main     #232   +/-   ##
=======================================
  Coverage   94.26%   94.26%           
=======================================
  Files          60       60           
  Lines        2441     2441           
  Branches      124      124           
=======================================
  Hits         2301     2301           
  Misses        101      101           
  Partials       39       39           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

could you please run black ?

could you please run black ?
Done

@sapkota-aayush still failed, https://github.com/numaproj/numaflow-python/actions/runs/15977556157/job/45079987225?pr=232

@sapkota-aayush You can use the inbuild make lint command for the repo

@sapkota-aayush You can use the inbuild make lint command for the repo
@vigith @kohlisid

I wasn’t able to run make lint directly because of some setup issues, but I manually fixed the formatting errors with Black and double-checked everything looks good now. That should take care of the lint errors.

Whenever you get a chance, maybe you could suggest the best way to run make lint on my setup?

Thanks!

looks like it ran successfully now.

You can try to setup a virtual env for the project, and install the required dependencies from the project toml file

Thanks @vigith Appreciate the tip @kohlisid — I’ll try setting up the virtual environment properly for smoother runs next time.

Also noticed that even after optimizing the Dockerfile size, each UDF still takes over 2 mins to build, which feels a bit much. Not sure if there’s a better way — maybe we could try caching or something else to speed it up?

Open to any suggestions — just trying to streamline things a bit. Appreciate the help as always!

did you look into #181 (comment) ?

@sapkota-aayush We can try to divide it into three multi stage layers

1. Base
2. Env setup
3. Builder

Then for each we can look at specific optimizations

@sapkota-aayush We can try to divide it into three multi stage layers

1. Base
2. Env setup
3. Builder

Then for each we can look at specific optimizations

Thanks for the suggestion Sid! Working on it, breaking it into Base, Env Setup, and Builder stages makes a lot of sense. Will follow up soon with progress and results. Appreciate the input!

@vigith @kohlisid
I’ve optimized all the example Dockerfiles using a 3-layer multi-stage build, fixed some file and formatting issues, and removed unnecessary dependencies and tested them as well. The only one I didn’t fully update is examples/reduce/batchmap/flatmap/Dockerfile since it’s missing a pyproject.toml and looks incomplete. If you could run the tests and let me know if anything else needs fixing or if you have any feedback, that’d be great. Thanks!

@sapkota-aayush Have you tested the new images end-to-end by running in a pipeline?

@sapkota-aayush Have you tested the new images end-to-end by running in a pipeline?

Yes, I have tested the new images end-to-end by running them in a Numaflow pipeline on Kubernetes.
The pipeline was deployed successfully, all pods were running and healthy, and the custom Docker image was used in the workflow.
Please see the attached screenshot for details.
Let me know if you need any more information!

Thanks for taking it up and the collaboration on this @sapkota-aayush :D

Thanks for taking it up and the collaboration on this @sapkota-aayush :D

Thank you @vigith and @kohlisid for your guidance. Learned a lot.