Thanks to visit codestin.com
Credit goes to github.com

Skip to content

chore: use same server info for all map modes and refactor #185

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 10 commits into from
Jul 18, 2024
Merged

chore: use same server info for all map modes and refactor #185

merged 10 commits into from
Jul 18, 2024

Conversation

kohlisid
Copy link
Contributor

@kohlisid kohlisid commented Jul 16, 2024 •
edited
Loading

Fixes : #1837

kohlisid added 3 commits July 15, 2024 21:03
@kohlisid
refactors
1e75a4b 
Signed-off-by: Sidhant Kohli <[email protected]>
@kohlisid
Merge branch 'main' into serverinfo
b117b3a
@kohlisid
refactors
19f1809 
Signed-off-by: Sidhant Kohli <[email protected]>
@kohlisid kohlisid changed the title chore: use same server info for all map modes and refactor chore: use same server info for all map modes and refactor [merge after batchmap PR] Jul 16, 2024
@codecov Codecov
Copy link

codecov bot commented Jul 16, 2024 •
edited
Loading

Codecov Report

Attention: Patch coverage is 78.37838% with 8 lines in your changes missing coverage. Please review.

Project coverage is 94.52%. Comparing base (5bbbe56) to head (f35c6bf).

Files Patch % Lines
pynumaflow/mapper/multiproc_server.py 33.33% 4 Missing ⚠️
pynumaflow/info/types.py 92.00% 1 Missing and 1 partial ⚠️
pynumaflow/mapper/sync_server.py 33.33% 2 Missing ⚠️
Additional details and impacted files 
@@            Coverage Diff             @@
##             main     #185      +/-   ##
==========================================
- Coverage   94.82%   94.52%   -0.30%     
==========================================
  Files          56       56              
  Lines        2184     2212      +28     
  Branches      130      131       +1     
==========================================
+ Hits         2071     2091      +20     
- Misses         83       90       +7     
- Partials       30       31       +1

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@KeranYang KeranYang requested a review from ayildirim21 July 16, 2024 15:58
kohlisid added 3 commits July 16, 2024 11:34
@kohlisid
refactor
6e8b352 
Signed-off-by: Sidhant Kohli <[email protected]>
@kohlisid
Merge branch 'main' into serverinfo
e4ce2ff
@kohlisid
batch-map
b4e47c5 
Signed-off-by: Sidhant Kohli <[email protected]>
@kohlisid kohlisid requested a review from KeranYang July 16, 2024 18:55
@kohlisid kohlisid changed the title chore: use same server info for all map modes and refactor [merge after batchmap PR] chore: use same server info for all map modes and refactor Jul 16, 2024
kohlisid added 2 commits July 16, 2024 11:59
@kohlisid
batch-map
fd75b95 
Signed-off-by: Sidhant Kohli <[email protected]>
@kohlisid
batch-map
7d63df7 
Signed-off-by: Sidhant Kohli <[email protected]>
@kohlisid
Copy link
Contributor Author

kohlisid commented Jul 16, 2024 •
edited
Loading

Map Stream with new map mode metadata

no annotations in spec

    - name: flatmap
      limits:
        readBatchSize: 1
      udf:
        container:
          image: quay.io/kohlisid/numaflow-python/map-flatmap-stream:stable1
          imagePullPolicy: Always

Map stream server with mapstream.sock

 skohli@macos-JQWR9T560R flatmap_stream % kubectl logs simple-pipeline-flatmap-0-nzluc -f  
+ python example.py
2024-07-16 21:03:01 INFO     Starting Map Stream Server
2024-07-16 21:03:01 INFO     Async GRPC Server listening on: unix:///var/run/numaflow/mapstream.sock with max threads: 4

Server info file - /var/run/numaflow/mapper-server-info

 kubectl logs simple-pipeline-flatmap-0-nzluc numa -f  | grep "mapper"
2024/07/16 21:02:43 Server info file /var/run/numaflow/mapper-server-info is not ready...

 kubectl logs simple-pipeline-flatmap-0-nzluc numa -f  | grep Info
2024/07/16 21:03:01 ServerInfo: &{uds python 1.2.0-rc4 0.7.3 map[MAP_MODE:stream-map]}

kubectl logs simple-pipeline-flatmap-0-nzluc numa -f  | grep "enabled"
2024-07-16T21:03:01.93051948Z   INFO    numaflow.MapUDF-processor       udf/map_udf.go:150      Map mode enabled: Stream Map    {"pipeline": "simple-pipeline", "vertex": "flatmap", "protocol": "uds-grpc-map-udf"}

@kohlisid
refactor
a5b2bab 
Signed-off-by: Sidhant Kohli <[email protected]>
KeranYang
KeranYang reviewed Jul 16, 2024
View reviewed changes
@kohlisid
Copy link
Contributor Author

Batch Map with new map mode metadata

no annotations in spec

    - name: batch-flatmap
      partitions: 2
      scale:
        min: 1
      udf:
        container:
          image: quay.io/kohlisid/numaflow-python/batch-map-flatmap:stable1
          imagePullPolicy: Always

2024/07/17 21:06:52 Server info file /var/run/numaflow/mapper-server-info is not ready...
2024/07/17 21:06:53 ServerInfo: &{uds python 1.2.0-rc4 0.7.3 map[MAP_MODE:batch-map]}
{"level":"info","ts":"2024-07-17T21:06:53.813018858Z","logger":"numaflow.MapUDF-processor","caller":"udf/map_udf.go:173","msg":"Map mode enabled: Batch Map","pipeline":"flatmap","vertex":"batch-flatmap","protocol":"uds-grpc-map-udf"}
2024/07/17 21:06:53 UDS Client: unix:/var/run/numaflow/batchmap.sock

@kohlisid
Copy link
Contributor Author

kohlisid commented Jul 17, 2024 •
edited
Loading

Unary Map with new map mode metadata

no annotations in spec

    - name: flatmap
      udf:
        container:
          image: quay.io/kohlisid/numaflow-python/map-flatmap:stable
          imagePullPolicy: Always
          env:
            - name: PYTHONDEBUG
              value: "true"


2024/07/17 21:13:39 Server info file /var/run/numaflow/mapper-server-info is not ready..
2024-07-17T21:13:55.67101085Z   INFO    numaflow.MapUDF-processor       udf/map_udf.go:196      Map mode enabled: Unary Map     {"pipeline": "simple-pipeline", "vertex": "flatmap", "protocol": "uds-grpc-map-udf"} 
2024/07/17 21:13:55 ServerInfo: &{uds python 1.2.0-rc4 0.7.3 map[MAP_MODE:unary-map]}
2024/07/17 21:13:55 UDS Client: unix:/var/run/numaflow/map.sock

@kohlisid
Copy link
Contributor Author

Unary Map with no metadata (older code + new numaflow backward)

no annotations in spec

    - name: flatmap
      udf:
        container:
          image: quay.io/kohlisid/numaflow-python/map-flatmap:stable
          imagePullPolicy: Always
          env:
            - name: PYTHONDEBUG
              value: "true"


2024/07/17 21:21:21 Server info file /var/run/numaflow/mapper-server-info is not ready...
2024/07/17 21:21:34 ServerInfo: &{uds python 1.2.0-rc4 0.7.3 map[]}
2024-07-17T21:21:34.368700528Z  INFO    numaflow.MapUDF-processor       udf/map_udf.go:196      Map mode enabled: Unary Map     {"pipeline": "simple-pipeline", "vertex": "flatmap", "protocol": "uds-grpc-map-udf"}
2024/07/17 21:21:34 UDS Client: unix:/var/run/numaflow/map.sock

@kohlisid
refactor
f35c6bf 
Signed-off-by: Sidhant Kohli <[email protected]>
@kohlisid kohlisid requested a review from KeranYang July 17, 2024 21:25
@kohlisid kohlisid marked this pull request as ready for review July 17, 2024 21:25
@kohlisid kohlisid requested review from ab93, vigith and whynowy as code owners July 17, 2024 21:25
@kohlisid
Copy link
Contributor Author

kohlisid commented Jul 17, 2024 •
edited
Loading

Multiproc Map

2024-07-17T21:27:54.898644917Z  INFO    numaflow.MapUDF-processor       udf/map_udf.go:196      Map mode enabled: Unary Map     {"pipeline": "simple-pipeline", "vertex": "mult", "protocol": "uds-grpc-map-udf"}
2024/07/17 21:27:54 ServerInfo: &{uds python 1.2.0-rc4 0.7.3 map[CPU_LIMIT:6 MULTIPROC:3]}
2024/07/17 21:27:54 Multiprocessing Servers : 3

@KeranYang
Copy link
Member

KeranYang commented Jul 18, 2024 •
edited
Loading

@kohlisid I am approving but it would be great if we make the codecov pass before merging.

@kohlisid kohlisid merged commit c5a46bb into numaproj:main Jul 18, 2024
@kohlisid kohlisid deleted the serverinfo branch July 18, 2024 20:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@vigith vigith vigith approved these changes

@KeranYang KeranYang KeranYang approved these changes

@ab93 ab93 Awaiting requested review from ab93 ab93 is a code owner

@whynowy whynowy Awaiting requested review from whynowy whynowy is a code owner

+1 more reviewer

@ayildirim21 ayildirim21 ayildirim21 approved these changes

Reviewers whose approvals may not affect merge requirements
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Python-SDK single-server info for map
4 participants
@kohlisid @KeranYang @vigith @ayildirim21