Updates YAML SDK to replace Kafka read/write transforms with equivalent managed transforms #34755

chamikaramj · 2025-04-28T01:57:34Z

To preserve update compatibility, we avoid replacement if the an 'updateCompatibilityBeamVersion' with an old Beam version is provided.

This fixes #34767

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
Update CHANGES.md with noteworthy changes.
If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.

github-actions · 2025-04-28T03:24:03Z

Checks are failing. Will not request review until checks are succeeding. If you'd like to override that behavior, comment assign set of reviewers

github-actions · 2025-04-28T07:48:02Z

Assigning reviewers. If you would like to opt out of this review, comment assign to next reviewer:

R: @shunping for label python.

Available commands:

stop reviewer notifications - opt out of the automated review tooling
remind me after tests pass - tag the comment author after tests pass
waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).

chamikaramj · 2025-04-28T19:43:42Z

PreCommit_Yaml_Xlang_Direct seems to be perma-red: #32603

chamikaramj · 2025-04-28T19:43:55Z

R: @ahmedabu98 @derrickaw

github-actions · 2025-04-28T19:45:24Z

Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control. If you'd like to restart, comment assign set of reviewers

ahmedabu98 · 2025-04-28T20:00:02Z

sdks/python/apache_beam/transforms/external.py


+ManagedReplacement = namedtuple(
+    'ManagedReplacement',
+    'underlying_transform_identifier update_compatibility_version')


Suggested change

'underlying_transform_identifier update_compatibility_version')

'underlying_transform_identifier, update_compatibility_version')

ahmedabu98 · 2025-04-28T20:08:37Z

sdks/python/apache_beam/yaml/yaml_provider.py

 import apache_beam.dataframe.io
 import apache_beam.io
 import apache_beam.transforms.util
+from apache_beam import ManagedReplacement


Curious how this is getting imported from top-level apache_beam

We include everything under the "transforms" package in the top level:

beam/sdks/python/apache_beam/__init__.py

Line 93 in b9fa49a

from apache_beam.transforms import *

ahmedabu98 · 2025-04-28T21:01:47Z

sdks/python/apache_beam/transforms/external.py

+
+    compatibility_version_match = True
+    if self._managed_replacement and compatibility_version_match:
+      compat_version_match = not is_compat_version_prior_to(


can we use a clearer name than compat_version_match? Not sure if this means the SDK version is before or after the specified update_compatibility_version

ahmedabu98 · 2025-04-28T21:02:54Z

sdks/python/apache_beam/transforms/managed.py

+  if identifier in MANAGED_TRANSFORM_URN_TO_JAR_TARGET_MAPPING:
+    default_target = MANAGED_TRANSFORM_URN_TO_JAR_TARGET_MAPPING.get(identifier)


Might as well simplify and throw an error if identifier is not in MANAGED_TRANSFORM_URN_TO_JAR_TARGET_MAPPING

chamikaramj

Thanks. PTAL.

chamikaramj · 2025-04-28T21:44:11Z

sdks/python/apache_beam/transforms/external.py


+ManagedReplacement = namedtuple(
+    'ManagedReplacement',
+    'underlying_transform_identifier update_compatibility_version')


chamikaramj · 2025-04-28T21:45:22Z

sdks/python/apache_beam/yaml/yaml_provider.py

 import apache_beam.dataframe.io
 import apache_beam.io
 import apache_beam.transforms.util
+from apache_beam import ManagedReplacement


We include everything under the "transforms" package in the top level:

beam/sdks/python/apache_beam/__init__.py

Line 93 in b9fa49a

from apache_beam.transforms import *

chamikaramj · 2025-04-28T21:46:44Z

sdks/python/apache_beam/transforms/external.py

+
+    compatibility_version_match = True
+    if self._managed_replacement and compatibility_version_match:
+      compat_version_match = not is_compat_version_prior_to(


chamikaramj · 2025-04-28T23:51:56Z

sdks/python/apache_beam/transforms/managed.py

+  if identifier in MANAGED_TRANSFORM_URN_TO_JAR_TARGET_MAPPING:
+    default_target = MANAGED_TRANSFORM_URN_TO_JAR_TARGET_MAPPING.get(identifier)


ahmedabu98 · 2025-04-30T15:03:14Z

Failing test: AttributeError: module 'apache_beam.transforms.managed' has no attribute '_EXPANSION_SERVICE_JAR_TARGETS'

ahmedabu98 · 2025-04-30T15:07:25Z

sdks/python/apache_beam/transforms/external.py

+_IO_EXPANSION_SERVICE_JAR_TARGET = "sdks:java:io:expansion-service:shadowJar"
+
+_GCP_EXPANSION_SERVICE_JAR_TARGET = (
+    "sdks:java:io:google-cloud-platform:expansion-service:shadowJar")


Maybe combine these in a list of "supported expansion jar targets" and replace this:

beam/sdks/python/gen_managed_doc.py

Line 102 in 74c514f

services_and_names = managed._EXPANSION_SERVICE_JAR_TARGETS

…nt managed transforms

chamikaramj · 2025-04-30T18:58:30Z

PTAL.

github-actions bot added python yaml labels Apr 28, 2025

chamikaramj force-pushed the beam_yaml_managed_kafka branch from b2787fd to 98de30b Compare April 28, 2025 06:29

github-actions bot added the Next Action: Reviewers label Apr 28, 2025

chamikaramj force-pushed the beam_yaml_managed_kafka branch from 98de30b to 8c91c8c Compare April 28, 2025 07:54

ahmedabu98 reviewed Apr 28, 2025

View reviewed changes

chamikaramj commented Apr 28, 2025

View reviewed changes

chamikaramj force-pushed the beam_yaml_managed_kafka branch from 8373652 to 03fb49c Compare April 29, 2025 18:08

ahmedabu98 reviewed Apr 30, 2025

View reviewed changes

chamikaramj added 6 commits April 30, 2025 09:28

Updates YAML SDK to replace Kafka read/write transforms with equivale…

5979487

…nt managed transforms

Addressing reviewer comments and adding unit tests

e996a3d

Resolves conflict

8c6d423

Fixes a test failure

25ecf87

Fix lint

cb336bc

Fixes a test

ae0fe9d

chamikaramj force-pushed the beam_yaml_managed_kafka branch from 6976ce5 to ae0fe9d Compare April 30, 2025 17:08

chamikaramj mentioned this pull request Apr 30, 2025

[Feature Request]: Automatically replace Beam YAML Kafka source/sink with managed I/O #34767

Closed

17 tasks

ahmedabu98 approved these changes Apr 30, 2025

View reviewed changes

chamikaramj merged commit f3e0f6d into apache:master Apr 30, 2025
90 checks passed

	'underlying_transform_identifier update_compatibility_version')
	'underlying_transform_identifier, update_compatibility_version')

		if identifier in MANAGED_TRANSFORM_URN_TO_JAR_TARGET_MAPPING:
		default_target = MANAGED_TRANSFORM_URN_TO_JAR_TARGET_MAPPING.get(identifier)

Updates YAML SDK to replace Kafka read/write transforms with equivalent managed transforms #34755

Updates YAML SDK to replace Kafka read/write transforms with equivalent managed transforms #34755

Uh oh!

Conversation

chamikaramj commented Apr 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

GitHub Actions Tests Status (on master branch)

Uh oh!

github-actions bot commented Apr 28, 2025

Uh oh!

github-actions bot commented Apr 28, 2025

Uh oh!

chamikaramj commented Apr 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

chamikaramj commented Apr 28, 2025

Uh oh!

github-actions bot commented Apr 28, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

chamikaramj left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ahmedabu98 commented Apr 30, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

chamikaramj commented Apr 30, 2025

Uh oh!

Uh oh!

Uh oh!

chamikaramj commented Apr 28, 2025 •

edited

Loading

chamikaramj commented Apr 28, 2025 •

edited

Loading