Drain Mode as WindowedValue extension #34764

stankiewicz · 2025-04-28T15:30:13Z

Changes:

new capability to runner - extended element metadata, set capability for java sdk
new proto definition for element metadata
add coder changes for windowedValue to detect and parse element metadata and vice versa, to encode them if we have them.
add changes to windowedValue to expose elementMetadata object (autovalue, not proto)

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
Update CHANGES.md with noteworthy changes.
If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.

stankiewicz · 2025-04-28T15:31:28Z

@kennknowles ptal.

github-actions · 2025-04-28T17:06:19Z

Assigning reviewers. If you would like to opt out of this review, comment assign to next reviewer:

R: @robertwb for label java.

Available commands:

stop reviewer notifications - opt out of the automated review tooling
remind me after tests pass - tag the comment author after tests pass
waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).

sjvanrossum · 2025-04-29T09:54:54Z

sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/PaneInfo.java

@@ -360,7 +387,8 @@ public PaneInfo decode(final InputStream inStream) throws CoderException, IOExce
      byte keyAndTag = (byte) inStream.read();
      PaneInfo base = Preconditions.checkNotNull(BYTE_TO_PANE_INFO.get((byte) (keyAndTag & 0x0F)));
      long index, onTimeIndex;
-      switch (Encoding.fromTag(keyAndTag)) {
+      boolean elementMetadata = (keyAndTag & 0x80) > 0;


Suggested change

boolean elementMetadata = (keyAndTag & 0x80) > 0;

boolean elementMetadata = keyAndTag < 0;

sjvanrossum · 2025-04-29T10:04:37Z

sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/PaneInfo.java

+      boolean elementMetadata = (keyAndTag & 0x80) > 0;
+      switch (Encoding.fromTag((byte) (keyAndTag & 0x7F))) {


Feel free to ignore my codegolfing...

Suggested change

boolean elementMetadata = (keyAndTag & 0x80) > 0;

switch (Encoding.fromTag((byte) (keyAndTag & 0x7F))) {

boolean elementMetadata = keyAndTag < (keyAndTag &= 0x7f);

switch (Encoding.fromTag(keyAndTag) {

codecov · 2025-04-29T13:38:05Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 56.38%. Comparing base (3b00517) to head (7e3e75a).
Report is 347 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff            @@
##             master   #34764   +/-   ##
=========================================
  Coverage     56.38%   56.38%           
- Complexity     3285     3288    +3     
=========================================
  Files          1174     1174           
  Lines        179401   179407    +6     
  Branches       3398     3399    +1     
=========================================
+ Hits         101158   101164    +6     
- Misses        74983    74984    +1     
+ Partials       3260     3259    -1

Flag	Coverage Δ
java	`70.50% <ø> (+<0.01%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

kennknowles

Largely looks good to me. I need to think about the protocol maybe slightly more. We can probably do it incrementally and get started with changes that are benign and not used yet.

One random cleanup issue (which I hit also since I'm doing similar things with WindowedValue but for different extensions) is I wonder if we can flatten all the silly inheritance in the layers of WindowedValue. For performance ultimately all that matters is whether or not we optimize the coder (such as for batch use case with no timestamps or windows we don't bother encoding min timestamp and global window). And for coding it is much better not to have such implementation inheritance that causes all this toil. Plus the anti-pattern of "superclass that just holds fields"

sjvanrossum · 2025-04-30T10:48:07Z

@kennknowles the default object layout on recent JVMs (64 bit, 8 bytes alignment, 12 bytes object header, compressed references and class pointers) leaves 4 bytes of padding in SimpleWindowedValue:

org.apache.beam.sdk.util.WindowedValue$SimpleWindowedValue object internals:
OFF  SZ                                                TYPE DESCRIPTION                 VALUE
  0   8                                                     (object header: mark)       N/A
  8   4                                                     (object header: class)      N/A
 12   4                                    java.lang.Object SimpleWindowedValue.value   N/A
 16   4   org.apache.beam.sdk.transforms.windowing.PaneInfo SimpleWindowedValue.pane    N/A
 20   4                                                     (object alignment gap)
Instance size: 24 bytes
Space losses: 0 bytes internal + 4 bytes external = 4 bytes total

MinTimestampWindowedValue and TimestampedValueInGlobalWindow also have the same size, but TimestampedValueInGlobalWindow packs the timestamp field in the alignment gap.

org.apache.beam.sdk.util.WindowedValue$MinTimestampWindowedValue object internals:
OFF  SZ                                                TYPE DESCRIPTION                 VALUE
  0   8                                                     (object header: mark)       N/A
  8   4                                                     (object header: class)      N/A
 12   4                                    java.lang.Object SimpleWindowedValue.value   N/A
 16   4   org.apache.beam.sdk.transforms.windowing.PaneInfo SimpleWindowedValue.pane    N/A
 20   4                                                     (object alignment gap)
Instance size: 24 bytes
Space losses: 0 bytes internal + 4 bytes external = 4 bytes total

org.apache.beam.sdk.util.WindowedValue$TimestampedValueInGlobalWindow object internals:
OFF  SZ                                                TYPE DESCRIPTION                          VALUE
  0   8                                                     (object header: mark)                N/A
  8   4                                                     (object header: class)               N/A
 12   4                                    java.lang.Object SimpleWindowedValue.value            N/A
 16   4   org.apache.beam.sdk.transforms.windowing.PaneInfo SimpleWindowedValue.pane             N/A
 20   4                               org.joda.time.Instant TimestampedWindowedValue.timestamp   N/A
Instance size: 24 bytes
Space losses: 0 bytes internal + 0 bytes external = 0 bytes total

And for TimestampedValueInSingleWindow and TimestampedValueInMultipleWindows it spills over into the next word (timestamp fills the alignment gap, window/windows spills into the next word):

org.apache.beam.sdk.util.WindowedValue$TimestampedValueInSingleWindow object internals:
OFF  SZ                                                     TYPE DESCRIPTION                             VALUE
  0   8                                                          (object header: mark)                   N/A
  8   4                                                          (object header: class)                  N/A
 12   4                                         java.lang.Object SimpleWindowedValue.value               N/A
 16   4        org.apache.beam.sdk.transforms.windowing.PaneInfo SimpleWindowedValue.pane                N/A
 20   4                                    org.joda.time.Instant TimestampedWindowedValue.timestamp      N/A
 24   4   org.apache.beam.sdk.transforms.windowing.BoundedWindow TimestampedValueInSingleWindow.window   N/A
 28   4                                                          (object alignment gap)
Instance size: 32 bytes
Space losses: 0 bytes internal + 4 bytes external = 4 bytes total

org.apache.beam.sdk.util.WindowedValue$TimestampedValueInMultipleWindows object internals:
OFF  SZ                                                TYPE DESCRIPTION                                 VALUE
  0   8                                                     (object header: mark)                       N/A
  8   4                                                     (object header: class)                      N/A
 12   4                                    java.lang.Object SimpleWindowedValue.value                   N/A
 16   4   org.apache.beam.sdk.transforms.windowing.PaneInfo SimpleWindowedValue.pane                    N/A
 20   4                               org.joda.time.Instant TimestampedWindowedValue.timestamp          N/A
 24   4                                java.util.Collection TimestampedValueInMultipleWindows.windows   N/A
 28   4                                                     (object alignment gap)
Instance size: 32 bytes
Space losses: 0 bytes internal + 4 bytes external = 4 bytes total

With default object layout settings the addition of an extensions field would only increase the size of TimestampedValueInGlobalWindow by 8 bytes since it would spill into the next word, all the other subclasses should pack that field into the alignment gap. I'm not sure whether users of Beam commonly disable UseCompressedPointers or UseCompressedOops, but it would seem somewhat unlikely in my opinion since both of them would generally increase memory usage.

Compile time annotation processing to produce flat extensions (without a separate extensions container,
treating timestamp, window and windows as standard extensions) of all subclasses of WindowedValue might help make this a bit more manageable. Something like the mechanism behind PipelineOptions, but at compile time and backed by fields instead of a Map<String, Object>.

kennknowles · 2025-05-02T16:51:02Z

Stating it here so that I remember - we should write a quite 1 pager separate from the DrainMode document about the extended metadata and have a top level thread on dev@. I think more people will be interested if it is something like "Extended per-element metadata proto". It is quite an alarming title that will bring out the performance experts :-)

github-actions · 2025-05-10T12:14:23Z

Reminder, please take a look at this pr: @robertwb

github-actions · 2025-05-14T12:15:05Z

Assigning new set of reviewers because Pr has gone too long without review. If you would like to opt out of this review, comment assign to next reviewer:

R: @chamikaramj for label java.

Available commands:

stop reviewer notifications - opt out of the automated review tooling
remind me after tests pass - tag the comment author after tests pass
waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

github-actions · 2025-05-23T12:15:02Z

Reminder, please take a look at this pr: @chamikaramj

robertwb · 2025-05-23T16:33:57Z

sdks/java/core/src/main/java/org/apache/beam/sdk/util/WindowedValue.java

+      Instant timestamp,
+      Collection<? extends BoundedWindow> windows,
+      PaneInfo pane,
+      @Nullable ElementMetadata elementMetadata) {
    checkArgument(pane != null, "WindowedValue requires PaneInfo, but it was null");
    checkArgument(windows.size() > 0, "WindowedValue requires windows, but there were none");



Perhaps also validate here that elementMetadata is null if and only if PaneInfo expects metadata?

stankiewicz · 2025-05-23T16:36:24Z

moving to draft, reviews are not needed at this point as it is not final PR.

github-actions · 2025-05-28T12:15:38Z

Assigning new set of reviewers because Pr has gone too long without review. If you would like to opt out of this review, comment assign to next reviewer:

R: @Abacn for label java.

Available commands:

stop reviewer notifications - opt out of the automated review tooling
remind me after tests pass - tag the comment author after tests pass
waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

github-actions · 2025-06-04T12:15:53Z

Reminder, please take a look at this pr: @Abacn

stankiewicz · 2025-06-04T13:57:37Z

stop reviewer notifications

github-actions · 2025-06-04T13:59:05Z

Stopping reviewer notifications for this pull request: requested by reviewer. If you'd like to restart, comment assign set of reviewers

stankiewicz · 2025-06-04T14:15:23Z

closing in favor of #35152

drain mode as extension

acf46fc

github-actions bot added java model labels Apr 28, 2025

github-actions bot added the Next Action: Reviewers label Apr 28, 2025

add withMetadata

2409fdf

github-actions bot added runners dataflow and removed runners dataflow labels Apr 29, 2025

sjvanrossum reviewed Apr 29, 2025

View reviewed changes

stankiewicz added 2 commits April 29, 2025 12:43

add tests

5473d67

fix ValueInEmptyWindows

932b53a

github-actions bot added runners dataflow and removed runners dataflow labels Apr 29, 2025

style

7e3e75a

github-actions bot added runners dataflow and removed runners dataflow labels Apr 29, 2025

kennknowles reviewed Apr 29, 2025

View reviewed changes

github-actions bot added the slow-review label May 10, 2025

github-actions bot added reassigned-reviewers and removed slow-review labels May 14, 2025

Merge branch 'apache:master' into wv_extensions

3082807

github-actions bot added runners dataflow and removed runners dataflow labels May 16, 2025

github-actions bot added the slow-review label May 23, 2025

robertwb reviewed May 23, 2025

View reviewed changes

stankiewicz marked this pull request as draft May 23, 2025 16:35

github-actions bot removed the slow-review label May 28, 2025

github-actions bot added the slow-review label Jun 4, 2025

stankiewicz closed this Jun 4, 2025

	boolean elementMetadata = (keyAndTag & 0x80) > 0;
	boolean elementMetadata = keyAndTag < 0;

		boolean elementMetadata = (keyAndTag & 0x80) > 0;
		switch (Encoding.fromTag((byte) (keyAndTag & 0x7F))) {

Drain Mode as WindowedValue extension #34764

Drain Mode as WindowedValue extension #34764

Uh oh!

Conversation

stankiewicz commented Apr 28, 2025

GitHub Actions Tests Status (on master branch)

Uh oh!

stankiewicz commented Apr 28, 2025

Uh oh!

github-actions bot commented Apr 28, 2025

Uh oh!

sjvanrossum Apr 29, 2025

Choose a reason for hiding this comment

Uh oh!

sjvanrossum Apr 29, 2025

Choose a reason for hiding this comment

Uh oh!

codecov bot commented Apr 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

kennknowles left a comment

Choose a reason for hiding this comment

Uh oh!

sjvanrossum commented Apr 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kennknowles commented May 2, 2025

Uh oh!

github-actions bot commented May 10, 2025

Uh oh!

github-actions bot commented May 14, 2025

Uh oh!

github-actions bot commented May 23, 2025

Uh oh!

robertwb May 23, 2025

Choose a reason for hiding this comment

Uh oh!

stankiewicz commented May 23, 2025

Uh oh!

github-actions bot commented May 28, 2025

Uh oh!

github-actions bot commented Jun 4, 2025

Uh oh!

stankiewicz commented Jun 4, 2025

Uh oh!

github-actions bot commented Jun 4, 2025

Uh oh!

stankiewicz commented Jun 4, 2025

Uh oh!

Uh oh!

codecov bot commented Apr 29, 2025 •

edited

Loading

sjvanrossum commented Apr 30, 2025 •

edited

Loading