Drain Mode as WindowedValue extension #34764

stankiewicz · 2025-04-28T15:30:13Z

Changes:

new capability to runner - extended element metadata, set capability for java sdk
new proto definition for element metadata
add coder changes for windowedValue to detect and parse element metadata and vice versa, to encode them if we have them.
add changes to windowedValue to expose elementMetadata object (autovalue, not proto)

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
Update CHANGES.md with noteworthy changes.
If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.

stankiewicz · 2025-04-28T15:31:28Z

@kennknowles ptal.

github-actions · 2025-04-28T17:06:19Z

Assigning reviewers. If you would like to opt out of this review, comment assign to next reviewer:

R: @robertwb for label java.

Available commands:

stop reviewer notifications - opt out of the automated review tooling
remind me after tests pass - tag the comment author after tests pass
waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).

sjvanrossum · 2025-04-29T09:54:54Z

sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/PaneInfo.java

@@ -360,7 +387,8 @@ public PaneInfo decode(final InputStream inStream) throws CoderException, IOExce
      byte keyAndTag = (byte) inStream.read();
      PaneInfo base = Preconditions.checkNotNull(BYTE_TO_PANE_INFO.get((byte) (keyAndTag & 0x0F)));
      long index, onTimeIndex;
-      switch (Encoding.fromTag(keyAndTag)) {
+      boolean elementMetadata = (keyAndTag & 0x80) > 0;


Suggested change

boolean elementMetadata = (keyAndTag & 0x80) > 0;

boolean elementMetadata = keyAndTag < 0;

sjvanrossum · 2025-04-29T10:04:37Z

sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/PaneInfo.java

+      boolean elementMetadata = (keyAndTag & 0x80) > 0;
+      switch (Encoding.fromTag((byte) (keyAndTag & 0x7F))) {


Feel free to ignore my codegolfing...

Suggested change

boolean elementMetadata = (keyAndTag & 0x80) > 0;

switch (Encoding.fromTag((byte) (keyAndTag & 0x7F))) {

boolean elementMetadata = keyAndTag < (keyAndTag &= 0x7f);

switch (Encoding.fromTag(keyAndTag) {

codecov · 2025-04-29T13:38:05Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 56.38%. Comparing base (3b00517) to head (7e3e75a).
Report is 247 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff            @@
##             master   #34764   +/-   ##
=========================================
  Coverage     56.38%   56.38%           
- Complexity     3285     3288    +3     
=========================================
  Files          1174     1174           
  Lines        179401   179407    +6     
  Branches       3398     3399    +1     
=========================================
+ Hits         101158   101164    +6     
- Misses        74983    74984    +1     
+ Partials       3260     3259    -1

Flag	Coverage Δ
java	`70.50% <ø> (+<0.01%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

kennknowles

Largely looks good to me. I need to think about the protocol maybe slightly more. We can probably do it incrementally and get started with changes that are benign and not used yet.

One random cleanup issue (which I hit also since I'm doing similar things with WindowedValue but for different extensions) is I wonder if we can flatten all the silly inheritance in the layers of WindowedValue. For performance ultimately all that matters is whether or not we optimize the coder (such as for batch use case with no timestamps or windows we don't bother encoding min timestamp and global window). And for coding it is much better not to have such implementation inheritance that causes all this toil. Plus the anti-pattern of "superclass that just holds fields"

sjvanrossum · 2025-04-30T10:48:07Z

@kennknowles the default object layout on recent JVMs (64 bit, 8 bytes alignment, 12 bytes object header, compressed references and class pointers) leaves 4 bytes of padding in SimpleWindowedValue:

org.apache.beam.sdk.util.WindowedValue$SimpleWindowedValue object internals:
OFF  SZ                                                TYPE DESCRIPTION                 VALUE
  0   8                                                     (object header: mark)       N/A
  8   4                                                     (object header: class)      N/A
 12   4                                    java.lang.Object SimpleWindowedValue.value   N/A
 16   4   org.apache.beam.sdk.transforms.windowing.PaneInfo SimpleWindowedValue.pane    N/A
 20   4                                                     (object alignment gap)
Instance size: 24 bytes
Space losses: 0 bytes internal + 4 bytes external = 4 bytes total

MinTimestampWindowedValue and TimestampedValueInGlobalWindow also have the same size, but TimestampedValueInGlobalWindow packs the timestamp field in the alignment gap.

org.apache.beam.sdk.util.WindowedValue$MinTimestampWindowedValue object internals:
OFF  SZ                                                TYPE DESCRIPTION                 VALUE
  0   8                                                     (object header: mark)       N/A
  8   4                                                     (object header: class)      N/A
 12   4                                    java.lang.Object SimpleWindowedValue.value   N/A
 16   4   org.apache.beam.sdk.transforms.windowing.PaneInfo SimpleWindowedValue.pane    N/A
 20   4                                                     (object alignment gap)
Instance size: 24 bytes
Space losses: 0 bytes internal + 4 bytes external = 4 bytes total

org.apache.beam.sdk.util.WindowedValue$TimestampedValueInGlobalWindow object internals:
OFF  SZ                                                TYPE DESCRIPTION                          VALUE
  0   8                                                     (object header: mark)                N/A
  8   4                                                     (object header: class)               N/A
 12   4                                    java.lang.Object SimpleWindowedValue.value            N/A
 16   4   org.apache.beam.sdk.transforms.windowing.PaneInfo SimpleWindowedValue.pane             N/A
 20   4                               org.joda.time.Instant TimestampedWindowedValue.timestamp   N/A
Instance size: 24 bytes
Space losses: 0 bytes internal + 0 bytes external = 0 bytes total

And for TimestampedValueInSingleWindow and TimestampedValueInMultipleWindows it spills over into the next word (timestamp fills the alignment gap, window/windows spills into the next word):

org.apache.beam.sdk.util.WindowedValue$TimestampedValueInSingleWindow object internals:
OFF  SZ                                                     TYPE DESCRIPTION                             VALUE
  0   8                                                          (object header: mark)                   N/A
  8   4                                                          (object header: class)                  N/A
 12   4                                         java.lang.Object SimpleWindowedValue.value               N/A
 16   4        org.apache.beam.sdk.transforms.windowing.PaneInfo SimpleWindowedValue.pane                N/A
 20   4                                    org.joda.time.Instant TimestampedWindowedValue.timestamp      N/A
 24   4   org.apache.beam.sdk.transforms.windowing.BoundedWindow TimestampedValueInSingleWindow.window   N/A
 28   4                                                          (object alignment gap)
Instance size: 32 bytes
Space losses: 0 bytes internal + 4 bytes external = 4 bytes total

org.apache.beam.sdk.util.WindowedValue$TimestampedValueInMultipleWindows object internals:
OFF  SZ                                                TYPE DESCRIPTION                                 VALUE
  0   8                                                     (object header: mark)                       N/A
  8   4                                                     (object header: class)                      N/A
 12   4                                    java.lang.Object SimpleWindowedValue.value                   N/A
 16   4   org.apache.beam.sdk.transforms.windowing.PaneInfo SimpleWindowedValue.pane                    N/A
 20   4                               org.joda.time.Instant TimestampedWindowedValue.timestamp          N/A
 24   4                                java.util.Collection TimestampedValueInMultipleWindows.windows   N/A
 28   4                                                     (object alignment gap)
Instance size: 32 bytes
Space losses: 0 bytes internal + 4 bytes external = 4 bytes total

With default object layout settings the addition of an extensions field would only increase the size of TimestampedValueInGlobalWindow by 8 bytes since it would spill into the next word, all the other subclasses should pack that field into the alignment gap. I'm not sure whether users of Beam commonly disable UseCompressedPointers or UseCompressedOops, but it would seem somewhat unlikely in my opinion since both of them would generally increase memory usage.

Compile time annotation processing to produce flat extensions (without a separate extensions container,
treating timestamp, window and windows as standard extensions) of all subclasses of WindowedValue might help make this a bit more manageable. Something like the mechanism behind PipelineOptions, but at compile time and backed by fields instead of a Map<String, Object>.

kennknowles · 2025-05-02T16:51:02Z

Stating it here so that I remember - we should write a quite 1 pager separate from the DrainMode document about the extended metadata and have a top level thread on dev@. I think more people will be interested if it is something like "Extended per-element metadata proto". It is quite an alarming title that will bring out the performance experts :-)

drain mode as extension

acf46fc

github-actions bot added java model labels Apr 28, 2025

github-actions bot added the Next Action: Reviewers label Apr 28, 2025

add withMetadata

2409fdf

github-actions bot added runners dataflow and removed runners dataflow labels Apr 29, 2025

sjvanrossum reviewed Apr 29, 2025

View reviewed changes

stankiewicz added 2 commits April 29, 2025 12:43

add tests

5473d67

fix ValueInEmptyWindows

932b53a

github-actions bot added runners dataflow and removed runners dataflow labels Apr 29, 2025

style

7e3e75a

github-actions bot added runners dataflow and removed runners dataflow labels Apr 29, 2025

kennknowles reviewed Apr 29, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Drain Mode as WindowedValue extension #34764

Drain Mode as WindowedValue extension #34764

stankiewicz commented Apr 28, 2025

stankiewicz commented Apr 28, 2025

github-actions bot commented Apr 28, 2025

sjvanrossum Apr 29, 2025

sjvanrossum Apr 29, 2025

codecov bot commented Apr 29, 2025

kennknowles left a comment

sjvanrossum commented Apr 30, 2025 •

edited

Loading

kennknowles commented May 2, 2025

	boolean elementMetadata = (keyAndTag & 0x80) > 0;
	boolean elementMetadata = keyAndTag < 0;

		boolean elementMetadata = (keyAndTag & 0x80) > 0;
		switch (Encoding.fromTag((byte) (keyAndTag & 0x7F))) {

Drain Mode as WindowedValue extension #34764

Are you sure you want to change the base?

Drain Mode as WindowedValue extension #34764

Conversation

stankiewicz commented Apr 28, 2025

GitHub Actions Tests Status (on master branch)

stankiewicz commented Apr 28, 2025

github-actions bot commented Apr 28, 2025

sjvanrossum Apr 29, 2025

Choose a reason for hiding this comment

sjvanrossum Apr 29, 2025

Choose a reason for hiding this comment

codecov bot commented Apr 29, 2025

Codecov Report

kennknowles left a comment

Choose a reason for hiding this comment

sjvanrossum commented Apr 30, 2025 • edited Loading

kennknowles commented May 2, 2025

sjvanrossum commented Apr 30, 2025 •

edited

Loading