Attempt prism for pipelines with unbounded PCollections. #34721
Conversation
Also fix error with terminal state checking.
c70404a to
f0f7a42
Compare
|
Checks are failing. Will not request review until checks are succeeding. If you'd like to override that behavior, comment |
640e8df to
d3a6dad
Compare
|
An attempt at fixing #34768 Note that several tests use periodic impulse which does work with the Python in-process runner as long as the number of impulses is finite. These tests also break on PrismRunner. |
d3a6dad to
fb543a4
Compare
|
cc: @shunping |
|
Assigning reviewers. If you would like to opt out of this review, comment R: @liferoad for label python. Available commands:
The PR bot will only process comments in the main thread (not review comments). |
| ''' | ||
| self.start_ts = start_timestamp | ||
| self.stop_ts = stop_timestamp | ||
| self.interval = fire_interval | ||
| self.apply_windowing = apply_windowing | ||
| self.is_bounded = ( | ||
| stop_timestamp - start_timestamp > 60 |
There was a problem hiding this comment.
From the code comment above, I think it should be "< 60" since it is considered as bounded for a small time range.
There was a problem hiding this comment.
You're right, I accidentally flipped the direction in my refactor. Fixed.
fb543a4 to
9cc1da4
Compare
|
Is it ready for re-review or still under revision? (just checking on 7d < last update < 14d PRs) |
|
Thanks for following up. This PR is plagued by the fact that it pushes more tests onto the Prism runner that used to run (just fine) on the old Python runner, but break on the Prism runner. Simply fixing the runner likely won't work well with the use of TestPipeline() to run these tests on a variety of runners. I haven't had time to figure out the best way to resolve this. |
|
FYI, I'm seeing similar things in #34612 - I just found your pr here, though, and this version might be more tractable. My plan right now is to make prism default and scope as many things out as I need to in order to get tests passing, then we can slowly add back capabilities/fix issues as we go. But I'm not sure how doable it will or won't be |
|
Reminder, please take a look at this pr: @liferoad |
|
stop reviewer notifications |
|
Stopping the bot since its not ready for review. FWIW, I am making steady progress on #34612 which may help here (or eventually we may be able to just take that instead) |
|
Stopping reviewer notifications for this pull request: requested by reviewer. If you'd like to restart, comment |
|
+1, happy to defer to #34612 to take care of this. I have some hesitation that unit tests now become much more heavyweight with multiple processes involved, but haven't done any actual measurements here. |
close this for now. |
Also fix error with terminal state checking.
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, commentfixes #<ISSUE NUMBER>instead.CHANGES.mdwith noteworthy changes.See the Contributor Guide for more tips on how to make review process smoother.
To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md
GitHub Actions Tests Status (on master branch)
See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.