[python] In platform tests, kill the pipeline after obtaining its out… #5521
Conversation
…put. Killing the pipeline then reading the output makes abrupt termination of the HTTP connection possible, which can cause test failures. This is an experiment to see whether it makes CI more reliable. Signed-off-by: Ben Pfaff <[email protected]>
There was a problem hiding this comment.
Pull request overview
Moves pipeline shutdown in platform tests to occur after consuming output, aiming to avoid abrupt HTTP connection termination and reduce CI flakiness.
Changes:
- Reordered
self.pipeline.stop(...)calls to happen afterout.to_pandas()/out.to_dict()across multiple tests. - Adjusted multi-phase tests (e.g.,
test_pandas_map) to stop the pipeline after verifying the first phase’s output.
| df = out.to_pandas() | ||
| assert df.shape[0] == 100 | ||
| self.pipeline.stop(force=True) |
There was a problem hiding this comment.
With stop() moved after assertions, a failing assert (or any exception in expected-value construction) will skip self.pipeline.stop(...), potentially leaking a running pipeline and causing follow-on test interference/flakiness. Consider ensuring cleanup always runs by wrapping the output/verification portion in a try/finally (or equivalent test cleanup hook), and/or moving self.pipeline.stop(...) to immediately after out.to_pandas()/out.to_dict() (before assertions) so shutdown still happens even when assertions fail.
|
We may need to update the documentation too; you should probably file an issue and assign it to @abhizer |
What kind of documentation update are you envisioning? |
|
The python SDK documentation is bound to have examples like this |
…put.
Killing the pipeline then reading the output makes abrupt termination of the HTTP connection possible, which can cause test failures.
This is an experiment to see whether it makes CI more reliable.