python: configure workers and hosts for tests #5474
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
mihaibudiu
approved these changes
Jan 20, 2026
blp
approved these changes
Jan 21, 2026
Member
blp
left a comment
There was a problem hiding this comment.
Seems reasonable to me.
I didn't look over all the details.
Contributor
Author
|
@blp The intent for this PR is primarily to help you in testing. When I ran it, several seemed to stall indefinitely, so I was unable to really paste an error. |
7ce5745 to
85ebb62
Compare
f791755 to
d689cf2
Compare
Contributor
There was a problem hiding this comment.
Pull request overview
This PR configures Python tests to use explicit worker and host counts via environment variables FELDERA_TEST_NUM_WORKERS and FELDERA_TEST_NUM_HOSTS, defaulting to 8 and 1 respectively. This ensures tests explicitly control these runtime parameters rather than relying on backend defaults.
Changes:
- Added environment variable support for configuring test worker and host counts
- Updated all test files to pass these values in RuntimeConfig
- Modified the RuntimeConfig class to accept a hosts parameter
Reviewed changes
Copilot reviewed 17 out of 17 changed files in this pull request and generated no comments.
Show a summary per file
| File | Description |
|---|---|
| python/feldera/testutils.py | Defines environment variables for test worker and host configuration |
| python/feldera/runtime_config.py | Adds hosts parameter to RuntimeConfig constructor |
| python/tests/shared_test_pipeline.py | Updates test base classes to use configured workers and hosts |
| python/tests/runtime_aggtest/aggtst_base.py | Adds worker and host configuration to aggregation tests |
| python/tests/runtime/test_udf.py | Adds worker and host configuration to UDF tests |
| python/tests/runtime/test_uda.py | Adds worker and host configuration to UDA tests |
| python/tests/runtime/test_transactions.py | Adds worker and host configuration to transaction tests |
| python/tests/runtime/lateness/test_issue4895.py | Adds worker and host configuration to lateness tests |
| python/tests/runtime/lateness/test_issue4457.py | Adds worker and host configuration to lateness tests |
| python/tests/platform/test_update_runtime.py | Adds worker and host configuration to runtime update tests |
| python/tests/platform/test_shared_pipeline.py | Adds worker and host configuration to shared pipeline tests |
| python/tests/platform/test_pipeline_builder.py | Adds worker and host configuration to pipeline builder tests |
| python/tests/platform/test_nowstream.py | Adds worker and host configuration to nowstream tests |
| python/tests/platform/test_checkpoint_sync.py | Adds worker and host configuration to checkpoint sync tests |
| python/tests/platform/test_bootstrapping.py | Adds worker and host configuration to bootstrapping tests |
| python/tests/platform/negative_test.py | Adds worker and host configuration to negative compilation tests |
| python/tests/platform/helper.py | Adds worker and host configuration to helper functions |
The number of workers and hosts can be set for several Python tests through the environment variables `FELDERA_TEST_NUM_WORKERS` and `FELDERA_TEST_NUM_HOSTS`. The only difference this PR results in, is that the tests no longer use the default number of workers or hosts that is set by the backend if no workers or hosts are specified. The environment variables are set by default to these current values (8 and 1 respectively), but if these defaults are ever changed, they also need to be updated in the Python tests. Signed-off-by: Simon Kassing <[email protected]>
d689cf2 to
9c94225
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The number of workers and hosts can be set for several Python tests through the environment variables
FELDERA_TEST_NUM_WORKERSandFELDERA_TEST_NUM_HOSTS.The only difference this PR results in, is that the tests no longer use the default number of workers or hosts that is set by the backend if no workers or hosts are specified. The environment variables are set by default to these current values (8 and 1 respectively), but if these defaults are ever changed, they also need to be updated in the Python tests.
Commands to try: