feat(vllm_performance): add support for benchmarking geospatial models #187
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- Add backend arg to execute benchmark - Add execute_geospatial_benchmark function This uses the geospatial_valencia.jsonl dataset
endpoint and full
Signed-off-by: Christian Pinto <[email protected]>
Signed-off-by: Christian Pinto <[email protected]>
Signed-off-by: Christian Pinto <[email protected]>
Signed-off-by: Christian Pinto <[email protected]>
Signed-off-by: Christian Pinto <[email protected]>
Signed-off-by: Christian Pinto <[email protected]>
Signed-off-by: Christian Pinto <[email protected]>
… template Signed-off-by: Christian Pinto <[email protected]>
Signed-off-by: Christian Pinto <[email protected]>
Signed-off-by: Christian Pinto <[email protected]>
Signed-off-by: Christian Pinto <[email protected]>
Signed-off-by: Christian Pinto <[email protected]>
Signed-off-by: Christian Pinto <[email protected]>
Signed-off-by: Christian Pinto <[email protected]>
Signed-off-by: Christian Pinto <[email protected]>
Signed-off-by: Christian Pinto <[email protected]>
…ix-run-experiment
validate_entity was incorrectly identifying Entities with optional properties as invalid
This was because validate_point_against_properties(allow_partial_matches=True) does not work as code expected.
Code expected that given {point props} and {required props}, if {required props}.issubset(point_props) it would return True. However, it was checking the opposite i.e.{point_probs}.issubset{required_props}
Signed-off-by: Christian Pinto <[email protected]>
Signed-off-by: Christian Pinto <[email protected]>
Signed-off-by: Christian Pinto <[email protected]>
Signed-off-by: Christian Pinto <[email protected]>
Member
|
Checks Summary Last run: 2025-11-07T11:54:47.427Z Code Risk Analyzer vulnerability scan found 2 vulnerabilities:
|
...m_performance/ado_actuators/vllm_performance/experiments/performance_testing_geospatial.yaml
Show resolved
Hide resolved
...m_performance/ado_actuators/vllm_performance/experiments/performance_testing_geospatial.yaml
Outdated
Show resolved
Hide resolved
...m_performance/ado_actuators/vllm_performance/experiments/performance_testing_geospatial.yaml
Show resolved
Hide resolved
...m_performance/ado_actuators/vllm_performance/experiments/performance_testing_geospatial.yaml
Outdated
Show resolved
Hide resolved
...ctuators/vllm_performance/ado_actuators/vllm_performance/k8/yaml_support/build_components.py
Outdated
Show resolved
Hide resolved
...s/vllm_performance/ado_actuators/vllm_performance/vllm_performance_test/execute_benchmark.py
Outdated
Show resolved
Hide resolved
plugins/actuators/vllm_performance/ado_actuators/vllm_performance/actuator.py
Outdated
Show resolved
Hide resolved
plugins/actuators/vllm_performance/ado_actuators/vllm_performance/actuator.py
Outdated
Show resolved
Hide resolved
plugins/actuators/vllm_performance/ado_actuators/vllm_performance/actuator.py
Outdated
Show resolved
Hide resolved
plugins/actuators/vllm_performance/ado_actuators/vllm_performance/actuator.py
Outdated
Show resolved
Hide resolved
Signed-off-by: Christian Pinto <[email protected]>
Signed-off-by: Christian Pinto <[email protected]>
Signed-off-by: Christian Pinto <[email protected]>
Signed-off-by: Christian Pinto <[email protected]>
Member
Author
|
@AlessandroPomponio @michael-johnston you can now have a look at thi PR |
Member
AlessandroPomponio
left a comment
There was a problem hiding this comment.
Mostly nitpicks - a couple of questions
...s/vllm_performance/ado_actuators/vllm_performance/vllm_performance_test/execute_benchmark.py
Outdated
Show resolved
Hide resolved
...s/vllm_performance/ado_actuators/vllm_performance/vllm_performance_test/execute_benchmark.py
Outdated
Show resolved
Hide resolved
...m_performance/ado_actuators/vllm_performance/experiments/performance_testing_geospatial.yaml
Outdated
Show resolved
Hide resolved
...m_performance/ado_actuators/vllm_performance/experiments/performance_testing_geospatial.yaml
Outdated
Show resolved
Hide resolved
...tuators/vllm_performance/ado_actuators/vllm_performance/experiments/performance_testing.yaml
Outdated
Show resolved
Hide resolved
...m_performance/ado_actuators/vllm_performance/experiments/performance_testing_geospatial.yaml
Outdated
Show resolved
Hide resolved
...m_performance/ado_actuators/vllm_performance/experiments/performance_testing_geospatial.yaml
Outdated
Show resolved
Hide resolved
...s/vllm_performance/ado_actuators/vllm_performance/vllm_performance_test/execute_benchmark.py
Outdated
Show resolved
Hide resolved
Signed-off-by: Christian Pinto <[email protected]>
Member
Author
|
@AlessandroPomponio you can have another look at this PR now. Thanks |
...s/vllm_performance/ado_actuators/vllm_performance/vllm_performance_test/execute_benchmark.py
Show resolved
Hide resolved
...s/vllm_performance/ado_actuators/vllm_performance/vllm_performance_test/execute_benchmark.py
Show resolved
Hide resolved
...s/vllm_performance/ado_actuators/vllm_performance/vllm_performance_test/execute_benchmark.py
Show resolved
Hide resolved
Comment on lines
167
to
174
| :param base_url: url for vllm endpoint | ||
| :param model: model | ||
| :param dataset: data set name ["sharegpt", "sonnet", "random", "hf"] | ||
| :param hf_token: huggingface token | ||
| :param benchmark_retries: number of benchmark execution retries | ||
| :param retries_timeout: timeout between initial retry | ||
| :param burstiness: burstiness factor of the request generation, 0 < burstiness < 1 | ||
| :param interpreter: python interpreter to use |
Member
There was a problem hiding this comment.
The docstring lacks:
- num_prompts
- request_rate
- max_concurrency
…t are irrelevant to geospatial tests Signed-off-by: Christian Pinto <[email protected]>
Signed-off-by: Christian Pinto <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR adds:
performance-testing-geospatial-endpoint,performance-testing-geospatial-full,performance-testing-geospatial-full-custom-dataset.randomfor LLM testing andindia_url_in_b64_out,valencia_url_in_b64_outfor geospatial tests.