Fix: align vllm bench serve ignore_eos behavior with legacy benchmark… #28780
Conversation
|
👋 Hi! Thank you for contributing to the vLLM project. 💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels. Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run You ask your reviewers to trigger select CI tests on top of Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging. To run CI, PR reviewers can either: Add If you have any questions, please reach out to us on Slack at https://slack.vllm.ai. 🚀 |
There was a problem hiding this comment.
Code Review
This pull request adds the sharegpt dataset to the condition that sets ignore_eos=True for certain benchmarks. This change aligns its behavior with that of random datasets, ensuring that generation runs to the full requested length for consistent throughput measurement. While the code change is correct, the comment explaining this block is now outdated as it only refers to 'random datasets'. I've added a comment to suggest an update to the comment to improve code clarity and prevent future confusion.
| # so generation runs to the requested length | ||
| if ( | ||
| args.dataset_name in ("random", "random-mm") | ||
| args.dataset_name in ("random", "random-mm", "sharegpt") |
There was a problem hiding this comment.
By adding sharegpt here, the comment on lines 1362-1363 has become inaccurate as it only mentions 'random datasets'. To prevent future confusion, please update the comment to clarify that this logic also applies to the sharegpt dataset. For example:
# For random and sharegpt datasets, default to ignoring EOS
# so generation runs to the requested length|
Thanks for the review and the pointer — appreciated! I've updated the comment in vllm/benchmarks/serve.py to explicitly mention both random and sharegpt datasets so the intent is clear and future confusion is avoided. Change included as a small docs commit:
Please let me know if you'd like the comment text tweaked further. If everything looks good, reviewers can trigger the full CI (or add the ready label) for a comprehensive run. Thanks! |
…vllm-project#24684) Signed-off-by: Amitjoiya <[email protected]>
…m-project#24684) Signed-off-by: Amitjoiya <[email protected]>
e2857c3 to
ace86e4
Compare
… (#24684)
Purpose
Test Plan
Test Result
Essential Elements of an Effective PR Description Checklist
supported_models.mdandexamplesfor a new model.