Add warm-ups for RTFX calculation in streaming ASR pipelines#15313
Conversation
Signed-off-by: naymaraq <[email protected]>
There was a problem hiding this comment.
Pull request overview
This pull request enhances the streaming ASR inference pipeline by adding warm-up runs before RTFx (Real-Time Factor) calculations and preserving the original file path order in output. The changes improve the reliability of performance measurements by allowing the system to stabilize before taking measurements.
Changes:
- Renamed
get_audio_filepaths()toprepare_audio_data()with extended return values including ASR request options and filepath ordering - Added warmup and run step loop to perform multiple inference runs and average the timing measurements
- Added filepath order preservation to maintain original input ordering in outputs
Reviewed changes
Copilot reviewed 6 out of 6 changed files in this pull request and generated 8 comments.
Show a summary per file
| File | Description |
|---|---|
| nemo/collections/asr/inference/utils/manifest_io.py | Renamed function to prepare_audio_data(), added options extraction from manifest and filepath order tracking |
| examples/asr/asr_streaming_inference/asr_streaming_infer.py | Implemented warmup/run loop for RTFx measurements, integrated with new prepare_audio_data function |
| examples/asr/conf/asr_streaming_inference/cache_aware_rnnt.yaml | Added warmup_steps and run_steps configuration parameters |
| examples/asr/conf/asr_streaming_inference/cache_aware_ctc.yaml | Added warmup_steps and run_steps configuration parameters |
| examples/asr/conf/asr_streaming_inference/buffered_rnnt.yaml | Added warmup_steps and run_steps configuration parameters |
| examples/asr/conf/asr_streaming_inference/buffered_ctc.yaml | Added warmup_steps and run_steps configuration parameters |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| def prepare_audio_data( | ||
| audio_file: str, sort_by_duration: bool = True | ||
| ) -> tuple[list[str], list[dict] | None, list[ASRRequestOptions] | None, dict[str, int]]: |
There was a problem hiding this comment.
The function get_audio_filepaths has been renamed to prepare_audio_data, but the file tests/functional_tests/asr_transcribe_boost_ground_truth.py (line 20 and line 52) still imports and uses the old function name. This will cause an ImportError. Either the old function should be kept as an alias for backward compatibility, or all usages should be updated.
Signed-off-by: naymaraq <[email protected]>
Signed-off-by: naymaraq <[email protected]>
|
[🤖]: Hi @naymaraq 👋, We wanted to let you know that a CICD pipeline for this PR just finished successfully. So it might be time to merge this PR or get some approvals. |
…NeMo#15313) * keep filepath order, add warmups for rtfx calculation Signed-off-by: naymaraq <[email protected]> * fix typos, minor fixes Signed-off-by: naymaraq <[email protected]> * fix broken tests Signed-off-by: naymaraq <[email protected]> --------- Signed-off-by: naymaraq <[email protected]> Co-authored-by: naymaraq <[email protected]>
Important
The
Update branchbutton must only be pressed in very rare occassions.An outdated branch is never blocking the merge of a PR.
Please reach out to the automation team before pressing that button.
What does this PR do ?
Collection: [ASR]
Impact of Warmup
Dataset - LS-Other
GPU - NVIDIA RTX 5000
Batch Size - 256
Changelog
Usage
# Add a code snippet demonstrating how to use thisGitHub Actions CI
The Jenkins CI system has been replaced by GitHub Actions self-hosted runners.
The GitHub Actions CI will run automatically when the "Run CICD" label is added to the PR.
To re-run CI remove and add the label again.
To run CI on an untrusted fork, a NeMo user with write access must first click "Approve and run".
Before your PR is "Ready for review"
Pre checks:
PR Type:
If you haven't finished some of the above items you can still open "Draft" PR.
Who can review?
Anyone in the community is free to review the PR once the checks have passed.
Contributor guidelines contains specific people who can review PRs to various areas.
Additional Information