Thanks to visit codestin.com
Credit goes to github.com

Skip to content
This repository was archived by the owner on Jun 3, 2025. It is now read-only.

StagedTimer + TimerManager Impl + Pipeline.__call__ integration#1062

Merged
markurtz merged 12 commits into
mainfrom
pipeline-timing
Jun 23, 2023
Merged

StagedTimer + TimerManager Impl + Pipeline.__call__ integration#1062
markurtz merged 12 commits into
mainfrom
pipeline-timing

Conversation

@markurtz

@markurtz markurtz commented Jun 9, 2023 •
edited by bfineran
Loading

Copy link
Copy Markdown
Member

adds StagedTimer and TimerManager classes to enable easier benchmarking of pipelines and eventually the end to end deepsparse server flow. See docstring and Pipeline.__call__ changes for intended workflow.

example usage:
code:

from deepsparse import Pipeline
from deepsparse.loggers import PythonLogger

pipeline = Pipeline.create("text-classification", engine_type="onnxruntime", logger=PythonLogger())
pipeline("test")
print(pipeline.timer_manager)

output (after data logs):

21/06/2023 15:47:53:271368 Identifier: text_classification/prediction_latency/total_inference_seconds | Category: system | Logged Data: 0.05413583107292652 | Additional Info: {'pipeline_name': 'text_classification'}
21/06/2023 15:47:53:271383 Identifier: text_classification/prediction_latency/pre_process_seconds | Category: system | Logged Data: 0.0015975041314959526 | Additional Info: {'pipeline_name': 'text_classification'}
21/06/2023 15:47:53:271396 Identifier: text_classification/prediction_latency/engine_forward_seconds | Category: system | Logged Data: 0.0512517262250185 | Additional Info: {'pipeline_name': 'text_classification'}
21/06/2023 15:47:53:271408 Identifier: text_classification/prediction_latency/post_process_seconds | Category: system | Logged Data: 0.00011516967788338661 | Additional Info: {'pipeline_name': 'text_classification'}
TimerManager({'engine_forward': 0.0512517262250185, 'post_process': 0.00011516967788338661, 'pre_process': 0.0015975041314959526, 'total_inference': 0.05413583107292652})

@markurtz markurtz requested a review from bfineran June 9, 2023 11:58
@markurtz markurtz self-assigned this Jun 9, 2023
bfineran
bfineran previously approved these changes Jun 9, 2023
View reviewed changes

@bfineran bfineran left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM overall to initially land- two things we need to think through more:

  • no clear design for nested pipelines as well as potentially timing bucketing
  • eventually need to handle race conditions once we integrate into the server (not worried about server timings since we can call directly into pipeline.inference_timer from the fast api route)

@bfineran bfineran requested a review from dbogunowicz June 9, 2023 20:30
dbogunowicz
dbogunowicz reviewed Jun 11, 2023
View reviewed changes
Comment thread src/deepsparse/utils/timer.py Outdated
dbogunowicz
dbogunowicz reviewed Jun 11, 2023
View reviewed changes
Comment thread src/deepsparse/pipeline.py
dbogunowicz
dbogunowicz reviewed Jun 11, 2023
View reviewed changes
Comment thread src/deepsparse/pipeline.py Outdated
dbogunowicz
dbogunowicz reviewed Jun 11, 2023
View reviewed changes
Comment thread src/deepsparse/utils/timer.py
@dbogunowicz

dbogunowicz commented Jun 11, 2023

Copy link
Copy Markdown
Contributor

@bfineran didn't we want to make a timer inherit from the Logger class? I am a bit confused with the current design.

@bfineran

bfineran commented Jun 21, 2023

Copy link
Copy Markdown
Contributor

@bfineran didn't we want to make a timer inherit from the Logger class? I am a bit confused with the current design.

both work as long as we can split out the timer in scenarios where logger should also be decoupled such as the server. this meets that requirement.

bfineran
bfineran previously approved these changes Jun 21, 2023
View reviewed changes
bfineran
bfineran previously approved these changes Jun 21, 2023
View reviewed changes
@bfineran bfineran changed the title Standardize pipeline timing to enable pipeline benchmarking StagedTimer + TimerManager Impl + Pipeline __call__ integration Jun 21, 2023
@bfineran bfineran changed the title StagedTimer + TimerManager Impl + Pipeline __call__ integration StagedTimer + TimerManager Impl + Pipeline.__call__ integration Jun 21, 2023
@markurtz markurtz mentioned this pull request Jun 22, 2023
Closed
KSGulin
KSGulin approved these changes Jun 23, 2023
View reviewed changes

@KSGulin KSGulin left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@markurtz markurtz merged commit 79c4ada into main Jun 23, 2023
@markurtz markurtz deleted the pipeline-timing branch June 23, 2023 18:26
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Reviewers

3 more reviewers

@dbogunowicz dbogunowicz dbogunowicz left review comments

@KSGulin KSGulin KSGulin approved these changes

@bfineran bfineran bfineran left review comments

Reviewers whose approvals may not affect merge requirements

Assignees

@markurtz markurtz

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@markurtz @dbogunowicz @bfineran @KSGulin