Adding a script for benchmarking pipelines, which reports the amount of compute time spent in each phase of the pipeline. This allows us to identify bottlenecks outside the engine in pre or post processing.

Example Usage

deepsparse.benchmark_pipeline "text_classification" "zoo:nlp/sentiment_analysis/distilbert-none/pytorch/huggingface/sst2/pruned90-none" -c "config.json"

Based on the pipeline argument, the script will infer what type of data to generate or search for (See get_input_schema_type for details)

Config File Documentation

Configurations for generating or loading data to the pipeline are specified as JSON, documented in the README. A quick summary:

• Set data_type to "real" to pull text or image data from data_folder, or set it to "dummy" to use randomly generated data. * In dummy mode, string lengths are set with gen_sequence_length and image shapes are set by input_image_shape
• In real mode, max_string_length will truncate input text if >0, set to -1 for no truncation
• data_folder is a path to either images(.jpg, .jpeg, .gif) or text(.txt) files, to be read in real mode
• set 'recursive_searchto true to recursively searchdata_folder`
• additional keyword arguments to pipeline.Pipeline() can be added to pipeline_kwargs
• additional keyword arguments to Pipeline.input_schema() can be added to input_schema_kwargs

Example:

{
    "data_type": "dummy",
    "gen_sequence_length": 100,
    "input_image_shape": [500,500,3],
    "data_folder": "/home/sadkins/imagenette2-320/",
    "recursive_search": true,
    "max_string_length": -1, 
    "pipeline_kwargs": {},
    "input_schema_kwargs": {}
} 

Testing

Added unit tests to test_pipeline_benchmark.py, and also manually tested the following pipelines:

• text_classification: deepsparse.benchmark_pipeline text_classification zoo:nlp/sentiment_analysis/distilbert-none/pytorch/huggingface/sst2/pruned90-none -c tests/test_data/pipeline_bench_config.json
• image_classification: deepsparse.benchmark_pipeline image_classification zoo:cv/classification/resnet_v1-50_2x/pytorch/sparseml/imagenet/base-none -c tests/test_data/pipeline_bench_config.json
• text_generation: deepsparse.benchmark_pipeline text_generation zoo:nlg/text_generation/codegen_mono-350m/pytorch/huggingface/bigpython_bigquery_thepile/base_quant-none -c tests/test_data/pipeline_bench_config.json
• yolo: deepsparse.benchmark_pipeline yolo zoo:cv/detection/yolov5-l/pytorch/ultralytics/coco/pruned_quant-aggressive_95-c tests/test_data/pipeline_bench_config.json
• question_answering: deepsparse.benchmark_pipeline question_answering zoo:nlp/question_answering/bert-base/pytorch/huggingface/squad/12layer_pruned80_quant-none-vnni -c tests/test_data/pipeline_bench_config.json
• token_generation: deepsparse.benchmark_pipeline token_classification zoo:nlp/token_classification/distilbert-none/pytorch/huggingface/conll2003/pruned90-none -c tests/test_data/pipeline_bench_config.json