Add resource usage performance tests for creating a span #1499

NathanielRN · 2020-12-21T23:59:09Z

Description

Adds resource usage tests (testing CPU & Memory) according to the performance tests guideline on the specifications.

This PR adds example files that can be run to see where CPU & Memory allocations are mostly made over simple test cases for the OTel SDK.

We don't have an automated solution for reporting on CPU & Memory usage right now, but I added the tests in this PR description just like how the Java SDK does it as a temporary solution.

Type of change

Please delete options that are not relevant.

New feature (non-breaking change which adds functionality)

How Has This Been Tested?

All tests were run on a m5.xlarge EC2 instance:

Install scalene using the following command

pip install -U scalene

2a. Run the SimpleExportSpanProcessor scalene test using

scalene opentelemetry-sdk/tests/performance/resource-usage/trace/profile_resource_usage_simple_export.py

Get the following results:

                                                                                                                                                                                            Memory usage: ▅▆▆▆▆▆▆▇▇▇▇▆▇▇▇████ (max:  25.00MB)
                                                                                                                                                    opentelemetry-sdk/tests/performance/resource-usage/trace/profile_resource_usage_simple_export.py: % of time = 100.00% out of  23.57s.
       ╷       ╷        ╷    ╷       ╷      ╷              ╷       ╷
  Line │Time % │Time %  │Sys │Mem %  │Net   │Memory usage  │Copy   │
       │Python │native  │%   │Python │(MB)  │over time / % │(MB/s) │opentelemetry-sdk/tests/performance/resource-usage/trace/profile_resource_usage_simple_export.py
╺━━━━━━┿━━━━━━━┿━━━━━━━━┿━━━━┿━━━━━━━┿━━━━━━┿━━━━━━━━━━━━━━┿━━━━━━━┿━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸
     1 │       │        │    │       │      │              │       │# Copyright The OpenTelemetry Authors
     2 │       │        │    │       │      │              │       │#
     3 │       │        │    │       │      │              │       │# Licensed under the Apache License, Version 2.0 (the "License");
     4 │       │        │    │       │      │              │       │# you may not use this file except in compliance with the License.
     5 │       │        │    │       │      │              │       │# You may obtain a copy of the License at
     6 │       │        │    │       │      │              │       │#
     7 │       │        │    │       │      │              │       │#     http://www.apache.org/licenses/LICENSE-2.0
     8 │       │        │    │       │      │              │       │#
     9 │       │        │    │       │      │              │       │# Unless required by applicable law or agreed to in writing, software
    10 │       │        │    │       │      │              │       │# distributed under the License is distributed on an "AS IS" BASIS,
    11 │       │        │    │       │      │              │       │# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    12 │       │        │    │       │      │              │       │# See the License for the specific language governing permissions and
    13 │       │        │    │       │      │              │       │# limitations under the License.
    14 │       │        │    │       │      │              │       │
    15 │       │        │    │       │      │              │       │import time
    16 │       │        │    │  100% │   17 │▂▂▂▂▃▂▃▃▃     │       │from unittest.mock import patch
    17 │       │        │    │       │      │              │       │
    18 │       │        │    │   97% │    8 │▄▄▅▅ 62%      │     1 │from opentelemetry.exporter.otlp.trace_exporter import OTLPSpanExporter
    19 │       │        │    │       │      │              │       │from opentelemetry.sdk.trace import TracerProvider, sampling
    20 │       │        │    │       │      │              │       │from opentelemetry.sdk.trace.export import SimpleExportSpanProcessor
    21 │       │        │    │       │      │              │       │
    22 │       │        │    │       │      │              │       │TEST_DURATION_SECONDS = 15
    23 │       │        │    │       │      │              │       │SPANS_PER_SECOND = 10_000
    24 │       │        │    │       │      │              │       │
    25 │       │        │    │       │      │              │       │
    26 │       │        │    │       │      │              │       │class MockTraceServiceStub(object):
    27 │       │        │    │       │      │              │       │    def __init__(self, channel):
    28 │       │        │    │       │      │              │       │        self.Export = lambda *args, **kwargs: None
    29 │       │        │    │       │      │              │       │
    30 │       │        │    │       │      │              │       │
    31 │       │        │    │       │      │              │       │old_stub = OTLPSpanExporter._stub
    32 │       │        │    │       │      │              │       │OTLPSpanExporter._stub = MockTraceServiceStub
    33 │       │        │    │       │      │              │       │
    34 │       │        │    │       │      │              │       │simple_span_processor = SimpleExportSpanProcessor(OTLPSpanExporter())
    35 │       │        │    │       │      │              │       │tracer = TracerProvider(
    36 │       │        │    │       │      │              │       │    active_span_processor=simple_span_processor, sampler=sampling.DEFAULT_ON,
    37 │       │        │    │       │      │              │       │).get_tracer("resource_usage_tracer")
    38 │       │        │    │       │      │              │       │
    39 │       │        │    │       │      │              │       │starttime = time.time()
    40 │       │        │    │       │      │              │       │for _ in range(TEST_DURATION_SECONDS):
    41 │       │        │    │       │      │              │       │    for _ in range(SPANS_PER_SECOND):
    42 │   26% │     1% │    │       │      │              │     3 │        span = tracer.start_span("benchmarkedSpan")
    43 │   71% │     2% │    │       │      │              │    10 │        span.end()
    44 │       │        │    │       │      │              │       │    time_to_finish_spans = time.time() - starttime
    45 │       │        │    │       │      │              │       │    time.sleep(1.0 - time_to_finish_spans if time_to_finish_spans < 1.0 else 0)
    46 │       │        │    │       │      │              │       │
    47 │       │        │    │       │      │              │       │OTLPSpanExporter._stub = old_stub
       ╵       ╵        ╵    ╵       ╵      ╵              ╵       ╵

2b. Run the BatchExportSpanProcessor scalene test using

scalene opentelemetry-sdk/tests/performance/resource-usage/trace/profile_resource_usage_batch_export.py

Get the following results:

                                                                                     Memory usage: ▆▇██▇▇█████ (max:  36.00MB)
                                         opentelemetry-sdk/tests/performance/resource-usage/trace/profile_resource_usage_batch_export.py: % of time = 100.00% out of  13.32s.
       ╷       ╷        ╷     ╷       ╷      ╷              ╷       ╷
  Line │Time % │Time %  │Sys  │Mem %  │Net   │Memory usage  │Copy   │
       │Python │native  │%    │Python │(MB)  │over time / % │(MB/s) │opentelemetry-sdk/tests/performance/resource-usage/trace/profile_resource_usage_batch_export.py
╺━━━━━━┿━━━━━━━┿━━━━━━━━┿━━━━━┿━━━━━━━┿━━━━━━┿━━━━━━━━━━━━━━┿━━━━━━━┿━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸
     1 │       │        │     │       │      │              │       │# Copyright The OpenTelemetry Authors
     2 │       │        │     │       │      │              │       │#
     3 │       │        │     │       │      │              │       │# Licensed under the Apache License, Version 2.0 (the "License");
     4 │       │        │     │       │      │              │       │# you may not use this file except in compliance with the License.
     5 │       │        │     │       │      │              │       │# You may obtain a copy of the License at
     6 │       │        │     │       │      │              │       │#
     7 │       │        │     │       │      │              │       │#     http://www.apache.org/licenses/LICENSE-2.0
     8 │       │        │     │       │      │              │       │#
     9 │       │        │     │       │      │              │       │# Unless required by applicable law or agreed to in writing, software
    10 │       │        │     │       │      │              │       │# distributed under the License is distributed on an "AS IS" BASIS,
    11 │       │        │     │       │      │              │       │# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    12 │       │        │     │       │      │              │       │# See the License for the specific language governing permissions and
    13 │       │        │     │       │      │              │       │# limitations under the License.
    14 │       │        │     │       │      │              │       │
    15 │       │        │     │       │      │              │       │import time
    16 │       │        │     │   98% │   16 │▁▁▁▁▁▁        │       │from unittest.mock import patch
    17 │       │        │     │       │      │              │       │
    18 │    1% │        │     │  100% │   17 │▆▆▆▆▆ 74%     │     2 │from opentelemetry.exporter.otlp.trace_exporter import OTLPSpanExporter
    19 │       │        │     │       │      │              │       │from opentelemetry.sdk.trace import TracerProvider, sampling
    20 │       │        │     │       │      │              │       │from opentelemetry.sdk.trace.export import BatchExportSpanProcessor
    21 │       │        │     │       │      │              │       │
    22 │       │        │     │       │      │              │       │TEST_DURATION_SECONDS = 15
    23 │       │        │     │       │      │              │       │SPANS_PER_SECOND = 10_000
    24 │       │        │     │       │      │              │       │
    25 │       │        │     │       │      │              │       │
    26 │       │        │     │       │      │              │       │class MockTraceServiceStub(object):
    27 │       │        │     │       │      │              │       │    def __init__(self, channel):
    28 │       │        │     │       │      │              │       │        self.Export = lambda *args, **kwargs: None
    29 │       │        │     │       │      │              │       │
    30 │       │        │     │       │      │              │       │
    31 │       │        │     │       │      │              │       │old_stub = OTLPSpanExporter._stub
    32 │       │        │     │       │      │              │       │OTLPSpanExporter._stub = MockTraceServiceStub
    33 │       │        │     │       │      │              │       │
    34 │       │        │     │       │      │              │       │simple_span_processor = BatchExportSpanProcessor(OTLPSpanExporter())
    35 │       │        │     │       │      │              │       │tracer = TracerProvider(
    36 │       │        │     │       │      │              │       │    active_span_processor=simple_span_processor, sampler=sampling.DEFAULT_ON,
    37 │       │        │     │       │      │              │       │).get_tracer("resource_usage_tracer")
    38 │       │        │     │       │      │              │       │
    39 │       │        │     │       │      │              │       │starttime = time.time()
    40 │       │        │     │       │      │              │       │for _ in range(TEST_DURATION_SECONDS):
    41 │       │        │     │       │      │              │       │    for _ in range(SPANS_PER_SECOND):
    42 │   72% │    15% │  1% │       │      │              │    15 │        span = tracer.start_span("benchmarkedSpan")
    43 │    9% │     2% │     │       │      │              │     1 │        span.end()
    44 │       │        │     │       │      │              │       │    time_to_finish_spans = time.time() - starttime
    45 │    1% │        │     │       │      │              │       │    time.sleep(1.0 - time_to_finish_spans if time_to_finish_spans < 1.0 else 0)
    46 │       │        │     │       │      │              │       │
    47 │       │        │     │       │      │              │       │OTLPSpanExporter._stub = old_stub
       ╵       ╵        ╵     ╵       ╵      ╵              ╵       ╵

Does This PR Require a Contrib Repo Change?

No.

Checklist:

Followed the style guidelines of this project
~~- [ ] Changelogs have been updated~~
~~- [ ] Unit tests have been added~~
~~- [ ] Documentation has been updated~~

...telemetry-sdk/tests/performance/resource-usage/trace/profile_resource_usage_simple_export.py

lzchen · 2021-01-04T18:06:41Z

Should we include a README on how to run these tests like you outlined in the description?

lzchen · 2021-01-04T18:07:01Z

All tests were run on a m5.xlarge EC2 instance:

Is this to fulfill the "out-of-process" requirement in the specs?

lzchen · 2021-01-04T18:07:36Z

Can you create an issue for finding a way to report these perf test results?

NathanielRN · 2021-01-11T05:29:15Z

@lzchen Created #1517 to track finding a way to report these performance tests.

NathanielRN · 2021-01-11T05:31:43Z

Is this to fulfill the "out-of-process" requirement in the specs?

No nothing to do with that. In fact we chose to not include the "receiver" at all and just made it a no-op right now for simplicity sake.

Using a m5.xlarge EC2 instance is just to be transparent about where we got these results which should hopefully make it easy to reproduce these results. i.e. if you spin up this type of instance you should get the same results.

NathanielRN · 2021-01-11T05:40:33Z

Should we include a README on how to run these tests like you outlined in the description?

I updated the README for opentelemetry-sdk with generic instructions from this PR!

lzchen · 2021-01-12T22:06:05Z

opentelemetry-sdk/README.rst

@@ -13,6 +13,23 @@ Installation

    pip install opentelemetry-sdk

+Running Performance Tests


Curious does this belong in the sdk README? I feel like it would be useful in the top level.

Also, unrelated but should be in the same place as a "HOW TO" for writing benchmark tests right?

Yes I agree I think it would be better at the top level! Since running the tests is different from developing tests, I can move it there.

Also, unrelated but should be in the same place as a "HOW TO" for writing benchmark tests right?

Hm, I guess we could write a "How to write Resource Utilization Tests" section like we did for benchmarks here. However, since it doesn't require a specific format like the benchmarks did (because you can just profile any Python program, I just provided examples here) I can't immediately think of any meaningful guidelines to provide.

codeboten

Nice! Thanks

NathanielRN force-pushed the add-resource-usage-tests branch from 4529423 to 9a7efe3 Compare December 22, 2020 02:05

NathanielRN marked this pull request as ready for review December 22, 2020 22:04

NathanielRN requested review from a team, owais and lzchen and removed request for a team December 22, 2020 22:04

codeboten added the Skip Changelog PRs that do not require a CHANGELOG.md entry label Dec 23, 2020

lzchen reviewed Jan 4, 2021

View reviewed changes

...telemetry-sdk/tests/performance/resource-usage/trace/profile_resource_usage_simple_export.py Show resolved Hide resolved

NathanielRN mentioned this pull request Jan 11, 2021

How to Report Resource Utilization Measurements (+ Automation) #1517

Open

NathanielRN force-pushed the add-resource-usage-tests branch from bbfcc4a to e888a36 Compare January 11, 2021 05:39

NathanielRN requested a review from lzchen January 11, 2021 05:40

lzchen reviewed Jan 12, 2021

View reviewed changes

lzchen approved these changes Jan 12, 2021

View reviewed changes

NathanielRN added 6 commits January 12, 2021 14:19

Add resource usage tests for creating a span

6c0afe2

Patch before we use the OTLPSpanExporter

dcd3325

Add batch export tests

3e82e98

Rename files to not get included in pytest

9a386c3

Explain how to run resource usage tests in readme

c94a790

Explain how to run resource utilization tests

6f17cc2

NathanielRN force-pushed the add-resource-usage-tests branch from e888a36 to 6f17cc2 Compare January 12, 2021 22:24

Merge branch 'master' into add-resource-usage-tests

975611a

codeboten approved these changes Jan 25, 2021

View reviewed changes

alrex added 2 commits January 25, 2021 15:31

Merge branch 'master' into add-resource-usage-tests

4fa0e02

Merge branch 'master' into add-resource-usage-tests

6512514

Merge branch 'master' into add-resource-usage-tests

35028b6

Base automatically changed from master to main January 29, 2021 16:49

alrex added 2 commits January 29, 2021 11:51

Merge branch 'main' into add-resource-usage-tests

6283fcb

Merge branch 'main' into add-resource-usage-tests

2e63753

lzchen merged commit 5bc23b0 into open-telemetry:main Jan 31, 2021

NathanielRN deleted the add-resource-usage-tests branch August 13, 2021 18:42

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add resource usage performance tests for creating a span #1499

Add resource usage performance tests for creating a span #1499

Uh oh!

NathanielRN commented Dec 21, 2020 •

edited

Loading

Uh oh!

Uh oh!

lzchen commented Jan 4, 2021

Uh oh!

lzchen commented Jan 4, 2021

Uh oh!

lzchen commented Jan 4, 2021

Uh oh!

NathanielRN commented Jan 11, 2021

Uh oh!

NathanielRN commented Jan 11, 2021

Uh oh!

NathanielRN commented Jan 11, 2021

Uh oh!

lzchen Jan 12, 2021

Uh oh!

NathanielRN Jan 12, 2021

Uh oh!

codeboten left a comment

Uh oh!

Uh oh!

		@@ -13,6 +13,23 @@ Installation

		pip install opentelemetry-sdk

		Running Performance Tests

Add resource usage performance tests for creating a span #1499

Add resource usage performance tests for creating a span #1499

Uh oh!

Conversation

NathanielRN commented Dec 21, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Type of change

How Has This Been Tested?

Does This PR Require a Contrib Repo Change?

Checklist:

Uh oh!

Uh oh!

lzchen commented Jan 4, 2021

Uh oh!

lzchen commented Jan 4, 2021

Uh oh!

lzchen commented Jan 4, 2021

Uh oh!

NathanielRN commented Jan 11, 2021

Uh oh!

NathanielRN commented Jan 11, 2021

Uh oh!

NathanielRN commented Jan 11, 2021

Uh oh!

lzchen Jan 12, 2021

Choose a reason for hiding this comment

Uh oh!

NathanielRN Jan 12, 2021

Choose a reason for hiding this comment

Uh oh!

codeboten left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

NathanielRN commented Dec 21, 2020 •

edited

Loading