[Perf -10%] System.Text.RegularExpressions.Tests.Perf_Regex_Common (2) #40094

DrewScoggins · 2020-07-29T17:57:05Z

Run Information

Architecture	x64
OS	Windows 10.0.18362
Changes	diff

Regressions in System.Text.RegularExpressions.Tests.Perf_Regex_Common

Benchmark	Baseline	Test	Test/Base	Modality	Baseline Outlier
Email_IsNotMatch	354.11 ns	380.00 ns	1.07		True
Email_IsMatch	257.82 ns	289.67 ns	1.12		True

Historical Data in Reporting System

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f netcoreapp5.0 --filter 'System.Text.RegularExpressions.Tests.Perf_Regex_Common*';

Histogram

System.Text.RegularExpressions.Tests.Perf_Regex_Common.Email_IsNotMatch(Options: IgnoreCase, Compiled)

[351.514 ; 362.555) | @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
[362.555 ; 375.809) | @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
[375.809 ; 394.434) | @@@@@@@@@@@@@@@@@@@@@@@@

System.Text.RegularExpressions.Tests.Perf_Regex_Common.Email_IsMatch(Options: IgnoreCase, Compiled)

[255.611 ; 262.054) | @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
[262.054 ; 267.995) | @@@@@@@@@@@@@@@@@@@@@@
[267.995 ; 274.438) | 
[274.438 ; 279.884) | 
[279.884 ; 286.328) | @@
[286.328 ; 295.294) | @@@@@@@@@@@@@@@@@@@@@

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

The text was updated successfully, but these errors were encountered:

ghost · 2020-07-29T17:57:09Z

Tagging subscribers to this area: @eerhardt, @pgovind
See info in area-owners.md if you want to be subscribed.

danmoseley · 2020-07-29T18:21:27Z

@DrewScoggins thanks for the tightly bounded diff. IThere were no regex changes since July 16th.

There are only two commits in that diff that are even slightly plausibly related

@davidwrighton Fix CanCastTo behavior (#39915) … 13fa14e
@erozenfeld Fix fgUpdateChangedFlowGraph. (#39878) … b179d69

eyeballing the raw graph, I get a slightly larger bracket. This change may also have been in those bounds:

@davidwrighton Fix issue in type equivalence involving arrays (#39914) … 90989b4

@davidwrighton @erozenfeld can you conceive of either of these causing this regression?

danmoseley · 2020-07-29T18:27:03Z

Looking at the 2nd test, I think @DrewScoggins tighter bracket is the correct one.

@DrewScoggins did you see any other regressions in other scenarios during those 3 hours, can you check?

DrewScoggins · 2020-07-29T18:30:35Z

I believe that this test also regressed during the same gap, #40088

danmoseley · 2020-07-29T18:35:25Z

OK, I'm inclined to put this in codegen area, but I am interested in @erozenfeld thoughts.

erozenfeld · 2020-07-29T19:52:38Z

#39878 had codegen diffs in System.Text.RegularExpressions so it's possible it caused this regression. I'll investigate.

danmoseley · 2020-07-29T20:57:54Z

OK thanks @erozenfeld

danmoseley · 2020-07-29T21:00:35Z

Hooray for (1) prompt detection of a performance regression (2) useful issue that allowed us to narrow it down quickly

In dotnet#39878 I switched fgUpdateChangedFlowGraph to call fgComputeReachability, which both removes unreachable blocks and calls fgComputeDoms. As mentioned in that PR, in addition to removing unreachable blocks fgRemoveUnreachableBlocks updates `BBF_LOOP_HEAD` flags even if no unreachable blocks were found. That resulted in some diffs, both positive and negative, from downstream effects: e.g., in some cases we now recognize more loops, which changes weights, etc. Some of the negative diffs affected benchmarks we are tracking, e.g., in dotnet#40094 `System.Text.RegularExpressions.Tests.Perf_Regex_Common` had a 10% regression because of codegen diffs in the large dynamic method created when compiling regular expressions. Because of these regressions, I decided to go with a more surgical fix for the original issue (assert when computing dominators after inlining GC polls). The downstream phases don't really need the dominator info so I changed fgUpdateChangedFlowGraph to not re-compute dominators after GC poll inlining. This reverses all diffs from dotnet#39878 and fixes dotnet#40094.

In #39878 I switched fgUpdateChangedFlowGraph to call fgComputeReachability, which both removes unreachable blocks and calls fgComputeDoms. As mentioned in that PR, in addition to removing unreachable blocks fgRemoveUnreachableBlocks updates `BBF_LOOP_HEAD` flags even if no unreachable blocks were found. That resulted in some diffs, both positive and negative, from downstream effects: e.g., in some cases we now recognize more loops, which changes weights, etc. Some of the negative diffs affected benchmarks we are tracking, e.g., in #40094 `System.Text.RegularExpressions.Tests.Perf_Regex_Common` had a 10% regression because of codegen diffs in the large dynamic method created when compiling regular expressions. Because of these regressions, I decided to go with a more surgical fix for the original issue (assert when computing dominators after inlining GC polls). The downstream phases don't really need the dominator info so I changed fgUpdateChangedFlowGraph to not re-compute dominators after GC poll inlining. This reverses all diffs from #39878 and fixes #40094.

danmoseley · 2020-07-31T00:38:05Z

Looking even closer at the graph and zooming in, we actually could get a diff bounding just 3 hours of time on July 24th:
f3f23b4...6ca6003

In dotnet#39878 I switched fgUpdateChangedFlowGraph to call fgComputeReachability, which both removes unreachable blocks and calls fgComputeDoms. As mentioned in that PR, in addition to removing unreachable blocks fgRemoveUnreachableBlocks updates `BBF_LOOP_HEAD` flags even if no unreachable blocks were found. That resulted in some diffs, both positive and negative, from downstream effects: e.g., in some cases we now recognize more loops, which changes weights, etc. Some of the negative diffs affected benchmarks we are tracking, e.g., in dotnet#40094 `System.Text.RegularExpressions.Tests.Perf_Regex_Common` had a 10% regression because of codegen diffs in the large dynamic method created when compiling regular expressions. Because of these regressions, I decided to go with a more surgical fix for the original issue (assert when computing dominators after inlining GC polls). The downstream phases don't really need the dominator info so I changed fgUpdateChangedFlowGraph to not re-compute dominators after GC poll inlining. This reverses all diffs from dotnet#39878 and fixes dotnet#40094.

Dotnet-GitSync-Bot added area-System.Text.RegularExpressions untriaged New issue has not been triaged by the area owner labels Jul 29, 2020

DrewScoggins added arch-x64 os-windows tenet-performance Performance related issue tenet-performance-benchmarks Issue from performance benchmark labels Jul 29, 2020

danmoseley mentioned this issue Jul 29, 2020

[Perf -22%] System.Xml.Linq.Perf_XDocument.GetRootElement #40088

Closed

erozenfeld self-assigned this Jul 29, 2020

danmoseley added area-CodeGen-coreclr CLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMI and removed area-System.Text.RegularExpressions labels Jul 29, 2020

erozenfeld added this to the 5.0.0 milestone Jul 29, 2020

AndyAyersMS removed the untriaged New issue has not been triaged by the area owner label Jul 30, 2020

erozenfeld mentioned this issue Jul 30, 2020

Re-work fix for flow graph update. #40162

Merged

erozenfeld closed this as completed in #40162 Jul 31, 2020

ghost locked as resolved and limited conversation to collaborators Dec 8, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[Perf -10%] System.Text.RegularExpressions.Tests.Perf_Regex_Common (2) #40094

[Perf -10%] System.Text.RegularExpressions.Tests.Perf_Regex_Common (2) #40094

DrewScoggins commented Jul 29, 2020

Histogram

System.Text.RegularExpressions.Tests.Perf_Regex_Common.Email_IsNotMatch(Options: IgnoreCase, Compiled)

System.Text.RegularExpressions.Tests.Perf_Regex_Common.Email_IsMatch(Options: IgnoreCase, Compiled)

Docs

ghost commented Jul 29, 2020

Uh oh!

danmoseley commented Jul 29, 2020 •

edited

Loading

Uh oh!

danmoseley commented Jul 29, 2020

Uh oh!

DrewScoggins commented Jul 29, 2020

Uh oh!

danmoseley commented Jul 29, 2020

Uh oh!

erozenfeld commented Jul 29, 2020

Uh oh!

danmoseley commented Jul 29, 2020

Uh oh!

danmoseley commented Jul 29, 2020

Uh oh!

danmoseley commented Jul 31, 2020

Uh oh!

[Perf -10%] System.Text.RegularExpressions.Tests.Perf_Regex_Common (2) #40094

[Perf -10%] System.Text.RegularExpressions.Tests.Perf_Regex_Common (2) #40094

Comments

DrewScoggins commented Jul 29, 2020

Run Information

Regressions in System.Text.RegularExpressions.Tests.Perf_Regex_Common

Repro

Histogram

System.Text.RegularExpressions.Tests.Perf_Regex_Common.Email_IsNotMatch(Options: IgnoreCase, Compiled)

System.Text.RegularExpressions.Tests.Perf_Regex_Common.Email_IsMatch(Options: IgnoreCase, Compiled)

Docs

ghost commented Jul 29, 2020

Uh oh!

danmoseley commented Jul 29, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

danmoseley commented Jul 29, 2020

Uh oh!

DrewScoggins commented Jul 29, 2020

Uh oh!

danmoseley commented Jul 29, 2020

Uh oh!

erozenfeld commented Jul 29, 2020

Uh oh!

danmoseley commented Jul 29, 2020

Uh oh!

danmoseley commented Jul 29, 2020

Uh oh!

danmoseley commented Jul 31, 2020

Uh oh!

danmoseley commented Jul 29, 2020 •

edited

Loading