-
-
Notifications
You must be signed in to change notification settings - Fork 1k
feat: Add validator for benchmarks that contains null runtime #2771
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
41b066f
to
4b61ffb
Compare
I think the UX needs to be improved around this, but this is ok as a stop-gap for now. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
See comment
I've updated runtime validation logics (457330d) Example validation error messages
|
ef841f8
to
ea68e83
Compare
It's not related to this PR thought.
|
Just thought about in-process toolchains after I merged. The current runtime behavior is correct on those toolchains, so they should not be included as part of the validation. @filzrev |
Updated [BenchmarkDotNet](https://github.com/dotnet/BenchmarkDotNet) from 0.14.0 to 0.15.3. <details> <summary>Release notes</summary> _Sourced from [BenchmarkDotNet's releases](https://github.com/dotnet/BenchmarkDotNet/releases)._ ## 0.15.3 Full changelog: https://benchmarkdotnet.org/changelog/v0.15.3.html ## Highlights Improvements: - Naot instruction set support for .NET 10+, migrate to ISA groupings in CPU summary [#2828](dotnet/BenchmarkDotNet#2828) - Support benchmark filtering for TestAdapter [#2662](dotnet/BenchmarkDotNet#2662) [#2788](dotnet/BenchmarkDotNet#2788) - Support non-primitive external types in `ArgumentsSource` [#2820](dotnet/BenchmarkDotNet#2820) - Enable MSBuild parallel build via `--nodeReuse:false` [#2693](dotnet/BenchmarkDotNet#2693) [#2814](dotnet/BenchmarkDotNet#2814) - Improve CPU detection [#2747](dotnet/BenchmarkDotNet#2747) [#2749](dotnet/BenchmarkDotNet#2749) - Enable assembly signing for debug build [#2774](dotnet/BenchmarkDotNet#2774) Deprecations: - Deprecated `WithNuget` [#2812](dotnet/BenchmarkDotNet#2812) Bug fixes: - Fix `InvalidOperationException` in diagnosers [#2758](dotnet/BenchmarkDotNet#2758) [#2805](dotnet/BenchmarkDotNet#2805) - Fix file detection in `NativeMemoryProfiler` [#2794](dotnet/BenchmarkDotNet#2794) [#2795](dotnet/BenchmarkDotNet#2795) - Fix long file paths issue in `EtwProfiler` [#2807](dotnet/BenchmarkDotNet#2807) [#2808](dotnet/BenchmarkDotNet#2808) - Fix log duplications in TestAdapter [#2790](dotnet/BenchmarkDotNet#2790) - Fix x86 disassembler error for net462 [#2792](dotnet/BenchmarkDotNet#2792) - Fix `IsNetCore` and `IsNativeAOT` for single-file apps without AOT [#2799](dotnet/BenchmarkDotNet#2799) - Fix density plot generation in `RPlotExporter` for latest version of R [#2809](dotnet/BenchmarkDotNet#2809) ## 0.15.2 Full changelog: https://benchmarkdotnet.org/changelog/v0.15.2.html ## Highlights - The most significant update in this release is the enhanced accuracy of the memory diagnoser ([#2562](dotnet/BenchmarkDotNet#2562)). This improvement resolves the issue of incorrectly reported memory allocations ([#1542](dotnet/BenchmarkDotNet#1542), [#2582](dotnet/BenchmarkDotNet#2582)). - We have introduced a new feature that allows users to sort benchmark jobs in numerical order ([#2768](dotnet/BenchmarkDotNet#2768), [#2770](dotnet/BenchmarkDotNet#2770)). - Benchmark validation has been improved ([#2771](dotnet/BenchmarkDotNet#2771)). - An issue with non-persistent auto-generated JobId has been fixed ([#2777](dotnet/BenchmarkDotNet#2777)). ## 0.15.1 Full changelog: https://benchmarkdotnet.org/changelog/v0.15.1.html ## Highlights - Added support for *.slnx ([#2763](dotnet/BenchmarkDotNet#2763), [#2764](dotnet/BenchmarkDotNet#2764)) - Enabled ArgumentsSource to reference methods in other types ([#2744](dotnet/BenchmarkDotNet#2744), [#2748](dotnet/BenchmarkDotNet#2748)) - Resolved fatal errors for ARM CPUs ([#2745](dotnet/BenchmarkDotNet#2745), [#2756](dotnet/BenchmarkDotNet#2756)) - Fixed bugs related to support for Android, browser, iOS, and tvOS ([#2739](dotnet/BenchmarkDotNet#2739), [#2741](dotnet/BenchmarkDotNet#2741), [#2740](dotnet/BenchmarkDotNet#2740), [#2742](dotnet/BenchmarkDotNet#2742)) ## 0.15.0 Full changelog: https://benchmarkdotnet.org/changelog/v0.15.0.html Commits viewable in [compare view](dotnet/BenchmarkDotNet@v0.14.0...v0.15.3). </details> [](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) </details> ###### Microsoft Reviewers: [Open in CodeFlow](https://microsoft.github.io/open-pr/?codeflow=https://github.com/Azure/bicep/pull/18084) Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
This PR add validator to handle issue that is reported #2609.
What's changed in this PR
RuntimeValidator.cs
for benchmarks that have multiple runtimes and contains null.Background
When benchmark don't contains
Runtime
characteristic. (e.g. When specifyWithToolChain() only
)BenchmarkCase::GetRuntime returns current runtime instead of actual runtime that used by benchmark.
Normally,
Runtime
column is hidden on table because all benchmarks have same runtime value.So it's not a problem. But when mixed
WithRuntime
settings. it display wrong runtime.So it need to add validator to check benchmarks that contains multiple runtimes and contains null.
RuntimeValidator's error message