Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Conversation

filzrev
Copy link
Contributor

@filzrev filzrev commented Jun 12, 2025

This PR add validator to handle issue that is reported #2609.

What's changed in this PR

  1. Add RuntimeValidator.cs for benchmarks that have multiple runtimes and contains null.
  2. Add RuntimeValidator to DefaultConfig's validator
  3. Add unit tests.

Background
When benchmark don't contains Runtime characteristic. (e.g. When specify WithToolChain() only)
BenchmarkCase::GetRuntime returns current runtime instead of actual runtime that used by benchmark.

Normally, Runtime column is hidden on table because all benchmarks have same runtime value.
So it's not a problem. But when mixed WithRuntime settings. it display wrong runtime.

So it need to add validator to check benchmarks that contains multiple runtimes and contains null.

RuntimeValidator's error message

// * There are benchmarks that job don't have a Runtime characteristic. It's recommended explicitly specify runtime by WithRuntime

@filzrev filzrev force-pushed the feat-add-runtimevalidator branch 2 times, most recently from 41b066f to 4b61ffb Compare June 12, 2025 11:18
@timcassell
Copy link
Collaborator

I think the UX needs to be improved around this, but this is ok as a stop-gap for now.

Copy link
Collaborator

@timcassell timcassell left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See comment

@filzrev
Copy link
Contributor Author

filzrev commented Jun 15, 2025

I've updated runtime validation logics (457330d)
Additionally, I've added JobId (or Job text representation) to validation error message.

Example validation error messages

// Validating benchmarks:
//    * Job(Toolchain=.NET 10.0) don't have a Runtime characteristic. It's recommended to specify runtime by using WithRuntime explicitly.

//    * Job(Dry) don't have a Runtime characteristic. It's recommended to specify runtime by using WithRuntime explicitly.

//    * Job(abc) don't have a Runtime characteristic. It's recommended to specify runtime by using WithRuntime explicitly.

@filzrev filzrev force-pushed the feat-add-runtimevalidator branch from ef841f8 to ea68e83 Compare June 15, 2025 04:50
@filzrev
Copy link
Contributor Author

filzrev commented Jun 15, 2025

It's not related to this PR thought. AllocationQuantumIsNotAnIssueForNetCore21Plus test seems flaky on CI.
Currently it's confirmed on mac-os (ARM64).

2025-06-15T04:21:35.2938130Z [xUnit.net 00:09:12.36]     AllocationQuantumIsNotAnIssueForNetCore21Plus(toolchain: .NET 8.0) [FAIL]
2025-06-15T04:21:35.2938900Z [xUnit.net 00:09:12.36]       Assert.Equal() Failure: Values differ
2025-06-15T04:21:35.2939300Z [xUnit.net 00:09:12.36]       Expected: 88
2025-06-15T04:21:35.2939630Z [xUnit.net 00:09:12.36]       Actual:   760

@timcassell timcassell merged commit ed5316b into dotnet:master Jun 15, 2025
14 of 16 checks passed
@timcassell timcassell added this to the v0.15.2 milestone Jun 15, 2025
@timcassell
Copy link
Collaborator

Just thought about in-process toolchains after I merged. The current runtime behavior is correct on those toolchains, so they should not be included as part of the validation. @filzrev

microsoft-github-policy-service bot pushed a commit to Azure/bicep that referenced this pull request Sep 21, 2025
Updated [BenchmarkDotNet](https://github.com/dotnet/BenchmarkDotNet)
from 0.14.0 to 0.15.3.

<details>
<summary>Release notes</summary>

_Sourced from [BenchmarkDotNet's
releases](https://github.com/dotnet/BenchmarkDotNet/releases)._

## 0.15.3

Full changelog: https://benchmarkdotnet.org/changelog/v0.15.3.html

## Highlights

Improvements:

- Naot instruction set support for .NET 10+, migrate to ISA groupings in
CPU summary
[#​2828](dotnet/BenchmarkDotNet#2828)
- Support benchmark filtering for TestAdapter
[#​2662](dotnet/BenchmarkDotNet#2662)
[#​2788](dotnet/BenchmarkDotNet#2788)
- Support non-primitive external types in `ArgumentsSource`
[#​2820](dotnet/BenchmarkDotNet#2820)
- Enable MSBuild parallel build via `--nodeReuse:false`
[#​2693](dotnet/BenchmarkDotNet#2693)
[#​2814](dotnet/BenchmarkDotNet#2814)
- Improve CPU detection
[#​2747](dotnet/BenchmarkDotNet#2747)
[#​2749](dotnet/BenchmarkDotNet#2749)
- Enable assembly signing for debug build
[#​2774](dotnet/BenchmarkDotNet#2774)

Deprecations:

- Deprecated `WithNuget`
[#​2812](dotnet/BenchmarkDotNet#2812)

Bug fixes:

- Fix `InvalidOperationException` in diagnosers
[#​2758](dotnet/BenchmarkDotNet#2758)
[#​2805](dotnet/BenchmarkDotNet#2805)
- Fix file detection in `NativeMemoryProfiler`
[#​2794](dotnet/BenchmarkDotNet#2794)
[#​2795](dotnet/BenchmarkDotNet#2795)
- Fix long file paths issue in `EtwProfiler`
[#​2807](dotnet/BenchmarkDotNet#2807)
[#​2808](dotnet/BenchmarkDotNet#2808)
- Fix log duplications in TestAdapter
[#​2790](dotnet/BenchmarkDotNet#2790)
- Fix x86 disassembler error for net462
[#​2792](dotnet/BenchmarkDotNet#2792)
- Fix `IsNetCore` and `IsNativeAOT` for single-file apps without AOT
[#​2799](dotnet/BenchmarkDotNet#2799)
- Fix density plot generation in `RPlotExporter` for latest version of R
[#​2809](dotnet/BenchmarkDotNet#2809)


## 0.15.2

Full changelog: https://benchmarkdotnet.org/changelog/v0.15.2.html

## Highlights

- The most significant update in this release is the enhanced accuracy
of the memory diagnoser
([#​2562](dotnet/BenchmarkDotNet#2562)). This
improvement resolves the issue of incorrectly reported memory
allocations
([#​1542](dotnet/BenchmarkDotNet#1542),
[#​2582](dotnet/BenchmarkDotNet#2582)).
- We have introduced a new feature that allows users to sort benchmark
jobs in numerical order
([#​2768](dotnet/BenchmarkDotNet#2768),
[#​2770](dotnet/BenchmarkDotNet#2770)).
- Benchmark validation has been improved
([#​2771](dotnet/BenchmarkDotNet#2771)).
- An issue with non-persistent auto-generated JobId has been fixed
([#​2777](dotnet/BenchmarkDotNet#2777)).


## 0.15.1

Full changelog: https://benchmarkdotnet.org/changelog/v0.15.1.html

## Highlights

- Added support for *.slnx
([#​2763](dotnet/BenchmarkDotNet#2763),
[#​2764](dotnet/BenchmarkDotNet#2764))
- Enabled ArgumentsSource to reference methods in other types
([#​2744](dotnet/BenchmarkDotNet#2744),
[#​2748](dotnet/BenchmarkDotNet#2748))
- Resolved fatal errors for ARM CPUs
([#​2745](dotnet/BenchmarkDotNet#2745),
[#​2756](dotnet/BenchmarkDotNet#2756))
- Fixed bugs related to support for Android, browser, iOS, and tvOS
([#​2739](dotnet/BenchmarkDotNet#2739),
[#​2741](dotnet/BenchmarkDotNet#2741),
[#​2740](dotnet/BenchmarkDotNet#2740),
[#​2742](dotnet/BenchmarkDotNet#2742))



## 0.15.0

Full changelog: https://benchmarkdotnet.org/changelog/v0.15.0.html



Commits viewable in [compare
view](dotnet/BenchmarkDotNet@v0.14.0...v0.15.3).
</details>

[![Dependabot compatibility
score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=BenchmarkDotNet&package-manager=nuget&previous-version=0.14.0&new-version=0.15.3)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)


</details>
###### Microsoft Reviewers: [Open in
CodeFlow](https://microsoft.github.io/open-pr/?codeflow=https://github.com/Azure/bicep/pull/18084)

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
This was referenced Sep 22, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants