Thanks to visit codestin.com
Credit goes to github.com

Skip to content

feat: Add filter support for TestAdapter #2788

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 1 commit into from
Jul 21, 2025
Merged

feat: Add filter support for TestAdapter #2788

merged 1 commit into from
Jul 21, 2025

Conversation

filzrev
Copy link
Contributor

@filzrev filzrev commented Jun 20, 2025 •
edited
Loading

This PR add --filter parameter support for dotnet test.
It intended to resolve #2767 issue (and some part of issue #2662)

What's changed in this PR.

1. Add benchmark filter logics
Add following files to support benchmark filtering.

  • TestCaseFilter.cs
  • LoggerHelper.cs

Note: These files are based on xUnit v2 TestAdapter Code (License: Apache 2.0)

2. Modify VSTestAdapter.cs
Add benchmark filter logics to DiscoverTests and RunTests

3. BenchmarkEnumerator.cs
Remove code for Debug configuration.
(By #2774 changes. It can load DLLs that are build with Debug configurations)

4. Disable TestTfmsInParallel MSBuild property
It's enabled by default on .NET 9 or later.
But it's not expected multiple benchmarks are executed in parallel.

I've added code to disable this setting to following files.

  • BenchmarkDotNet.Samples.csproj
  • BenchmarkDotNet.Samples.FSharp.csproj
  • BenchmarkDotNet.TestAdapter/build/BenchmarkDotNet.TestAdapter.props

5. Temporary comment out IntroVisualStudioProfiler benchmark
Because it cause errors when benchmarks are enumerated/evaluated twice. (#2758)

Tasks that is not scope of this PR

  • Benchmark progress is not shown when running benchmark with dotnet test by default
    -> It need to specify --logger:"console;verbosity=normal" explicitly
  • DisplayName filter is same as FullyQualifiedName
    -> Currently there is no attribute that can be used for DisplayName

What's Tested

Currently it's hard to write unit tests that use TestAdapter.
So it need to run tests manually.

TestExplorer

1. All benchmarks(net462/net80) are displayed when using Release configuration.

  1. It can run single selected test and multiple selected tests.
  2. When running multiple TFMs benchmarks. It's sequentially executed. (Not executed in parallel)

dotnet test

  1. Verify default behavior is not changed when run command without --filter parameter

    dotnet test -c Release -tl:off --no-build --framework net8.0 --logger:"console;verbosity=detailed" --list-tests
    dotnet test -c Release -tl:off --no-build --framework net8.0 --logger:"console;verbosity=detailed"

  2. Filter by partial query works as expected.

    dotnet test -c Release --filter IntroBasic -tl:off --no-build --framework net8.0 --logger:"console;verbosity=detailed" --list-tests
    dotnet test -c Release --filter IntroBasic -tl:off --no-build --framework net8.0 --logger:"console;verbosity=detailed"

  3. Filter by property value works as expected.

    dotnet test -c Release --filter FullyQualifiedName=BenchmarkDotNet.Samples.IntroBasic.Sleep -tl:off --no-build --framework net8.0 --logger:"console;verbosity=detailed" --list-tests
    dotnet test -c Release --filter DisplayName=BenchmarkDotNet.Samples.IntroBasic.Sleep -tl:off --no-build --framework net8.0 --logger:"console;verbosity=detailed" --list-tests

    dotnet test -c Release --filter FullyQualifiedName=BenchmarkDotNet.Samples.IntroBasic.Sleep -tl:off --no-build --framework net8.0 --logger:"console;verbosity=detailed"
    dotnet test -c Release --filter DisplayName=BenchmarkDotNet.Samples.IntroBasic.Sleep -tl:off --no-build --framework net8.0 --logger:"console;verbosity=detailed"

  4. Filter by Category works as expected

    dotnet test -c Release --filter Category=Fast -tl:off --no-build --framework net8.0 --logger:"console;verbosity=normal" --list-tests
    dotnet test -c Release --filter Category=Fast -tl:off --no-build --framework net8.0 --logger:"console;verbosity=normal"

  5. Verify it can run .NET462 benchmarks also

    dotnet test -c Release --filter IntroBasic -tl:off --no-build --framework net462 --logger:"console;verbosity=detailed" --list-tests
    dotnet test -c Release --filter IntroBasic -tl:off --no-build --framework net462 --logger:"console;verbosity=detailed"

@timcassell
Copy link
Collaborator

  1. All benchmarks(net462/net8.0) are displayed on Debug/Release configuration.

In the original PR, not including Debug without in-process was on purpose. See comment chain #2438 (comment). We decided to leave it as a future improvement to include debugger attachment. I think that should be done in a separate PR.

@filzrev
Copy link
Contributor Author

filzrev commented Jun 20, 2025

In the original PR, not including Debug without in-process was on purpose.

Thanks for pointed out it's expected behavior that display only InProcess jobs on Debug configuration.
I've add commit to revert related changes.

@filzrev
Copy link
Contributor Author

filzrev commented Jun 22, 2025

I've added commit(8551dbf) to resolve issue console logs are outputted twice.

It use reflection to modify ImmutableConfig's logger field with following reasons

  • ImmutableConfig member can't modify fields after construction
  • To use existing ImmutableConfigBuilder. It requires BenchmarkDotNet side code modification. or need to a lot of duplicated code copy.

When it failed to get logger fields of ImmutableConfig. It skip steps to remove ConsoleLogger.

@timcassell
Copy link
Collaborator

timcassell commented Jun 22, 2025 •
edited
Loading

Is that related to filter? Can it be done in a separate PR?

Also, I don't like using reflection to do that. We can instead construct a new config object, copying all the properties except ones we want to exclude. We can also add APIs to ManualConfig to do this kind of thing.

@filzrev
Copy link
Contributor Author

filzrev commented Jun 22, 2025 •
edited
Loading

Is that related to filter? Can it be done in a separate PR?
It's existing TestAdapter's problem. (Duplicated logs are outputted to Output Window and Console)

So it might better to create separated PR.

@filzrev filzrev force-pushed the feat-add-vstest-filter-supports branch from e89ded7 to 2391e11 Compare June 23, 2025 13:38
@filzrev
Copy link
Contributor Author

filzrev commented Jun 23, 2025

I've added commit (2391e11)

It set <TestTfmsInParallel>false</TestTfmsInParallel> setting automatically if value is not explicitly set. when package is referenced by using <PackageReference>.

Note:
It's not works when reference TestAdapter project with ProjectReference.
So BenchmarkDotNet.Samples.csproj settings are remained as is)

@filzrev filzrev force-pushed the feat-add-vstest-filter-supports branch from 2391e11 to dd4ac03 Compare June 25, 2025 12:33
@filzrev filzrev force-pushed the feat-add-vstest-filter-supports branch from dd4ac03 to d2af178 Compare July 5, 2025 04:03
@filzrev filzrev force-pushed the feat-add-vstest-filter-supports branch from d6b203a to 553522c Compare July 5, 2025 05:07
@filzrev filzrev marked this pull request as ready for review July 5, 2025 06:13
timcassell
timcassell approved these changes Jul 21, 2025
View reviewed changes
Copy link
Collaborator

@timcassell timcassell left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @filzrev!

@timcassell timcassell merged commit 8b62787 into dotnet:master Jul 21, 2025
9 checks passed
@timcassell timcassell added this to the v0.15.3 milestone Jul 21, 2025
@timcassell timcassell linked an issue Jul 21, 2025 that may be closed by this pull request
Unable to filter benchmarks with TestAdapter #2662
Closed
This was referenced Sep 17, 2025
Open
Open
Closed
Merged
Merged
Merged
Open
Merged
Merged
Open
Open
Closed
Closed
CurtHagenlocher pushed a commit to apache/arrow-dotnet that referenced this pull request Sep 22, 2025
@dependabot
chore: Bump BenchmarkDotNet from 0.15.2 to 0.15.3 (#76)
425bc83 
Updated [BenchmarkDotNet](https://github.com/dotnet/BenchmarkDotNet)
from 0.15.2 to 0.15.3.

<details>
<summary>Release notes</summary>

_Sourced from [BenchmarkDotNet's
releases](https://github.com/dotnet/BenchmarkDotNet/releases)._

## 0.15.3

Full changelog: https://benchmarkdotnet.org/changelog/v0.15.3.html

## Highlights

Improvements:

- Naot instruction set support for .NET 10+, migrate to ISA groupings in
CPU summary
[#​2828](dotnet/BenchmarkDotNet#2828)
- Support benchmark filtering for TestAdapter
[#​2662](dotnet/BenchmarkDotNet#2662)
[#​2788](dotnet/BenchmarkDotNet#2788)
- Support non-primitive external types in `ArgumentsSource`
[#​2820](dotnet/BenchmarkDotNet#2820)
- Enable MSBuild parallel build via `--nodeReuse:false`
[#​2693](dotnet/BenchmarkDotNet#2693)
[#​2814](dotnet/BenchmarkDotNet#2814)
- Improve CPU detection
[#​2747](dotnet/BenchmarkDotNet#2747)
[#​2749](dotnet/BenchmarkDotNet#2749)
- Enable assembly signing for debug build
[#​2774](dotnet/BenchmarkDotNet#2774)

Deprecations:

- Deprecated `WithNuget`
[#​2812](dotnet/BenchmarkDotNet#2812)

Bug fixes:

- Fix `InvalidOperationException` in diagnosers
[#​2758](dotnet/BenchmarkDotNet#2758)
[#​2805](dotnet/BenchmarkDotNet#2805)
- Fix file detection in `NativeMemoryProfiler`
[#​2794](dotnet/BenchmarkDotNet#2794)
[#​2795](dotnet/BenchmarkDotNet#2795)
- Fix long file paths issue in `EtwProfiler`
[#​2807](dotnet/BenchmarkDotNet#2807)
[#​2808](dotnet/BenchmarkDotNet#2808)
- Fix log duplications in TestAdapter
[#​2790](dotnet/BenchmarkDotNet#2790)
- Fix x86 disassembler error for net462
[#​2792](dotnet/BenchmarkDotNet#2792)
- Fix `IsNetCore` and `IsNativeAOT` for single-file apps without AOT
[#​2799](dotnet/BenchmarkDotNet#2799)
- Fix density plot generation in `RPlotExporter` for latest version of R
[#​2809](dotnet/BenchmarkDotNet#2809)


Commits viewable in [compare
view](dotnet/BenchmarkDotNet@v0.15.2...v0.15.3).
</details>

[![Dependabot compatibility
score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=BenchmarkDotNet&package-manager=nuget&previous-version=0.15.2&new-version=0.15.3)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)


</details>

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
This was referenced Sep 22, 2025
Open
Open
Merged
Open
Open
Open
Merged
Open
Open
Open
Open
Open
Open
Open
Merged
Open
Open
Merged
Open
Merged
Open
Open
Open
Open
Open
Open
Open
Open
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@timcassell timcassell timcassell approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
v0.15.3
Development

Successfully merging this pull request may close these issues.

[Feature Request] Support filtering feature on BenchmarkDotNet.TestAdapter Unable to filter benchmarks with TestAdapter
2 participants
@filzrev @timcassell