Fix ArgumentsSource on external types not working if the argument type is not primitive #2820

JimmyCushnie · 2025-08-14T04:50:34Z

Followup to #2748. As of BDN 0.15.2, you can't use [ArgumentsSource] with an external type providing the arguments if the argument type is not primitive. This happens because the generated code has compiler errors.

The issue

If you have code like this:

[Benchmark]
[ArgumentsSource(typeof(DiagnosticsCases), nameof(DiagnosticsCases.MethodACases))]
public MethodAResults BenchmarkMethodA(MethodACase @case)
    => CodeBeingBenchmarked.MethodA(@case.Param1, @case.Param2, @case.Param3);

Then BDN currently generates code like this:

public Runnable_0()
{
    globalSetupAction = () => { };
    globalCleanupAction = () => { };
    iterationSetupAction = () => { };
    iterationCleanupAction = () => { };
    overheadDelegate = __Overhead;
    workloadDelegate = WithChildrenAdded;
    __argField0 = (ExampleNamespace.MethodACase)BenchmarkDotNet.Parameters.ParameterExtractor.GetParameter(MethodACases(), 0);;
}

The problem is that MethodACases is not in the type that the generated class (Runnable_0) inherits from, it's in an external type (DiagnosticsCases). So there's a compiler error here ("error CS0103: The name 'MethodACases' does not exist in the current context")

This only happens if the argument type is non-primitive (like MethodACase). If it's a primitive type (like int) then the argument value is inlined as a literal, instead of using ParameterExtractor.

The fix

In this example, we need to put the fully qualified type name before the method call. The last line of the generated code should instead read:

__argField0 = (ExampleNamespace.MethodACase)BenchmarkDotNet.Parameters.ParameterExtractor.GetParameter(ExampleNamespace.DiagnosticsCases.MethodACases(), 0);;

However, we can't always add the fully qualified type name, because ArgumentsSource can also be used with non-static methods/properties.

So, this PR checks if ArgumentsSource is being used with a static type. If so, the fully qualified type name is added at this location; otherwise, the generated code is unchanged compared to before this PR.

I am worried about external instance methods

Using ArgumentsSource for non-primitive arguments with different kinds of members:

	Does it work before this PR	Does it work after this PR
Static member from the same class	✔	✔
Static member from an external class	❌	✔
Instance member from the same class	✔	✔
Instance member from an external class	❌	❌

As you can see from this helpful table, this PR fixes one broken case but there is still one broken case remaining. How best to fix this? Should the generated code create an instance of the member's type, and call the member from the new instance?

To be honest, I think there's no benefit to supporting instance members at all. @timcassell also dislikes that we support instance members and we discussed it a bit here: #2744 (comment).

Maybe we should take this opportunity to just drop support for instance members in ArgumentsSource and ParamsSource?

Other things I am worried about

The way this PR checks if the member is static is kind of brittle and makes me nervous. Is there a better way of doing it? (If we just drop support for instance members, this problem goes away!!)
Is the code I wrote the correct way to fix the described issue? I would appreciate some feedback from someone more familiar with the codegen parts of BDN. It's very complex code and I couldn't fully wrap my head around it.

… type

filzrev · 2025-08-15T08:22:01Z

I thought sample benchmark IntroArgumentsSource.cs also needs to be fixed.

Currently IntroArgumentsSource benchmark failed with following errors.
(And it seems to be fixed by adding static modifier)

BenchmarkDotNet.Samples-DefaultJob-1.notcs(2565,103): error CS0103: The name 'TimeSpans' does not exist in the current context

JimmyCushnie · 2025-08-15T19:07:47Z

Thanks, nice catch. Should be fixed now!

timcassell · 2025-08-19T19:18:40Z

I'm trying to understand what was broken before. When I run the tests, they succeed. Can you add tests that failed previously that pass with this change?

JimmyCushnie · 2025-08-22T07:16:35Z

I'm trying to understand what was broken before.

What is currently broken is that if you try to use ArgumentsSource with external types, the benchmarks fail to run. This happens due to a compiler error in the generated code. I explain this and give an example in the PR description.

When I run the tests, they succeed. Can you add tests that failed previously that pass with this change?

I address this in the PR description as well. I was also confused by the tests passing, since if you create an example project using my example code it does not work. Does our test infra not currently check the generated code at all? If it does, why did it miss this compiler error, and if it doesn't, how should we go about adding that capability?

Before I add tests to this PR, I was hoping somebody with knowledge about our test infra would chime in with some insight. Such a person giving quick answers to the above questions would save me hours of investigating our very complicated test infra myself, which would be nice since it already took me hours of investigation to figure out the cause of this issue and figure out how to fix it.

It seems like you didn't understand the information in the PR description. I'm sorry if I presented the information unclearly. Do you have any advice for how I can make my writing easier to understand?

timcassell · 2025-08-22T07:27:31Z

There doesn't seem to be anything wrong with the test infrastructure. I copied the existing tests to a new benchmark project, and they still succeed. It appears to be due to the arg type MethodACase, because the existing tests only use primitive types. So just add test cases with that custom type.

Catches some issues not previously caught.

JimmyCushnie · 2025-08-23T04:19:48Z

Aha, you're completely correct. Thanks a lot! The primitive types got inlined as literals in the generated code which is why the bug wasn't triggered by them.

I added some more test cases that use non-primitive types. I've confirmed that the new tests fail before this PR and pass after this PR.

I've also updated the PR title and description to reflect this improved understanding of the bug.

timcassell · 2025-08-23T05:03:06Z

Thanks. Can you also update the ParamsSource tests?

JimmyCushnie · 2025-08-23T06:29:12Z

Done 😎

src/BenchmarkDotNet/Parameters/SmartParamBuilder.cs

Co-authored-by: Tim Cassell <[email protected]>

Updated [BenchmarkDotNet](https://github.com/dotnet/BenchmarkDotNet) from 0.15.2 to 0.15.3. <details> <summary>Release notes</summary> _Sourced from [BenchmarkDotNet's releases](https://github.com/dotnet/BenchmarkDotNet/releases)._ ## 0.15.3 Full changelog: https://benchmarkdotnet.org/changelog/v0.15.3.html ## Highlights Improvements: - Naot instruction set support for .NET 10+, migrate to ISA groupings in CPU summary [#2828](dotnet/BenchmarkDotNet#2828) - Support benchmark filtering for TestAdapter [#2662](dotnet/BenchmarkDotNet#2662) [#2788](dotnet/BenchmarkDotNet#2788) - Support non-primitive external types in `ArgumentsSource` [#2820](dotnet/BenchmarkDotNet#2820) - Enable MSBuild parallel build via `--nodeReuse:false` [#2693](dotnet/BenchmarkDotNet#2693) [#2814](dotnet/BenchmarkDotNet#2814) - Improve CPU detection [#2747](dotnet/BenchmarkDotNet#2747) [#2749](dotnet/BenchmarkDotNet#2749) - Enable assembly signing for debug build [#2774](dotnet/BenchmarkDotNet#2774) Deprecations: - Deprecated `WithNuget` [#2812](dotnet/BenchmarkDotNet#2812) Bug fixes: - Fix `InvalidOperationException` in diagnosers [#2758](dotnet/BenchmarkDotNet#2758) [#2805](dotnet/BenchmarkDotNet#2805) - Fix file detection in `NativeMemoryProfiler` [#2794](dotnet/BenchmarkDotNet#2794) [#2795](dotnet/BenchmarkDotNet#2795) - Fix long file paths issue in `EtwProfiler` [#2807](dotnet/BenchmarkDotNet#2807) [#2808](dotnet/BenchmarkDotNet#2808) - Fix log duplications in TestAdapter [#2790](dotnet/BenchmarkDotNet#2790) - Fix x86 disassembler error for net462 [#2792](dotnet/BenchmarkDotNet#2792) - Fix `IsNetCore` and `IsNativeAOT` for single-file apps without AOT [#2799](dotnet/BenchmarkDotNet#2799) - Fix density plot generation in `RPlotExporter` for latest version of R [#2809](dotnet/BenchmarkDotNet#2809) Commits viewable in [compare view](dotnet/BenchmarkDotNet@v0.15.2...v0.15.3). </details> Updated [BenchmarkDotNet.Annotations](https://github.com/dotnet/BenchmarkDotNet) from 0.15.2 to 0.15.3. <details> <summary>Release notes</summary> _Sourced from [BenchmarkDotNet.Annotations's releases](https://github.com/dotnet/BenchmarkDotNet/releases)._ ## 0.15.3 Full changelog: https://benchmarkdotnet.org/changelog/v0.15.3.html ## Highlights Improvements: - Naot instruction set support for .NET 10+, migrate to ISA groupings in CPU summary [#2828](dotnet/BenchmarkDotNet#2828) - Support benchmark filtering for TestAdapter [#2662](dotnet/BenchmarkDotNet#2662) [#2788](dotnet/BenchmarkDotNet#2788) - Support non-primitive external types in `ArgumentsSource` [#2820](dotnet/BenchmarkDotNet#2820) - Enable MSBuild parallel build via `--nodeReuse:false` [#2693](dotnet/BenchmarkDotNet#2693) [#2814](dotnet/BenchmarkDotNet#2814) - Improve CPU detection [#2747](dotnet/BenchmarkDotNet#2747) [#2749](dotnet/BenchmarkDotNet#2749) - Enable assembly signing for debug build [#2774](dotnet/BenchmarkDotNet#2774) Deprecations: - Deprecated `WithNuget` [#2812](dotnet/BenchmarkDotNet#2812) Bug fixes: - Fix `InvalidOperationException` in diagnosers [#2758](dotnet/BenchmarkDotNet#2758) [#2805](dotnet/BenchmarkDotNet#2805) - Fix file detection in `NativeMemoryProfiler` [#2794](dotnet/BenchmarkDotNet#2794) [#2795](dotnet/BenchmarkDotNet#2795) - Fix long file paths issue in `EtwProfiler` [#2807](dotnet/BenchmarkDotNet#2807) [#2808](dotnet/BenchmarkDotNet#2808) - Fix log duplications in TestAdapter [#2790](dotnet/BenchmarkDotNet#2790) - Fix x86 disassembler error for net462 [#2792](dotnet/BenchmarkDotNet#2792) - Fix `IsNetCore` and `IsNativeAOT` for single-file apps without AOT [#2799](dotnet/BenchmarkDotNet#2799) - Fix density plot generation in `RPlotExporter` for latest version of R [#2809](dotnet/BenchmarkDotNet#2809) Commits viewable in [compare view](dotnet/BenchmarkDotNet@v0.15.2...v0.15.3). </details> Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) </details> Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Fix generated code breaking when using ArgumentsSource on an external…

2ed2eb6

… type

IntroArgumentsSource: use static type for external arguments

cbc6b0a

Add more test cases for ArgumentsSource in external class

9438017

Catches some issues not previously caught.

JimmyCushnie changed the title ~~Fix ArgumentsSource on external types not actually working~~ Fix ArgumentsSource on external types not working if the argument type is not primitive Aug 23, 2025

Add test cases for ParamsSource with non-primitive param types

527d598

timcassell reviewed Aug 23, 2025

View reviewed changes

src/BenchmarkDotNet/Parameters/SmartParamBuilder.cs Outdated Show resolved Hide resolved

timcassell added the Area:CodeGen label Aug 23, 2025

Fix casts

97bd914

Co-authored-by: Tim Cassell <[email protected]>

timcassell approved these changes Aug 23, 2025

View reviewed changes

timcassell merged commit e704adb into dotnet:master Aug 23, 2025
14 of 16 checks passed

timcassell added this to the v0.15.3 milestone Aug 23, 2025

JimmyCushnie deleted the fix-broken-crap branch August 23, 2025 13:12

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

Fix ArgumentsSource on external types not working if the argument type is not primitive #2820

Fix ArgumentsSource on external types not working if the argument type is not primitive #2820

JimmyCushnie commented Aug 14, 2025 •

edited

Loading

Uh oh!

filzrev commented Aug 15, 2025

Uh oh!

JimmyCushnie commented Aug 15, 2025

Uh oh!

timcassell commented Aug 19, 2025

Uh oh!

JimmyCushnie commented Aug 22, 2025

Uh oh!

timcassell commented Aug 22, 2025

Uh oh!

JimmyCushnie commented Aug 23, 2025 •

edited

Loading

Uh oh!

timcassell commented Aug 23, 2025

Uh oh!

JimmyCushnie commented Aug 23, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Fix ArgumentsSource on external types not working if the argument type is not primitive #2820

Fix ArgumentsSource on external types not working if the argument type is not primitive #2820

Conversation

JimmyCushnie commented Aug 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

The issue

The fix

I am worried about external instance methods

Other things I am worried about

Uh oh!

filzrev commented Aug 15, 2025

Uh oh!

JimmyCushnie commented Aug 15, 2025

Uh oh!

timcassell commented Aug 19, 2025

Uh oh!

JimmyCushnie commented Aug 22, 2025

Uh oh!

timcassell commented Aug 22, 2025

Uh oh!

JimmyCushnie commented Aug 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

timcassell commented Aug 23, 2025

Uh oh!

JimmyCushnie commented Aug 23, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

JimmyCushnie commented Aug 14, 2025 •

edited

Loading

JimmyCushnie commented Aug 23, 2025 •

edited

Loading