@dotnet/dotnet-testing I was talking with @Evangelink and he noticed that we had a bunch of old microsoft.net.test.sdk references so I tried to update them all.

Microsoft.NET.ToolPack.Tests.GivenThatWeWantToTestPackAToolProject.Microsoft.NET.ToolPack.Tests.GivenThatWeWantToTestPackAToolProject.When_app_project_reference_a_library_it_flows_to_test_project is failing with an odd error now that I updated the test sdk version. Anything you've seen before? Any suggestions? I'm not sure why it's timing out.
Failed to negotiate protocol, waiting for response timed out after 90 seconds. This may occur due to machine slowness, please set environment variable VSTEST_CONNECTION_TIMEOUT to increase timeout.

Failed to negotiate protocol, waiting for response timed out after 90 seconds. This may occur due to machine slowness, please set environment variable VSTEST_CONNECTION_TIMEOUT to increase timeout.

This is a quite common issue with VSTest but it can have many sources. I'll try to read the logs next week.

@Evangelink pinging after the holiday. any suggestions?

From discussion with @nohwnd, it usually happens when the testhost process has crashed. We would need the diagnostic logs from when it happens (please refer to https://github.com/microsoft/vstest/blob/main/docs/troubleshooting.md)

 C:\h\w\B6040957\t\dotnetSdkTests\blslzlhx.jch\When_app_proj---CDDE0EAB\Test\XUnitTestProject.csproj : warning NU1604: Project dependency 

Microsoft.NET.Test.Sdk does not contain an inclusive lower bound. 

Include a lower bound in the dependency version to ensure consistent restore results.
[xUnit.net 00:02:01.42]         Restored C:\h\w\B6040957\t\dotnetSdkTests\blslzlhx.jch\When_app_proj---CDDE0EAB\Test\XUnitTestProject.csproj (in 25.32 sec).
[xUnit.net 00:02:01.42]       C:\h\w\B6040957\t\dotnetSdkTests\blslzlhx.jch\When_app_proj---CDDE0EAB\Test\XUnitTestProject.csproj : 
warning NU1604: 
Project dependency Microsoft.NET.Test.Sdk does not contain an inclusive lower bound. 
Include a lower bound in the dependency version to ensure consistent restore results.

log says that you did not set the lower bound for the package, so the property used for the version is not propagating.

This will reference a super old version of test.sdk, which is probably not compatible anymore and so it cannot respond to the handshake

Linking dotnet/arcade#15402

@Evangelink @nohwnd coming back to this, I pushed a change that fixes the two test failures but, it might be worth someone from test looking into the When_app_project_reference_a_library_it_flows_to_test_project testcase.

That test just downloads PortableToolWithTestProject and then runs dotnet test against it. That's an xunit test project with a Microsoft.NET.Test.Sdk package reference that references a .net tool project. When the test sdk version is 16.11.0, running "dotnet test" loads the project and runs the test (with it passing) all in a few seconds. When I had the version of Microsoft.NET.Test.Sdk updated to current, it never finishes. Not sure what it's doing during that time but it's stuck so there's something broken here.

Ahh, I figured it out. The value for the test package is used in the sdk build and not available in the tests so when I used it there, it was actually resolving as empty. When it had an empty value, this is hanging the dotnet test call and triggering the NU1604 warning. Not sure why it's hanging but much lower priority as the customer would have to mess up their test version to hit this. As long as there is a value for the Version, it seems to work fine.

I'll let @nohwnd investigate if needs be.

The failing tests look like actual failing tests that don't have anything to do with the vstest infrastructure, e.g.

Microsoft.DotNet.Cli.New.IntegrationTests.DotnetNewSearchTests.CanFilterAuthor_WithoutName(testCase: "--search --columns author --author micro")
'Author' column does not contain the criteria
at Microsoft.DotNet.Cli.New.IntegrationTests.DotnetNewSearchTests.CanFilterAuthor_WithoutName(String testCase) in D:\a_work\1\s\test\dotnet-new.Tests\DotnetNewSearchTests.cs:line 244
at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
at System.Reflection.MethodBaseInvoker.InvokeDirectByRefWithFewArgs(Object obj, Span`1 copyOfArgs, BindingFlags invokeAttr)

hopefully those can be solved by their respective owners. Or merged as is if that is known issue.