File-based programs editor features

Related discussion: #79275 (comment) and below in that PR

This PR adds a project Microsoft.CodeAnalysis.FileBasedPrograms.Features. This DLL ships with the language server only, not with Visual Studio. The new editor features are being added to the existing MS.CA.CSharp.Features.dll. I created a small completion provider for #:property to prove out the viability of the approach.

This is planned to include stuff like:

• CompletionProviders for #:package/#:property/#:project, etc., including the directive "kinds" as well as file path and package name/version completion.
• DiagnosticAnalyzers/fixers for the same. See Live file-based program directive diagnostics #80006.

There has been some concern about where the types that provide this functionality should physically live. In the discussion linked above I previously outlined some options. I'll try to reiterate here as that discussion was a while ago.

Where to put FileBasedPrograms.Features

I think the Roslyn repo is the best place for this project to live at this point. Shipping from Roslyn means all we need to do is add the ProjectReference in the LanguageServer, and the components we want will simply land in the language server. This minimizes the need for cross-repo dependency flow.

The other candidate location we had in mind was the dotnet/sdk repo. Shipping from there would make it difficult to ship updates to these features. When we fix bugs for these features, we would have to start considering whether to take fixes for them thru SDK servicing and so on. This represents a difficulty/risk which seems avoidable. See also #79275 (comment).

One reason dotnet/sdk was interesting as a candidate was that we could share the implementation code which reports directive diagnostics for dotnet run app.cs. Due to the difficulty represented by servicing the editor features in the SDK, I think the best way forward for sharing, would be to publish a source-only package from the SDK, which is consumed in the Roslyn repo, and includes API we can call from an analyzer defined in the FileBasedPrograms.Features to diagnose problems with the directives. This is something previously discussed with @jjonescz.

The aforementioned source package may also provide the ability to obtain the virtual project itself and remove the need to spawn a subprocess for dotnet run-api from FileBasedProgramsProjectSystem in the language server. In other words, this is a dependency which we have reason for wanting to create, even without considering the editor features scenario. This is also an area where we previously expressed being comfortable with IDE aligning with the "latest SDK behavior" for this feature area, rather than attempting to support as many combinations as possible of supported SDKs and C# extension versions. Tagging @jasonmalinowski @jaredpar regarding that particular line of thinking.

Use of internal APIs

It was previously suggested that the file-based programs editor features should use the Roslyn public API surface only: #79275 (comment)

This approach poses a few problems, outlined below. This PR opts to add IVTs to some similar areas where CSharp.Features project already holds IVT.

Testing is difficult

• There are no public helpers for testing completion providers. Enable testing of custom CompletionProvider roslyn-sdk#880
• aspnetcore is one of our primary examples for how to ship third-party completion providers. It looks like they copy-paste-edited parts of Roslyn's internals in order to do so. See for example Roslyn's MarkupTestFile versus aspnetcore's MarkupTestFile.

The solution in this PR just tests the new features in the same CSharp.Features.UnitTests project as the built-in ones. I did this because identifying and copying the subset of internal API needed for the tests did not feel like a productive activity when we are already shipping out of the Roslyn repo. We can also consider breaking out a separate test project, which has IVT to the helpers that it needs.

Useful implementation APIs are internal

Here are some examples of APIs which would either require copy-pasting or searching for an alternative solution.

• internal class LSPCompletionProvider provides TriggerCharacters which are aggregated and forwarded to the LSP client.
  • Per discussion with @dibarbet we think we could get away with not subtyping this, because there are already built-in providers which trigger completion on :. We may want to consider exposing some public API for completion providers to contribute TriggerCharacters, but, the main reason for that would perhaps be the principle of it.
• SyntaxToken FindTokenOnLeftOfPosition(SyntaxNode, int)
• CompletionUtilities.IsStartingNewWord
• Possibly some or all of these APIs are available via source packages a la what are used in Have RoslynAnalyzers reference shared utilities/extensions from Roslyn layer itself. #79030.

As implementation work continues more of these cases are likely to be found. In general these can be solved, but, it's not clear to me if doing that work "up front" offers a compelling benefit. Essentially, in the future, if there is hypothetically a need to migrate the features to a different repo, then we would have to do the work to move from IVTs to source packages or similar solution.

I am hoping to get your feedback on whether this general direction is reasonable, and to address any concerns, so that we can move forward with this work. Thanks!

I would also like to emphasize that finding the ideal home for things, has been particularly difficult with this feature. We have "project system" code in the language server, which is used for FBPs, which theoretically would be better placed in the actual project system repo. We have code for parsing/diagnosing directives and creating the virtual project, which we might prefer to live in msbuild rather than dotnet/sdk. For all of these, and the editor features components, we have some ability to migrate where these components live over time, particularly as we move from building out the core end-to-end experience, to making things work more broadly toward the corners of the .NET platform.