Fail on invalid search options, add test

dotnet · elinor-fung · Jul 18, 2024 · Jul 10, 2024 · Jul 10, 2024 · Jul 11, 2024
commit 899e71c36565a339932dd0be41d967092d3932fc
diff --git a/docs/design/features/host-error-codes.md b/docs/design/features/host-error-codes.md
@@ -31,7 +31,7 @@ Note that the exit code returned by running an application via `dotnet.exe` or `
 | `LibHostInvalidArgs`                  | `0x80008092` | `-2147450734`          | `146`         | Arguments to `hostpolicy` are invalid. This is used in three unrelated places in the `hostpolicy`, but in all cases it means the component calling `hostpolicy` did something wrong: <ul><li> Command line arguments for the app - the failure would typically mean that wrong argument was passed or such. For example if the application main assembly is not specified on the command line. On its own this should not happen as `hostfxr` should have parsed and validated all command line arguments. </li><li> `hostpolicy` context's `get_delegate` - if the requested delegate enum value is not recognized. Again this would mean `hostfxr` passed the wrong value. </li><li> `corehost_resolve_component_dependencies` - if something went wrong initializing `hostpolicy` internal structures. Would happen for example when the `component_main_assembly_path` argument is wrong. </li></ul> |
 | `InvalidConfigFile`                   | `0x80008093` | `-2147450733`          | `147`         | The `.runtimeconfig.json` file is invalid. The reasons for this failure can be among these: <ul><li> Failure to read from the file </li><li> Invalid JSON </li><li> Invalid value for a property (for example number for property which requires a string) </li><li> Missing required property </li><li> Other inconsistencies (for example `rollForward` and `applyPatches` are not allowed to be specified in the same config file) </li><li> Any of the above failures reading the `.runtimecofig.dev.json` file </li><li> Self-contained `.runtimeconfig.json` used in `hostfxr_initialize_for_runtime_config`. Note that missing `.runtimconfig.json` is not an error (means self-contained app). This error code is also used when there is a problem reading the CLSID map file in `comhost`. </li></ul> |
 | `AppArgNotRunnable`                   | `0x80008094` | `-2147450732`          | `148`         | Used internally when the command line for `dotnet.exe` doesn't contain path to the application to run. In such case the command line is considered to be a CLI/SDK command. This error code should never be returned to external caller. |
-| `AppHostExeNotBoundFailure`           | `0x80008095` | `-2147450731`          | `149`         | `apphost` failed to determine which application to run. This can mean: <ul><li> The `apphost` binary has not been imprinted with the path to the app to run (so freshly built `apphost.exe` from the branch will fail to run like this) </li><li> The `apphost` is a bundle (single-file exe) and it failed to extract correctly. </li></ul> |
+| `AppHostExeNotBoundFailure`           | `0x80008095` | `-2147450731`          | `149`         | `apphost` failed to determine which application to run. This can mean: <ul><li> The `apphost` binary has not been imprinted with the path to the app to run (so freshly built `apphost.exe` from the branch will fail to run like this) </li><li> The `apphost` is a bundle (single-file exe) and it failed to extract correctly. </li><li>The `apphost` binary has been imprinted with invalid .NET search options</li></ul> |
 | `FrameworkMissingFailure`             | `0x80008096` | `-2147450730`          | `150`         | It was not possible to find a compatible framework version. This originates in `hostfxr` (`resolve_framework_reference`) and means that the app specified a reference to a framework in its `.runtimeconfig.json` which could not be resolved. The failure to resolve can mean that no such framework is available on the disk, or that the available frameworks don't match the minimum version specified or that the roll forward options specified excluded all available frameworks. Typically this would be used if a 3.0 app is trying to run on a machine which has no 3.0 installed. It would also be used for example if a 32bit 3.0 app is running on a machine which has 3.0 installed but only for 64bit. |
 | `HostApiFailed`                       | `0x80008097` | `-2147450729`          | `151`         | Returned by `hostfxr_get_native_search_directories` if the `hostpolicy` could not calculate the `NATIVE_DLL_SEARCH_DIRECTORIES`. |
 | `HostApiBufferTooSmall`               | `0x80008098` | `-2147450728`          | `152`         | Returned when the buffer specified to an API is not big enough to fit the requested value. Can be returned from: <ul><li> `hostfxr_get_runtime_properties` </li><li> `hostfxr_get_native_search_directories` </li><li> `get_hostfxr_path` </li></ul> |

diff --git a/src/installer/managed/Microsoft.NET.HostModel/AppHost/HostWriter.cs b/src/installer/managed/Microsoft.NET.HostModel/AppHost/HostWriter.cs
@@ -293,7 +293,7 @@ private static byte[] GetSearchOptionBytes(DotNetSearchOptions searchOptions)
                 throw new AppRelativePathTooLongException(searchOptions.AppRelativeDotNet, MaxAppRelativeDotNetSizeInBytes);
 
             // <search_location> 0 <app_relative_dotnet_root> 0
-            byte[] searchOptionsBytes = new byte[pathBytes.Length + 3]; // +2 for search location + null, +2 for null terminator
+            byte[] searchOptionsBytes = new byte[pathBytes.Length + 3]; // +2 for search location + null, +1 for null terminator
             searchOptionsBytes[0] = (byte)searchOptions.Location;
             searchOptionsBytes[1] = 0;
             searchOptionsBytes[searchOptionsBytes.Length - 1] = 0;

diff --git a/src/installer/tests/HostActivation.Tests/FrameworkDependentAppLaunch.cs b/src/installer/tests/HostActivation.Tests/FrameworkDependentAppLaunch.cs
@@ -317,10 +317,8 @@ public void AppHost_CLI_MissingRuntimeFramework_ErrorReportedInStdErr(bool missi
                 result.Should().Fail()
                     .And.HaveStdErrContaining($"https://aka.ms/dotnet-core-applaunch?{expectedUrlQuery}")
                     .And.HaveStdErrContaining($"&rid={TestContext.BuildRID}")
-                    .And.HaveStdErrContaining(expectedStdErr);
-
-                // Some Unix systems will have 8 bit exit codes.
-                Assert.True(result.ExitCode == expectedErrorCode || result.ExitCode == (expectedErrorCode & 0xFF));
+                    .And.HaveStdErrContaining(expectedStdErr)
+                    .And.ExitWith(expectedErrorCode);
             }
         }
 

diff --git a/src/installer/tests/HostActivation.Tests/InstallLocation.cs b/src/installer/tests/HostActivation.Tests/InstallLocation.cs
@@ -344,6 +344,23 @@ public void SearchOptions(SearchLocation searchLocation)
             }
         }
 
+        [Fact]
+        public void AppHost_AppRelative_MissingPath()
+        {
+            TestApp app = sharedTestState.App.Copy();
+            app.CreateAppHost(dotNetRootOptions: new HostWriter.DotNetSearchOptions()
+            {
+                Location = SearchLocation.AppRelative
+            });
+            Command.Create(app.AppExe)
+                .CaptureStdErr()
+                .CaptureStdOut()
+                .Execute()
+                .Should().Fail()
+                .And.HaveStdErrContaining("The app-relative .NET path is not embedded.")
+                .And.ExitWith(Constants.ErrorCode.AppHostExeNotBoundFailure);
+        }
+
         [Theory]
         [InlineData("./dir")]
         [InlineData("../dir")]

diff --git a/src/installer/tests/HostActivation.Tests/InvalidHost.cs b/src/installer/tests/HostActivation.Tests/InvalidHost.cs
@@ -30,19 +30,8 @@ public void AppHost_NotBound()
                 .Execute(expectedToFail: true);
 
             result.Should().Fail()
-                .And.HaveStdErrContaining("This executable is not bound to a managed DLL to execute.");
-
-            int exitCode = result.ExitCode;
-            const int AppHostExeNotBoundFailure = unchecked((int)0x80008095);
-            if (OperatingSystem.IsWindows())
-            {
-                exitCode.Should().Be(AppHostExeNotBoundFailure);
-            }
-            else
-            {
-                // Some Unix flavors filter exit code to ubyte.
-                (exitCode & 0xFF).Should().Be(AppHostExeNotBoundFailure & 0xFF);
-            }
+                .And.HaveStdErrContaining("This executable is not bound to a managed DLL to execute.")
+                .And.ExitWith(Constants.ErrorCode.AppHostExeNotBoundFailure);
         }
 
         [Fact]
@@ -84,19 +73,8 @@ public void DotNet_Renamed()
                 .Execute(expectedToFail: true);
 
             result.Should().Fail()
-                .And.HaveStdErrContaining($"Error: cannot execute dotnet when renamed to {Path.GetFileNameWithoutExtension(sharedTestState.RenamedDotNet)}");
-
-            int exitCode = result.ExitCode;
-            const int EntryPointFailure = unchecked((int)0x80008084);
-            if (OperatingSystem.IsWindows())
-            {
-                exitCode.Should().Be(EntryPointFailure);
-            }
-            else
-            {
-                // Some Unix flavors filter exit code to ubyte.
-                (exitCode & 0xFF).Should().Be(EntryPointFailure & 0xFF);
-            }
+                .And.HaveStdErrContaining($"Error: cannot execute dotnet when renamed to {Path.GetFileNameWithoutExtension(sharedTestState.RenamedDotNet)}")
+                .And.ExitWith(Constants.ErrorCode.EntryPointFailure);
         }
 
         public class SharedTestState : IDisposable

diff --git a/src/installer/tests/TestUtils/Assertions/CommandResultAssertions.cs b/src/installer/tests/TestUtils/Assertions/CommandResultAssertions.cs
@@ -21,6 +21,10 @@ public CommandResultAssertions(CommandResult commandResult)
 
         public AndConstraint<CommandResultAssertions> ExitWith(int expectedExitCode)
         {
+            // Some Unix systems will have 8 bit exit codes
+            if (!OperatingSystem.IsWindows())
+                expectedExitCode = expectedExitCode & 0xFF;
+
             Execute.Assertion.ForCondition(Result.ExitCode == expectedExitCode)
                 .FailWith($"Expected command to exit with {expectedExitCode} but it did not.{GetDiagnosticsInfo()}");
             return new AndConstraint<CommandResultAssertions>(this);

diff --git a/src/installer/tests/TestUtils/Constants.cs b/src/installer/tests/TestUtils/Constants.cs
@@ -116,11 +116,13 @@ public static class ErrorCode
             public const int Success = 0;
             public const int InvalidArgFailure = unchecked((int)0x80008081);
             public const int CoreHostLibMissingFailure = unchecked((int)0x80008083);
+            public const int EntryPointFailure = unchecked((int)0x80008084);
             public const int ResolverInitFailure = unchecked((int)0x8000808b);
             public const int ResolverResolveFailure = unchecked((int)0x8000808c);
             public const int LibHostInvalidArgs = unchecked((int)0x80008092);
             public const int InvalidConfigFile = unchecked((int)0x80008093);
             public const int AppArgNotRunnable = unchecked((int)0x80008094);
+            public const int AppHostExeNotBoundFailure = unchecked((int)0x80008095);
             public const int FrameworkMissingFailure = unchecked((int)0x80008096);
             public const int FrameworkCompatFailure = unchecked((int)0x8000809c);
             public const int BundleExtractionFailure = unchecked((int)0x8000809f);

diff --git a/src/native/corehost/apphost/standalone/hostfxr_resolver.cpp b/src/native/corehost/apphost/standalone/hostfxr_resolver.cpp
@@ -17,6 +17,8 @@ namespace
     // <fxr_resolver::search_location_default> \0 <app_relative_dotnet_placeholder>
     #define EMBED_DOTNET_SEARCH_FULL_UTF8    ("\0\0" EMBED_DOTNET_SEARCH_HI_PART_UTF8 EMBED_DOTNET_SEARCH_LO_PART_UTF8)
 
+    // Get the .NET search options that should be used
+    // Returns false if options are invalid - for example, app-relative search was specified, but the path is invalid or not embedded
     bool try_get_dotnet_search_options(fxr_resolver::search_location& out_search_location, pal::string_t& out_app_relative_dotnet)
     {
         constexpr int EMBED_SIZE = 512;
@@ -27,46 +29,46 @@ namespace
 
         out_search_location = (fxr_resolver::search_location)embed[0];
         assert(embed[1] == 0); // NUL separates the search location and embedded .NET root value
-        bool is_configured = out_search_location != fxr_resolver::search_location_default;
-        if ((out_search_location & fxr_resolver::search_location_app_relative) != 0)
+        if ((out_search_location & fxr_resolver::search_location_app_relative) == 0)
+            return true;
+
+        // Get the embedded app-relative .NET path
+        std::string binding(&embed[2]); // Embedded path is null-terminated
+
+        // Check if the path exceeds the max allowed size
+        constexpr int EMBED_APP_RELATIVE_DOTNET_MAX_SIZE = EMBED_SIZE - 3; // -2 for search location + null, -1 for null terminator
+        if (binding.size() > EMBED_APP_RELATIVE_DOTNET_MAX_SIZE)
         {
-            std::string binding(&embed[2]); // Embedded path is null-terminated
-
-            // Check if the path exceeds the max allowed size
-            constexpr int EMBED_APP_RELATIVE_DOTNET_MAX_SIZE = EMBED_SIZE - 3; // -2 for search location + null, -1 for null terminator
-            if (binding.size() > EMBED_APP_RELATIVE_DOTNET_MAX_SIZE)
-            {
-                trace::verbose(_X("The app-relative .NET path is longer than the max allowed length (%d)"), EMBED_APP_RELATIVE_DOTNET_MAX_SIZE);
-                return is_configured;
-            }
-
-            // Check if the value is the same as the placeholder
-            // Since the single static string is replaced by editing the executable, a reference string is needed to do the compare.
-            // So use two parts of the string that will be unaffected by the edit.
-            static const char hi_part[] = EMBED_DOTNET_SEARCH_HI_PART_UTF8;
-            static const char lo_part[] = EMBED_DOTNET_SEARCH_LO_PART_UTF8;
-            size_t hi_len = (sizeof(hi_part) / sizeof(hi_part[0])) - 1;
-            size_t lo_len = (sizeof(lo_part) / sizeof(lo_part[0])) - 1;
-            if (binding.size() >= (hi_len + lo_len)
+            trace::error(_X("The app-relative .NET path is longer than the max allowed length (%d)"), EMBED_APP_RELATIVE_DOTNET_MAX_SIZE);
+            return false;
+        }
+
+        // Check if the value is empty or the same as the placeholder
+        // Since the single static string is replaced by editing the executable, a reference string is needed to do the compare.
+        // So use two parts of the string that will be unaffected by the edit.
+        static const char hi_part[] = EMBED_DOTNET_SEARCH_HI_PART_UTF8;
+        static const char lo_part[] = EMBED_DOTNET_SEARCH_LO_PART_UTF8;
+        size_t hi_len = (sizeof(hi_part) / sizeof(hi_part[0])) - 1;
+        size_t lo_len = (sizeof(lo_part) / sizeof(lo_part[0])) - 1;
+        if (binding.empty()
+            || (binding.size() >= (hi_len + lo_len)
                 && binding.compare(0, hi_len, &hi_part[0]) == 0
-                && binding.compare(hi_len, lo_len, &lo_part[0]) == 0)
-            {
-                trace::verbose(_X("The app-relative .NET path is not embedded."));
-                return is_configured;
-            }
-
-            pal::string_t app_relative_dotnet;
-            if (!pal::clr_palstring(binding.c_str(), &app_relative_dotnet))
-            {
-                trace::error(_X("The app-relative .NET path could not be retrieved from the executable image."));
-                return is_configured;
-            }
-
-            trace::info(_X("Embedded app-relative .NET path: '%s'"), app_relative_dotnet.c_str());
-            out_app_relative_dotnet = std::move(app_relative_dotnet);
+                && binding.compare(hi_len, lo_len, &lo_part[0]) == 0))
+        {
+            trace::error(_X("The app-relative .NET path is not embedded."));
+            return false;
         }
 
-        return is_configured;
+        pal::string_t app_relative_dotnet;
+        if (!pal::clr_palstring(binding.c_str(), &app_relative_dotnet))
+        {
+            trace::error(_X("The app-relative .NET path could not be retrieved from the executable image."));
+            return false;
+        }
+
+        trace::info(_X("Embedded app-relative .NET path: '%s'"), app_relative_dotnet.c_str());
+        out_app_relative_dotnet = std::move(app_relative_dotnet);
+        return true;
     }
 }
 
@@ -99,14 +101,17 @@ hostfxr_resolver_t::hostfxr_resolver_t(const pal::string_t& app_root)
     fxr_resolver::search_location search_location = fxr_resolver::search_location_default;
     pal::string_t app_relative_dotnet;
     pal::string_t app_relative_dotnet_path;
-    if (try_get_dotnet_search_options(search_location, app_relative_dotnet))
+    if (!try_get_dotnet_search_options(search_location, app_relative_dotnet))
     {
-        trace::info(_X(".NET root search location options: %d"), search_location);
-        if (!app_relative_dotnet.empty())
-        {
-            app_relative_dotnet_path = app_root;
-            append_path(&app_relative_dotnet_path, app_relative_dotnet.c_str());
-        }
+        m_status_code = StatusCode::AppHostExeNotBoundFailure;
+        return;
+    }
+
+    trace::info(_X(".NET root search location options: %d"), search_location);
+    if (!app_relative_dotnet.empty())
+    {
+        app_relative_dotnet_path = app_root;
+        append_path(&app_relative_dotnet_path, app_relative_dotnet.c_str());
     }
 
     if (!fxr_resolver::try_get_path(app_root, search_location, &app_relative_dotnet_path, &m_dotnet_root, &m_fxr_path))

diff --git a/src/native/corehost/corehost.cpp b/src/native/corehost/corehost.cpp
@@ -64,7 +64,7 @@ bool is_exe_enabled_for_execution(pal::string_t* app_dll)
     // Check if the path exceeds the max allowed size
     if (binding.size() > EMBED_MAX - 1) // -1 for null terminator
     {
-        trace::verbose(_X("The managed DLL bound to this executable is longer than the max allowed length (%d)"), EMBED_MAX - 1);
+        trace::error(_X("The managed DLL bound to this executable is longer than the max allowed length (%d)"), EMBED_MAX - 1);
         return false;
     }