Thanks to visit codestin.com
Credit goes to github.com

Skip to content

BinaryFormatter serialization and deserialization warning which throws an error in .net8 #2282

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
RobertClapham opened this issue Nov 16, 2023 · 13 comments

Comments

@RobertClapham
Copy link

RobertClapham commented Nov 16, 2023

Environment

  • Pythonnet version: 3.0.3
  • Python version: 3.9
  • Operating System: Windows 11
  • .NET Runtime: .net8 (release version 16/11/23)

Details

  • When shutting down the the PythonEngine via a Dispose method on the impermentation class, the error occurs at the PythonEngine.Shutdown(); method and throws the error System.NotSupportedException: BinaryFormatter serialization and deserialization are disabled within this application (see trace below). The same code worked in .net7 without any issues, the error message leads to this description

  • https://learn.microsoft.com/en-us/dotnet/standard/serialization/binaryformatter-security-guide

  • Proir to the Shutdown error the pythonnet works correctly and calls python functions and returns the expected results

  • The resolution is to add true to the .csproj file

  • <PropertyGroup>
      <TargetFramework>net8.0</TargetFramework>
      <EnableUnsafeBinaryFormatterSerialization>true</EnableUnsafeBinaryFormatterSerialization>
     </PropertyGroup>
    

    TODO

  • _logger.Log is to to trace the point of failure

  • _logger.Log(LogLevel.Error, "Shutting down Python engine"); runs

  • it crashes at the PythonEngine.Shutdown(); method

  • _logger.Log(LogLevel.Error, "Python engine shut down"); does not run

    private void Dispose(bool disposing)
    {
    _logger.Log(LogLevel.Error, "Dispose");
    if (_disposed) return;
    if (disposing)
    {
    _logger.Log(LogLevel.Error, "Shutting down Python engine");
    PythonEngine.Shutdown();
    _logger.Log(LogLevel.Error, "Python engine shut down");
    }
    _disposed = true;
    }

    print('TODO')
  • System.NotSupportedException: BinaryFormatter serialization and deserialization are disabled within this application. See https://aka.ms/binaryformatt...

System.NotSupportedException
BinaryFormatter serialization and deserialization are disabled within this application. See https://aka.ms/binaryformatter for more information.
at System.Runtime.Serialization.Formatters.Binary.BinaryFormatter.Serialize(Stream serializationStream, Object graph)
at Python.Runtime.RuntimeData.Stash()
at Python.Runtime.Runtime.Shutdown()
at Python.Runtime.PythonEngine.Shutdown()
at EcgAi.Training.Infrastructure.Python.PythonEnvironment.Dispose(Boolean disposing) in C:\Data\Rider\EcgAi.Training\Src\EcgAi.Training.Infrastructure.Python\PythonEnvironment.cs:line 113
at EcgAi.Training.Infrastructure.Python.PythonEnvironment.Dispose() in xxxxxxxxxxxxxxxxxxxxxxxxx

    print('TODO')
@TimIrwin26
Copy link

The issue happens in RuntimeData.Stash --

    var runtimeStorage = new PythonNetState
    {
        Metatype = MetaType.SaveRuntimeData(),
        ImportHookState = ImportHook.SaveRuntimeData(),
        Types = TypeManager.SaveRuntimeData(),
        Classes = ClassManager.SaveRuntimeData(),
        SharedObjects = SaveRuntimeDataObjects(),
    };

    IFormatter formatter = CreateFormatter();
    var ms = new MemoryStream();
    formatter.Serialize(ms, runtimeStorage);
   internal static IFormatter CreateFormatter()
   {
       return FormatterType != null ?
           (IFormatter)Activator.CreateInstance(FormatterType)
           : new BinaryFormatter();
   }

While there is an option to set another formatter, it's not used and expected to be of type IFormatter which is problematic.

@fsz1987
Copy link

fsz1987 commented Jan 1, 2024

Pythonnet use BinaryFormatter. and it is deprecated in .net 8.
create an instance of BinaryFormatter will cause an error.

you can set RuntimeData.FormatterType to your custom formatter to get it work.

if you dont want write a custom formatter, then add this line

AppContext.SetSwitch("System.Runtime.Serialization.EnableUnsafeBinaryFormatterSerialization", true);

before you call PythonEngine.Shutdown to temporary enable BinaryFormatter.

after PythonEngine.Shutdown. call

AppContext.SetSwitch("System.Runtime.Serialization.EnableUnsafeBinaryFormatterSerialization", false);

to disable BinaryFormatter

@amishratnasthapit
Copy link

Facing same issue and using work around currently. But .NET 9 is getting rid of Binary Formatter altogether. So this needs to be fixed.

@AtwoodTM
Copy link

This is a big issue as the binary formatter was deprecated as it is a serious security issue.

To replace the formatter, there looks to be two high performance/low latency alternatives... Protobuf and MessagePack. Any preference which to use (MessagePack will be slightly faster than Protobuf, but Protobuf is more well known)?

@lostmsu
Copy link
Member

lostmsu commented Aug 31, 2024

No formatter is really needed unless you are using .NET Framework. The only place where formatter is used is app domain restart, and app domains are not available in later .NETs

@filmor
Copy link
Member

filmor commented Sep 19, 2024

This should be fixed now.

@filmor filmor closed this as completed Sep 19, 2024
@nwoolls
Copy link

nwoolls commented Oct 21, 2024

@filmor is this fix available on NuGet? I'm still seeing this issue with the latest 3.1 preview version on NuGet.

@Lemur8063
Copy link

@filmor is this fix available on NuGet? I'm still seeing this issue with the latest 3.1 preview version on NuGet.

no

mmsbld pushed a commit to mmsbld/LLMR that referenced this issue Nov 3, 2024
…nEngine.Shutdown() method from python.net relies on the deprecated (and highly risky!) BinaryFormatter. Until this is fixxed, we can only initialize once, presumably this will change in the very near future: pythonnet/pythonnet#2282
@Baumx1234
Copy link

I'm using Nuget Version 3.0.5 and .Net Version 8.0 and still got the issue.

@filmor filmor reopened this Dec 17, 2024
@filmor
Copy link
Member

filmor commented Dec 17, 2024

Yeah, this is more difficult to fix than anticipated.

@WallucePinkham
Copy link

Is there an update on this issues, by any chance?

@leftos
Copy link

leftos commented Feb 10, 2025

I was struggling with this, tried to use all sorts of BinaryFormatter NuGet packages to get pythonnet to be happy, and nothing worked...

But then I came across this:
https://learn.microsoft.com/en-us/dotnet/core/tutorials/creating-app-with-plugin-support

All I had to do was implement that PluginLoadContext class...
(note that I changed it ever so slightly from the example in the linked article to enable unloading which is off by default, because of its associated cost)

    class PluginLoadContext : AssemblyLoadContext
    {
        private AssemblyDependencyResolver _resolver;

        public PluginLoadContext(string pluginPath) : base(true) // passing true to the base constructor enables unloading
        {
            _resolver = new AssemblyDependencyResolver(pluginPath);
        }

        protected override Assembly Load(AssemblyName assemblyName)
        {
            string assemblyPath = _resolver.ResolveAssemblyToPath(assemblyName);
            if (assemblyPath != null)
            {
                return LoadFromAssemblyPath(assemblyPath);
            }

            return null;
        }

        protected override IntPtr LoadUnmanagedDll(string unmanagedDllName)
        {
            string libraryPath = _resolver.ResolveUnmanagedDllToPath(unmanagedDllName);
            if (libraryPath != null)
            {
                return LoadUnmanagedDllFromPath(libraryPath);
            }

            return IntPtr.Zero;
        }
    }

...and then change up my code a bit to have a PluginLoadContext instance and use that to load the assembly and unload it (using IDisposable), and I was finally able to keep reusing the same assembly, without pythonnet throwing an exception that it's still initialized when I would invoke the assembly's constructor on the very first Runtime.PythonDLL = ... line.

    public class PythonTTSProvider : ITTSProvider // ITTSProvider : IDisposable
    {
        private readonly dynamic _pythonTTSLib;
        private static PluginLoadContext? _pluginLoadContext;

        private PythonTTSProvider(dynamic pythonTTSLib)
        {
            _pythonTTSLib = pythonTTSLib;

           // ...
        }

        public static ITTSProvider? New()
        {
            string assemblyPath = Path.Combine(AppDomain.CurrentDomain.BaseDirectory, @"PythonTTS\PythonTTS.dll");
            _pluginLoadContext = new(assemblyPath);
            dynamic? ttsLib = null;
            if (File.Exists(assemblyPath))
            {
                var assembly = _pluginLoadContext.LoadFromAssemblyPath(assemblyPath);
                var type = assembly.GetType("PythonTTS.KokoroTTS");
                if (type != null)
                {
                    ttsLib = Activator.CreateInstance(type, [Path.Combine(AppDomain.CurrentDomain.BaseDirectory, @"PythonTTS\python3.12.9\python312.dll")]);
                }
            }
            else
            {
                throw new FileNotFoundException($"PythonTTS library not found. Please ensure you have extracted the PythonTTS archive.\n\nFile path expected: {assemblyPath}");
            }
            return ttsLib != null ? new PythonTTSProvider(ttsLib) : null;
        }

        // ...

        public void Dispose()
        {
            _pluginLoadContext?.Unload();
            GC.SuppressFinalize(this);
        }

I was so happy to finally get this to work after 2 days of struggling with it.

Hopefully this helps someone else as well.

@sharpSteff
Copy link

sharpSteff commented Feb 11, 2025

I created a custom JsonBinaryFormater, based on good old Newtonsoft.Json for this issue:

public class JsonBinaryFormatter : IFormatter
{
    private readonly JsonSerializer_serializer;

    public JsonFormatter()
    {
        _serializer = new JsonSerializer();
    }

    /// <inheritdoc />
    public SerializationBinder Binder { get; set; }

    /// <inheritdoc />
    public StreamingContext Context { get; set; }

    /// <inheritdoc />
    public ISurrogateSelector SurrogateSelector { get; set; }

    /// <inheritdoc />
    public object Deserialize(Stream serializationStream)
    {
         using var sr = new StreamReader(stream, Encoding.UTF8, true, 1024, true);
         using var jsonTextReader = new JsonTextReader(sr);
         return _serializer.Deserialize(jsonTextReader);
    }

    /// <inheritdoc />
    public void Serialize(Stream serializationStream, object graph)
    {
        using var sr = new StreamWriter(serializationStream, Encoding.UTF8, 1024, true);
        _serializer.Serialize(sr, graph);
    }
}

It can be used via

Python.Runtime.RuntimeData.FormatterType = typeof(JsonFormatter);

So far I have no issues

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests