Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Bindingsgenerator stops recognizing C# types #543

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
MFinkBK opened this issue Dec 18, 2019 · 10 comments
Closed

Bindingsgenerator stops recognizing C# types #543

MFinkBK opened this issue Dec 18, 2019 · 10 comments
Milestone

Comments

@MFinkBK
Copy link

MFinkBK commented Dec 18, 2019

I have a bindings project where we bind a Brother Printer SDK library. Starting with VS 2019 16.4.0 we get warnings about missing managed types. This is the first line of about 30:

warning BG8800: Unknown parameter type System.Collections.Generic.IList<Java.Util.Locale.LanguageRange> in method Filter in managed type Java.Util.Locale.

Compiling and linking with the app project works fine, no further warnings. I can attach the whole log if needed.

I checked the msbuild output and saw the call to the generator.exe in the Xamarin Android folder (formatted for readability):

"C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\MSBuild\Xamarin\Android\generator.exe" ^
 obj\Debug\api.xml ^
 --codegen-target=XAJavaInterop1 ^
 --csdir=obj\Debug\generated\src ^
 --enumdir=obj\Debug\generated\enums ^
 --enummetadata=obj\Debug\generated\metadata ^
 --assembly=BrotherSDK.Android ^
 --ref="C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\Common7\IDE\ReferenceAssemblies\Microsoft\Framework\MonoAndroid\v1.0\mscorlib.dll" ^
 --ref="C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\Common7\IDE\ReferenceAssemblies\Microsoft\Framework\MonoAndroid\v1.0\Java.Interop.dll" ^
 --ref="C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\Common7\IDE\ReferenceAssemblies\Microsoft\Framework\MonoAndroid\v9.0\Mono.Android.dll" ^
 --ref="C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\Common7\IDE\ReferenceAssemblies\Microsoft\Framework\MonoAndroid\v1.0\System.Core.dll" ^
 --ref="C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\Common7\IDE\ReferenceAssemblies\Microsoft\Framework\MonoAndroid\v1.0\System.dll" ^
 --ref="C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\Common7\IDE\ReferenceAssemblies\Microsoft\Framework\MonoAndroid\v1.0\Facades\System.Runtime.dll" ^
 --ref="C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\Common7\IDE\ReferenceAssemblies\Microsoft\Framework\MonoAndroid\v1.0\System.Xml.dll" ^
 --fixup=Transforms\Metadata.xml ^
 --enumfields=Transforms\EnumFields.xml ^
 --enummethods=Transforms\EnumMethods.xml ^
 --api-level=28 ^
 --type-map-report=obj\Debug\generated\type-mapping.txt ^
 --global ^
 --public

When I run this command line in the project folder, I get (as expected) the same errors as in Visual Studio 2019. Now when I'm changing the path to generator.exe to use the VS 2017 generator.exe (but not changing the references to MonoAndroid assemblies), I get no warnings.

How can I fix these warnings? Or is this a regression in the generator.exe tool? Thanks!

@jpobst
Copy link
Contributor

jpobst commented Dec 18, 2019

I'm not aware of anything that should have changed this in 16.4. Can you provide your api.xml so I can try to reproduce?

@MFinkBK
Copy link
Author

MFinkBK commented Dec 18, 2019

Here's the api.xml from the obj/Debug folder.
api.zip

@jpobst
Copy link
Contributor

jpobst commented Dec 18, 2019

When you say Starting with VS 2019 16.4.0, what version were you on previously that works correctly? Are you jumping from VS2017 or were you on an earlier version of VS2019?

@jpobst
Copy link
Contributor

jpobst commented Dec 18, 2019

I think this is caused by this change: #450.

We are attempting to resolve the type

System.Collections.Generic.IList<Java.Util.Locale.LanguageRange>

but the key in the symbol table is

System.Collections.Generic.IList`1

We need to either strip more arity or less.

Is this just a warning in your project? Does it actually keep any bindings from being generated?

@MFinkBK
Copy link
Author

MFinkBK commented Dec 19, 2019

I was upgrading from VS 2019 16.3.10. It's just warning messages, but I try to be as warnings-free as possible. At the moment I have added a Jenkins console parse rule to not mark the messages as warnings.

jonpryor pushed a commit that referenced this issue Jan 7, 2020
Fixes: #543

In commit 262743b we began stripping the arity from Cecil imported
types so that we wouldn't write invalid types like
``System.Collections.Generic.IList`1<string>``, in order to correctly
emit `System.Collections.Generic.IList<string>`.

However when we try to resolve these types in the `SymbolTable` we
only look at the type name, ignoring the generic type parameters.  In
this case we are looking for `System.Collections.Generic.IList` but
the type is stored in the symbol table with the arity to disambiguate
types like ``Action`1`` and ``Action`2``.  Therefore our lookup fails
because the table key is ``System.Collections.Generic.IList`1``.

For example, if we have `LibraryA.dll` with:

	public class Foo : Java.Lang.Object {
	    public virtual void Bar (IList<string> p0) {…}
	}

And we attempt to bind a `LibraryB.jar` which references
`LibraryA.dll`, overriding `Foo.Bar()`:

	public class Foo2 : Foo {
	    public override void Bar (IList<string> p0) {…}
	}

Then `Foo2.Bar()` would elicit a BG8800 warning:

	warning BG8800: Unknown parameter type System.Collections.Generic.IList<System.String> in method Bar in managed type Foo2.

Fix type lookup so that if the type is not found in the `SymbolTable`
and there are generic type parameters, calculate the arity from the
generic type parameters and look in the table again.

That is, `System.Collections.Generic.IList<string>` is rechecked as
``System.Collections.Generic.IList`1``, which allows the type to be
found.
@MFinkBK
Copy link
Author

MFinkBK commented Jan 7, 2020

I see the commits in the master and the d16-5 branches. Is the fix contained in VS 2019 version 16.5.x then?

@jpobst
Copy link
Contributor

jpobst commented Jan 7, 2020

Yes, it will be available beginning in 16.5 Preview 2.

jonpryor added a commit to dotnet/android that referenced this issue Jan 8, 2020
jonpryor pushed a commit to dotnet/android that referenced this issue Jan 8, 2020
Fixes: dotnet/java-interop#525
Fixes: dotnet/java-interop#543

Context: dotnet/java-interop@476597b...4565369
Context: dotnet/java-interop@71afce5
Context: https://github.com/Kotlin/KEEP/blob/13b67668ccc5b4741ecc37d0dd050fd77227c035/proposals/unsigned-types.md
Context: https://kotlinlang.org/docs/reference/basic-types.html#unsigned-integers

Bumps to xamarin/Java.Interop/master@45653697

  * dotnet/java-interop@4565369: [generator] Improve generic type lookup (#552)
  * dotnet/java-interop@71afce5: [generator] Support Kotlin's unsigned types (#539)
  * dotnet/java-interop@f26bc27: [Java.Interop] use Dictionary<string, *>(StringComparer.Ordinal) (#550)

Kotlin has experimental (!) support for unsigned types, providing the
following value types:

  * `kotlin.UByte`: marshals as a Java `byte`/JNI `B`;
    equivalent to C# `byte`
  * `kotlin.UInt`: marshals as an Java `int`/JNI `I`;
    equivalent to C# `uint`
  * `kotlin.ULong`: marshals as a Java `long`/JNI `J`;
    equivalent to C# `ulong`
  * `kotlin.UShort`: marshals as a Java `short`/JNI `S`;
    equivalent to C# `ushort`

Kotlin also provides arrays of these types:

  * `kotlin.UByteArray`: marshals as a Java `byte[]`/JNI `[B`;
    equivalent to C# `byte[]`, 
  * `kotlin.UIntArray`: marshals as an Java `int[]`/JNI `[I`;
    equivalent to C# `uint[]`.
  * `kotlin.ULongArray`: marshals as a Java `long[]`/JNI `[J`;
    equivalent to C# `ulong[]`.
  * `kotlin.UShortArray`: marshals as a Java `short[]`/JNI `[S`;
    equivalent to C# `ushort[]`.

Kotlin methods which contain unsigned types are "mangled", containing
characters which cannot be part of a valid Java identifier.  As such,
they cannot be invoked from Java code.

They *can* be invoked via JNI.

To bind these, we have two options:

 1. We could ignore all members which use unsigned types, or
 2. We could present a "Java-esque" view of the methods.

(1) is potentially problematic, as it means that abstract classes and
interfaces which contain them could become unbindable, making life
more complicated than is necessarily ideal.

(2) is viable, but ugly.

Consider this Kotlin type:

	// Kotlin
	package example;
	public open class ExampleBase {
	  public fun foo(value : UInt) {}
	}

Run `javap` on the resulting output, and we observe:

	Compiled from "hello.kt"
	public class example.ExampleBase {
	  public final void foo-WZ4Q5Ns(int);
	  public example.ExampleBase();
	}

We could bind `example.ExampleBase.foo-WZ4Q5Ns(int)` in C# by
replacing invalid characters with `_`, e.g.:

	// C# Binding
	partial class ExampleBase : Java.Lang.Object {
	  [Register ("foo-WZ4Q5Ns", …)]
	  public void Foo_WZ4Q5Ns (int value) {…}
	}

...but that would be *really* ugly.

We could instead just take everything before the `-` as the method
name for the C# method name -- resulting in `ExampleBase.Foo()` --
which results in a friendlier name from C#, but opens us up to
"collisions" with method overloads:

	// Kotlin
	public open class ExampleBase2 {
	  public fun foo(value : Int) {}
	  public fun foo(value : UInt) {}
	}

The C# `ExampleBase` type can't bind *both* `ExampleBase2.foo()`
methods as `Foo(int)`.

The chosen solution is to "more natively" support Kotlin unsigned
types, which allows reasonable disambiguation:

	// C# binding
	partial class ExampleBase2 : Java.Lang.Object {
	  [Register ("foo", …)]
	  public void Foo (int value) {…}

	  [Register ("foo-WZ4Q5Ns", …)]
	  public void Foo (uint value) {…}
	}

Java.Interop added support for emitting C# binding code which supports
Kotlin unsigned types.

For `kotlin.UByte`, `kotlin.UInt`, `kotlin.ULong`, and `kotlin.UShort`,
no additional runtime changes are needed to support the binding.

The `*Array` counterparts *do* require additional runtime changes.

Consider this Kotlin type:

	// Kotlin
	class UnsignedInstanceMethods {
	  public open fun uintArrayInstanceMethod (value: UIntArray) : UIntArray {
	    return value;
	  }
	}

The binding for this method would be:

	// C# binding
	partial class UnsignedInstanceMethods {
	  [Register ("uintArrayInstanceMethod--ajY-9A", "([I)[I", "")]
	  public unsafe uint[] UintArrayInstanceMethod (uint[] value)
	  {
	    const string __id = "uintArrayInstanceMethod--ajY-9A.([I)[I";
	    IntPtr native_value = JNIEnv.NewArray ((int[])(object)value);
	    try {
	      JniArgumentValue* __args = stackalloc JniArgumentValue [1];
	      __args [0] = new JniArgumentValue (native_value);
	      var __rm = _members.InstanceMethods.InvokeNonvirtualObjectMethod (__id, this, __args);
	      return (uint[]) JNIEnv.GetArray (__rm.Handle, JniHandleOwnership.TransferLocalRef, typeof (uint));
	    } finally {
	      if (value != null) {
	        JNIEnv.DeleteLocalRef (native_value);
	      }
	    }
	  }
	}

The problem is the `JNIEnv.GetArray(…, typeof(uint))` invocation.
This eventually hits various dictionaries within `JNIEnv`, which
requires additional support so that `uint` can be marshaled properly.

Previous versions of Xamarin.Android do not contain support for
marshaling arrays of unsigned types.

As such, support for using Kotlin bindings which contain unsigned
types will require Xamarin.Android 10.2.0 and later.


~~ Installer ~~

Add `protobuf-net.dll` to our installers, as it is required to read
the metadata that Kotlin puts within `.class` files.


~~ Unit Tests ~~

On-device tests for calling Kotlin code that uses unsigned types have
been added.

As we don't currently have a way of acquiring the Kotlin compiler for
builds, the test `.jar` and Kotlin's
`org.jetbrains.kotlin.kotlin-stdlib.jar` are copied into the repo.
They are *not* redistributed.

The source of the test `.jar` is provided for future use.


~~ Warnings ~~

Kotlin unsigned types are still experimental.  If Kotlin changes how
they are represented in Java bytecode, bindings which use them will
break.

Methods which use unsigned types *cannot* be virtual, nor can they
be used in bound interfaces.  This is because Java source code is
currently used as an "intermediary" for Java <-> Managed transitions,
and Java cannot override Kotlin "mangled" methods.  If the virtual
method is declared on a class, it will be bound as a non-`virtual`
method.  If the method is on an interface, it will be bound, but the
interface will not actually be implementable; a
[XA4213 build-time error][0] will be generated.

[0]: dotnet/java-interop@3bf5333
jonpryor pushed a commit to dotnet/android that referenced this issue Jan 8, 2020
Fixes: dotnet/java-interop#525
Fixes: dotnet/java-interop#543

Context: dotnet/java-interop@71afce5
Context: https://github.com/Kotlin/KEEP/blob/13b67668ccc5b4741ecc37d0dd050fd77227c035/proposals/unsigned-types.md
Context: https://kotlinlang.org/docs/reference/basic-types.html#unsigned-integers

Kotlin has experimental (!) support for unsigned types, providing the
following value types:

  * `kotlin.UByte`: marshals as a Java `byte`/JNI `B`;
    equivalent to C# `byte`
  * `kotlin.UInt`: marshals as an Java `int`/JNI `I`;
    equivalent to C# `uint`
  * `kotlin.ULong`: marshals as a Java `long`/JNI `J`;
    equivalent to C# `ulong`
  * `kotlin.UShort`: marshals as a Java `short`/JNI `S`;
    equivalent to C# `ushort`

Kotlin also provides arrays of these types:

  * `kotlin.UByteArray`: marshals as a Java `byte[]`/JNI `[B`;
    equivalent to C# `byte[]`,
  * `kotlin.UIntArray`: marshals as an Java `int[]`/JNI `[I`;
    equivalent to C# `uint[]`.
  * `kotlin.ULongArray`: marshals as a Java `long[]`/JNI `[J`;
    equivalent to C# `ulong[]`.
  * `kotlin.UShortArray`: marshals as a Java `short[]`/JNI `[S`;
    equivalent to C# `ushort[]`.

Kotlin methods which contain unsigned types are "mangled", containing
characters which cannot be part of a valid Java identifier.  As such,
they cannot be invoked from Java code.

They *can* be invoked via JNI.

To bind these, we have two options:

 1. We could ignore all members which use unsigned types, or
 2. We could present a "Java-esque" view of the methods.

(1) is potentially problematic, as it means that abstract classes and
interfaces which contain them could become unbindable, making life
more complicated than is necessarily ideal.

(2) is viable, but ugly.

Consider this Kotlin type:

	// Kotlin
	package example;
	public open class ExampleBase {
	  public fun foo(value : UInt) {}
	}

Run `javap` on the resulting output, and we observe:

	Compiled from "hello.kt"
	public class example.ExampleBase {
	  public final void foo-WZ4Q5Ns(int);
	  public example.ExampleBase();
	}

We could bind `example.ExampleBase.foo-WZ4Q5Ns(int)` in C# by
replacing invalid characters with `_`, e.g.:

	// C# Binding
	partial class ExampleBase : Java.Lang.Object {
	  [Register ("foo-WZ4Q5Ns", …)]
	  public void Foo_WZ4Q5Ns (int value) {…}
	}

...but that would be *really* ugly.

We could instead just take everything before the `-` as the method
name for the C# method name -- resulting in `ExampleBase.Foo()` --
which results in a friendlier name from C#, but opens us up to
"collisions" with method overloads:

	// Kotlin
	public open class ExampleBase2 {
	  public fun foo(value : Int) {}
	  public fun foo(value : UInt) {}
	}

The C# `ExampleBase` type can't bind *both* `ExampleBase2.foo()`
methods as `Foo(int)`.

The chosen solution is to "more natively" support Kotlin unsigned
types, which allows reasonable disambiguation:

	// C# binding
	partial class ExampleBase2 : Java.Lang.Object {
	  [Register ("foo", …)]
	  public void Foo (int value) {…}

	  [Register ("foo-WZ4Q5Ns", …)]
	  public void Foo (uint value) {…}
	}

Java.Interop added support for emitting C# binding code which supports
Kotlin unsigned types.

For `kotlin.UByte`, `kotlin.UInt`, `kotlin.ULong`, and `kotlin.UShort`,
no additional runtime changes are needed to support the binding.

The `*Array` counterparts *do* require additional runtime changes.

Consider this Kotlin type:

	// Kotlin
	class UnsignedInstanceMethods {
	  public open fun uintArrayInstanceMethod (value: UIntArray) : UIntArray {
	    return value;
	  }
	}

The binding for this method would be:

	// C# binding
	partial class UnsignedInstanceMethods {
	  [Register ("uintArrayInstanceMethod--ajY-9A", "([I)[I", "")]
	  public unsafe uint[] UintArrayInstanceMethod (uint[] value)
	  {
	    const string __id = "uintArrayInstanceMethod--ajY-9A.([I)[I";
	    IntPtr native_value = JNIEnv.NewArray ((int[])(object)value);
	    try {
	      JniArgumentValue* __args = stackalloc JniArgumentValue [1];
	      __args [0] = new JniArgumentValue (native_value);
	      var __rm = _members.InstanceMethods.InvokeNonvirtualObjectMethod (__id, this, __args);
	      return (uint[]) JNIEnv.GetArray (__rm.Handle, JniHandleOwnership.TransferLocalRef, typeof (uint));
	    } finally {
	      if (value != null) {
	        JNIEnv.DeleteLocalRef (native_value);
	      }
	    }
	  }
	}

The problem is the `JNIEnv.GetArray(…, typeof(uint))` invocation.
This eventually hits various dictionaries within `JNIEnv`, which
requires additional support so that `uint` can be marshaled properly.

Previous versions of Xamarin.Android do not contain support for
marshaling arrays of unsigned types.

As such, support for using Kotlin bindings which contain unsigned
types will require Xamarin.Android 10.2.0 and later.

~~ Installer ~~

Add `protobuf-net.dll` to our installers, as it is required to read
the metadata that Kotlin puts within `.class` files.

~~ Unit Tests ~~

On-device tests for calling Kotlin code that uses unsigned types have
been added.

As we don't currently have a way of acquiring the Kotlin compiler for
builds, the test `.jar` and Kotlin's
`org.jetbrains.kotlin.kotlin-stdlib.jar` are copied into the repo.
They are *not* redistributed.

The source of the test `.jar` is provided for future use.

~~ Warnings ~~

Kotlin unsigned types are still experimental.  If Kotlin changes how
they are represented in Java bytecode, bindings which use them will
break.

Methods which use unsigned types *cannot* be virtual, nor can they
be used in bound interfaces.  This is because Java source code is
currently used as an "intermediary" for Java <-> Managed transitions,
and Java cannot override Kotlin "mangled" methods.  If the virtual
method is declared on a class, it will be bound as a non-`virtual`
method.  If the method is on an interface, it will be bound, but the
interface will not actually be implementable; a
[XA4213 build-time error][0] will be generated.

[0]: dotnet/java-interop@3bf5333
@awattar
Copy link

awattar commented Jan 26, 2020

What is the ETA of the 16.5 going stable?

For me this is critical as it broke my binding project due to chain of reactions that ending with whole types being removed from the generated binding.

BINDINGSGENERATOR : warning BG8800: Unknown parameter type System.Collections.Generic.ICollection<System.String> in method GetStringSet in managed type Android.Content.ISharedPreferences.
BINDINGSGENERATOR : warning BG8503: Invalidating Android.Content.ISharedPreferences and all nested types because some of its methods were invalid.

@jpobst
Copy link
Contributor

jpobst commented Jan 26, 2020

I don't believe an RTM date has been announced for 16.5, however this fix is included in the 16.5 Preview 2 that came out this week. The preview installs side-by-side with your existing VS so you can use it without messing up your current VS.

https://docs.microsoft.com/en-us/visualstudio/releases/2019/release-notes-preview#16.5.0-pre.2.0

@awattar
Copy link

awattar commented Jan 26, 2020

Unfortunately, I'm working on Mac and side by side preview doesn't work from some time.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants