[runtime] Compare custom modifiers in type equality #7012

alexanderkyte · 2018-02-13T22:25:11Z

Custom modifiers are an odd feature of .net which allow types to
be tagged. These tags are semantically relevant. A method with a
modifier should not override a method without a modifier, for
instance (github bug #6936).

We don't put enough information into the modifier to uniquely find the
Image associated with the token we have in the type. Modifiers from
different referring assemblies will have different tokens to refer to
the same modifier class, making it important to resolve the tokens to
MonoClass pointers at runtime. Pointer equality of these classes is the
way to determine two custom modifiers the same, or unique.

We cannot resolve the image just from the MonoType because it is
possible to put modifiers on primitive, builtin types. These builtin
types lack any filled-out pointers in the data block, meaning we have
zero metadata associated with our floating token.

Other places in the codebase fixed this by passing around MonoImages,
but adding a MonoImage argument to our type comparison function for
this single bug would be a very poor decision with much code churn all
throughout the runtime.

The bug above resolves to two I4 types, one of which has a modifier, one
of which does not.

In order to pass this MonoImage around, we need to put a reference to it
into something reachable from the MonoType. We can't put it in the
custom module type because that is exported. It is part of the public
API.

MonoTypes do not adhere to pointer equality, and custom modules are structs
baked directly into the MonoType. We need therefore to make MonoType
point to a MonoCustomModInternal, a new type. This type embeds the
public type as a struct and carries around the image as another element
in the struct.

All public interfaces still see the same metadata endpoints for parsing the
modules, including the icalls. It is only our type comparison code that
accesses our new field.

lambdageek · 2018-02-13T22:27:14Z

mono/metadata/metadata.h

If you define it here, it's not internal. It's part of the public API.

We shouldn't store then on each cmod, but change MonoType to have an optional cmod container.

IE, change MonoType to this:

struct _MonoType { union { MonoClass *klass; /* for VALUETYPE and CLASS */ MonoType *type; /* for PTR */ MonoArrayType *array; /* for ARRAY */ MonoMethodSignature *method; MonoGenericParam *generic_param; /* for VAR and MVAR */ MonoGenericClass *generic_class; /* for GENERICINST */ } data; unsigned int attrs : 16; /* param attributes or field flags */ MonoTypeEnum type : 8; unsigned int has_cmods : 1; /* has a cmod appended at the end */ unsigned int byref : 1; unsigned int pinned : 1; /* valid when included in a local var signature */ };

Then have this:

typedef struct CustomModContainer { int count; MonoImage *image; MonoCustomMod modifiers [1]; /*the actual size of this array is count*/ }

You'd be able to fetch it using something like this:

CustomModContainer* mono_type_get_cmods (MonoType *t) { if (!t->has_cmods) return null; return (CustomModContainer*)(((char*)t) + sizeof (MonoType)); }

Maybe a bit nicer:

typedef struct _MonoTypeWithModifiers { MonoType type; CustomModContainer cmods; } MonoTypeWithModifiers;

So you can just do &((MonoTypeWithModifiers*)type)->cmods in mono_type_get_cmods

That looks good @lambdageek, that works just like our other prefixed struct types

lambdageek · 2018-02-13T22:29:32Z

mono/metadata/metadata.c

Shouldn't this only be comparing required modifiers, not optional ones?

Are two integers, one with an optional modifier, one without any modifiers, are they equal?

It looks like the main use of optional modifiers are to differentiate between functions taking a long and functions taking an int (or other bitness) on C++/CLI. Since they use the same underlying CLR type for both, they use modopts to prevent one function from being used where the other is expected. If we treat them as equal, that defeats that use.

From 335, 7.1.1:

The CLI itself shall treat required and optional modifiers in the same manner. Two signatures that differ only by the addition of a custom modifier (required or optional) shall not be considered to match.

lambdageek · 2018-02-13T22:30:29Z

mono/metadata/metadata.c

What if the two signatures have the modifiers in different orders? (Is there an ordering rule in ECMA?)

That's worrying, let me look into that.

kumpera

It makes no sense to store the same data on every cmod, since they all will have the same image.

kumpera · 2018-02-14T17:21:10Z

mono/metadata/metadata.h

We shouldn't store then on each cmod, but change MonoType to have an optional cmod container.

IE, change MonoType to this:

struct _MonoType { union { MonoClass *klass; /* for VALUETYPE and CLASS */ MonoType *type; /* for PTR */ MonoArrayType *array; /* for ARRAY */ MonoMethodSignature *method; MonoGenericParam *generic_param; /* for VAR and MVAR */ MonoGenericClass *generic_class; /* for GENERICINST */ } data; unsigned int attrs : 16; /* param attributes or field flags */ MonoTypeEnum type : 8; unsigned int has_cmods : 1; /* has a cmod appended at the end */ unsigned int byref : 1; unsigned int pinned : 1; /* valid when included in a local var signature */ };

Then have this:

typedef struct CustomModContainer { int count; MonoImage *image; MonoCustomMod modifiers [1]; /*the actual size of this array is count*/ }

You'd be able to fetch it using something like this:

CustomModContainer* mono_type_get_cmods (MonoType *t) { if (!t->has_cmods) return null; return (CustomModContainer*)(((char*)t) + sizeof (MonoType)); }

marek-safar · 2018-02-23T14:34:31Z

@alexanderkyte could you rebase it and address @kumpera review

luhenry · 2018-03-09T20:21:58Z

@alexanderkyte what's the status of that?

alexanderkyte · 2018-03-20T17:55:10Z

It was temporarily put on hold because I was asked to prioritize more urgent things. The current status is that I am making the requested modifications while investigating worries we have about memory unsafety around MonoType copying. This bug brought to our attention that they are variably sized types, and that we have been placing them on the stack / copying them in a number of places. Thankfully they are few in number.

lambdageek

Mostly looks good.

Just a couple of concerns about type equality.

Does custom modifier order matter?
Should modreq(mod1) and modopt(mod1) compare unequal?

Also I'm not sure about making MonoCustomModContainer public right away.

lambdageek · 2018-03-21T20:36:05Z

mono/metadata/metadata.h

this can be MonoImage*, I think.

lambdageek · 2018-03-21T20:49:25Z

mono/mini/aot-runtime.c

lambdageek · 2018-03-21T20:53:53Z

mono/metadata/metadata.c

Can you add a comment here referencing ECMA - I keep stumbling over this line.

(From the last review)

From 335, 7.1.1:

The CLI itself shall treat required and optional modifiers in the same manner. Two signatures that differ only by the addition of a custom modifier (required or optional) shall not be considered to match.

lambdageek · 2018-03-21T20:59:14Z

mono/metadata/metadata.c

We should probably have a FIXME here to propagate the error out to the caller.

lambdageek · 2018-03-21T21:00:08Z

mono/metadata/metadata.c

Does custom modifier order matter? if I have t modreq(mod1) modreq(mod2) t vs t modreq(mod2) modreq(mod1) are those the same type or different?

lambdageek · 2018-03-21T21:06:09Z

mono/metadata/metadata.h

Also does MonoCustomModContainer really need to be in the public API already?

Custom modifiers are used by monodis. I'm not sure how the public API consumers would be able to interact with the mods without the image, count, and mods provided here.

lambdageek · 2018-03-21T21:11:01Z

mono/metadata/metadata.c

I think you need to also compare cm1->modifiers[i].required != cm2->modifiers[i].required

This was added by a line earlier in the loop body

lambdageek · 2018-03-21T21:25:36Z

If ECMA is underspecified or ambiguous about some cases, we should do what .NET Framework does.

We should add tests to lock down the behavior we believe we're implementing.

alexanderkyte · 2018-03-24T02:40:28Z

Any suggestions as to the tests? We use types in a great many places, and this significantly changes only inequalities with modreqs.

I think that the fact that signatures treat modifiers as inequalities significantly narrows down the behaviors they can have. The typeof for two arguments that differ in the modifiers has to return false (otherwise C++/CLI would see typeof(int) and typeof(long) to be the same). That means that since these types can be reflected, we'd have to compare for the inequality in the most general place. This is in our type inequality here.

One test would be the actual bug filed with roslyn, the inheritance with a single modreq.

We could add one test to see if the order matters, using a 3-modreq type.

Lastly, we want one that checks if modreqs and modopts with the same parameters are considered equivalent.

marek-safar · 2018-03-28T08:50:04Z

@kumpera could you review this again

lambdageek · 2018-03-29T20:28:09Z

@monojenkins build Linux x64 Checked Private Types Build

Custom modifiers are an odd feature of .net which allow types to be tagged. These tags are semantically relevant. A method with a modifier should not override a method without a modifier, for instance (github bug mono#6936). We don't put enough information into the modifier to uniquely find the Image associated with the token we have in the type. Modifiers from different referring assemblies will have different tokens to refer to the same modifier class, making it important to resolve the tokens to MonoClass pointers at runtime. Pointer equality of these classes is the way to determine two custom modifiers the same, or unique. We cannot resolve the image just from the MonoType because it is possible to put modifiers on primitive, builtin types. These builtin types lack any filled-out pointers in the data block, meaning we have zero metadata associated with our floating token. Other places in the codebase fixed this by passing around MonoImages, but adding a MonoImage argument to our type comparison function for this single bug would be a very poor decision with much code churn all throughout the runtime. The bug above resolves to two I4 types, one of which has a modifier, one of which does not. In order to pass this MonoImage around, we need to put a reference to it into something reachable from the MonoType. We can't put it in the custom module type because that is exported. It is part of the public API. MonoTypes do not adhere to pointer equality, and custom modules are structs baked directly into the MonoType. We need therefore to make MonoType point to a MonoCustomModInternal, a new type. This type embeds the public type as a struct and carries around the image as another element in the struct. All public interfaces still see the same metadata endpoints for parsing the modules, including the icalls. It is only our type comparison code that accesses our new field.

alexanderkyte · 2018-04-06T17:41:15Z

@lambdageek I'm not very confident in my understanding of what I did in sre.c. We have a lot of functions that all state that they do the same general thing, but some "wrap" others. Could you check that I'm not using anything too-unwrapped here?

lambdageek

SRE stuff looks alright.

Fixup the static function's name. And add some comments to the test.

lambdageek · 2018-04-06T17:51:25Z

mono/tests/custom-modifiers-inheritance.cs

Please add a comment at the top of test that summarizes what is being tested.

For SRE stuff, I also try to put a comment with C#-ish pseudocode of what's being constructed and a comment of which bit is expected to fail or do a tricky thing, etc. It'll save someone time in the future from having to mentally execute SRE.

I added a number of blocks of comments

lambdageek · 2018-04-06T17:56:24Z

mono/metadata/sre.c

How about add_custom_modifiers_to_type?

marek-safar · 2018-04-09T08:27:04Z

@monojenkins build failed

lambdageek · 2018-04-09T15:51:25Z

@kumpera I think this addresses all your concerns now - the image is stored once per MonoType when there are custom modifiers present. Could you review again or dismiss your old review.

kumpera

LGTM

alexanderkyte requested review from kumpera and vargaz as code owners February 13, 2018 22:25

lambdageek reviewed Feb 13, 2018

View reviewed changes

alexanderkyte force-pushed the custommod_type_equality branch from 24db715 to 70ae9f7 Compare February 13, 2018 22:32

kumpera suggested changes Feb 14, 2018

View reviewed changes

alexanderkyte force-pushed the custommod_type_equality branch from 70ae9f7 to 66fcd13 Compare February 15, 2018 20:06

alexanderkyte requested review from akoeplinger, alexischr, luhenry and marek-safar as code owners February 15, 2018 20:06

luhenry self-assigned this Feb 20, 2018

alexanderkyte mentioned this pull request Mar 21, 2018

Mono runtime doesn't check modreqs for overriding methods #6936

Closed

alexanderkyte force-pushed the custommod_type_equality branch 2 times, most recently from 840eac8 to 5e8eedc Compare March 21, 2018 20:01

lambdageek suggested changes Mar 21, 2018

View reviewed changes

alexanderkyte force-pushed the custommod_type_equality branch 2 times, most recently from df5250e to dea3e5f Compare March 24, 2018 02:42

lambdageek approved these changes Mar 29, 2018

View reviewed changes

luhenry assigned alexanderkyte and unassigned luhenry Apr 2, 2018

alexanderkyte force-pushed the custommod_type_equality branch from dea3e5f to 0e08b6d Compare April 6, 2018 17:35

alexanderkyte force-pushed the custommod_type_equality branch from 0e08b6d to 9d505af Compare April 6, 2018 17:35

lambdageek approved these changes Apr 6, 2018

View reviewed changes

alexanderkyte force-pushed the custommod_type_equality branch 4 times, most recently from c543fb6 to 48c33f1 Compare April 7, 2018 01:39

alexanderkyte added 2 commits April 7, 2018 03:15

[runtime] Add sre-based modreq tests

023d5e8

[runtime] Add CustomModifiers support to sre

7efbdf1

alexanderkyte force-pushed the custommod_type_equality branch from 48c33f1 to 7efbdf1 Compare April 7, 2018 07:15

marek-safar approved these changes Apr 9, 2018

View reviewed changes

kumpera approved these changes Apr 9, 2018

View reviewed changes

lambdageek merged commit 8eb6dae into mono:master Apr 9, 2018

lambdageek mentioned this pull request Sep 24, 2018

[marshal] Use MonoClass getters in marshal.c #7253

Merged

[runtime] Compare custom modifiers in type equality #7012

[runtime] Compare custom modifiers in type equality #7012

Uh oh!

Conversation

alexanderkyte commented Feb 13, 2018

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lambdageek Feb 14, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

alexanderkyte Feb 13, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

alexanderkyte Feb 13, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kumpera left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

marek-safar commented Feb 23, 2018

Uh oh!

luhenry commented Mar 9, 2018

Uh oh!

alexanderkyte commented Mar 20, 2018

Uh oh!

lambdageek left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lambdageek commented Mar 21, 2018

Uh oh!

alexanderkyte commented Mar 24, 2018

Uh oh!

marek-safar commented Mar 28, 2018

Uh oh!

lambdageek commented Mar 29, 2018

Uh oh!

alexanderkyte commented Apr 6, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

lambdageek left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lambdageek Feb 14, 2018 •

edited

Loading

alexanderkyte Feb 13, 2018 •

edited

Loading

alexanderkyte Feb 13, 2018 •

edited

Loading

alexanderkyte commented Apr 6, 2018 •

edited

Loading