[runtime] Add support for weak fields. #5972

vargaz · 2017-11-09T00:58:19Z

Fields marked with the newly added [Weak] attribute will have weakref semantics, they will be null-ed out when
the object they point to is collected.

This is implemented as follows:

We check for fields with the attribute named WeakAttribute.
To avoid repeated scans for custom attributes, we scan the customattr table once per image,
collecting the field table indexes of weak fields.
We clear the gc bit of these fields so they are not considered gc references.
When an object with weak fields is allocated, we create a gc handle for it with type WEAK_FIELDS.
When processing gc handles, we process handles of type WEAK_FIELDS by checking that
the weak fields point to a live object. If not, we set them to null.

kumpera

There are multiple blockers with this PR:

The GC code gets both generational and moving heap assumptions wrong.
It's missing boehm support.
The metadata code can be potentially slow due to scanning a few big tables upfront. We should measure the memory and time costs of this PR before merging it.
There's no AOT caching of that info nor does the patch handles cached AOT metadata.
It doesn't handle generic instances, structs, static or thread-local variables. It doesn't fail the type either when finding those unsupported types.
The JIT incorrectly generates write barriers to weak field stores. Either that or make use of remset info to scan those old gen objects.
The test suite needs to be expanded to include at least this scenarios:

old gen obj with weak field pointing to young gen object. Have the young gen object get promoted and verify that it remains ok.
LOS obj with weak field (array of structs with a weak field) pointing to young gen object. Get the young gen object promoted and verify that it remains ok.
Pass a weak field byref, load/store from it and verify.
A weak field in a struct or a generic class.

kumpera · 2017-11-09T20:19:42Z

mono/metadata/class.c

Did you check that interfaces_inited, has_finalize_inited, fields_inited and has_failure are all modified with the loader lock held?

MonoClass has this comment:
/* ALL BITFIELDS SHOULD BE WRITTEN WHILE HOLDING THE LOADER LOCK */

kumpera · 2017-11-09T20:20:57Z

mono/metadata/custom-attrs.c

Missing a mono_error_cleanup.

kumpera · 2017-11-09T20:34:54Z

mono/metadata/class.c

This will recompute the weak layout of base types multiple times and this could be quite inefficient.
Since we compute the layout of the parent type first, we could just check klass->has_weak_fields and bootstrap from that bitmap instead of doing all of this heavy weight lookup.

How will this interact with AOT cached info that won't call mono_class_layout_fields for cached classes?

How will this work for generic instances?

How will this handle a struct with a weak field?

kumpera · 2017-11-09T20:43:30Z

mono/metadata/custom-attrs.c

This is quite inefficient. The member ref table can be as big as 10x of the typedef table.
We should first scan the typeref table for a type in corlib or "Mono.Runtime.Testing" assembly before delving into the memberef table.

Will add that.

kumpera · 2017-11-09T20:44:23Z

mono/metadata/custom-attrs.c

This check is assumes that WeakAttribute will only have one constructor. This must be documented in the type definition otherwise things will magically break for no good reason.

kumpera · 2017-11-09T20:48:21Z

mono/metadata/null-gc.c

Can't null-gc behavior be of just asserting?

kumpera · 2017-11-09T20:51:25Z

mono/sgen/sgen-gchandles.c

This is wrong. An old gen object might have a weak field pointing to an young gen object that got promoted.

kumpera · 2017-11-09T20:52:05Z

mono/sgen/sgen-gchandles.c

This is wrong, the object can move and we must update the value in the gchandle table

kumpera · 2017-11-09T20:53:30Z

mono/sgen/sgen-gchandles.c

This is wrong, since weak fields are not scanned as part of remset processing, we must scan all of those objects.

kumpera · 2017-11-09T21:23:25Z

mono/metadata/class-accessors.c

We should embed the bitmap at the end of WeakBitmapData and avoid the extra allocation and indirection.

The assumption is that weak fields are not common, so the storage doesn't need to be optimized too much.

kumpera · 2017-11-09T21:27:01Z

mono/sgen/sgen-gchandles.c

We must check if the object moved and update the field.

evincarofautumn · 2017-11-10T22:22:17Z

mono/sgen/sgen-gchandles.c

+{
+	//
+	// We use a gc handle to be able to do some processing for these objects at every gc
+	//


I would add a note here that this handle is freed by scan_for_weak.

joncham · 2017-11-10T22:52:27Z

Is this a new feature in .NET, or a mono specific extension?

vargaz · 2017-11-11T08:21:19Z

Currently, its a mono extension intended to be used on ios, similar to the weak references in swift.

marek-safar · 2017-11-21T15:18:52Z

mono/metadata/class.c

+				guint32 field_idx = first_field_idx + (field - p->fields);
+				if (MONO_TYPE_IS_REFERENCE (field->type) && mono_assembly_is_weak_field (p->image, field_idx + 1)) {
+					has_weak_fields = TRUE;
+					mono_trace_message (MONO_TRACE_TYPE, "Field %s:%s at offset %x is weak.", field->parent->name, field->name, field->offset);


Why does it have to keep scanning?

A class can have multiple weak fields.

sure but we are looking only for the first one, right?

We are also printing out the trace message. Since these fields are not common, I don't think continuing the loop is a problem.

There's still the issue that we're iterating over the parents multiple times.
This info should be cached in class and not recomputed multiple times for the same class.

Plus, this breaks SRE if it has a parent with a Weak field.

marek-safar · 2017-11-21T15:25:43Z

mcs/class/corlib/System/WeakAttribute.cs

+namespace System {
+
+[AttributeUsage(AttributeTargets.Field)]
+public sealed class WeakAttribute : Attribute


I still think the name is very common, this can be easily breaking change due to ambiguity. We should rename it to WeakReferenceAttribute

It might also be useful to add sample into this file how it's intended to be used

marek-safar · 2017-11-21T15:30:40Z

mono/metadata/object.c

+
+					while ((field = mono_class_get_fields (p, &iter))) {
+						guint32 field_idx = first_field_idx + (field - p->fields);
+						if (MONO_TYPE_IS_REFERENCE (field->type) && mono_assembly_is_weak_field (p->image, field_idx + 1)) {


We should also check for readonly flag

why the readonly flag matters for this?

It just feels wrong to me to have Weak on readonly field, the value should not be ever unset

This PR doesn't include any form of error handling for bad field configurations. It's left for the C# compiler to eventually do it.

A readonly weak ref will work as advertised, only inited in the constructor and eventually zero'd when the object is gone. It's weird but kinda makes sense?

I don't think C# work will be done anytime soon if ever. Maybe I misunderstand but [Weak] readonly Foo f = Other.Value; can be set back to null after initialization and Other.Value as an unique reference is gone which just looks odd to me.

Hey Marek,

This PR is not the right medium to discuss semantics and scope. I support us discussing it all just not here.

marek-safar · 2017-11-29T11:22:14Z

commit apidiff

marek-safar · 2017-11-29T11:36:34Z

@vargaz there are many tests failures could you investigate

vargaz · 2017-11-29T15:48:22Z

build

marek-safar · 2017-11-30T08:41:10Z

@vargaz it's still failing with following on all lanes

MONO_PATH=D:/j/workspace/x/mcs/class/lib/net_4_x /cygdrive/d/j/workspace/x/tools/pedump/pedump --verify error test-runner.exe
make[4]: *** [Makefile:2341: test-pedump] Error 255

Fields marked with the newly added [Weak] attribute will have weakref semantics, they will be null-ed out when the object they point to is collected. This is implemented as follows: - We check for fields with the attribute named WeakAttribute. To avoid repeated scans for custom attributes, we scan the customattr table once per image, collecting the field table indexes of weak fields. - We clear the gc bit of these fields so they are not considered gc references. - When an object with weak fields is allocated, we create a gc handle for it with type WEAK_FIELDS. - When processing gc handles, we process handles of type WEAK_FIELDS by checking that the weak fields point to a live object. If not, we set them to null.

… smaller. Fix leaking a MonoError.

…criptor (), only computed klass->has_weak_fields in mono_class_layout_fields ().

…he 'normal' GCHandleType values so it matches the managed definition.

…allback is not set, happens in pedump.

marek-safar · 2017-12-01T09:37:56Z

@monojenkins build Linux x64 Acceptance Tests

@lewurm

It was introduced by #5972 Filed bug: https://bugzilla.xamarin.com/show_bug.cgi?id=60973 Reviewed manually by @lewurm.

@lewurm

It was introduced by mono#5972 Filed bug: https://bugzilla.xamarin.com/show_bug.cgi?id=60973 Reviewed manually by @lewurm.

Commit migrated from mono/mono@5bdaef7

@lewurm

It was introduced by mono/mono#5972 Filed bug: https://bugzilla.xamarin.com/show_bug.cgi?id=60973 Reviewed manually by @lewurm. Commit migrated from mono/mono@58618de

vargaz requested review from BrzVlad, akoeplinger, kumpera, luhenry and marek-safar as code owners November 9, 2017 00:58

dnfclas added the cla-already-signed label Nov 9, 2017

vargaz force-pushed the weak-fields branch 2 times, most recently from ecf62ac to 7e629a2 Compare November 9, 2017 15:54

kumpera suggested changes Nov 9, 2017

View reviewed changes

kumpera reviewed Nov 9, 2017

View reviewed changes

mono/sgen/sgen-gchandles.c Outdated

Copy link

Contributor

kumpera Nov 9, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We must check if the object moved and update the field.

vargaz force-pushed the weak-fields branch from 7e629a2 to a87baf1 Compare November 10, 2017 20:10

evincarofautumn reviewed Nov 10, 2017

View reviewed changes

ghost removed the cla-already-signed label Nov 16, 2017

marek-safar requested changes Nov 21, 2017

View reviewed changes

kumpera approved these changes Nov 21, 2017

View reviewed changes

marek-safar approved these changes Nov 29, 2017

View reviewed changes

monojenkins added a commit to mono/api-snapshot that referenced this pull request Nov 29, 2017

Apply changes from mono/mono#5972 to API snapshot

8482d12

vargaz force-pushed the weak-fields branch from 1e857e7 to e2650fc Compare November 29, 2017 15:04

vargaz force-pushed the weak-fields branch from 9f4df61 to 30053bf Compare November 29, 2017 15:56

vargaz and others added 5 commits November 30, 2017 23:44

Scan the TYPEREF table before scanning the MEMBERREF table, since its…

aec5efb

… smaller. Fix leaking a MonoError.

Move the weak bitmap construction code into mono_class_compute_gc_des…

cb72e05

…criptor (), only computed klass->has_weak_fields in mono_class_layout_fields ().

Add AOT caching for the weak fields hash.

3fbd420

[boehm] Fix boehm compilation

c0ee75b

kumpera and others added 7 commits November 30, 2017 23:46

[sgen] Move sgen_is_object_alive to sgen-gc.h

009eee9

[sgen] Fix [Weak] scanning and make it behave like short weak refs.

562d878

[sgen] Add one more case to the [Weak] test.

e56c86f

Bump API snapshot submodule

42bab3b

[sgen] Fix the definition of HANDLE_WEAK_FIELDS, it should be after t…

543b3e2

…he 'normal' GCHandleType values so it matches the managed definition.

[runtime] Avoid a crash in mono_assembly_init_weak_fields () if the c…

5f8a2dd

…allback is not set, happens in pedump.

[runtime] Fix a rebase problem.

1bb9d9b

vargaz force-pushed the weak-fields branch from 8d7eb32 to 1bb9d9b Compare December 1, 2017 04:49

marek-safar merged commit 5bdaef7 into mono:master Dec 3, 2017

akoeplinger added a commit that referenced this pull request Dec 4, 2017

[tests] Disable weak-fields.exe on ARM

58618de

It was introduced by #5972 Filed bug: https://bugzilla.xamarin.com/show_bug.cgi?id=60973 Reviewed manually by @lewurm.

alexrp mentioned this pull request Dec 6, 2017

[profiler] Rework GC roots reporting. #6144

Merged

marek-safar pushed a commit to marek-safar/mono that referenced this pull request Jan 10, 2018

[runtime] Add support for weak fields. (mono#5972)

2eaf41d

marek-safar pushed a commit to marek-safar/mono that referenced this pull request Jan 11, 2018

[tests] Disable weak-fields.exe on ARM

a1d87c3

It was introduced by mono#5972 Filed bug: https://bugzilla.xamarin.com/show_bug.cgi?id=60973 Reviewed manually by @lewurm.

picenka21 pushed a commit to picenka21/runtime that referenced this pull request Feb 18, 2022

[runtime] Add support for weak fields. (mono/mono#5972)

c5bf0de

Commit migrated from mono/mono@5bdaef7

rolfbjarne mentioned this pull request Apr 29, 2022

[API Proposal]: (Re)introduce WeakAttribute that makes a field behave as if it were a WeakReference. dotnet/runtime#68702

Open

[runtime] Add support for weak fields. #5972

[runtime] Add support for weak fields. #5972

Uh oh!

Conversation

vargaz commented Nov 9, 2017

Uh oh!

kumpera left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kumpera Nov 9, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

joncham commented Nov 10, 2017

Uh oh!

vargaz commented Nov 11, 2017

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

marek-safar commented Nov 29, 2017

kumpera Nov 9, 2017 •

edited

Loading