[profiler] GC roots revamp. #5126

kumpera · 2017-06-27T01:39:50Z

This is not ready for review, tons of line damage, debug spew and half done changes.

kumpera · 2017-07-13T23:32:13Z

@alexrp the PR is read for review / merge.
I don't expect to work further on the runtime side for now.
Further work will happen in the managed library to integrate my local tests.

rodrmoya · 2017-07-14T08:05:02Z

mono/profiler/log.c

doesn't this need a MLPD version bump?

We already bumped it in master and haven't shipped it yet AFAIK.

alexrp · 2017-07-14T11:20:15Z

mono/metadata/boehm-gc.c

I don't think we should raise these events in Boehm since we don't report roots anyway.

I did it out of completeness. Boehm doesn't really matter.

alexrp · 2017-07-14T11:24:40Z

mcs/class/Mono.Profiler.Log/Mono.Profiler.Log/LogEnums.cs

A few things:

Please add a comment above the enum that points to the C enum (as for the others here).

I think LogHeapRootSource would be a better name, since e.g. Threading is not a kind, it's the subsystem of the runtime (source) it comes from.

I prefer to use explicit enum values when we're mapping to a C enum, especially if that C enum is using explicit values too.

I think we should use singular names here, i.e. StaticVariable, GCHandle, etc.

Marshalling -> Marshal for consistency with the rest of .NET.

alexrp · 2017-07-14T11:26:57Z

mcs/class/Mono.Profiler.Log/Mono.Profiler.Log/LogEnums.cs

Should be HeapRootUnregister.

alexrp · 2017-07-14T11:27:58Z

mono/metadata/gc-internals.h

Don't forget that this will require changes in the products.

How so? I don't think they use mono_gc_register_root directly.

You're right. For some reason, I was under the impression that they did. Why on earth are these MONO_API then?

Maybe they use in a sneaky way, I'll check more carefully before merging.

alexrp · 2017-07-14T11:30:51Z

mono/metadata/marshal.c

Minor point: 0 -> NULL for clarity.

alexrp · 2017-07-14T11:31:32Z

mcs/class/Mono.Profiler.Log/Mono.Profiler.Log/LogEnums.cs

No separate source value for critical finalizers?

I wonder if it matters, the distinction today exists as each get a separate pseudo-address when their roots get registered.

Given that the vast majority of C# users don't know that critical finalizers exists, I think it's ok to keep them all on the sabe basket until it turns to be an issue.

alexrp · 2017-07-14T11:33:10Z

mono/profiler/log.h

Should be TYPE_HEAP_ROOT_REGISTER and TYPE_HEAP_ROOT_UNREGISTER.

alexrp · 2017-07-14T11:34:13Z

mono/profiler/log.h

This line should go in the changelog for the profiler version, not the data version, further up the file.

alexrp · 2017-07-14T11:35:33Z

mono/profiler/log.h

We should document specifically which event it used to be emitted after and which one we emit it after now.

Yes, reorganized

alexrp · 2017-07-14T11:35:48Z

mono/profiler/log.h

Also, indentation is off. We use spaces here for whatever reason.

Odd, but fixed

alexrp · 2017-07-14T11:36:51Z

mono/profiler/log.c

A bit of line damage here.

alexrp · 2017-07-14T11:37:11Z

mono/profiler/log.c

Missing TYPE_HEAP_ROOT_UNREGISTER.

alexrp · 2017-07-14T11:38:52Z

mono/metadata/profiler.h

What's up with this one?

garbage, removed.

alexrp · 2017-07-14T11:40:05Z

null-gc.c will need to be updated for some of these changes.

alexrp · 2017-07-14T11:41:58Z

mono/metadata/sgen-mono.c

Isn't the handle stack allocated with mono_gc_alloc_fixed or something?

No, it uses g_malloc.

alexrp · 2017-07-14T11:42:20Z

@BrzVlad could you review the SGen parts?

alexrp · 2017-07-14T11:43:06Z

mcs/class/Mono.Profiler.Log/Mono.Profiler.Log/LogProcessor.cs

Minor point: You could use object initializers for these.

alexrp · 2017-07-14T11:57:03Z

mono/metadata/threads.c

This cast looks wrong. In the thread_started event above, we pass the result of MONO_NATIVE_THREAD_ID_TO_UINT (tid) (see the call site of this function), but here, we convert it back to native thread ID before passing it.

Same for most of this file. We should be consistent about passing one kind of TID.

yes, it's messy. I'll see if I can just pass in a void*

alexrp · 2017-07-14T12:01:42Z

mono/metadata/object.c

In what situations do we expect thread->internal_thread to not be set here? I'm not too familiar with this code.

During attach/detach all goes. It's defensive programing. :(

alexrp · 2017-07-14T12:02:46Z

mono/metadata/profiler-events.h

I generally try to keep these events 'namespaced', so these should be gc_roots, gc_root_register, and gc_root_unregister.

They should also be in the section above with the other gc_* events.

GcRoots2 -> GCRoots, especially since we're just removing the old event entirely.

gc_root_register should be typed as const mono_byte *start, uintptr_t size, MonoGCRootSource source, const void *key, const char *msg for consistency with the rest of the events.

gc_root_unregister should be typed as const mono_byte *start.

gc_roots should be typed as uint64_t count, const mono_byte *const *addresses, MonoObject *const *objects.

Fixed the signature

alexrp · 2017-07-14T12:07:22Z

The comments for MonoGCRootSource in mono/metadata/mono-gc.h should be updated since we use these values differently than I originally planned to.

kumpera · 2017-07-14T23:09:34Z

The comments on MonoGCRootSource are still correct. I updated the comments that those values now leak into the profiler API.

I think it's ok that the null-gc.c gets no profiling support, I don't think it needs anything else.

Reduced the amount of casting in threads.c around tid.

alexrp · 2017-07-14T23:16:23Z

I think it's ok that the null-gc.c gets no profiling support, I don't think it needs anything else.

I was referring to the changed signature of mono_gc_register_root.

This change re-implement GC roots reporting in a way that allows users to correctly track the source of a GC root. The new design is based on emitting GC root information upfront and only report addresses during heap dumps. When a GC root is registered we emit an event that is an address range, the root kind, a key and a text description. A decoder can use this information to match a reported root address with all registration addresses to figure out what they mean. A GC Root key is used to further its meaning. For example, if kind is thread, key is a tid, if kind is static variables, key is the class pointer and so on. The build of the change is introducing this key argument across all root registration code and rework our root reporting code to encode their addresses. Some roots use pseudo-addresses when they are not registrable. This is the case of thread stacks/registers and the finalizer queue. Finally, root reporting was changed to happen only once per collection and at the end, leading to correct and useful data being produced.

kumpera · 2017-07-17T23:44:29Z

close

kumpera · 2017-07-17T23:44:39Z

build profiler stress

alexrp · 2017-07-18T19:01:27Z

mono/profiler/log.c

+gc_root_register (MonoProfiler *prof, const mono_byte *start, size_t size, MonoGCRootSource kind, const void *key, const char *msg)
+{
+	int msg_len = msg ? strlen (msg) + 1 : 0;
+	ENTER_LOG (&heap_roots_ctr, logbuffer,


I forgot to mention this in my last review: Each kind of event should have its own kind of counter, so you'll want to add one for root register and root unregister events.

alexrp · 2017-07-19T22:11:32Z

mono/metadata/sgen-mono.c

-	if (profile_roots)
-		notify_gc_roots (&major_root_report);
+	if (!MONO_PROFILER_ENABLED (gc_roots))
+		return;


Would be a good idea to put a TODO comment here so it isn't forgotten.

alexrp · 2017-07-19T22:11:50Z

mono/metadata/sgen-mono.c

+void
+sgen_client_pinning_end (void)
+{
+	if (!MONO_PROFILER_ENABLED (gc_roots))


It's unclear what purpose this check serves.

luhenry · 2017-10-05T14:49:41Z

@kumpera I created kumpera#4 for merging master in this PR to resolve all the conflicts. What else would block it getting merged?

kumpera · 2017-10-05T15:46:39Z

@luhenry if you're interested in working on it, rebase and resubmit it. I don't think merge commits with tons of changes is something we want.

I don't remember what was missing from this PR beyond what's stated on it.

dnfclas added the cla-already-signed label Jun 27, 2017

alexrp mentioned this pull request Jul 7, 2017

[profiler] Report main thread name after attaching. #5179

Merged

kumpera requested a review from alexrp as a code owner July 13, 2017 23:30

kumpera force-pushed the gc_roots_experiments branch from 79b2960 to 4e24dc9 Compare July 13, 2017 23:30

rodrmoya reviewed Jul 14, 2017

View reviewed changes

alexrp reviewed Jul 14, 2017

View reviewed changes

alexrp requested a review from BrzVlad July 14, 2017 11:42

alexrp reviewed Jul 14, 2017

View reviewed changes

kumpera requested a review from luhenry as a code owner July 14, 2017 23:28

kumpera force-pushed the gc_roots_experiments branch from e6b7b16 to 129ca06 Compare July 17, 2017 23:22

kumpera closed this Jul 17, 2017

kumpera reopened this Jul 17, 2017

dnfclas added the cla-already-signed label Jul 17, 2017

BrzVlad approved these changes Jul 18, 2017

View reviewed changes

alexrp reviewed Jul 18, 2017

View reviewed changes

alexrp reviewed Jul 19, 2017

View reviewed changes

kumpera closed this Oct 5, 2017

luhenry mentioned this pull request Oct 5, 2017

[profiler] Rework GC roots reporting. #5710

Closed

[profiler] GC roots revamp. #5126

[profiler] GC roots revamp. #5126

Uh oh!

Conversation

kumpera commented Jun 27, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kumpera commented Jul 13, 2017

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

alexrp Jul 14, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

alexrp Jul 14, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

alexrp commented Jul 14, 2017

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

kumpera commented Jun 27, 2017 •

edited

Loading

alexrp Jul 14, 2017 •

edited

Loading

alexrp Jul 14, 2017 •

edited

Loading

alexrp Jul 14, 2017 •

edited

Loading