Profiler changes to enable better support for dynamic event flags changes #3546

alexrp · 2016-09-09T17:23:25Z

No description provided.

BrzVlad · 2016-09-12T18:16:49Z

sgen commits look ok

…ION is set.

…mode.

…rofiling. This new MANAGED_ALLOCATOR_PROFILER works much like the regular managed allocator, but has a check at the end before returning: if (G_UNLIKELY (mono_profiler_events & MONO_PROFILE_ALLOCATIONS)) { mono_profiler_allocation (p); } There are two main reasons why we need this: 1. Running with the slow path managed allocator as a means to ensure that all allocations are reported to the profiler is overkill. It slows down allocation-heavy programs massively. This matters a lot if a user is doing sampling at the same time as allocation profiling. 2. We would like to be able to dynamically switch allocation profiling on or off at runtime. To do so with the old managed allocator setup, we would have to always use the slow path managed allocator when profiling at all, even if the user never decides to enable allocation profiling. Having a variant of the managed allocator that is instrumented with a profiling check solves both of these issues. It won't be quite as fast as the regular managed allocator, but the slowdown from the check is only about 2-3%.

It could previously fail to process a newly allocated buffer.

…ered.

…llocators. Traces only show 6 frames by default, but when the new managed allocator is in use, we need 7 frames to see the full call chain. Also change check_alloc_traces () to skip over any amount of wrapper frames.

The runtime itself should access the mono_profiler_events variable directly.

This has not actually been necessary since allocation reporting was moved into the GC implementations.

* mono_gc_enable_events: This did nothing for SGen and we called it unconditionally for Boehm. So no point in having it around. * mono_gc_enable_alloc_events: The performance difference between checking a static alloc_events variable versus checking mono_profiler_events is marginal at best and doesn't justify the indirection, especially now that we support managed allocators while profiling.

The sampling thread is now always started when profiling, but sits idle if statistical sampling is not enabled.

…untime. This avoids doing calls in the sampling thread all the time.

…more cases.

… value.

It was never documented and is no longer functional.

Also refactor the heap walk code a bit.

…API + MONO_RT_EXTERNAL_ONLY. Also make it work with MSVC.

This new API has two goals: 1. Explicitly specify the profiler descriptor in all callback installation functions so that they can be changed at any point, even when multiple profilers are loaded. 2. Make sure that a profiler can change its own event flags at any point without interfering with other profilers that may be loaded. The old APIs have been marked with MONO_DEPRECATED.

…_LEAVE rather than MONO_PROFILE_EXCEPTIONS. This event isn't that useful at profiling exceptions. It's mainly meant to keep track of call depth (for enter/leave profiling purposes) when an exception is thrown.

…s instead of MONO_PROFILE_GC_ROOTS.

These have no value outside of heapshots, and in large apps, tend to contribute significantly to log file bloat.

This variable can be changed at any time by any thread. It makes no sense to assert that it has a particular flag.

kumpera · 2016-09-14T22:11:32Z

Ignore the Ok above. This is github being terrible as usual.

I was on the per commit view as the whole diff is just too big.

kumpera · 2016-09-14T22:14:50Z

mono/mini/driver.c

 	mono_native_thread_set_name (mono_native_thread_id_get (), "Main");

-	if (enable_profile) {
+	if (mono_profiler_enabled ()) {


Do we need to enable it at arg parsing time?

Can't we simply let mono_profiler_load set it? This reduces the amount of state we expose.

Yeah, I like that way of doing it more.

kumpera · 2016-09-14T23:09:01Z

mono/profiler/proflog.c

 			InterlockedIncrement (&thread_ends_ctr);

-			buffer = ensure_logbuf_inner (buffer,
+			thread->buffer = ensure_logbuf_inner (thread->buffer,


Should we backport this one to 4.8?

This plus a few other crash/deadlock fixes in this PR are definitely worth backporting.

kumpera · 2016-09-14T23:14:06Z

mono/profiler/proflog.c

 } MonoCounterAgent;

 static MonoCounterAgent* counters;
-static gboolean counters_initialized = FALSE;


Why this check is no longer needed?

Because in the same commit, I moved helper thread initialization to just after counters initialization, and made sure only the helper thread samples counters. So counters will always be initialized when we sample them.

kumpera · 2016-09-14T23:20:23Z

mono/metadata/boehm-gc.c

 	}

-	if (G_UNLIKELY (alloc_events))
+	if (G_UNLIKELY (mono_profiler_events & MONO_PROFILE_ALLOCATIONS))


I think we should wrap this into a any macro that says whether a given profiler event it enabled. Including the G_UNLIKELY.

Yep, good point. I completely forgot to apply the G_UNLIKELY to many of the checks.

kumpera · 2016-09-14T23:26:41Z

mono/mini/mini-runtime.c

 #endif

-	if (mono_profiler_events & MONO_PROFILE_STATISTICAL)
+	if (mono_profiler_enabled ())


This is a no go change. The products install profilers during regular operations. This would cause a background thread to be created. Worse, that thread always spins, which is a power disaster.

Yeah, this is a really good argument for why we should not be calling mono_profiler_enable in mono_profiler_new/mono_profiler_install, only in mono_profiler_load.

kumpera · 2016-09-14T23:33:21Z

mono/mini/mini-posix.c

 	InterlockedWrite (&sampling_thread_running, 0);

-#ifdef HAVE_CLOCK_NANOSLEEP
+#ifndef PLATFORM_MACOSX


How about iOS?

For whatever strange reason, this macro is defined for all Apple platforms. It should be renamed at some point.

kumpera · 2016-09-14T23:40:59Z

mono/utils/mono-compiler.h

 #define MONO_LLVM_INTERNAL 
 #endif

-#if HAVE_DEPRECATED


We should remove the check from configure.

kumpera · 2016-09-14T23:58:39Z

mono/metadata/profiler.c

 mono_profiler_enable (void)
 {
+	if (!profiling_enabled)
+		mono_os_mutex_init_recursive (&profiler_mutex);


Does it need to be recursive?

kumpera · 2016-09-15T00:02:19Z

mono/metadata/profiler.c

-mono_profiler_install_module      (MonoProfileModuleFunc start_load, MonoProfileModuleResult end_load,
-                                   MonoProfileModuleFunc start_unload, MonoProfileModuleFunc end_unload)
+void
+mono_profiler_set_class_cb (MonoProfilerDesc *desc, MonoProfileClassFunc start_load, MonoProfileClassResult end_load,


This is a good chance to document all those undocumented callbacks

kumpera · 2016-09-15T00:11:36Z

mono/profiler/proflog.c

 			do_heap_walk = TRUE;
+
+		if (do_heap_shot && do_heap_walk)
+			mono_profiler_set_event_flags (profiler->desc, mono_profiler_get_event_flags (profiler->desc) | MONO_PROFILE_GC_ROOTS);


What about mono_profiler_enable_events / mono_profiler_disable_event macros?

kumpera · 2016-09-15T00:26:51Z

mono/metadata/appdomain.c

 	data->friendly_name = g_strdup (friendly_name);

-	mono_profiler_appdomain_name (data, data->friendly_name);
+	if (mono_profiler_events & MONO_PROFILE_APPDOMAIN_EVENTS)


I really don't like this change.

The profiler events are checked anyways on the calls. Plus we're not hinting the compiler that it's a non taken branch.

We should use something like this:

#define GEN_EVT2(EVENT, FUNC, ARG0, ARG1) do { \ if (G_UNLIKELY (mono_profiler_events & MONO_PROFILE_ ## EVENT)) \ mono_profiler_ ## FUNC (ARG0, ARG1); \ } while (0) #define RAISE_APPDOMAIN_NAME(ARG0, ARG1) GEN_EVT2( APPDOMAIN_EVENTS, appdomain_name, ARG0, ARG1);

…read ID. This was introduced in 928b840.

kumpera · 2016-09-16T23:15:44Z

mono/metadata/sgen-mono.c

-	if (moved_objects_idx == MOVED_OBJECTS_NUM) {
-		mono_profiler_gc_moves (moved_objects, moved_objects_idx);
-		moved_objects_idx = 0;
+	/*


@BrzVlad can you review this one?

Vlad already OK'd the SGen changes here.

kumpera · 2016-09-19T21:57:59Z

mono/profiler/mono-profiler-log.c

+// Avoid calling this directly if possible. Use the functions below.
 static void
-safe_send (MonoProfiler *profiler, gboolean if_needed, gboolean threadless)
+send_log_unsafe (MonoProfiler *profiler, gboolean lock, gboolean if_needed)


More than one boolean per function is a code smell, It's better to use a single flags argument.

kumpera · 2016-09-19T22:56:59Z

mono/profiler/mono-profiler-log.c

+	 */
+	if (fd >= FD_SETSIZE) {
+		fprintf (stderr, "File descriptor is out of bounds for fd_set: %d\n", fd);
+		exit (1);


I'm not sure that the profiler aborting the whole process is a good idea. We don't do that in other cases IIRC.

We do. For example, if we can't set up the server socket for whatever reason, we exit (1). We also do it for command line syntax errors, e.g. log:sample=,alloc. Whether that's good or bad is arguable, but I'm just following precedent here.

I don't think there's much value in continuing to execute if the profiler is in an unusable state, though. Note that this particular check will only fail at early startup if either the pipe or server socket FD is larger than FD_SETSIZE for some reason.

kumpera · 2016-09-19T22:58:23Z

mono/profiler/mono-profiler-log.c

+				continue;
+
+			fprintf (stderr, "Error in mono-profiler-log server: %s", strerror (errno));
+			exit (1);


This used to be non fatal

select can return these errors: EINTR (signal, we handle this one), EBADF (bad FD, indicates a programming error, because the server socket and pipes should not be closed at this point), EINVAL (bad arguments, programming error), ENOMEM (OOM, nothing we can do at that point).

kumpera · 2016-09-19T23:01:18Z

mono/profiler/mono-profiler-log.c

 		}
 		if ((opt = match_option (p, "heapshot", &val)) != p) {
 			events &= ~MONO_PROFILE_ALLOCATIONS;
+			events &= ~MONO_PROFILE_GC_MOVES;


Couple of things. I believe the commit message is wrong.. The serve no purpose when allocation profiling is disabled.

More importantly, have you discussed this change with the profiler team?

The serve no purpose when allocation profiling is disabled.

Oops.

More importantly, have you discussed this change with the profiler team?

AFAIK, they always enable allocation events anyway, so this shouldn't affect them. But I'll check with Moya to be sure.

We only enable allocations if the user loads the 'Allocations' instrument.

I guess this is about what @alexrp asked yesterday ("would it be a problem for you guys if the profiler stopped producing gc move events when allocation events are disabled?")? If so, yes, if we don't enable allocations, not having GC_MOVES is ok for us. In fact, getting GC_MOVES events when we are not tracking allocations just gives us some unneeded extra work (reading those events just to find out we know nothing about those allocations being moved).

kumpera · 2016-09-19T23:44:17Z

Besides the cosmetic suggestions, the only issue is on always starting the sampler thread when a profiler module is installed.

As mentioned, the SDKs install profilers and that will screw with power consumption.

dnfclas added the cla-already-signed label Sep 9, 2016

alexrp changed the title ~~Profiler changes to enable better support for dynamic event flags changes~~ [WIP] Profiler changes to enable better support for dynamic event flags changes Sep 12, 2016

alexrp changed the title ~~[WIP] Profiler changes to enable better support for dynamic event flags changes~~ Profiler changes to enable better support for dynamic event flags changes Sep 12, 2016

Alex Rønne Petersen added 26 commits September 13, 2016 17:52

[profiler] Only emit code chunk events when MONO_PROFILE_JIT_COMPILAT…

53ce76f

…ION is set.

[profiler] Add a new internal API to switch the runtime to profiling …

3c273ab

…mode.

[profiler] Add some documentation for event flags.

6c60be2

[profiler] Add counters for all event types.

1b950f0

[profiler] Fix the code that synthesizes thread end events.

6c730c5

It could previously fail to process a newly allocated buffer.

[profiler] Don't emit a perf counters sample event if none are regist…

2bcc30f

…ered.

[profiler] Simplify counters initialization.

d12e7bb

[profiler] Don't leak counter data structures on shutdown.

4703e01

[profiler] Mark mono_profiler_get_events () as MONO_RT_EXTERNAL_ONLY.

54ba5a2

The runtime itself should access the mono_profiler_events variable directly.

[profiler] Lift profiler limitation in mono_class_get_allocation_ftn ().

0abe70c

This has not actually been necessary since allocation reporting was moved into the GC implementations.

[profiler] Allow changing sampling parameters at runtime.

174e554

The sampling thread is now always started when profiling, but sits idle if statistical sampling is not enabled.

[profiler] Expose sampling mode/frequency variables directly to the r…

2a8c6d0

…untime. This avoids doing calls in the sampling thread all the time.

[profiler] Use the signal shutdown logic for the sampling thread for …

461d0aa

…more cases.

[man] Update mprof-report.1 to indicate the correct default maxframes…

2f45c5f

… value.

[profiler] Remove the countersonly option.

2a475ca

It was never documented and is no longer functional.

[profiler] Document and fix the onlycoverage option.

c97d2e7

[profiler] Remove the long-obsolete hsmode option.

4082074

Also refactor the heap walk code a bit.

[utils] Move MONO_DEPRECATED to mono-publib.h and make it imply MONO_…

80a2fba

…API + MONO_RT_EXTERNAL_ONLY. Also make it work with MSVC.

[profiler] Fire method exceptional exit events for MONO_PROFILE_ENTER…

29fd30e

…_LEAVE rather than MONO_PROFILE_EXCEPTIONS. This event isn't that useful at profiling exceptions. It's mainly meant to keep track of call depth (for enter/leave profiling purposes) when an exception is thrown.

[profiler] Add MONO_PROFILE_GC_HANDLES and use it for GC handle event…

3153441

…s instead of MONO_PROFILE_GC_ROOTS.

[profiler] Only emit heap root events when we do a heapshot.

66c74c3

These have no value outside of heapshots, and in large apps, tend to contribute significantly to log file bloat.

[sgen] Remove bogus mono_profiler_events assert.

eeb2d37

This variable can be changed at any time by any thread. It makes no sense to assert that it has a particular flag.

Alex Rønne Petersen added 8 commits September 14, 2016 21:49

[utils/threads] Export all native thread functions.

b8ef38c

[utils/threads] Add and export a mono_native_thread_join () function.

5dc9108

[profiler] Use the native thread wrapper functions to manage threads.

b83a537

[profiler] Warn if we can't detect the CPU count.

22d6698

[profiler] Use eglib alloc/free functions everywhere.

a867af5

[profiler] Address a couple of FIXMEs in the counters sampling code.

6c36051

[profiler] Add a nocounters option.

d1fe4ae

[profiler] Don't emit enter/leave events in onlycoverage mode.

5e1e678

kumpera approved these changes Sep 14, 2016

View reviewed changes

kumpera reviewed Sep 14, 2016

View reviewed changes

mono/utils/mono-compiler.h

#define MONO_LLVM_INTERNAL

#endif

#if HAVE_DEPRECATED

Copy link

Contributor

kumpera Sep 14, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should remove the check from configure.

kumpera reviewed Sep 14, 2016

View reviewed changes

kumpera reviewed Sep 15, 2016

View reviewed changes

[profiler] Fix a regression that caused all buffers to have a zero th…

474e374

…read ID. This was introduced in 928b840.

kumpera reviewed Sep 16, 2016

View reviewed changes

kumpera reviewed Sep 19, 2016

View reviewed changes

alexrp mentioned this pull request Oct 2, 2016

Profiler fixes for Cycle 9 #3688

Merged

alexrp closed this Jun 22, 2017

Profiler changes to enable better support for dynamic event flags changes #3546

Profiler changes to enable better support for dynamic event flags changes #3546

Uh oh!

Conversation

alexrp commented Sep 9, 2016

Uh oh!

BrzVlad commented Sep 12, 2016

Uh oh!

kumpera commented Sep 14, 2016

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

alexrp Sep 14, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kumpera Sep 19, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kumpera commented Sep 19, 2016

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

alexrp Sep 14, 2016 •

edited

Loading

kumpera Sep 19, 2016 •

edited

Loading