For the use-case, see here.

This is also useful for us at Datadog for the profiler in the ddtrace gem.

In fact, I ended up copy-pasting rb_profile_frames and keeping a separate copy inside the gem to be able to support sampling other threads: https://github.com/DataDog/dd-trace-rb/blob/f9b0f45ed93aa6ec9d36afa7c9a3c4b5b4a4706d/ext/ddtrace_profiling_native_extension/private_vm_api_access.c#L410 (among some other fixes to make it behave exactly like Thread#backtrace, etc).

On an additional related note, the backtracie gem (authored by me and @KJTsanaktsidis) also reimplements rb_profile_frames, but changes the API entirely, see our RubyKaigi 2022 talk for a discussion of why ;)

Edit: Our use-case for ddtrace is to build an accurate wall-time profiler, e.g. be also able to look at stack traces for threads that are not currently executing.

I'm very much in favor of this. A few notes.

1: This is slightly different from https://bugs.ruby-lang.org/issues/10602, as it accept VALUE thread instead of rb_thread_t *th. No big deal but it would need to be clarified if this is meant to be discussed at a developer meeting.

2: I see the need to profile only a particular thread, but I also see the need to profile all threads Thread.list.each { |th| capture(th) }. To do this efficiently in stackprof and other profilers, I think we'd need an efficient way to iterate over the live threads from C, and get some identifier for them. Alternatively, profilers can snapshot Thread.list when starting, but then it won't work well if the code being profiled spawns new threads.

3: There is the question of what to report when a thread is waiting for the GVL or stopped because GC is running. Currently since you only profile the running thread, it's not much of a consideration. But if you start profiling a specific thread, you need to properly report when it wasn't executing. Perhaps with a special frame? Perhaps with another API?

Great points!

2: I see the need to profile only a particular thread, but I also see the need to profile all threads Thread.list.each { |th| capture(th) }. To do this efficiently in stackprof and other profilers, I think we'd need an efficient way to iterate over the live threads from C, and get some identifier for them. Alternatively, profilers can snapshot Thread.list when starting, but then it won't work well if the code being profiled spawns new threads.

The ddtrace gem actually ends up shipping with its own implementation of thread list, so that it can get an up-to-date list on every sample. Link to current impl

Having a thread list API visible would be great! (And yeah we're careful when and how we call this, e.g. making sure we're in the main ractor)

Thank you all for comments!

This is slightly different from https://bugs.ruby-lang.org/issues/10602, as it accept VALUE thread instead of rb_thread_t *th.

This change is intentional, as rb_thread_t is not exposed outside of CRuby.

I think we'd need an efficient way to iterate over the live threads from C, and get some identifier for them.

Do you mean that calling Thread.list via rb_funcall would not be performant enough?

There is the question of what to report when a thread is waiting for the GVL or stopped because GC is running.

In such cases, I believe the current implementation will simply return the last stack trace executed. While I think that a rb_profile_frames()-ish API returning information about the execution state would be easy to use, I'm not sure if that would be the best design. Maybe checking the global GC state through rb_during_gc() before calling rb_thread_profile_frames() might suffice?

One more point to add: Allowing profiling of threads which don't have the GVL lets us build a more precise wall-time profiler. Profiles obtained through the current API tends to show I/O time less than actual. This is because rb_profile_frames() returns information about the thread which had the GVL - threads performing I/O have a lesser chance to be the targeted one.

A accurate wall-time profiler is wanted especially when you are profiling web apps, where I/O is frequent (HTTP responses and RDBMS calls).

Do you mean that calling Thread.list via rb_funcall would not be performant enough?

It's not so much that it would be slow. It's more that an important behavior of rb_profile_frames is that it doesn't allocate.

rb_funcall(rb_cThread, rb_intern("list"), 0) would and may trigger GC, release the GVL etc, which is very undesirable for profilers.

Maybe checking the global GC state through rb_during_gc() before calling rb_thread_profile_frames() might suffice?

For GC yes. But thinking about the "profile alls threads" use case, I'd need to know for every given thread what "state" they're in (waiting for GVL, running, etc).

All this may be considered a bit unrelated, but if we're to propose some new profiling APIs, I think we might as well consider that too.

A accurate wall-time profiler is wanted especially when you are profiling web apps, where I/O is frequent

Absolutely, hence why I'd like to expose thread status as well. If you want to profile a multi-threaded web app, you need to properly report if the thread was doing IOs or was waiting on the GVL. Because that's a major difference that will prompt you to implement radically different solutions to optimize your application.

To clarify: I don't think the concerns I raised are really blocking, I just think it would make sense to come with a new feature request that attempt to solve all these at once.

There's interest for a new profiler in ruby-core: https://github.com/rubygsoc/rubygsoc/wiki/Ideas-List-%282023%29#make-a-new-profiler, so we can probably propose more than a simple incremental improvement.

I've renamed this to rb_profile_thread_frames() from rb_thread_profile_frames() (flipped the profile and thread), assuming that these APIs should be categorized under the profile category.
Also I've added some tests.

@ko1 Can you take a look?