Fix racy initialization with KINETO_USE_DAEMON=1 #1157
Closed
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Summary:
TIThe current state of `initProfiler` leads to data races in several different places during kineto's initialization.
```
static void initProfilers() {
if (!initialized) {
libkineto::api().initProfilerIfRegistered();
libkineto::api().configLoader().initBaseConfig();
initialized = true;
VLOG(0) << "libkineto profilers activated";
}
}
```
The root-cause is `libkineto::api().initProfilerIfRegistered()` spawns a thread on `updateConfigThread` that periodically calls `updateBaseConfig`
while `libkineto::api().configLoader().initBaseConfig()` on the main thread also calls `updateBaseConfig` (this is basically all it does afaict)
this can lead to racing in the initialization of "singletons" in multiple spots, the most relevant I've seen so far:
1. `DaemonConfigLoader::getConfigClient()` (cause of [this reported crash](pytorch/pytorch#163545)
2. `ConfigLoader::daemonConfigLoader()`
It appears `initBaseConfig` was added >4 years ago as a temporary workaround for an event profiler bug ([ref](pytorch@61a7083)).
Given that the event profiler is no longer supported ([ref](pytorch@61a7083)) we should be able to remove this.
Differential Revision: D84663094
|
cc: @sanrise @sraikund16 @briancoutinho @staugust from discussion on pytorch/pytorch#163545 |
aaronenyeshi
approved these changes
Oct 15, 2025
|
This pull request has been merged in 6fcbc53. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary:
The current state of
initProfilersleads to data races in several different places during kineto's initialization.The root-cause is
libkineto::api().initProfilerIfRegistered()spawns a thread onupdateConfigThreadthat periodically callsupdateBaseConfigwhile
libkineto::api().configLoader().initBaseConfig()on the main thread also callsupdateBaseConfig(this is basically all it does afaict)this can lead to racing in the initialization of "singletons" in multiple spots, the most relevant I've seen so far:
DaemonConfigLoader::getConfigClient()(cause of this reported crashConfigLoader::daemonConfigLoader()It appears
initBaseConfigwas added >4 years ago as a temporary workaround for an event profiler bug (ref).Given that the event profiler is no longer supported (ref) we should be able to remove this.
Differential Revision: D84663094
Test Plan
Observe registration of process in dynolog's logs
I ran this 100 times and observed 0 crashes (previously was seeing crashes ~80-90% of the time)