fix(ngcc): do not spawn unlocker processes on cluster workers #36569
Conversation
|
You can preview 038c341 at https://pr36569-038c341.ngbuilds.io/. |
There was a problem hiding this comment.
The later the unlocker child process is created the more chance there is of it not being in-place in case the main process dies (or is killed). When I was testing different positions for the creation of the child process, it was quite possible to Ctrl-C before the unlocker had been created. Can you test this? Especially for the sync version of ngcc.
|
The code between instantiating the I tested it (which always beats "assuming" 😛) and I am seeing a delay between 40ms and 90ms (which is significantly higher than I thought. I'll reword the PR. Thx for the feedback ❤️
We could, but I don't want to sprinkle the code with Sounds like we might need a |
This is specifically why I did not implement that previously. (I did know about the extra unlockers but I kept quiet about it 🙀). Are there any cases where |
The current ngcc lock-file strategy spawns a new process in order to capture a potential `SIGINT` and remove the lock-file. For more information see angular#35861. Previously, this unlocker process was spawned as soon as the `LockFile` was instantiated in order to have it available as soon as possible (given that spawning a process is an asynchronous operation). Since the `LockFile` was instantiated and passed to the `Executor`, this meant that an unlocker process was spawned for each cluster worker, when running ngcc in parallel mode. These processes were not needed, since the `LockFile` was not used in cluster workers, but we still had to pay the overhead of each process' own memory and V8 instance. (NOTE: This overhead was small compared to the memory consumed by ngcc's normal operations, but still unnecessary.) This commit avoids the extra processes by only spawning an unlocker process when running on the cluster master process and not on worker processes.
On Windows, the output of a detached process (such as the unlocker process used by `LockFileWithChildProcess`) is not shown in the parent process' stdout. This commit addresses this by piping the spawned process' stdin/stdout and manually writing to the parent process' stdout.
038c341 to
f6304ea
Compare
|
I think I changed the first commit to use a |
|
You can preview f6304ea at https://pr36569-f6304ea.ngbuilds.io/. |
| * A `LockFileWithChildProcess` that is `cluster`-aware and does not spawn unlocker processes from | ||
| * worker processes (only from the master process, which does the locking). | ||
| */ | ||
| export class ClusterLockFileWithChildProcess extends LockFileWithChildProcess { |
There was a problem hiding this comment.
Rather than extending LockFileWithChildProcess, how about using composition instead...
export class ClusterLockFile implements LockFile {
constructor(private delegate: LockFileWithChildProcess) {
if (cluster.isMaster) {
delegate.createUnlocker();
}
}
... etc
}Obviously we would need to make createUnlocker() public, but this would make testing a lot cleaner, avoiding all those spies, since you could just pass a mock delegate in for testing.
There was a problem hiding this comment.
It might need a bit of tweaking.... to work properly but you get the idea...
There was a problem hiding this comment.
Or perhaps we could move this logic into the Locker classes themselves? Saving the middleman?
There was a problem hiding this comment.
Rather than extending
LockFileWithChildProcess, how about using composition instead...
That would be my preference too, but the delegate would have spawned the process in its constructor. There are ways around that but they either make things more complicated than they already are or end up delaying the creation of the unlocker (which we are trying to avoid here).
Or perhaps we could move this logic into the Locker classes themselves? Saving the middleman?
Not sure what you mean by that 😅
There was a problem hiding this comment.
On further thought, over the weekend, I am happy to approve this.
My big gripe, outside of this PR, is that we are kicking off the worker processes with the same entry-point as the master process, which requires us to jump through these hoops.
I would like to propose that we land this but then refactor the whole clustering approach to avoid having to do this.
On Windows, the output of a detached process (such as the unlocker process used by `LockFileWithChildProcess`) is not shown in the parent process' stdout. This commit addresses this by piping the spawned process' stdin/stdout and manually writing to the parent process' stdout. PR Close #36569
The current ngcc lock-file strategy spawns a new process in order to capture a potential `SIGINT` and remove the lock-file. For more information see #35861. Previously, this unlocker process was spawned as soon as the `LockFile` was instantiated in order to have it available as soon as possible (given that spawning a process is an asynchronous operation). Since the `LockFile` was instantiated and passed to the `Executor`, this meant that an unlocker process was spawned for each cluster worker, when running ngcc in parallel mode. These processes were not needed, since the `LockFile` was not used in cluster workers, but we still had to pay the overhead of each process' own memory and V8 instance. (NOTE: This overhead was small compared to the memory consumed by ngcc's normal operations, but still unnecessary.) This commit avoids the extra processes by only spawning an unlocker process when running on the cluster master process and not on worker processes. PR Close #36569
On Windows, the output of a detached process (such as the unlocker process used by `LockFileWithChildProcess`) is not shown in the parent process' stdout. This commit addresses this by piping the spawned process' stdin/stdout and manually writing to the parent process' stdout. PR Close #36569
…angular#36569)" This reverts commit 66effde.
|
This issue has been automatically locked due to inactivity. Read more about our automatic conversation locking policy. This action has been performed automatically by a bot. |
The current ngcc lock-file strategy spawns a new process in order to capture a potential
SIGINTand remove the lock-file. For more information see #35861.Previously, this unlocker process was spawned as soon as the
LockFilewas instantiated in order to have it available as soon as possible (given that spawning a process is an asynchronous operation). Since theLockFilewas instantiated and passed to theExecutor, this meant that an unlocker process was spawned for each cluster worker, when running ngcc in parallel mode. These processes were not needed, since theLockFilewas not used in cluster workers, but we still had to pay the overhead of each process' own memory and V8 instance.(NOTE: This overhead was small compared to the memory consumed by ngcc's normal operations, but still unnecessary.)
This commit avoids the extra processes by only spawning an unlocker process when first creating a lock-file (which only happens on the cluster master process). Since instantiating the
LockFileand attempting to lock happen on the same tick, this change is unlikely to cause any difference in actual behavior.