It looks like this pull request may not have tests. Please make sure to add tests or get an explicit test exemption before merging.

If you are not sure if you need tests, consider this rule of thumb: the purpose of a test is to make sure someone doesn't accidentally revert the fix. Ask yourself, is there anything in your PR that you feel it is important we not accidentally revert back to how it was before your fix?

Reviewers: Read the Tree Hygiene page and make sure this patch meets those guidelines before LGTMing.If you believe this PR qualifies for a test exemption, contact "@test-exemption-reviewer" in the #hackers channel in Discord (don't just cc them here, they won't see it!). The test exemption team is a small volunteer group, so all reviewers should feel empowered to ask for tests, without delegating that responsibility entirely to the test exemption group.

FYI @DanTup, I wonder if Dart-Code needs some updates to acomodate for it. I think there is some reliance on the fact that you get pause event in each reloaded isolate at the end of the reload.

FYI @DanTup, I wonder if Dart-Code needs some updates to acomodate for it. I think there is some reliance on the fact that you get pause event in each reloaded isolate at the end of the reload.

I'm not sure I understand this change well enough to know if anything needs changing, but the reason we use pause events when isolates are reloaded (PausePostRequest?) is so we can re-send any breakpoints and configure break-on-exception behaviour for that isolate, before finally resuming.

With this change, will each isolates still pause after the reload and send this event for each?

Edit: FWIW, the debug adapter implementation is now in the SDK so the related code is around here:

https://github.com/dart-lang/sdk/blob/6d6da849ba3dedd99b3e57448bc92761dba6c80e/pkg/dds/lib/src/dap/isolate_manager.dart#L677

Although at a glance, I think this code says "hot restart" in a few places where it means "hot reload" (IIRC, a hot restart creates new isolates and the old ones exit, and the PausePostRequest is just for reload?)

Can you shed a bit more light on why we need to pause and resend breakpoints after reload?

So here is what is happening now. Imagine you have two isolates in the same isolate group: I1 and I2. We trigger hot reload with pause: true from the flutter tool (or daemon). It will send reload requests to both I1 and I2 in some order. Lets say the order is I1 and then I2. I1 will reload, will be paused after reload, PauseAfterRequest will be dispatched and vm-service client will do whatever it wants with the isolate and resume it. Then the same will happen to I2: it will reload, will be paused after reload, PauseAfterRequest will be dispatched and vm-service client will do whatever it wants with the isolate and resume it. Note, however, that after I1 was reloaded I2 is not paused and already runs new code because they are in the same group. So there is a gap where I2 runs new code and whatever needs to happen in PauseAfterRequest has not happened.

What is going to happen after this change: one of two isolates get the reload request, gets reloaded and paused. There is a single PauseAfterRequest dispatched and only one isolate is paused in that moment.

Does it help? So I would like to understand what is going to break (that is not broken now - because there is certainly some strangeness already) and what we can do to address that? (And for that I would like to understand why pause after reload is even necessary in the first place).

Finally, I think it is also worth for you to look at: #169517. There is strange stuff happening there with rather long (multisecond) debugger pauses after reload.

Can you shed a bit more light on why we need to pause and resend breakpoints after reload?

It was my understanding that when we hot reload, any code that was replaced would lose the breakpoints (eg. if there was a breakpoint inside a function that was replaced, after a hot reload there would be no breakpoint in the VM in the new version of the function). However, I'm not all that familiar with the internals of hot reload or the debugger - perhaps @bkonyi can confirm if this is the case or not (it would be fantastic if we did not need to pause and resend breakpoints during a hot reload).

What is going to happen after this change: one of two isolates get the reload request, gets reloaded and paused. There is a single PauseAfterRequest dispatched and only one isolate is paused in that moment.
So I would like to understand what is going to break

If the above is true, then my guess is that breakpoints would be "lost" in any new code in the isolates that do not trigger the PauseAfterRequest event.

Finally, I think it is also worth for you to look at: #169517. There is strange stuff happening there with rather long (multisecond) debugger pauses after reload.

Reloads/restarts will definitely take longer with more isolates because of the pause/work/resume we do noted above (and in some cases setting up those breakpoints might involve more requests to the VM to map file:/// URIs into Dart VM URIs). In VS Code we have a Dart: Capture Debugging Logs command that will capture both Debug Adapter and VM Service traffic during a reload that might help indicate if it's these requests adding to the time).

Thanks @DanTup. Now that I think about it - it makes sense that we need to resend breakpoints because source locations could have shifted as IDE makes breakpoints stick to the line and updates them as user is adding and removing lines above.

I have looked around in the VM code and I don't see reload itself doing anything to breakpoints - we just keep the old list around.

This does however mean I can't really land this change as is, because it will cause issues with breakpoints. Here is what I am going to do instead:

• Step 1: I will update VM to include pauseGroup parameter for reloadSources. If pauseGroup is true then it will pause all isolates in the group after the reload (this will also eliminate the race which I have described above, where some isolates start running before their breakpoints are updated).
• Step 2: We roll VM and related package into Flutter.
• Step 3: I update this PR to pass pauseGroup instead of pause

That sounds reasonable to me :-)

I have looked around in the VM code and I don't see reload itself doing anything to breakpoints - we just keep the old list around.

From what I recall, breakpoints are set on Code objects in the VM, so any breakpoints associated with reloaded code are invalidated. This effectively meant that we needed to reset breakpoints for each changed library (@derekxu16 might remember more details than me).

@bkonyi I think that's should not be a problem, when we recompile new code for the function compiler will then notify the debugger (GroupDebugger::NotifyCompilation) and that should reinstall the breakpoint into the new version of the code.

I think the problem is:

1. locations change due to editing, as I have mentioned.
2. it seems GroupDebugger::code_breakpoints_ seems to be holding to Code objects strongly, if I am not missing anything. Which means we are effectively leaking code objects which were discarded by reload.

Clearing all breakpoints and reinstalling them helps with both problems.

I have been looking at this more and I am starting to think that properly implementing this (i.e. atomically pausing the whole group without giving it a chance to run after the reload) is going to be non-trivial. Main challenge here is that isolates participating in the reload might be in different parts of their lifecycle (e.g. there are RawReloadParticipationScope in FreeActiveThread and ExitIsolate). I would need to be extremely careful on how I handle this - because just trying to do PausePostRequest might be a recipe for disaster.

(I think we should have made breakpoint reinstallation part of reloadSources API, then we could have atomically reinstalled breakpoints after the reload without requiring round trips through the service - but alas the ship has sailed on that one, introducing this now require too much plumbing).

(I think we should have made breakpoint reinstallation part of reloadSources API, then we could have atomically reinstalled breakpoints after the reload without requiring round trips through the service - but alas the ship has sailed on that one, introducing this now require too much plumbing).

Out of curiosity, how would that have worked? Would we just reset the breakpoints at their existing source location, even if they're no longer valid, and send events for the IDE to update their state? I guess that's effectively what the IDEs are doing anyway.

I was told eons ago as a junior engineer (maybe by @rmacnak-google or @johnmccutchan) that there was a reason that the VM couldn't accurately reset the breakpoints after a reload and that the IDEs should be responsible for resetting them since they know exactly where they should live. I'm only just realizing that I don't remember exactly why this is the case, and if there's a way that we can handle breakpoint reinstallation automatically from within the runtime I think that's something we should investigate doing in the future as part of a major version bump to the service protocol. It'd greatly reduce complexity in some of our tooling if it's something we could do.

Out of curiosity, how would that have worked? Would we just reset the breakpoints at their existing source location, even if they're no longer valid, and send events for the IDE to update their state? I guess that's effectively what the IDEs are doing anyway.

We would need to pass new breakpoint locations to the reloadSources. So effectively instead of reloadSources(pause: true) followed by deleting and adding breakpoints during subsequent pause we would have reloadSources(breakpoints: </*Isolate Id*/ String, List<BreakpointInfo>>{}).

I think the API with pause makes sense in the pre-isolate-groups world, but with isolate groups you would like to update all breakpoints in all isolates in one go and you have capability to do so (because all isolates are safepointed anyway), but the current API does not actually allow it.

As I have mentioned above with the current API (and how it is currently used by flutter CLI) there is actually a race between isolate group source changing, isolates resuming running for a bit with old breakpoint locations. Imagine two isolates in the same group:

                                        bps updates in I1
                       Group reloaded   | 
                       |                |
Isolate 1 -------------|##[reload][pause]----
                       |  
Isolate 2 ------[reload][pause]--------------
                              | bps updated in I2

Here Isolate 1 runs for a bit with reloaded source but with old breakpoints.

there was a reason that the VM couldn't accurately reset the breakpoints

That is correct. VM can't really update breakpoints in the same way IDE can. I did not immediately realize it (hence I stared asking why we do it), because I don't actually debug from IDEs all that often... When you edit source in the IDE (e.g. edit or remove lines), IDE actually moves breakpoints accordingly: e.g. if you add a line about breakpoint at line 10, IDE will shift breakpoint together with the line and breakpoint will now resides at line 11. That makes total sense from user perspective. VM can't really do that because it does not have the history of edits.

Out of curiosity, how would that have worked? Would we just reset the breakpoints at their existing source location, even if they're no longer valid, and send events for the IDE to update their state? I guess that's effectively what the IDEs are doing anyway.

We would need to pass new breakpoint locations to the reloadSources. So effectively instead of reloadSources(pause: true) followed by deleting and adding breakpoints during subsequent pause we would have reloadSources(breakpoints: </*Isolate Id*/ String, List<BreakpointInfo>>{}).

Ah, of course! That makes a lot of sense.

I think the API with pause makes sense in the pre-isolate-groups world, but with isolate groups you would like to update all breakpoints in all isolates in one go and you have capability to do so (because all isolates are safepointed anyway), but the current API does not actually allow it.

As I have mentioned above with the current API (and how it is currently used by flutter CLI) there is actually a race between isolate group source changing, isolates resuming running for a bit with old breakpoint locations. Imagine two isolates in the same group:

                                        bps updates in I1
                       Group reloaded   | 
                       |                |
Isolate 1 -------------|##[reload][pause]----
                       |  
Isolate 2 ------[reload][pause]--------------
                              | bps updated in I2

Here Isolate 1 runs for a bit with reloaded source but with old breakpoints.

Thanks for the explanation, that's an unfortunate race. Given the change in reload behavior introduced with isolate groups, I think it's worth filing an issue to make a breaking change to the reloadSources signature so that we can make the change when we have enough reason to do a major version bump for the service protocol spec.