Fix fewest modules OOM #4986
Fix fewest modules OOM #4986
Conversation
|
TY for taking a stab at this. I'll start working through it. For starters:
IIUC But indeed, this means we might perform a traversal merely for the purpose of tracking the excluded hop count (which we will not need in |
|
Yep, you are right. I was incorrectly thinking that it was possible to get it to not propagate the new excluded hop count to a previously visited path but that is impossible. It does mean that it has to traverse all paths to find the maximum but we can simply do that as a second pass without worrying about the actual path and blowing up the heap. |
|
Some rough benchmarks with 6G of memory running
I'm fairly confident that it is doing the right thing. Here is what our modules look like on our master branch with 1.6.0 before I ran into this bug: Here is after my big refactor which started triggering the OOMes with the changes in this PR: |
| // Revisit static dependencies again when the module id changes. | ||
| // We do not need to revisit dynamic dependencies because by definition they are not used by public | ||
| // modules, the first time we tag them is the shortest path so that path end never changes and only the | ||
| // ends are used for the module ids. |
There was a problem hiding this comment.
This is the key. If we revisit dynamic dependencies then it blows up. I'm pretty sure that this holds up. I tried coming with a counter example but I couldn't.
|
Dunno what's up with those two targets that fail to find the logs. Is that flaky? |
|
Logs are automatically deleted after a while (one week I think?) because they're really huge and fill up our hard disks pretty quickly. The pass/fail information is kept and is still relevant. I'll rerun the tests so you get the logs back. If you want to look at them later, make a copy. ;) |
|
Ah Scalastyle. Pretty cryptic error message |
|
Oh nope there were trailing commas. Lovely... |
| assertNotEquals(initialStaticModuleID, finalStaticModuleID) | ||
| assertNotEquals(finalPublicModuleID, finalStaticModuleID) | ||
| // TODO: Why are there 3 modules and not 2? | ||
| assertNotEquals(finalDynamicModuleID, finalStaticModuleID) |
There was a problem hiding this comment.
I thought there should have been 2 modules but there are 3. Is that right?
There was a problem hiding this comment.
Yes:
- The public module with the public class
- The dynamic module with the dynamic class
- The core module with the shared dependencies of the public and the dynamic class (this cannot be in the public module, because it must be imported by the dynamic module).
Fixes #4985
This is a first attempt. I'm not sure if it is doing the right thing yet but its fast and doesn't throw an OOME and it passes the various assertions in
ModuleSplitter(which caught one bug).I ended up creating a separate tagger as keeping track of the
maxExcludedHopCountwas getting pretty complicated and I was struggling to keep the logic correct. It isn't even used forFewestModules.maxExcludedHopCountactually doesn't look right to me. AFAICT the value you end up with depends on the order that the paths are traversed (which is undefined). If a longer path with more exclusions is traversed first you might get a longer path than if a shorter path is traversed first because theif (updated)check will prevent the dependencies being traversed if a shorter or equal path already exists.