@jasonk000,

Last week, we rolled out a new build of MAT from the latest commit to IBM support servers and we're getting reports of very slow parses up to 9 hours. The issue is reproducible and the majority of time is spent in:

Subtask: Removing unreachable objects
Marking reachable objects

Doing some basic profiling with thread dumps every 30 seconds during this step shows a flame graph with most samples in ObjectMarker.countMarked (the percentage on each frame is of all threads in each thread dump though only the Worker-X threads are relevant; the key point is that across 30 thread dumps, almost all Worker threads were in ObjectMarker.countMarked):

org/eclipse/mat/parser/internal/snapshot/ObjectMarker.countMarked (29 samples, 1.83%)
org/eclipse/mat/parser/internal/snapshot/ObjectMarker.markSingleThreaded (30 samples, 1.89%)
org/eclipse/mat/parser/internal/GarbageCleaner.markUnreachableAsGCRoots (30 samples, 1.89%)
org/eclipse/mat/parser/internal/GarbageCleaner.clean (30 samples, 1.89%)
org/eclipse/mat/parser/internal/SnapshotFactoryImpl.parse (30 samples, 1.89%)
org/eclipse/mat/parser/internal/SnapshotFactoryImpl.openSnapshot (30 samples, 1.89%)
org/eclipse/mat/snapshot/SnapshotFactory.openSnapshot (30 samples, 1.89%)
org/eclipse/mat/ui/snapshot/ParseHeapDumpJob.run (30 samples, 1.89%)
org/eclipse/core/internal/jobs/Worker.run (30 samples, 1.89%)
Worker-X (30 samples, 1.89%)

Taking a dump of MAT itself during this step shows that the boolean[] bits array has about 77 million entries. I guess it's just incredibly slow to iterate over so many elements in our support server environment. The support servers are virtualized Windows servers. The boxes don't appear to be particularly loaded. 2 CPU sockets and 8 virtual processors, about 25% utilized. 256GB RAM.

I then uploaded a test build of MAT from commit 35b3ff7 and re-parsed from scratch, and the parse completed in about 30 minutes.

I'll be reverting our MAT builds to that commit but let me know if you want me to run any debug builds as I can upload custom builds and I can re-try on this dump.