Thanks to visit codestin.com
Credit goes to github.com

Skip to content

perf: convert _walk_js_tree from recursive generator to iterative#1294

Merged
safishamsi merged 1 commit into
safishamsi:v8from
sirphilliptubell:perf/fix4
Jun 13, 2026
Merged

perf: convert _walk_js_tree from recursive generator to iterative#1294
safishamsi merged 1 commit into
safishamsi:v8from
sirphilliptubell:perf/fix4

Conversation

@sirphilliptubell

@sirphilliptubell sirphilliptubell commented Jun 12, 2026

Copy link
Copy Markdown

Eliminates O(depth) generator-frame chain overhead. Each leaf's value previously propagated through 26+ yield-from frames; now a single stack loop handles the full traversal.

On my corporate codebase this reduced module>main>main>extract>_augment_symbol_resolution_edges from 195s -> 84s

Before:

[graphify extract] scanning C:\myrepo
[graphify extract] found 36456 code, 0 docs, 0 papers, 0 images
[graphify extract] AST extraction on 36456 code files...
  AST extraction: 100/36456 uncached files (0%) [20 workers]
  AST extraction: 200/36456 uncached files (0%) [20 workers]
<trimmed>
  AST extraction: 36300/36456 uncached files (99%) [20 workers]
  AST extraction: 36400/36456 uncached files (99%) [20 workers]
  AST extraction: 36456/36456 files (100%) [20 workers]
[graphify] Deduplicated 125511 node(s) (124857 exact, 580 fuzzy).
[graphify extract] wrote C:\myrepo\graphify-out\graph.json: 510992 nodes, 970955 edges, 29659 communities
[graphify extract] wrote C:\myrepo\graphify-out\.graphify_analysis.json
[graphify extract] next: run `graphify cluster-only C:\myrepo` to generate GRAPH_REPORT.md and name communities

  _     ._   __/__   _ _  _  _ _/_   Recorded: 13:57:57  Samples:  981477
 /_//_/// /_\ / //_// / //_'/ //     Duration: 1198.852  CPU time: 1087.500
/   _/                      v5.1.2

Program: graphify .

1198.866 <module>  ..\graphify\__main__.py:1
└─ 1198.833 main  ..\graphify\__main__.py:2070
   └─ 1198.600 main  ..\graphify\__main__.py:2070
      ├─ 446.998 extract  ..\graphify\extract.py:11293
      │  ├─ 195.316 _augment_symbol_resolution_edges  ..\graphify\extract.py:7827
      │  │  ├─ 172.086 _collect_js_symbol_resolution_facts  ..\graphify\extract.py:7487
      │  │  │  ├─ 144.229 _walk_js_tree  ..\graphify\extract.py:7194
      │  │  │  │  └─ 138.517 _walk_js_tree  ..\graphify\extract.py:7194
      │  │  │  │     └─ 133.040 _walk_js_tree  ..\graphify\extract.py:7194
      │  │  │  │        └─ 127.268 _walk_js_tree  ..\graphify\extract.py:7194
      │  │  │  │           └─ 121.598 _walk_js_tree  ..\graphify\extract.py:7194
      │  │  │  │              └─ 115.943 _walk_js_tree  ..\graphify\extract.py:7194
      │  │  │  │                 └─ 109.930 _walk_js_tree  ..\graphify\extract.py:7194
      │  │  │  │                    └─ 104.084 _walk_js_tree  ..\graphify\extract.py:7194
      │  │  │  │                       └─ 97.903 _walk_js_tree  ..\graphify\extract.py:7194
      │  │  │  │                          └─ 91.412 _walk_js_tree  ..\graphify\extract.py:7194
      │  │  │  │                             └─ 84.709 _walk_js_tree  ..\graphify\extract.py:7194
      │  │  │  │                                └─ 78.958 _walk_js_tree  ..\graphify\extract.py:7194
      │  │  │  │                                   └─ 73.395 _walk_js_tree  ..\graphify\extract.py:7194
      │  │  │  │                                      └─ 65.729 _walk_js_tree  ..\graphify\extract.py:7194
      │  │  │  │                                         └─ 60.322 _walk_js_tree  ..\graphify\extract.py:7194
      │  │  │  │                                            └─ 54.653 _walk_js_tree  ..\graphify\extract.py:7194
      │  │  │  │                                               └─ 49.202 _walk_js_tree  ..\graphify\extract.py:7194
      │  │  │  │                                                  └─ 43.343 _walk_js_tree  ..\graphify\extract.py:7194
      │  │  │  │                                                     └─ 38.016 _walk_js_tree  ..\graphify\extract.py:7194
      │  │  │  │                                                        └─ 32.767 _walk_js_tree  ..\graphify\extract.py:7194
      │  │  │  │                                                           └─ 28.526 _walk_js_tree  ..\graphify\extract.py:7194
      │  │  │  │                                                              └─ 23.547 _walk_js_tree  ..\graphify\extract.py:7194
      │  │  │  │                                                                 └─ 19.959 _walk_js_tree  ..\graphify\extract.py:7194
      │  │  │  │                                                                    └─ 16.834 _walk_js_tree  ..\graphify\extract.py:7194
      │  │  │  │                                                                       └─ 14.309 _walk_js_tree  ..\graphify\extract.py:7194
      │  │  │  └─ 15.618 [self]  ..\graphify\extract.py
      │  │  └─ 22.532 _apply_symbol_resolution_facts  ..\graphify\extract.py:6958
      │  ├─ 149.658 _extract_parallel  ..\graphify\extract.py:11173
      │  │  └─ 141.965 as_completed  concurrent\futures\_base.py:193
      │  │        [3 frames hidden]  threading, <built-in>
      │  ├─ 41.410 load_cached  ..\graphify\cache.py:226
      │  │  └─ 24.127 file_hash  ..\graphify\cache.py:97
      │  │     └─ 13.407 WindowsPath.resolve  pathlib\__init__.py:937
      │  │        └─ 12.881 realpath  <frozen ntpath>:705
      │  └─ 31.424 _disambiguate_colliding_node_ids  ..\graphify\extract.py:6759
      │     └─ 15.351 _source_key  ..\graphify\extract.py:6748
      ├─ 392.513 cluster  ..\graphify\cluster.py:86
      │  ├─ 317.511 _partition  ..\graphify\cluster.py:22
      │  │  └─ 282.928 func  networkx\utils\decorators.py:783
      │  │     └─ 282.928 argmap_louvain_communities_5  <class 'networkx.utils.decorators.argmap'> compilation 9:1
      │  │        └─ 282.928 _dispatchable._call_if_no_backends_installed  networkx\utils\backends.py:541
      │  │           └─ 282.928 louvain_communities  networkx\algorithms\community\louvain.py:14
      │  │              └─ 282.861 louvain_partitions  networkx\algorithms\community\louvain.py:133
      │  │                 └─ 251.421 _one_level  networkx\algorithms\community\louvain.py:227
      │  │                    ├─ 124.533 [self]  networkx\algorithms\community\louvain.py
      │  │                    └─ 113.433 _neighbor_weights  networkx\algorithms\community\louvain.py:335
      │  └─ 67.669 _split_community  ..\graphify\cluster.py:191
      │     └─ 63.805 _partition  ..\graphify\cluster.py:22
      │        └─ 36.579 argmap_louvain_communities_5  <class 'networkx.utils.decorators.argmap'> compilation 9:1
      │           └─ 36.571 _dispatchable._call_if_no_backends_installed  networkx\utils\backends.py:541
      │              └─ 36.569 louvain_communities  networkx\algorithms\community\louvain.py:14
      │                 └─ 36.538 louvain_partitions  networkx\algorithms\community\louvain.py:133
      │                    └─ 16.220 _one_level  networkx\algorithms\community\louvain.py:227
      ├─ 193.522 detect  ..\graphify\detect.py:997
      │  ├─ 167.288 _is_ignored  ..\graphify\detect.py:760
      │  │  └─ 156.078 _eval  ..\graphify\detect.py:783
      │  │     └─ 135.357 _matches  ..\graphify\detect.py:787
      │  │        ├─ 119.444 fnmatch  fnmatch.py:22
      │  │        │     [5 frames hidden]  <frozen ntpath>, <built-in>, fnmatch
      │  │        └─ 12.114 [self]  ..\graphify\detect.py
      │  └─ 12.945 count_words  ..\graphify\detect.py:624
      ├─ 79.453 to_json  ..\graphify\export.py:484
      │  └─ 68.974 dump  json\__init__.py:120
      │        [4 frames hidden]  json
      └─ 56.839 build  ..\graphify\build.py:276
         └─ 45.248 build_from_json  ..\graphify\build.py:107

To view this report with different options, run:
    pyinstrument --load-prev 2026-06-12T13-57-57 [options]

Elapsed: 1420.9s

After:

[graphify extract] scanning C:\myrepo
[graphify extract] found 36456 code, 0 docs, 0 papers, 0 images
[graphify extract] AST extraction on 36456 code files...
  AST extraction: 100/36456 uncached files (0%) [20 workers]
  AST extraction: 200/36456 uncached files (0%) [20 workers]
<trimmed>
  AST extraction: 36300/36456 uncached files (99%) [20 workers]
  AST extraction: 36400/36456 uncached files (99%) [20 workers]
  AST extraction: 36456/36456 files (100%) [20 workers]
[graphify] Deduplicated 125510 node(s) (124857 exact, 580 fuzzy).
[graphify extract] wrote C:\myrepo\graphify-out\graph.json: 510993 nodes, 970955 edges, 29755 communities
[graphify extract] wrote C:\myrepo\graphify-out\.graphify_analysis.json
[graphify extract] next: run `graphify cluster-only C:\myrepo` to generate GRAPH_REPORT.md and name communities

  _     ._   __/__   _ _  _  _ _/_   Recorded: 14:50:25  Samples:  913401
 /_//_/// /_\ / //_// / //_'/ //     Duration: 1368.850  CPU time: 987.984
/   _/                      v5.1.2

Program: graphify .

1368.865 <module>  ..\graphify\__main__.py:1
└─ 1368.834 main  ..\graphify\__main__.py:2070
   └─ 1368.566 main  ..\graphify\__main__.py:2070
      ├─ 420.841 cluster  ..\graphify\cluster.py:86
      │  ├─ 315.731 _partition  ..\graphify\cluster.py:22
      │  │  └─ 281.434 func  networkx\utils\decorators.py:783
      │  │     └─ 281.433 argmap_louvain_communities_5  <class 'networkx.utils.decorators.argmap'> compilation 9:1
      │  │        └─ 281.433 _dispatchable._call_if_no_backends_installed  networkx\utils\backends.py:541
      │  │           └─ 281.433 louvain_communities  networkx\algorithms\community\louvain.py:14
      │  │              └─ 281.367 louvain_partitions  networkx\algorithms\community\louvain.py:133
      │  │                 └─ 247.689 _one_level  networkx\algorithms\community\louvain.py:227
      │  │                    ├─ 124.306 [self]  networkx\algorithms\community\louvain.py
      │  │                    └─ 110.314 _neighbor_weights  networkx\algorithms\community\louvain.py:335
      │  └─ 96.767 _split_community  ..\graphify\cluster.py:191
      │     └─ 91.448 _partition  ..\graphify\cluster.py:22
      │        └─ 59.034 argmap_louvain_communities_5  <class 'networkx.utils.decorators.argmap'> compilation 9:1
      │           └─ 59.020 _dispatchable._call_if_no_backends_installed  networkx\utils\backends.py:541
      │              └─ 59.019 louvain_communities  networkx\algorithms\community\louvain.py:14
      │                 └─ 58.968 louvain_partitions  networkx\algorithms\community\louvain.py:133
      │                    ├─ 26.548 _one_level  networkx\algorithms\community\louvain.py:227
      │                    └─ 14.976 argmap_modularity_19  <class 'networkx.utils.decorators.argmap'> compilation 22:1
      │                       └─ 14.968 _dispatchable._call_if_no_backends_installed  networkx\utils\backends.py:541
      │                          └─ 14.920 modularity  networkx\algorithms\community\quality.py:144
      ├─ 328.208 save_manifest  ..\graphify\detect.py:1229
      │  └─ 323.129 _md5_file  ..\graphify\detect.py:1149
      │     └─ 320.476 WindowsPath.open  pathlib\__init__.py:768
      │        └─ 320.433 open  <built-in>
      ├─ 269.743 extract  ..\graphify\extract.py:11298
      │  ├─ 107.038 _extract_parallel  ..\graphify\extract.py:11178
      │  │  └─ 102.795 as_completed  concurrent\futures\_base.py:193
      │  │        [3 frames hidden]  threading, <built-in>
      │  ├─ 83.613 _augment_symbol_resolution_edges  ..\graphify\extract.py:7832
      │  │  ├─ 60.454 _collect_js_symbol_resolution_facts  ..\graphify\extract.py:7492
      │  │  │  └─ 43.993 _walk_js_tree  ..\graphify\extract.py:7194
      │  │  └─ 22.442 _apply_symbol_resolution_facts  ..\graphify\extract.py:6958
      │  ├─ 36.282 load_cached  ..\graphify\cache.py:226
      │  │  └─ 21.613 file_hash  ..\graphify\cache.py:97
      │  └─ 20.552 _disambiguate_colliding_node_ids  ..\graphify\extract.py:6759
      ├─ 168.004 detect  ..\graphify\detect.py:997
      │  └─ 149.078 _is_ignored  ..\graphify\detect.py:760
      │     └─ 139.281 _eval  ..\graphify\detect.py:783
      │        └─ 121.024 _matches  ..\graphify\detect.py:787
      │           └─ 107.143 fnmatch  fnmatch.py:22
      │                 [5 frames hidden]  <frozen ntpath>, <built-in>, fnmatch
      ├─ 84.587 to_json  ..\graphify\export.py:484
      │  └─ 70.835 dump  json\__init__.py:120
      │        [4 frames hidden]  json
      ├─ 50.783 build  ..\graphify\build.py:276
      │  └─ 40.163 build_from_json  ..\graphify\build.py:107
      ├─ 26.108 surprising_connections  ..\graphify\analyze.py:119
      │  └─ 25.544 _cross_file_surprises  ..\graphify\analyze.py:263
      │     └─ 16.311 _is_file_node  ..\graphify\analyze.py:50
      └─ 16.402 score_all  ..\graphify\cluster.py:220
         └─ 16.355 cohesion_score  ..\graphify\cluster.py:209
            └─ 14.254 Graph.number_of_edges  networkx\classes\graph.py:1961
               └─ 14.227 Graph.size  networkx\classes\graph.py:1918
                  └─ 13.712 <genexpr>  networkx\classes\graph.py:1954

To view this report with different options, run:
    pyinstrument --load-prev 2026-06-12T14-50-25 [options]

Elapsed: 1529.1s

@ptubell-gs
perf: convert _walk_js_tree from recursive generator to iterative
a7c6980 
Eliminates O(depth) generator-frame chain overhead. Each leaf's value previously propagated through 26+ yield-from frames; now a single stack loop handles the full traversal.
@safishamsi safishamsi merged commit dbbf0a4 into safishamsi:v8 Jun 13, 2026
3 checks passed
@sirphilliptubell sirphilliptubell deleted the perf/fix4 branch June 16, 2026 19:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@sirphilliptubell @safishamsi @ptubell-gs