feat: support Dataform SQLX models - #3526
Conversation
There was a problem hiding this comment.
Graphify reviewed this change.
Worth a look — the grounded gate found no coupling regressions or blocking issues, but 3 advisory finding(s) below merit a look before merge.
Graphify review — findings
Adds a deterministic Dataform .sqlx extractor (extract_dataform) that parses the config {} block for model type/schema/dependencies and scans ref()/self() calls for dependency edges, deliberately skipping SQL and JavaScript templating. Registers .sqlx in CODE_EXTENSIONS, the extractor dispatch table, and the LANGUAGE_EXTRACTORS map, and documents the new file type in the README. Comment- and string-masking helpers keep byte offsets intact so config-key and dependency scans don't trip over quoted contents or comments.
Worth a look
- _parse_config array value uses body slice with unbounded _value_end matching brackets inside strings incorrectly —
graphify/extractors/dataform.py· Escalate · medium- agreed by 2 of 2 members but NOT verified (no proof, no reproducing execution) — consensus is not a verdict; needs human review
- Config block detection matches commented-out config headers —
graphify/extractors/dataform.py:69· Escalate · medium- agreed by 2 of 2 members but NOT verified (no proof, no reproducing execution) — consensus is not a verdict; needs human review
- Ref scan extracts dependencies from SQL string literals —
graphify/extractors/dataform.py:271· Escalate · medium- agreed by 2 of 2 members but NOT verified (no proof, no reproducing execution) — consensus is not a verdict; needs human review
Analysis details — impact, health, verification
Impact & health
Graphify review
Impact — 2535 functions depend on the 411 functions this change touches.
Health — this change adds coupling hotspots:
- new:
extract()— 591 callers, 44 callees - new:
_rebuild_code()— 116 callers, 51 callees - new:
detect()— 112 callers, 15 callees - new:
_extract_generic()— 18 callers, 26 callees - new:
save_manifest()— 40 callers, 11 callees - new:
extract_files_direct()— 17 callers, 20 callees - new:
extract_js()— 85 callers, 4 callees - new:
extract_xaml()— 19 callers, 17 callees - …and 56 more — each is listed as a finding
Verification — 2535 functions in the blast radius were not formally verified this run (proofs are advisory here).
Gate & verification
graphify gate
PASS — objectively clean (no health regressions, tests not run — proofs not run this pass (advisory)). Grounded, not self-assessed.
Advisory (not blocking):
- verification_scope: 2284 function(s) in the blast radius were not formally verified this run
Test selection
Test selection
276 of 276 test file(s) selected (100%) via static blast radius.
Escalated to a full run for safety — the selection is not trustworthy on its own (see below). CI should run the whole suite.
tests/test_affected_cli.py— full-run-safetytests/test_affected_member_seed.py— full-run-safetytests/test_agents_platform.py— full-run-safetytests/test_analyze.py— full-run-safetytests/test_anthropic_custom_endpoint.py— full-run-safetytests/test_antigravity_install.py— full-run-safetytests/test_apm_fallback_version.py— full-run-safetytests/test_architecture_doc.py— full-run-safetytests/test_astro_extraction.py— impact, full-run-safetytests/test_astro_import_ids.py— impact, full-run-safetytests/test_atomic_canvas_export.py— full-run-safetytests/test_atomic_version_stamp.py— full-run-safetytests/test_atomic_writes.py— impact, full-run-safetytests/test_backend_env_isolation.py— full-run-safetytests/test_backend_extras.py— full-run-safetytests/test_benchmark.py— full-run-safetytests/test_benchmark_raw_graph.py— full-run-safetytests/test_build.py— impact, full-run-safetytests/test_build_merge_dedup_scope.py— full-run-safetytests/test_build_merge_hyperedges_and_prune.py— full-run-safetytests/test_build_merge_shrink_guard.py— full-run-safetytests/test_builtin_global_type_refs.py— impact, full-run-safetytests/test_cache.py— full-run-safetytests/test_callflow_html.py— full-run-safetytests/test_cargo_introspect.py— full-run-safetytests/test_carried_hyperedge_remap.py— full-run-safetytests/test_case_sensitive_resolution.py— impact, full-run-safetytests/test_charmap_encoding.py— impact, full-run-safetytests/test_chunking.py— impact, full-run-safetytests/test_cjs_module_extension.py— impact, full-run-safetytests/test_claude_cli_backend.py— impact, full-run-safetytests/test_claude_md.py— full-run-safetytests/test_cli_broken_pipe.py— full-run-safetytests/test_cli_export.py— full-run-safetytests/test_cli_help.py— full-run-safetytests/test_cluster.py— full-run-safetytests/test_codebuddy.py— full-run-safetytests/test_community_hub_labels.py— full-run-safetytests/test_community_labels_skill.py— full-run-safetytests/test_confidence.py— full-run-safetytests/test_corrupt_graph_json.py— full-run-safetytests/test_cpp_nested_and_cli.py— impact, full-run-safetytests/test_cpp_objc_cross_file_calls.py— impact, full-run-safetytests/test_cpp_preprocess.py— full-run-safetytests/test_cross_extension_reexport_self_cycle.py— impact, full-run-safetytests/test_cross_language_call_resolution.py— impact, full-run-safetytests/test_cross_repo_member_calls.py— impact, full-run-safetytests/test_cross_repo_shared_types.py— full-run-safetytests/test_csharp_call_site_generic_args.py— impact, full-run-safetytests/test_csharp_enum_members.py— impact, full-run-safety- … and 226 more
non-code file(s) changed (
README.md,tests/fixtures/dataform_basic.sqlx) → running the full suite for safety (a code graph can't see config/fixture/data deps)
changed code file(s) with no mapped test (
README.md,graphify/extractors/__init__.py) — a coverage gap or a missing link — running the full suite rather than only the selected tests
Selection is safe under the controlled-regression assumption; always-run tests + a periodic full run are the backstops. Advisory — it never changes the check verdict.
· 1 grounded finding(s) anchored inline below; 63 more finding(s) on lines outside this diff (see the check run).
| return None | ||
|
|
||
|
|
||
| def extract_dataform(path: Path) -> dict: |
There was a problem hiding this comment.
extract_dataform()
fans out to 7 callees (efferent coupling).
Grounded coupling-delta finding (deterministic), not an LLM guess.
There was a problem hiding this comment.
Graphify reviewed this change.
Worth a look — the grounded gate found no coupling regressions or blocking issues, but 3 advisory finding(s) below merit a look before merge.
Graphify review — findings
Adds a deterministic Dataform .sqlx extractor (extract_dataform) that pulls model, config, and dependency structure from config blocks and ref/self calls without parsing the embedded SQL or JavaScript templating, masking comments and string contents so those scans stay within the config body and ignore commented-out fields. Registers .sqlx in CODE_EXTENSIONS, the extractor dispatch table, and the LANGUAGE_EXTRACTORS map, and documents the new file type in the README.
Worth a look
- extract_dataform falls through to undefined name instead of returning result —
graphify/extractors/dataform.py:354· Escalate · high- agreed by 2 of 2 members but NOT verified (no proof, no reproducing execution) — consensus is not a verdict; needs human review
- extract_dataform raises NameError instead of returning graph —
graphify/extractors/dataform.py:354· Escalate · high- agreed by 2 of 2 members but NOT verified (no proof, no reproducing execution) — consensus is not a verdict; needs human review
- dependency offset via text.find matches first occurrence anywhere, giving wrong line number —
graphify/extractors/dataform.py:330· Escalate · medium- agreed by 2 of 2 members but NOT verified (no proof, no reproducing execution) — consensus is not a verdict; needs human review
Analysis details — impact, health, verification
Impact & health
Graphify review
Impact — 2539 functions depend on the 415 functions this change touches.
Health — this change adds coupling hotspots:
- new:
extract()— 591 callers, 44 callees - new:
_rebuild_code()— 116 callers, 51 callees - new:
detect()— 112 callers, 15 callees - new:
_extract_generic()— 18 callers, 26 callees - new:
save_manifest()— 40 callers, 11 callees - new:
extract_files_direct()— 17 callers, 20 callees - new:
extract_js()— 85 callers, 4 callees - new:
extract_xaml()— 19 callers, 17 callees - …and 56 more — each is listed as a finding
Verification — 2539 functions in the blast radius were not formally verified this run (proofs are advisory here).
Gate & verification
graphify gate
PASS — objectively clean (no health regressions, tests not run — proofs not run this pass (advisory)). Grounded, not self-assessed.
Advisory (not blocking):
- verification_scope: 2288 function(s) in the blast radius were not formally verified this run
Test selection
Test selection
276 of 276 test file(s) selected (100%) via static blast radius.
Escalated to a full run for safety — the selection is not trustworthy on its own (see below). CI should run the whole suite.
tests/test_affected_cli.py— full-run-safetytests/test_affected_member_seed.py— full-run-safetytests/test_agents_platform.py— full-run-safetytests/test_analyze.py— full-run-safetytests/test_anthropic_custom_endpoint.py— full-run-safetytests/test_antigravity_install.py— full-run-safetytests/test_apm_fallback_version.py— full-run-safetytests/test_architecture_doc.py— full-run-safetytests/test_astro_extraction.py— impact, full-run-safetytests/test_astro_import_ids.py— impact, full-run-safetytests/test_atomic_canvas_export.py— full-run-safetytests/test_atomic_version_stamp.py— full-run-safetytests/test_atomic_writes.py— impact, full-run-safetytests/test_backend_env_isolation.py— full-run-safetytests/test_backend_extras.py— full-run-safetytests/test_benchmark.py— full-run-safetytests/test_benchmark_raw_graph.py— full-run-safetytests/test_build.py— impact, full-run-safetytests/test_build_merge_dedup_scope.py— full-run-safetytests/test_build_merge_hyperedges_and_prune.py— full-run-safetytests/test_build_merge_shrink_guard.py— full-run-safetytests/test_builtin_global_type_refs.py— impact, full-run-safetytests/test_cache.py— full-run-safetytests/test_callflow_html.py— full-run-safetytests/test_cargo_introspect.py— full-run-safetytests/test_carried_hyperedge_remap.py— full-run-safetytests/test_case_sensitive_resolution.py— impact, full-run-safetytests/test_charmap_encoding.py— impact, full-run-safetytests/test_chunking.py— impact, full-run-safetytests/test_cjs_module_extension.py— impact, full-run-safetytests/test_claude_cli_backend.py— impact, full-run-safetytests/test_claude_md.py— full-run-safetytests/test_cli_broken_pipe.py— full-run-safetytests/test_cli_export.py— full-run-safetytests/test_cli_help.py— full-run-safetytests/test_cluster.py— full-run-safetytests/test_codebuddy.py— full-run-safetytests/test_community_hub_labels.py— full-run-safetytests/test_community_labels_skill.py— full-run-safetytests/test_confidence.py— full-run-safetytests/test_corrupt_graph_json.py— full-run-safetytests/test_cpp_nested_and_cli.py— impact, full-run-safetytests/test_cpp_objc_cross_file_calls.py— impact, full-run-safetytests/test_cpp_preprocess.py— full-run-safetytests/test_cross_extension_reexport_self_cycle.py— impact, full-run-safetytests/test_cross_language_call_resolution.py— impact, full-run-safetytests/test_cross_repo_member_calls.py— impact, full-run-safetytests/test_cross_repo_shared_types.py— full-run-safetytests/test_csharp_call_site_generic_args.py— impact, full-run-safetytests/test_csharp_enum_members.py— impact, full-run-safety- … and 226 more
non-code file(s) changed (
README.md,tests/fixtures/dataform_basic.sqlx) → running the full suite for safety (a code graph can't see config/fixture/data deps)
changed code file(s) with no mapped test (
README.md,graphify/extractors/__init__.py) — a coverage gap or a missing link — running the full suite rather than only the selected tests
Selection is safe under the controlled-regression assumption; always-run tests + a periodic full run are the backstops. Advisory — it never changes the check verdict.
· 1 grounded finding(s) anchored inline below; 63 more finding(s) on lines outside this diff (see the check run).
| yield match | ||
|
|
||
|
|
||
| def extract_dataform(path: Path) -> dict: |
There was a problem hiding this comment.
extract_dataform()
fans out to 8 callees (efferent coupling); 6 callers depend on it (afferent coupling).
Grounded coupling-delta finding (deterministic), not an LLM guess.
Summary
Closes #3496.
Dataform
.sqlxfiles combine SQL with Dataform configuration and JavaScript templating. They were previously unclassified and contributed no graph nodes or relationships.This PR adds a deterministic, dependency-free first-pass extractor that:
.sqlxas code;type,schema,database,description,tags, anddependenciesmetadata fromconfig { ... };depends_onedges for literalref(model),ref(schema, model), and config dependencies;self()but suppresses the meaningless self-edge;Unresolved external model references are emitted as canonical targets and are handled by the existing graph build behavior; no fabricated source node is created.
Validation
python -m pytest tests/test_dataform.py tests/test_extractors_registry.py -q: 6 passedpython -m pytest tests/test_dataform.py tests/test_extract.py tests/test_build.py tests/test_cache.py -q: 404 passed, 4 skipped, 1 unrelated Windows cache-path test failurepython -m graphify update .: passedThe regression fixture covers model detection, config metadata, cross-model references,
self(), comments, quoted SQL literals, nonliteral calls, member-call false positives, commented-out config headers, bounded/comment-aware config arrays, and inline config fields.