DATEX Formatter #461

jonasstrehle · 2025-10-21T20:15:58Z

This PR adds a formatter (see https://github.com/unyt-org/datex-core/tree/feat/fmt/src/fmt) for DATEX

Additional:

Finalize visitor pattern of feat/compiler-workspace
AST + grammar enhancement and refactoring
Implement TypeExpressionData
Implement the visitor trait for the new precompiler (WIP)
Some more fucking refactoring fuck shit fuck

Copilot

Pull Request Overview

This PR introduces a proof-of-concept formatter implementation for DATEX code, along with significant AST restructuring to support better formatting and source location tracking.

Key Changes:

Implements a new pretty-printing formatter using the pretty crate with configurable formatting options
Restructures the AST to separate expression data from spans for better source tracking
Refactors the decompiler to use the new AST structure
Updates type system integration and precompiler for the new AST format

Reviewed Changes

Copilot reviewed 38 out of 39 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
src/fmt/mod.rs	New formatter implementation with pretty-printing support
src/ast/tree.rs	New AST structure separating data and spans
src/decompiler/ast_to_source_code.rs	Refactored AST-to-source-code converter with new formatting modes
src/compiler/precompiler.rs	Updated to work with new AST structure and span tracking
src/compiler/type_inference.rs	Updated type inference for new AST data structure
src/compiler/mod.rs	Compilation pipeline updates for AST restructuring
src/decompiler/mod.rs	Decompiler updates for new formatting modes
tests/compiler.rs	Test updates for new AST structure
Multiple AST parsers	Updates to generate spans for all expression types

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

src/decompiler/mod.rs

src/compiler/workspace.rs

github-actions · 2025-10-25T14:32:08Z

Test Results

566 tests +20 566 ✅ +20 1m 8s ⏱️ +5s
6 suites ± 0 0 💤 ± 0
1 files ± 0 0 ❌ ± 0

Results for commit ffe3b14. ± Comparison against base commit bf225ab.

♻️ This comment has been updated with latest results.

github-actions · 2025-10-25T15:19:24Z

Bencher Report

Branch	feat/fmt
Testbed	ubuntu-latest

Click to view all benchmark results

Benchmark	Latency	Benchmark Result microseconds (µs) (Result Δ%)	Upper Boundary microseconds (µs) (Limit %)
dxb to json/test1.json	📈 view plot 🚷 view threshold	62.48 µs (-2.17%) Baseline: 63.86 µs	66.85 µs (93.45%)
dxb to json/test2.json	📈 view plot 🚷 view threshold	13,254.00 µs (-0.27%) Baseline: 13,290.29 µs	13,716.74 µs (96.63%)
dxb to json/test3.json	📈 view plot 🚷 view threshold	140,940.00 µs (-1.14%) Baseline: 142,560.00 µs	147,948.89 µs (95.26%)
dxb to runtime value/test1.json	📈 view plot 🚷 view threshold	61.55 µs (+51.32%) Baseline: 40.68 µs	67.59 µs (91.07%)
dxb to runtime value/test2.json	📈 view plot 🚷 view threshold	16,326.00 µs (+46.12%) Baseline: 11,172.86 µs	17,910.38 µs (91.15%)
dxb to runtime value/test3.json	📈 view plot 🚷 view threshold	178,040.00 µs (+79.68%) Baseline: 99,084.71 µs	200,518.25 µs (88.79%)
json to dxb/test1.json	📈 view plot 🚷 view threshold	7,970.90 µs (+372.09%) Baseline: 1,688.44 µs	9,748.96 µs (81.76%)
json to dxb/test2.json	📈 view plot 🚷 view threshold	8,381,400.00 µs (+516.75%) Baseline: 1,358,965.71 µs	10,368,679.88 µs (80.83%)
json to dxb/test3.json	📈 view plot 🚷 view threshold	48,336,000.00 µs (+465.57%) Baseline: 8,546,357.14 µs	59,596,051.17 µs (81.11%)
json to runtime value datex auto static detection/test1.json	📈 view plot 🚷 view threshold	8,030.40 µs (+371.12%) Baseline: 1,704.54 µs	9,820.57 µs (81.77%)
json to runtime value datex auto static detection/test2.json	📈 view plot 🚷 view threshold	8,383,200.00 µs (+514.46%) Baseline: 1,364,310.00 µs	10,369,478.83 µs (80.84%)
json to runtime value datex auto static detection/test3.json	📈 view plot 🚷 view threshold	48,529,000.00 µs (+461.92%) Baseline: 8,636,271.43 µs	59,818,248.45 µs (81.13%)
json to runtime value datex forced static/test1.json	📈 view plot 🚷 view threshold	7,993.10 µs (+372.08%) Baseline: 1,693.16 µs	9,775.93 µs (81.76%)
json to runtime value datex forced static/test2.json	📈 view plot 🚷 view threshold	8,371,500.00 µs (+515.70%) Baseline: 1,359,661.43 µs	10,355,782.78 µs (80.84%)
json to runtime value datex forced static/test3.json	📈 view plot 🚷 view threshold	48,367,000.00 µs (+464.35%) Baseline: 8,570,371.43 µs	59,629,036.33 µs (81.11%)
json to runtime value datex/test1.json	📈 view plot 🚷 view threshold	7,961.50 µs (+368.57%) Baseline: 1,699.11 µs	9,733.70 µs (81.79%)
json to runtime value datex/test2.json	📈 view plot 🚷 view threshold	8,384,500.00 µs (+513.05%) Baseline: 1,367,670.00 µs	10,370,197.63 µs (80.85%)
json to runtime value datex/test3.json	📈 view plot 🚷 view threshold	48,515,000.00 µs (+460.63%) Baseline: 8,653,685.71 µs	59,795,343.14 µs (81.14%)
json to runtime value json_syntax/test1.json	📈 view plot 🚷 view threshold	21.17 µs (+4.47%) Baseline: 20.26 µs	21.74 µs (97.37%)
json to runtime value json_syntax/test2.json	📈 view plot 🚷 view threshold	6,594.90 µs (+0.49%) Baseline: 6,562.81 µs	6,986.64 µs (94.39%)
json to runtime value json_syntax/test3.json	📈 view plot 🚷 view threshold	70,337.00 µs (-14.48%) Baseline: 82,243.57 µs	97,685.77 µs (72.00%)
json to runtime value serde_json/test1.json	📈 view plot 🚷 view threshold	6.25 µs (-0.66%) Baseline: 6.29 µs	6.53 µs (95.66%)
json to runtime value serde_json/test2.json	📈 view plot 🚷 view threshold	2,070.70 µs (-2.76%) Baseline: 2,129.53 µs	2,214.75 µs (93.50%)
json to runtime value serde_json/test3.json	📈 view plot 🚷 view threshold	14,444.00 µs (-10.68%) Baseline: 16,170.57 µs	18,882.48 µs (76.49%)
runtime init	📈 view plot 🚷 view threshold	16.92 µs (-87.61%) Baseline: 136.53 µs	1,198.58 µs (1.41%)
runtime value to dxb/test1.json	📈 view plot 🚷 view threshold	8.35 µs (+107.74%) Baseline: 4.02 µs	9.57 µs (87.20%)
runtime value to dxb/test2.json	📈 view plot 🚷 view threshold	2,576.70 µs (+132.31%) Baseline: 1,109.15 µs	2,992.04 µs (86.12%)
runtime value to dxb/test3.json	📈 view plot 🚷 view threshold	23,925.00 µs (+120.16%) Baseline: 10,867.21 µs	27,646.81 µs (86.54%)
runtime value to json datex/test1.json	📈 view plot 🚷 view threshold	71.47 µs (+3.95%) Baseline: 68.75 µs	73.31 µs (97.50%)
runtime value to json datex/test2.json	📈 view plot 🚷 view threshold	15,398.00 µs (+6.52%) Baseline: 14,454.86 µs	15,762.45 µs (97.69%)
runtime value to json datex/test3.json	📈 view plot 🚷 view threshold	163,660.00 µs (+5.40%) Baseline: 155,278.57 µs	167,242.19 µs (97.86%)
runtime value to json json_syntax/test1.json	📈 view plot 🚷 view threshold	10.90 µs (+1.25%) Baseline: 10.76 µs	11.10 µs (98.16%)
runtime value to json json_syntax/test2.json	📈 view plot 🚷 view threshold	4,634.90 µs (+1.48%) Baseline: 4,567.29 µs	4,739.70 µs (97.79%)
runtime value to json json_syntax/test3.json	📈 view plot 🚷 view threshold	59,867.00 µs (-5.27%) Baseline: 63,196.43 µs	71,005.61 µs (84.31%)
runtime value to json serde_json/test1.json	📈 view plot 🚷 view threshold	2.86 µs (-2.24%) Baseline: 2.93 µs	3.31 µs (86.41%)
runtime value to json serde_json/test2.json	📈 view plot 🚷 view threshold	923.16 µs (-5.84%) Baseline: 980.43 µs	1,161.28 µs (79.49%)
runtime value to json serde_json/test3.json	📈 view plot 🚷 view threshold	9,143.60 µs (-8.01%) Baseline: 9,939.50 µs	12,270.99 µs (74.51%)

🐰 View full continuous benchmarking report in Bencher

src/ast/data/visitor.rs

jonasstrehle · 2025-10-28T20:50:48Z

AWS down again? xd

# Conflicts: # src/compiler/precompiler.rs

src/runtime/mod.rs

jonasstrehle · 2025-11-01T07:57:25Z

PR is okay from my POV. I'd suggest to wait for #493 and merge once this one is ready to avoid tractor conflicts.
@benStre @janiejestemja

jonasstrehle self-assigned this Oct 21, 2025

Copilot AI review requested due to automatic review settings October 21, 2025 20:15

jonasstrehle added enhancement New feature or request non-blocking labels Oct 21, 2025

jonasstrehle changed the base branch from main to release/0.0.7 October 21, 2025 20:16

Copilot AI reviewed Oct 21, 2025

View reviewed changes

src/decompiler/mod.rs Outdated Show resolved Hide resolved

src/compiler/workspace.rs Outdated Show resolved Hide resolved

jonasstrehle changed the base branch from release/0.0.7 to feat/compiler-workspace October 21, 2025 20:16

jonasstrehle changed the base branch from feat/compiler-workspace to release/0.0.7 October 25, 2025 10:27

jonasstrehle changed the base branch from release/0.0.7 to main October 25, 2025 10:38

jonasstrehle changed the base branch from main to release/0.0.7 October 25, 2025 10:38

jonasstrehle force-pushed the feat/fmt branch from e289374 to c6cd240 Compare October 25, 2025 10:42

jonasstrehle changed the title ~~Formatter POC (WIP)~~ DATEX Formatter Oct 25, 2025

jonasstrehle added this to the Release 0.0.7 milestone Oct 25, 2025

jonasstrehle marked this pull request as draft October 25, 2025 14:24

jonasstrehle assigned benStre, TeeB3utel and janiejestemja Oct 26, 2025

benStre reviewed Oct 27, 2025

View reviewed changes

src/ast/data/visitor.rs Outdated Show resolved Hide resolved

src/ast/data/visitor.rs Outdated Show resolved Hide resolved

src/ast/data/visitor.rs Outdated Show resolved Hide resolved

jonasstrehle added 9 commits October 31, 2025 14:07

work on proper formatter (WIP)

9547e02

work on fmt (WIP)

4991e1b

work on fmt (WIP)

cc490c4

work on fmt (WIP)

ae521cc

work on fmt (WIP)

5daa6e9

fix new map / list struct

7b4046a

add bracketing to ast

8730b90

work on fmt (WIP)

dadf1ca

work on fmt (WIP)

5289fa8

# Conflicts: # src/compiler/precompiler.rs

benStre added 2 commits October 31, 2025 14:12

🗑️ cleanup

4ef6568

🐛 fix precompile methods

4d6d163

benStre force-pushed the feat/fmt branch from ef962db to 4d6d163 Compare October 31, 2025 13:14

benStre added 6 commits October 31, 2025 14:19

🐛 fix binary operation precompiler

d69f534

🎨 clippy

acecb3b

♻️ refactor precompiler

4eb4241

🐛 handle variable redeclaration errors

012a1e3

🐛 handle variable redeclaration errors

1102e35

🔥 remove final references

562071a

This was referenced Oct 31, 2025

Remove final refs unyt-org/datex-core-js#30

Merged

Deprecate final in favour of permission system #275

Closed

fmt

841626f

unyt-org deleted a comment from benStre Nov 1, 2025

jonasstrehle commented Nov 1, 2025

View reviewed changes

src/runtime/mod.rs Outdated Show resolved Hide resolved

remove println

02af154

jonasstrehle marked this pull request as ready for review November 1, 2025 07:54

jonasstrehle added scope:compiler scope:decompiler scope:type-system scope:formatter scope:visitor scope:ast scope:parser labels Nov 1, 2025

This was referenced Nov 1, 2025

Release (0.0.7) #457

Draft

Type inference #495

Merged

Merge branch 'feat/fmt' of github.com:unyt-org/datex-core into feat/fmt

0ce78e8

benStre approved these changes Nov 4, 2025

View reviewed changes

Merge branch 'release/0.0.7' into feat/fmt

ffe3b14

benStre merged commit 626486a into release/0.0.7 Nov 5, 2025
3 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

DATEX Formatter #461

DATEX Formatter #461

Uh oh!

jonasstrehle commented Oct 21, 2025 •

edited by benStre

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Oct 25, 2025 •

edited

Loading

Uh oh!

github-actions bot commented Oct 25, 2025 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jonasstrehle commented Oct 28, 2025

Uh oh!

Uh oh!

jonasstrehle commented Nov 1, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Uh oh!

DATEX Formatter #461

DATEX Formatter #461

Uh oh!

Conversation

jonasstrehle commented Oct 21, 2025 • edited by benStre Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Oct 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Test Results

Uh oh!

github-actions bot commented Oct 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Bencher Report

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jonasstrehle commented Oct 28, 2025

Uh oh!

Uh oh!

jonasstrehle commented Nov 1, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

jonasstrehle commented Oct 21, 2025 •

edited by benStre

Loading

github-actions bot commented Oct 25, 2025 •

edited

Loading

github-actions bot commented Oct 25, 2025 •

edited

Loading