Codestin Search App

weaversam8 · 2024-10-17T17:50:58Z

This is a follow up to #169, to more fully address the use case discussed in #23.

Along the way, I also found and fixed #171, #172, and #173. (To view those changes in isolation, please look at the diffs of the commits linked to those issues specifically.)

I also believe that this supersedes @zabacad's PR #143 to support nested interpolation. (Sorry Ian! I didn't see your PR and recreated your work - but we landed on similar approaches!)

I'm submitting this PR as a draft right now, because while this is technically working, there are a few more things I'd like to add. Namely:

Better documentation about how to use this feature
Better whitespace and formatting embedded in the AST generated from the dictionary
- (this draft implementation produces syntactically equivalent output but it's very ugly)
Helper functions to control the formatting of the generated output by embedding metadata in the Python dictionary
Better error messages when parsing an invalid dictionary
More tests for all of the above.

This PR adds:

New API methods:
- reverse_transform(hcl_dict) which returns an AST generated from a Python dictionary
Lark grammar changes:
- Changes to handle nested string interpolation (similar to Allow expressions in string interpolation #143)
- Renamed several terminals to make them easier to reference when reconstructing
BREAKING: changes to the output of the DictTransformer class to ensure it generates valid HCL expressions. (See HCL interpolations within dict output contain single quotes #172, I hope it's OK to include this since we just did a major version bump.)
hcl2.reconstructor changes:
- I added the HCLReverseTransformer class which aims to reverse the process that hcl2.transformer.DictTransformer takes (the bread and butter of this PR)
- I overhauled the whitespace/formatting handling that I implemented in Support full reconstruction of HCL from parse tree #169 to make those decisions based on individual tokens/rules, rather than simply looking at the characters only
Tests:
- I added a simple test for HCL escape sequences for Terraform escape sequences are not processed when parsing strings #171
- I updated the test files for the fixes I made for HCL interpolations within dict output contain single quotes #172 (see BREAKING note above)
- I added a test for deeply nested string interpolation (Lark grammar does not support interpolations nested more than 2 times #173)
- I modified test_load to print the full diff when a file does not match its expected JSON
- I added test_reconstruct_dict to ensure that the HCL reconstructed from a Python dictionary is syntactically equivalent (whitespace checks will be added to this file soon.)

Still a draft PR, as described above, but open to hear any comments or suggestions!

…k trees

… type

)

…o match Terraform syntax instead of Python (fixes amplify-education#172)

(breaks some reconstruction tests, tackling that next)

…ons to make rule-aware

weaversam8 · 2024-10-24T15:09:15Z

OK, I think this is ready for a proper review. Since the draft, I added:

More extensive documentation on how to use these features
A rudimentary hcl2.Builder class allowing users to create fresh HCL documents from Python (the reason why this is necessary is described in the docs I added)
Better formatting defaults on output from hcl2.reverse_transform(...).
Tests for the hcl2.Builder class.

weaversam8 · 2024-11-04T21:44:54Z

@kkozik-amplify - still reviewing this? any questions I can answer about my changes?

kkozik-amplify · 2024-11-05T10:14:52Z

@weaversam8 Yeah, need some more time. In the meantime - could you look into the errors that Codacy Analysis PR check reported?

Nfsaavedra · 2024-11-15T16:11:42Z

Hi!

While trying to use the code in this branch I found that when parsing the following file:

locals {
  tags = {
    application = "Modernisation Platform: ${terraform.workspace}"
  }
}

The program gets stuck in a regex match. I am almost sure it is related to the changes to better support string interpolation (that's why I'm using this branch).

Nfsaavedra · 2024-11-15T17:27:04Z

+STRING_LIT : "\"" (STRING_CHARS)* "\""
+STRING_CHARS : /(?:(?!\${)([^"\\]|\\.))+/+ // any character except '"' unless inside a interpolation string


Suggested change

STRING_LIT : "\"" (STRING_CHARS)* "\""

STRING_CHARS : /(?:(?!\${)([^"\\]|\\.))+/+ // any character except '"' unless inside a interpolation string

STRING_LIT : "\"" STRING_CHARS? "\""

STRING_CHARS : /(?:(?!\${)([^"\\]|\\.))+/ // any character except '"'

This seems to fix the problem, but I did not run the tests to check it. It does parse this file correctly tho.

@Nfsaavedra Hey! Thanks for the suggestion. Since it's been some time already, I added your fix to the PR myself, all tests are passing.

guilinden · 2025-01-08T17:04:33Z

@weaversam8 any plans when this will be merged?

amplify-education#177 (comment)

Nfsaavedra

Based on my understanding of the code, LGTM. Do the new tests consider the case where the previous version of the program would get stuck as described here? It's important to avoid regressions. If so, then I have nothing to add :)

…ring

kkozik-amplify · 2025-01-16T13:51:53Z

@@ -1,3 +1,3 @@
 locals {
-  embedded_interpolation = "${module.special_constants.aws_accounts["aaa-${local.foo}-${local.bar}"]}/us-west-2/key_foo"
+  embedded_interpolation = "(long substring without interpolation); ${module.special_constants.aws_accounts["aaa-${local.foo}-${local.bar}"]}/us-west-2/key_foo"


@Nfsaavedra I updated this test case to take issue you found into consideration.
It seems like the original regex would take longer to match non-interpolated string with each character, to the point where for 20 characters it would basically get stuck.

Makes sense! :)

@weaversam8

…es (originally implemented by @weaversam8 in #177) and fixed `STRING_CHARS` regex to treat escaped interpolation as a regular string

@weaversam8

…es (originally implemented by @weaversam8 in #177) and fixed `STRING_CHARS` regex to treat escaped interpolation as a regular string

weaversam8 mentioned this pull request Oct 17, 2024

Allow expressions in string interpolation #143

Closed

weaversam8 added 15 commits October 24, 2024 10:34

Initial commit of a "reverse transformer" to turn HCL2 dicts into Lar…

78c264a

…k trees

Add tests for the reverse reconstructor

d7350f6

Add different handling to the reverse reconstructor depending on data…

0b2282f

… type

Add support for multiple block labels

ddfbb12

Fix accidentally escaping quotes within interpolated strings

23c63c3

Properly handle escapes within HCL strings (closes amplify-education#171

373aa78

)

Standardize string output from transformer within nested structures t…

613afad

…o match Terraform syntax instead of Python (fixes amplify-education#172)

Fix block labels and booleans during reconstruction

093db9a

Better handle nested interpolation (fixes amplify-education#173)

d3f8760

(breaks some reconstruction tests, tackling that next)

Begin refactor of whitespace handling (more to come)

98f7fd8

Midway thru whitespace handling refactor, need to complete modificati…

6c86b49

…ons to make rule-aware

Finish overhaul of whitespace handling, remove old logic.

a0291c4

Fix Pylint warnings

281b706

Fix a few formatting issues in reconstruction

1ad6758

Add a "builder" class for constructing HCL files from Python

8652de4

weaversam8 force-pushed the feature/hcl2-reverse-transformer branch from a28991c to 8652de4 Compare October 24, 2024 14:34

Update the docs for reconstruction

92071ea

weaversam8 marked this pull request as ready for review October 24, 2024 15:09

weaversam8 requested a review from a team as a code owner October 24, 2024 15:09

kkozik-amplify self-assigned this Oct 25, 2024

weaversam8 mentioned this pull request Oct 25, 2024

Fix Symbol.__eq__ to return false when comparing with None lark-parser/lark#1481

Merged

Nfsaavedra reviewed Nov 15, 2024

View reviewed changes

kkozik-amplify added 2 commits January 13, 2025 14:47

fix suggested by Nfsaavedra

d780f4f

amplify-education#177 (comment)

batch amplify-education#1 of fixes to issues reported by codacy

a18a8b2

batch amplify-education#2 of fixes to issues reported by codacy

d9639af

kkozik-amplify force-pushed the feature/hcl2-reverse-transformer branch from 2c6d36a to d9639af Compare January 14, 2025 13:14

a bit of refactoring

993ef6b

kkozik-amplify force-pushed the feature/hcl2-reverse-transformer branch from a367ac2 to 993ef6b Compare January 15, 2025 14:39

kkozik-amplify requested a review from Nfsaavedra January 15, 2025 14:46

Nfsaavedra reviewed Jan 15, 2025

View reviewed changes

update interpolation test case to include long non-interpolated subst…

9e49e8a

…ring

kkozik-amplify force-pushed the feature/hcl2-reverse-transformer branch from 880ad1c to 9e49e8a Compare January 16, 2025 13:40

kkozik-amplify reviewed Jan 16, 2025

View reviewed changes

kkozik-amplify approved these changes Jan 16, 2025

View reviewed changes

rddimon approved these changes Jan 16, 2025

View reviewed changes

kkozik-amplify merged commit f8a2c88 into amplify-education:main Jan 16, 2025

kkozik-amplify added a commit that referenced this pull request Apr 16, 2025

reintroduce parsing of string interpolations nested deeper than 2 tim…

ae2f5c1

…es (originally implemented by @weaversam8 in #177) and fixed `STRING_CHARS` regex to treat escaped interpolation as a regular string

kkozik-amplify mentioned this pull request Apr 16, 2025

reintroduce parsing of string interpolations nested deeper than 2 times #223

Merged

kkozik-amplify added a commit that referenced this pull request Apr 16, 2025

reintroduce parsing of string interpolations nested deeper than 2 tim…

458817b

…es (originally implemented by @weaversam8 in #177) and fixed `STRING_CHARS` regex to treat escaped interpolation as a regular string

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support full reconstruction of HCL from output dictionary#177

Support full reconstruction of HCL from output dictionary#177
kkozik-amplify merged 21 commits into
amplify-education:mainfrom
weaversam8:feature/hcl2-reverse-transformer

weaversam8 commented Oct 17, 2024

Uh oh!

weaversam8 commented Oct 24, 2024

Uh oh!

weaversam8 commented Nov 4, 2024 •

edited

Loading

Uh oh!

kkozik-amplify commented Nov 5, 2024 •

edited

Loading

Uh oh!

Nfsaavedra commented Nov 15, 2024

Uh oh!

Nfsaavedra Nov 15, 2024 •

edited

Loading

Uh oh!

kkozik-amplify Jan 15, 2025

Uh oh!

guilinden commented Jan 8, 2025

Uh oh!

Nfsaavedra left a comment

Uh oh!

kkozik-amplify Jan 16, 2025

Uh oh!

Nfsaavedra Jan 16, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

		STRING_LIT : "\"" (STRING_CHARS)* "\""
		STRING_CHARS : /(?:(?!\${)([^"\\]\|\\.))+/+ // any character except '"' unless inside a interpolation string

Conversation

weaversam8 commented Oct 17, 2024

Uh oh!

weaversam8 commented Oct 24, 2024

Uh oh!

weaversam8 commented Nov 4, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kkozik-amplify commented Nov 5, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Nfsaavedra commented Nov 15, 2024

Uh oh!

Nfsaavedra Nov 15, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kkozik-amplify Jan 15, 2025

Choose a reason for hiding this comment

Uh oh!

guilinden commented Jan 8, 2025

Uh oh!

Nfsaavedra left a comment

Choose a reason for hiding this comment

Uh oh!

kkozik-amplify Jan 16, 2025

Choose a reason for hiding this comment

Uh oh!

Nfsaavedra Jan 16, 2025

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

weaversam8 commented Nov 4, 2024 •

edited

Loading

kkozik-amplify commented Nov 5, 2024 •

edited

Loading

Nfsaavedra Nov 15, 2024 •

edited

Loading