add raw json/yaml template support #530

troyready · 2018-02-07T01:31:47Z

Fixes #444

codecov-io · 2018-02-07T01:34:23Z

Codecov Report

Merging #530 into master will decrease coverage by 0.22%.
The diff coverage is 76.81%.

@@            Coverage Diff             @@
##           master     #530      +/-   ##
==========================================
- Coverage   87.94%   87.72%   -0.23%     
==========================================
  Files          91       93       +2     
  Lines        5875     6003     +128     
==========================================
+ Hits         5167     5266      +99     
- Misses        708      737      +29

Impacted Files	Coverage Δ
stacker/config/__init__.py	`92.68% <100%> (+0.57%)`	⬆️
stacker/actions/build.py	`92.19% <100%> (ø)`	⬆️
stacker/tests/test_config.py	`99.39% <100%> (+0.03%)`	⬆️
stacker/tests/blueprints/test_raw.py	`100% <100%> (ø)`
stacker/actions/diff.py	`57.46% <16.66%> (-1%)`	⬇️
stacker/blueprints/base.py	`93.75% <50%> (-0.43%)`	⬇️
stacker/stack.py	`85.71% <58.33%> (-5.2%)`	⬇️
stacker/blueprints/raw.py	`66.12% <66.12%> (ø)`

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b9242f6...aefc378. Read the comment docs.

troyready · 2018-02-07T01:39:06Z

@phobologic @ejholmes I'm mildly surprised at how easy this ended up being; I've tested it with a couple of yaml & json templates without issue.

Could use some help figuring out how to use schematics to model the option to specify exactly one of class_path or template_path. I looked at the Model docs a bit but it didn't jump out at me -- curious if one of you knows or could figure it out faster than I can?

ejholmes

This is cool. I think this is a good start, but I wonder if we can implement everything within RawTemplateBlueprint, without changing any stacker internals.

ejholmes · 2018-02-07T04:52:09Z

stacker/actions/build.py

-        required_parameters = stack.required_parameter_definitions.keys()
-        parameters = _handle_missing_parameters(resolved, required_parameters,
-                                                provider_stack)
+        if hasattr(stack.blueprint, 'raw_template_path') and (


By the looks of it, the changes in this file shouldn't really be necessary. I think all you would need to do is implement the get_parameter_definitions() and get_required_parameter_definitions() method on the blueprint.

If we go down this path, the contract between _launch_stack and a "blueprint" starts to get muddled and brittle.

ejholmes · 2018-02-07T04:54:30Z

stacker/blueprints/raw.py

+                                                   description)
+        self._raw_template_path = raw_template_path
+
+    def create_template(self):


It's not really necessary to override this, since it won't ever actually be called. I think it's actually better to remove it, because then the base class will raise an exception if it does get called.

ejholmes · 2018-02-07T04:57:11Z

stacker/util.py

    return client._client_config.region_name


+def get_template_default_params(template_path):


In the CloudFormation API, there's a ValidateTemplate call that will parse the parameters from the template, and return them.

With that said, idk what's actually better here, since that would require another slow network call, which sucks.

ejholmes · 2018-02-07T04:58:05Z

stacker/blueprints/raw.py

+        """Load template and generate its md5 hash."""
+        if self.description:
+            self.set_template_description(self.description)
+        with open(self.raw_template_path, 'r') as myfile:


Can we call this something other than myfile? I'm having flashbacks of php.

ejholmes · 2018-02-07T04:59:22Z

stacker/util.py

+    """
+    params = {}
+
+    template_ext = os.path.splitext(template_path)[1]


It seems odd that we read the file in 2 separate locations. Can we read the file once, then call this method with the in memory content? I think that'll be easier of you implement those get_parameter_definitions() and get_required_parameter_definitions() methods on the blueprint.

ejholmes · 2018-02-07T05:02:04Z

docs/config.rst

-  The python class path to the Blueprint to be used.
+  The python class path to the Blueprint to be used. Specify this or
+  ``template_path`` for the stack.
+**template_path:**


I kinda feel like this could all be done without even adding a new config option. The RawTemplateBlueprint could just use a variable to specify the template path.

For example:

stacks: - name: vpc class_path: stacker.blueprints.RawTemplateBlueprint variables: template_path: templates/vpc.yaml

Whether or not that's better is probably debatable, but it keeps stacker config's more generic, which is good.

I think it's worth of the pain of class_path vs template_path to avoid any other config values on where the template is.

troyready · 2018-02-07T22:54:52Z

@ejholmes thanks for the feedback; you're right, it's much nicer fixing the blueprint to do the right thing.

I've gone through and tested build/destroy/diff now and think this is in a good place.

ejholmes

Ok, another round of review. This is a looking better!

I'll let @phobologic chime in as well.

ejholmes · 2018-02-07T23:48:10Z

stacker/actions/diff.py

            old_params = {}

        stack.resolve(self.context, self.provider)
+        yaml_stack = hasattr(stack.blueprint, 'raw_template_format') and (


Kinda the same comment as before about changes to stacker.actions.build. I don't think actions should have any coupling to different types of blueprints.

Instead, we should just change this so that the action calls a to_json method on the blueprint, which should always return the JSON format. In the case of a yaml raw template, it would just yaml parse -> json dump it.

ejholmes · 2018-02-07T23:49:45Z

stacker/blueprints/raw.py

+    return value
+
+
+class RawTemplateBlueprint(Blueprint):  # pylint: disable=abstract-method


I'm starting to feel like maybe inheriting from Blueprint isn't the right thing for this. It might be easier to maintain of it just inherits from object, since there doesn't really seem to be much that's re-usable from Blueprint.

Yeah, agreed that it doesn't seem to re-use much. I like it having a override/subset relationship, but I could be persuaded otherwise.

ejholmes · 2018-02-07T23:54:31Z

stacker/stack.py

-                mappings=self.mappings,
-                description=self.definition.description,
-            )
+            if self.definition.class_path:


this could probably be simplified a bit to:

kwargs = {} blueprint_class = None if self.definition.class_path: class_path = self.definition.class_path blueprint_class = util.load_object_from_string(class_path) if not hasattr(blueprint_class, "rendered"): raise AttributeError("Stack class %s does not have a " "\"rendered\" " "attribute." % (class_path,)) elif self.definition.template_path: blueprint_class = RawTemplateBlueprint kwargs["raw_template_path"] = self.definition.template_path else: raise AttributeError("Stack does not have a defined class or " "template path.") self._blueprint = blueprint_class( name=self.name, context=self.context, mappings=self.mappings, description=self.definition.description, **kwargs )

ejholmes · 2018-02-07T23:58:59Z

stacker/tests/test_config.py

-        self.assertEquals(
-            error.__str__(),
-            "This field is required.")
+    # TODO: add test for class_path or template_path (and not both)


I would check the schematics documentation, but I think you'd likely need to perform this validation in stacker.config.Config#validate.

ejholmes · 2018-02-08T00:00:54Z

stacker/util.py

    return client._client_config.region_name


+def get_template_file_format(template_path):


Can we move these methods to stacker.blueprints.raw, since they're only used there? At the very least, it'll make reading RawTemplateBlueprint easier.

ejholmes · 2018-02-08T00:02:07Z

stacker/blueprints/raw.py

+
+    def render_template(self):
+        """Load template and generate its md5 hash."""
+        with open(self.raw_template_path, 'r') as template:


We're still reading the template twice. Once here, and again in util.get_template_params. We should only need to read the template from disk once into memory here, and then use the in memory contents to get params.

ejholmes · 2018-02-08T00:02:43Z

stacker/blueprints/raw.py

+        version = hashlib.md5(rendered).hexdigest()[:8]
+        parsed_template = yaml.load(rendered)
+        if 'Transform' in parsed_template:
+            self.template.add_transform(parsed_template['Transform'])


Why do we do this? What significance does the troposphere Template play in this blueprint?

ejholmes · 2018-02-08T00:04:26Z

stacker/util.py

+    params = {}
+
+    if not os.path.isfile(template_path):
+        raise EnvironmentError("Could not find CFN template at path "


Do we have a test for this? It seems like this isn't a defined exception:

stacker $ git grep EnvironmentError

troyready · 2018-02-08T05:04:56Z

@ejholmes good thoughts, thanks. I think I've addressed each comment.

ejholmes · 2018-02-09T01:53:43Z

@troyready I opened a PR against yours with a few small changes, and some more tests: troyready#1.

Fixes cloudtools#444

* update diff action to support regular & raw blueprints without conditions * fix validation & test of class / template path use in Config * move raw blueprint specific function from util to blueprint module * cleanup class selection in stack.py

troyready · 2018-02-09T03:29:59Z

Thanks! Merged and rebased.

ejholmes · 2018-02-09T03:57:28Z

I'll let @phobologic give the final 👍. Otherwise, this looks awesome to me.

russellballestrini

Wow great work on this @troyready !

Very cool, looks good to me!

troyready · 2018-02-09T18:01:46Z

I've confirmed this can be rebased into a standalone change on v1.1.4 for a non-DAG release without issue.

So, if it makes sense to delay a DAG release until #531 , then a v1_2 release branch (branched off 1.1.4) could cherry-pick 1d222ec to get this branch's changes only.

* add raw json/yaml template support Fixes cloudtools#444 * fix diff; fix transform; remove build action hack * add raw template tests * additional updates for raw template support * update diff action to support regular & raw blueprints without conditions * fix validation & test of class / template path use in Config * move raw blueprint specific function from util to blueprint module * cleanup class selection in stack.py * Handle mutual exclusion validation better * Add functional test for template_path * Remove dependency on troposphere template * Make RawTemplateBlueprint inherit from object. * cleanup pylint/pydocstyle messages in raw blueprint

ejholmes requested changes Feb 7, 2018

View reviewed changes

ejholmes requested changes Feb 8, 2018

View reviewed changes

ejholmes mentioned this pull request Feb 9, 2018

Add raw template support troyready/stacker#1

Merged

troyready and others added 9 commits February 8, 2018 19:10

add raw json/yaml template support

ccd57b1

Fixes cloudtools#444

fix diff; fix transform; remove build action hack

c402462

add raw template tests

f09cb04

Handle mutual exclusion validation better

711d571

Add functional test for template_path

37a0ade

Remove dependency on troposphere template

c5339be

Make RawTemplateBlueprint inherit from object.

fb5024c

cleanup pylint/pydocstyle messages in raw blueprint

9cb66ef

troyready force-pushed the add_raw_template_support branch from afb9bf2 to 9cb66ef Compare February 9, 2018 03:10

ejholmes approved these changes Feb 9, 2018

View reviewed changes

russellballestrini self-requested a review February 9, 2018 14:09

russellballestrini approved these changes Feb 9, 2018

View reviewed changes

Merge branch 'master' into add_raw_template_support

aefc378

phobologic merged commit 4d0fad8 into cloudtools:master Feb 11, 2018

troyready deleted the add_raw_template_support branch February 11, 2018 23:54

troyready mentioned this pull request Feb 23, 2018

fix diff with local yaml templates #541

Merged

		return client._client_config.region_name


		def get_template_default_params(template_path):

		return value


		class RawTemplateBlueprint(Blueprint): # pylint: disable=abstract-method

		return client._client_config.region_name


		def get_template_file_format(template_path):

add raw json/yaml template support #530

add raw json/yaml template support #530

Uh oh!

Conversation

troyready commented Feb 7, 2018

Uh oh!

codecov-io commented Feb 7, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

troyready commented Feb 7, 2018

Uh oh!

ejholmes left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

troyready commented Feb 7, 2018

Uh oh!

ejholmes left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ejholmes Feb 8, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ejholmes Feb 8, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

troyready commented Feb 8, 2018

Uh oh!

ejholmes commented Feb 9, 2018

Uh oh!

troyready commented Feb 9, 2018

Uh oh!

ejholmes commented Feb 9, 2018

Uh oh!

russellballestrini left a comment

Choose a reason for hiding this comment

Uh oh!

troyready commented Feb 9, 2018

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

codecov-io commented Feb 7, 2018 •

edited

Loading

ejholmes Feb 8, 2018 •

edited

Loading

ejholmes Feb 8, 2018 •

edited

Loading