Implement Variables and Lookups #194

mwildehahn · 2016-08-12T03:45:40Z

These will deprecate PARAMETERS and LOCAL_PARAMETERS.

Advantages:

No need to use CF parameters
Leverage yaml lists and dicts
Blueprint subclasses can easily add/overwrite parameters

Fixes: #175

I added: remind101/stacker_blueprints@93db0d2 to stacker_blueprints as a POC for showing how we can leverage the new BLUEPRINT_PARAMETERS

These will deprecate PARAMETERS and LOCAL_PARAMETERS. Advantages: - No need to use CF parameters - Leverage yaml lists and dicts - Blueprint subclasses can easily add/overwrite parameters

mwildehahn · 2016-08-12T17:29:30Z

stacker/blueprints/base.py

+        """Resolve the values of the blueprint parameters.
+
+        resolve_parameters will resolve the values of the
+        `BLUEPRINT_PARAMETERS` with values from the command lie, the env file,


mwildehahn · 2016-08-12T19:31:45Z

Now supports doing the following:

DATABASE_URL: postgres://${empiredb_user}:${empiredb_password}@${empireDB::DBCname}/empire

empireDB::DBCname will get resolved at build time, empiredb_user will get resolved when we parse the config

phobologic · 2016-08-15T16:37:21Z

stacker/blueprints/base.py

+            dict: parameters available to the template
+
+        """
+        if self.resolved_parameters is None:


Why not make this do the parameter resolution when it's called if resolved_parameters is None?

I didn't want get_parameters to take any arguments and was invoking resolve_parameters where we were before: https://github.com/remind101/stacker/pull/194/files#diff-92ed81f5017042692188db7b5961c619R203. We need stack.parameters, provider and context which was readily available there, i'm not sure if we can get all of those values when we init the blueprint to make this resolve when called.

phobologic · 2016-08-15T17:27:42Z

👍 other than the small comments I made. Thanks!

phobologic · 2016-08-16T20:04:40Z

stacker/tests/blueprints/test_base.py

+        provider = MagicMock()
+        context = MagicMock()
+        provider.get_output.return_value = "Test Output"
+        output = resolve_string("some-stack::Output, other-stack::Output",


We should ditch this old syntax when using blueprint_parameters in preference for the new template syntax.

phobologic · 2016-08-21T17:52:48Z

One thing I thought of - what if instead of ${stack::output} being the generic syntax, what if we combined this with the concept of translators? Like, what if the output lookup was a sort of translator, but with the new syntax? Something like ${output MyStack::StackOutput}? Seems like that'd be more flexible (maybe it'd replace translators entirely?) and it'd make things more explicit. I suppose, since the output lookup is such a common thing, we could keep generic ${MyStack::StackOutput} to default to the output lookup.

mwildehahn · 2016-08-22T19:35:49Z

we talked about this a little on slack, but to repeat for the record:

my concern with this is just that it gets confusing when the translator gets resolved. right now translators get resolved after the config is parsed, whereas outputs need to be resolved at build time. i think it would get confusing to have some translators resolved pre-build and others during build.

troyready · 2016-08-23T17:13:19Z

Curious how this change would impact the generated templates' CFN parameters? Sounds like there won't be any, so I'm wondering if we could still add them to templates (using troposphere.add_parameter) while still using this new functionality?

It may be an errand of folly, but I'd like to keep adding CFN parameters for templates that I might share in built json form.

phobologic · 2016-08-24T05:20:59Z

@troyready The idea was to eventually remove stacker support for parameters & local parameters in favor of these. That wouldn't mean that you couldn't still add them directly via troposphere, it just wouldn't have all the helper logic around building them into the blueprints.

The reason for that is that using real CFN parameters in stacker has sort of proven to be pointless. In theory you use parameters to make the templates re-usable, but in stacker you never actually re-use the templates (at least in the use cases I've seen - different environments usually have different buckets/namespaces - we end up with copies of the same template in two different buckets).

Do you often share the generated templates with others?

troyready · 2016-08-24T17:23:37Z

It's definitely not something used in a normal stacker workflow, but it is nice to be able to share the json templates at times. It's something that I've done with colleagues (e.g. "even though you don't have stacker setup, you can just grab the template from this stack and use it with regular CFN").

Don't get me wrong, I'm not trying to argue for some crazy workflow that depends on that. I just have some reservation about a change that significantly alters what stacker can output (maybe that does mean I depend on spacebar heating?).

mwildehahn · 2016-08-26T00:32:01Z

@troyready 😆 re: https://xkcd.com/1172/

We're now calling these variables so it isn't out of the realm of possibility that we keep PARAMETERS around in some form, but I'd like to remove them and allow people to just use add_parameter in a blueprint if they really want it.

mwildehahn · 2016-08-26T00:41:01Z

After talking with @phobologic about the initial implementation of blueprint parameters, we decided to go with a concept of "variables" that wouldn't be as easy to confuse with CF parameters. Variables are only related to how you can build templates within stacker.

This PR now introduces two new classes to stacker:

Variable and Lookup. A Variable is a python class that represents the variable passed within the stacker config, ie:

  # Create a simple app within empire named "acme-inc"
  - name: acmeApp
    class_path: stacker_blueprints.empire.app.App
    variables:
      ServiceToken: ${empireDaemon::CustomResourcesTopic}
      Name: "acme-inc"

ServiceToken and Name are Variables.

A Variable can have zero or many Lookups within its value that will be resolved at build time and will then be available within the Blueprint as a dictionary of <variable name>: <resolved value>.

A Lookup is defined as ${<lookup type> <lookup input>} within the stacker config. We default to the output lookup since this is the most common use case. ie ${empireDaemon::CustomResourcesTopic} could be more verbosely defined as ${output empireDaemon::CustomResourcesTopic}.

Custom lookups can be registered within the config: https://github.com/remind101/stacker_blueprints/pull/24/files#diff-8f84670404b5e0466057d169e260c840R58, https://github.com/remind101/stacker_blueprints/pull/24/files#diff-7cb05c06f16d1eea2c363ecf8e0d1463R3

They can also be nested: https://github.com/remind101/stacker_blueprints/pull/24/files#diff-8f84670404b5e0466057d169e260c840R207

When we parse the config, we register any custom lookups that have been defined. At build time, we resolve all of the variables, invoking the lookup handlers that have been defined. When the Blueprint renders its template, variables will have been translated into a dictionary with any lookups resolved.

In the near future, we'll collapse the concept of "translators" into lookups given that their execution at build time and access to context and provider makes them much more powerful.

mwildehahn · 2016-08-26T00:43:08Z

stacker/blueprints/base.py

@@ -98,13 +104,15 @@ class Blueprint(object):
            template.

    """
-    def __init__(self, name, context, mappings=None):
+
+    def __init__(self, name, context, variables=None, mappings=None):


this isn't used

mwildehahn · 2016-08-26T00:44:37Z

stacker/blueprints/base.py

+                raise UnresolvedVariable(self, variable)
+
+            if variable.value is None:
+                logger.debug("Got None value for parameter %s, not submitting "


s/parameter/variable

phobologic · 2016-08-28T21:25:43Z

stacker/blueprints/base.py

+        env file, the config, and any lookups resolved.
+
+        Args:
+            variables (list): list of variables


This is a list of a specific class, right? Can you use the :class: syntax to point at the actual class that you want a list of? Right now it's hard to tell what a list of variables means.

phobologic · 2016-08-28T21:49:03Z

This is looking so awesome. A couple of nits, and a few questions, but I'm really excited for this.

phobologic · 2016-08-29T19:19:17Z

Also, once you're feeling comfortable with this, can you add some documentation? Thanks!

mwildehahn · 2016-08-29T21:59:45Z

going to add docs in a separate PR, merging this in now!

Implement new BLUEPRINT_PARAMETERS

6413960

These will deprecate PARAMETERS and LOCAL_PARAMETERS. Advantages: - No need to use CF parameters - Leverage yaml lists and dicts - Blueprint subclasses can easily add/overwrite parameters

mwildehahn reviewed Aug 12, 2016
View reviewed changes

Support resolving references to outputs within a parameter value

ea70b1a

phobologic reviewed Aug 15, 2016
View reviewed changes

mwildehahn added 4 commits August 16, 2016 11:00

Fix typo

f164637

Use correct docstring syntax

5ed13af

Add comment about safe_substitute followed by replace

f5b46be

Break out private functions and add specific tests

bc69fe4

phobologic reviewed Aug 16, 2016
View reviewed changes

mwildehahn added 5 commits August 25, 2016 16:46

Add support for stack Variables and Lookups

f26d2d6

Raise an exception if a variable is missing

c585709

Add docstrings

ad9160f

Use a verbose regex

776eb8e

Resolve variables before diff

4b857fe

mwildehahn reviewed Aug 26, 2016
View reviewed changes

mwildehahn added 4 commits August 25, 2016 18:01

Adjust some wording

bc6e7aa

Stack requires should only consider output lookups

c405a7f

Add docstring

8c11648

Use lookup terminology

b298947

mwildehahn changed the title ~~Implement new BLUEPRINT_PARAMETERS~~ Implement new Variables and Lookups Aug 26, 2016

mwildehahn changed the title ~~Implement new Variables and Lookups~~ Implement Variables and Lookups Aug 26, 2016

phobologic reviewed Aug 28, 2016
View reviewed changes

mwildehahn added 3 commits August 29, 2016 12:50

Use :class: syntax in docstrings

afd7a94

Update README

6a97876

Move lookup handlers into a subdirectory for clarity

6667432

mwildehahn merged commit d9c4452 into cloudtools:master Aug 29, 2016

mwildehahn deleted the blueprint-parameters branch August 29, 2016 21:59

This was referenced Aug 29, 2016

Added support for loading files as parameter values, including interpolation #185

Merged

support mixed parameters referencing / not referencing other stacks #183

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement Variables and Lookups #194

Implement Variables and Lookups #194

mwildehahn commented Aug 12, 2016

mwildehahn Aug 12, 2016

mwildehahn commented Aug 12, 2016

phobologic Aug 15, 2016

mwildehahn Aug 16, 2016

phobologic commented Aug 15, 2016

phobologic Aug 16, 2016

phobologic commented Aug 21, 2016

mwildehahn commented Aug 22, 2016

troyready commented Aug 23, 2016

phobologic commented Aug 24, 2016

troyready commented Aug 24, 2016

mwildehahn commented Aug 26, 2016

mwildehahn commented Aug 26, 2016

mwildehahn Aug 26, 2016

mwildehahn Aug 26, 2016

phobologic Aug 28, 2016

phobologic commented Aug 28, 2016

phobologic commented Aug 29, 2016

mwildehahn commented Aug 29, 2016

Implement Variables and Lookups #194

Implement Variables and Lookups #194

Conversation

mwildehahn commented Aug 12, 2016

mwildehahn Aug 12, 2016

Choose a reason for hiding this comment

mwildehahn commented Aug 12, 2016

phobologic Aug 15, 2016

Choose a reason for hiding this comment

mwildehahn Aug 16, 2016

Choose a reason for hiding this comment

phobologic commented Aug 15, 2016

phobologic Aug 16, 2016

Choose a reason for hiding this comment

phobologic commented Aug 21, 2016

mwildehahn commented Aug 22, 2016

troyready commented Aug 23, 2016

phobologic commented Aug 24, 2016

troyready commented Aug 24, 2016

mwildehahn commented Aug 26, 2016

mwildehahn commented Aug 26, 2016

mwildehahn Aug 26, 2016

Choose a reason for hiding this comment

mwildehahn Aug 26, 2016

Choose a reason for hiding this comment

phobologic Aug 28, 2016

Choose a reason for hiding this comment

phobologic commented Aug 28, 2016

phobologic commented Aug 29, 2016

mwildehahn commented Aug 29, 2016