Support a configurable namespace delimiter

[acmcelwee]

We're trying to adapt a lot of our existing troposphere stacks to stacker, and we've namespaced all of our stacks, but we've omitted the delimiter between the namespace and the stack name, in favor of simply camel casing our stack names. Some flexibility in the delimiter used there (including a blank string) would go a long way to providing some flexibility to start using stacker with stacks that might've been created by other methods.

[phobologic]

Thanks @acmcelwee - can you add some documentation about this to https://github.com/remind101/stacker/blob/master/docs/config.rst since this adds a new top level key word?

[phobologic]

BTW, probably worth mentioning what characters are allowed for delimiters, since CloudFormation can be picky.

[phobologic]

Thanks!

[acmcelwee]

@phobologic quick question related to namespacing. Today, we're using a two-tiered approach to our namespacing in each of our AWS accounts. For our dev account, everything lives in the d namespace, so we have stacks like dVPC, dLogging, etc. We have the ability to spin up an entire parallel set of infrastructure for various needs within the account, where each parallel infra stack depends on the more account-wide resources. To further clarify, here's a more concrete example:

• Account-wide dVPC and dLogging stacks that contain resources related to the stack name
• More granular-scoped stacks, say dptAPI for the API stack in the dev environment (d prefix), owned by the Platform Team (pt portion of the prefix) that depend on outputs of the account-wide stacks...think security groups, subnets, etc.

Given the way stacker namespacing works today, it's impossible to do cross-namespace stack dependencies, so I've worked around it using a combination of yaml anchors and references plus environment files that contain those desired stack outputs. Would y'all be interested in a contribution to open up that functionality? So far, I'm thinking something along the lines of Ruby's double-colon operator. It fits the style that y'all use today, but would open up the possibility to have more dynamic dependencies that don't require manually capturing the outputs in a yaml file somewhere. I could see referencing things outside of your namespace with a leading ::, so to refer to our VPC stack, it would be something like ::dVPC::PublicSubnets.

[phobologic]

Hey @acmcelwee - this is definitely a known weakness of stacker that I'd been thinking about, and I'd definitely be interested in seeing a PR for it. Are you thinking of just extending the double colon operator that we already use in Parameters? So that in a stack with the namespace 'myNamespace'

vpc::ExampleOutput would get the ExampleOutput output from myNamespace-vpc

and

other-namespace::vpc::ExampleOutput would get that output from other-namespace-vpc?

@mhahn @ejholmes might have some opinions as well.

[mwildehahn]

The tricky part about this to me is making it so if I'm building a specific namespace, say dpt, stacker is recognizing dependencies in another stack, and if they aren't built yet, stacker handles submitting and waiting for them to finish before moving on with dpt.

The current way of invoking build isn't going to handle this unless you make a lot of assumptions about where to look for the file specifying the resources in the other namespace.

I would imagine you could have something like the following:

conf/acme-inc/
├── common
│   ├── config.yaml
│   ├── dev.env
│   └── prod.env
└── individual
    ├── config.yaml
    ├── dev-pt-team.env
    └── prod.env

where i would invoke build with something like:

stacker build conf/acme-inc/individual/dev-pt-team.env conf/acme-inc/individual/config.yaml --namespace-dir conf/acme-inc

or some such way to indicate that stacker should build the dev-pt-team.env environment/namespace of the individual stack and resolve other namespace dependencies within that directory.

if a stack within individual/config.yaml depends on dvpc::PrivateSubnets, stacker would read in the other config files and construct the correct dependency tree.

is that what you're going after?

[phobologic]

I don't think it has to be that complicated - the contract could be that if you specify a full stack name for an Output that it no longer guarantees much other than it's in a 'consistent' state (ie: UPDATE_COMPLETE, CREATE_COMPLETE, etc).

[phobologic]

@acmcelwee I'm going to go ahead and open an issue for this, since discussing it here isn't the best place.