v0.21.0

Improvements and new features @aknysh (#40)

what

• Improvements and new features

why

• Add remote_state_backend and remote_state_backend_type YAML config sections. They can be used to specify a different backend (different attributes, or even a completely different backend type) for remote state. It's useful for the privileged components where we use different attributes (e.g. role_arn or profile) during the cold-start and during the normal operations to get the remote state of the privileged components

      # Override backend for this component
      backend_type: static # s3, remote, vault, static, etc.
      backend:
        static:
          val1: 1
          val2: "2"
          val3: true
          val4: ""
          val5: null
      # Override remote state backend for this component
      remote_state_backend_type: static # s3, remote, vault, static, etc.
      remote_state_backend:
        static:
          val1: 1
          val2: "2"
          val3: true
          val4: ""
          val5: 5

The new remote_state_backend section in YAML is similar to the backend section - you can specify it in globals, stage globals, or override per component (which all gets deep-merged into the final remote state backend for the component). The two sections are separate and the attributes from them are not merged together

v0.20.0

Update to use latest stacks and components features. Add `static` backend type. Improve examples and tests @aknysh (#39)

what

• Update to use latest stacks and components features
• Add static backend type
• Improve examples and tests

why

• Support the new terraform-provider-utils
• Support tenant
• Support YAML stack config folders
• Support components folders
• static backend type is used to find the remote state outputs of components that are already provisioned and their backend config (outputs) is statically defined in YAML config

v0.19.0

Update to new `null-label`. Add `stack` sub-module to calculate stack names @aknysh (#35)

what

• Update to new null-label
• Add stack sub-module to calculate stack names

why

• Use new features from null-label module:

  • Additional label and id component: tenant
  • New input labels_as_tags controls which labels are exported as tags
  • New input descriptor_formats generates new output descriptors - use it to calculate the stack names

• stack submodule is used to calculate stack names in one place and use the same logic in all other sub-modules

references

• cloudposse/terraform-null-label#132

v0.18.4

🚀 Enhancements

Update Terraform cloudposse/label/null to v0.25.0 @renovate (#34)

This PR contains the following updates:

Package	Type	Update	Change
cloudposse/label/null (source)	module	minor	0.24.1 -> 0.25.0

---

Release Notes

cloudposse/terraform-null-label

v0.25.0

Compare Source

Add "tenant", "labels_as_tags", and "descriptors" @​Nuru (#​132)

##### what - Add additional label and `id` component: `tenant` - New input `labels_as_tags` controls which labels are exported as tags - New input `descriptor_formats` generates new output `descriptors` - Update README, remove link to obsolete `terraform-terraform-label` ##### why - Support users that host resources on behalf of and/or dedicated to single customers - Supersedes and closes #​131, giving people control over which tags the module generates - Simple mechanism for creating multiple identifiers from the same inputs, reducing the need to create multiple instances of `null-label` - Document `tenant`, `labels_as_tags`, `descriptor_formats`, add additional clarification, stop promoting obsolete module

Fix: Update README Snippets @​korenyoni (#​130)

##### what * Update README snippets to reflect use of Terraform Registry. ##### why * Including snippets that reflect use of the Terraform Registry make it easier for users to quickly instantiate a null_label module. * README is out of date and does not include snippets that reflect use of the Terraform Registry. ##### references * N/A

Bridgecrew compliance @​Nuru (#​125)

##### what - Resolve Bridgecrew compliance complaint about example Autoscaling Group (BC_AWS_GENERAL_31) - Fix typo in README - Include Terraform lock file in `.gitignore` ##### why - Get clean Bridgecrew badge - Correct confusing error - Ensure lock files are not checked into GitHub ##### note The PR can and should be merged into `master` to update README and Bridgecrew without triggering a new release/version. These changes have no effect on the actual module in use and a release will create unnecessary ripple effects. However, merging to `master` will update the README and badges, so is worthwhile, and the changes will move forward into the next release.

---

Configuration

📅 Schedule: At any time (no schedule defined).

🚦 Automerge: Disabled by config. Please merge this manually once you are satisfied.

♻ Rebasing: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.

🔕 Ignore: Close this PR and you won't be reminded about this update again.

---

• If you want to rebase/retry this PR, check this box.

---

This PR has been generated by WhiteSource Renovate. View repository job log here.

🤖 Automatic Updates

Update Terraform cloudposse/label/null to v0.25.0 @renovate (#34)

This PR contains the following updates:

Package	Type	Update	Change
cloudposse/label/null (source)	module	minor	0.24.1 -> 0.25.0

---

Release Notes

cloudposse/terraform-null-label

v0.25.0

Compare Source

Add "tenant", "labels_as_tags", and "descriptors" @​Nuru (#​132)

##### what - Add additional label and `id` component: `tenant` - New input `labels_as_tags` controls which labels are exported as tags - New input `descriptor_formats` generates new output `descriptors` - Update README, remove link to obsolete `terraform-terraform-label` ##### why - Support users that host resources on behalf of and/or dedicated to single customers - Supersedes and closes #​131, giving people control over which tags the module generates - Simple mechanism for creating multiple identifiers from the same inputs, reducing the need to create multiple instances of `null-label` - Document `tenant`, `labels_as_tags`, `descriptor_formats`, add additional clarification, stop promoting obsolete module

Fix: Update README Snippets @​korenyoni (#​130)

##### what * Update README snippets to reflect use of Terraform Registry. ##### why * Including snippets that reflect use of the Terraform Registry make it easier for users to quickly instantiate a null_label module. * README is out of date and does not include snippets that reflect use of the Terraform Registry. ##### references * N/A

Bridgecrew compliance @​Nuru (#​125)

##### what - Resolve Bridgecrew compliance complaint about example Autoscaling Group (BC_AWS_GENERAL_31) - Fix typo in README - Include Terraform lock file in `.gitignore` ##### why - Get clean Bridgecrew badge - Correct confusing error - Ensure lock files are not checked into GitHub ##### note The PR can and should be merged into `master` to update README and Bridgecrew without triggering a new release/version. These changes have no effect on the actual module in use and a release will create unnecessary ripple effects. However, merging to `master` will update the README and badges, so is worthwhile, and the changes will move forward into the next release.

---

Configuration

📅 Schedule: At any time (no schedule defined).

🚦 Automerge: Disabled by config. Please merge this manually once you are satisfied.

♻ Rebasing: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.

🔕 Ignore: Close this PR and you won't be reminded about this update again.

---

• If you want to rebase/retry this PR, check this box.

---

This PR has been generated by WhiteSource Renovate. View repository job log here.

v0.18.3

🚀 Enhancements

Set remote-state.outputs to null @nitrocode (#33)

what

• Set remote-state.outputs to null

To test

module "sso" {
  source = "git::ssh://git@github.com:cloudposse/terraform-yaml-stack-config.git/modules/remote-state?ref=remote-state-outputs-null"

  # ...
}

why

• Avoid Error: Inconsistent conditional result types

references

• Previous PR #32

v0.18.2

🚀 Enhancements

Fix backend lookup error @nitrocode (#32)

what

• Fix backend lookup error

why

• Awful awful awful error

│ Error: Invalid index
│
│   on .terraform/modules/sso/modules/backend/main.tf line 14, in locals:
│   14:   backend_type    = local.config["components"][var.component_type][local.final_component]["backend_type"]
│     ├────────────────
│     │ local.config["components"] is object with 2 attributes
│     │ local.final_component is "sso"
│     │ var.component_type is "terraform"
│
│ The given key does not identify an element in this collection value.

This comes up when you have to plan the iam-delegated-roles component in a stage that doesn't have the tfstate-backend (defaultregion-root) or the sso (gbl-identity) remote states.

references

N/A

v0.18.1

🚀 Enhancements

Remove unused and deprecated template provider @mmuehlberger (#31)

what

• Remove any references to the template provider

why

• The template provider is deprecated and no binaries exist for darwin/arm64
• The provider is not used in the module or any submodules

references

• Provider documentation

v0.18.0

Add `backend.privileged` attribute to the backend config in YAML stack config @aknysh (#30)

what

• Add backend.privileged attribute to the backend config in YAML stack config

why

• Allow making a component privileged or not by just changing the attribute in YAML stack config
• Simplify cold-start

notes

NOTE: component types
- Privileged components are those that require elevated (root-level) permissions to provision and access 
their remote state.
- For example: `tfstate-backend`, `account`, `account-map`, `account-settings`, `iam-primary`.
- Privileged components are usually provisioned during cold-start (when we don't have any IAM roles provisioned yet) 
by using an admin user credentials.
- To access the remote state of privileged components, the caller needs to have permissions to access the backend 
and the remote state without assuming roles.
- Regular components, on the other hand, don't require root-level permissions and are provisioned and their 
remote state is accessed by assuming IAM roles (or using profiles).
- For example: `vpc`, `eks`, `rds`

NOTE: global `backend` config
- The global `backend` config should be declared in a global YAML stack config file (e.g. `globals.yaml`)
where all stacks can import it and have access to it (note that the global `backend` config is organization-wide 
and will not change after cold-start).
- The global `backend` config in the global config file should always have the `role_arn` or `profile` specified 
(added after the cold-start).

NOTE: components `backend` config
- The `backend` portion for each individual component should be declared in a catalog file (e.g. 
`stacks/catalog/<component>.yaml`) along with all the default values for a component.
- The `privileged` attribute should always be declared in the `backend` portion for each individual 
component in the catalog.
- Top-level stacks where a component is provisioned import the component's catalog 
  (the default values and the component's backend config portion) and can override the default value.

NOTE: `cold-start`
- During cold-start we don't have any IAM roles provisioned yet, so we use an admin user credentials to 
provision the privileged components.
- The `privileged` attribute for the privileged components should be set to `true` in the components'  catalog,
   and the privileged components should be provisioned using an admin user credentials.

NOTE: after `cold-start`
- After the privileged components (including the primary IAM roles) are provisioned, we update the 
global `backend` config in the global config file
to add the IAM role or profile to access the backend (after this, the global `backend` config should never change).
- For some privileged components we can change the `privileged` attribute in the YAML config from `true` to `false`
to allow the regular components to access their remote state 
(e.g. we set the `privileged` attribute to `false` in the `account-map` component
since we use `account-map` in almost all regular components.
- For each regular component, set the `privileged` attribute to `false` in the components'  portion of `backend` 
config (in `stacks/catalog/<component>.yaml`)

# Advantages:
- The global `backend` config is specified just once in the global config file, IAM role or profile is added to it 
after the cold start, and after that the global `backend` config never changed.
- We can make a component privileged or not any time by just updating its `privileged` attribute in the 
component's catalog file.
- We can change a component's `backend` portion any time without touching/affection the backend configs 
  of all other components (e.g. when we add a new component, we don't touch the `globals.yaml` file at all, 
 and we don't update the component's `role_arn`  and `profile` settings).

v0.17.0

Update `spacelift` module @aknysh (#29)

what

• Update spacelift module
• Add stack_config_path variable

why

• Need a separate variable with a relative path to the stack config folder
• stack_config_path_template is used to create Spacelift stack labels, it does not specify the relative path to the YAML config files

v0.16.0

Add Spacelift module and example. Bump `terraform-provider-utils` versions @aknysh (#27)

what

• Add Spacelift module and example
• Bump terraform-provider-utils version
• Add process_component_deps variable

why

• Directly convert infrastructure YAML stack configs into Spacelift stack configs

• Speed up YAML stack processing (in many cases, from tens of minutes to tens of seconds)

• Reduce unnecessary stack runs in CI/CD systems like Spacelift

• The provider already calculates these parameters:

  • imports - a list of all imports (all levels) for the stack
  • stacks - a list of all stacks in the infrastructure where the component is defined

• Both imports and stacks are not 100% suitable to correctly determine the YAML config files that a component depends on

  • imports is too broad. The provider returns all direct and indirect imports for the stack, even those that don't define any variables for the component. This will trigger the component's stack in Spacelift even if the unrelated imports are modified resulting unrelated stack runs in Spacelift
  • stacks is too broad and too narrow at the same time. On the one hand, it detects all stacks in the infrastructure where the component is defined, but we don't need to trigger a particular stack in Spacelift if some other top-level YAML stack configs are modified. On the other hand, it misses the cases where a YAML stack config file specifies global variables, which are applied to the component as well

For example, in examples/data-sources/utils_stack_config_yaml/stacks/catalog/eks-defaults.yaml we define eks-iam terraform component. The provider returns the following:

stacks:
  - catalog/eks-defaults
  - globals
  - uw2-dev
  - uw2-globals
  - uw2-prod
  - uw2-staging
  - uw2-uat

imports:
  - catalog/eks-defaults
  - catalog/rds-defaults
  - catalog/s3-defaults
  - globals
  - uw2-globals

imports returns the unrelated catalog/s3-defaults and catalog/rds-defaults which will unnecessary trigger this component stack in Spacelift.

stacks returns all top-level YAML stack configs in the infrastructure all of which will unnecessary trigger this component stack in Spacelift.

We need to return only those YAML config files if any of the following conditions is true:

1. The imported config file has the global vars section and it's not empty
2. The imported config file has the component type section, which has a vars section which is not empty
3. The imported config file has the "components" section, which has the component type section, which has the component section
4. The imported config file has the "components" section, which has the component type section, which has the base component section

For example:

vars:
  stage: dev
terraform:
  vars: {}
components:
  terraform:
    aurora-postgres:
      vars:
        instance_type: db.r4.large
        cluster_size: 1
    aurora-postgres-2:
      component: aurora-postgres
      vars:
        instance_type: db.r4.xlarge
  helmfile: {}

vars and terraform.vars are global vars and component-type vars sections
components is the components section
components.terraform is the component type section (as is components.helmfile) - those are component types and get processed similarly
components.terraform.aurora-postgres is the base component section
components.terraform.aurora-postgres-2 is the component section

The changes in this PR return the correct YAML config dependencies for the components, e.g. for eks-iam:

eks-iam:
  deps:
    - catalog/eks-defaults
    - globals
    - uw2-globals
    - uw2-dev

because globals and uw2-globals have global variables for the component, and catalog/eks-defaults defines the component itself.

This will trigger the Spacelift component stack only if these files are modified (in addition to the component project files and the top-level stack, e.g. uw2-prod, themselves).

references

• cloudposse/terraform-provider-utils#32
• cloudposse/terraform-provider-utils#40