Create flag for specifying backend config

[mittz]

Updates

• Add --backend-config flag for ghpc create and ghpc expand.
• Update examples/README.md on the usage and note.

For this PR, I checked the following test cases.

CLI Usage

$ ./ghpc create --help
Create a new blueprint based on a provided YAML config.

Usage:
  ghpc create FILENAME [flags]

Flags:
      --backend-config strings    Comma-separated list of name=value variables to set Terraform backend configuration. Can be invoked multiple times.
...

Case 1: No backend config by Yaml and CLI

This checks that backend config is not created if it's not specified by Yaml or CLI.
Just run the following command with the default examples/hpc-cluster-small.yaml:

$ ./ghpc create --vars "project_id=${PROJECT_ID}" examples/hpc-cluster-small.yaml
Terraform group was successfully created in directory hpc-cluster-small/primary
To deploy, run the following commands:
  terraform -chdir=hpc-cluster-small/primary init
  terraform -chdir=hpc-cluster-small/primary validate
  terraform -chdir=hpc-cluster-small/primary apply

Success if Terraform Backend statement is not present in hpc-cluster-small/primary/main.tf.

Case 2: Backend config by Yaml file

Add the following section in examples/hpc-cluster-small.yaml.

terraform_backend_defaults:
  type: gcs
  configuration:
    bucket: a_bucket
    impersonate_service_account: a_bucket_reader@project.iam.gserviceaccount.com

Run the following command:

$ ./ghpc create --vars "project_id=${PROJECT_ID}" examples/hpc-cluster-small.yaml
Terraform group was successfully created in directory hpc-cluster-small/primary
To deploy, run the following commands:
  terraform -chdir=hpc-cluster-small/primary init
  terraform -chdir=hpc-cluster-small/primary validate
  terraform -chdir=hpc-cluster-small/primary apply

Success if Terraform Backend statement is present in hpc-cluster-small/primary/main.tf like below.

$ cat hpc-cluster-small/primary/main.tf
...
terraform {
  backend "gcs" {
    impersonate_service_account = "a_bucket_reader@project.iam.gserviceaccount.com"
    prefix                      = "hpc-cluster-small/hpc-small/primary"
    bucket                      = "a_bucket"
  }
}
...

Case 3: Backend config by CLI

No backend section in Yaml file which is the default examples/hpc-cluster-small.yaml, run the following command:

With "type=gcs":

$ ./ghpc create --vars "project_id=${PROJECT_ID}" examples/hpc-cluster-small.yaml --backend-config "type=gcs,bucket=a_bucket,impersonate_service_account=a_bucket_reader@project.iam.gserviceaccount.com"
Terraform group was successfully created in directory hpc-cluster-small/primary
To deploy, run the following commands:
  terraform -chdir=hpc-cluster-small/primary init
  terraform -chdir=hpc-cluster-small/primary validate
  terraform -chdir=hpc-cluster-small/primary apply

Without "type=gcs":
The tool sets "gcs" as backend type by default.

$ ./ghpc create --vars "project_id=${PROJECT_ID}" examples/hpc-cluster-small.yaml --backend-config "bucket=a_bucket,impersonate_service_account=a_bucket_reader@project.iam.gserviceaccount.com"
Terraform group was successfully created in directory hpc-cluster-small/primary
To deploy, run the following commands:
  terraform -chdir=hpc-cluster-small/primary init
  terraform -chdir=hpc-cluster-small/primary validate
  terraform -chdir=hpc-cluster-small/primary apply

Success if Terraform Backend statement is present in hpc-cluster-small/primary/main.tf like below.

$ cat hpc-cluster-small/primary/main.tf
...
terraform {
  backend "gcs" {
    bucket                      = "a_bucket"
    impersonate_service_account = "a_bucket_reader@project.iam.gserviceaccount.com"
    prefix                      = "hpc-cluster-small/hpc-small/primary"
  }
}
...

Case 4: Backend config by Yaml file & CLI

The toolkit overrides the backend config in the yaml file by variables at CLI.

Add the following section in examples/hpc-cluster-small.yaml.

terraform_backend_defaults:
  type: gcs
  configuration:
    bucket: a_bucket
    impersonate_service_account: a_bucket_reader@project.iam.gserviceaccount.com

Run the following command:

$ ./ghpc create --vars "project_id=${PROJECT_ID}" examples/hpc-cluster-small.yaml --backend-config "type=gcs,bucket=a_bucket_cli,impersonate_service_account=a_bucket_reader_cli@project.iam.gserviceaccount.com"

Success if Terraform Backend statement is present in hpc-cluster-small/primary/main.tf and it has the variables set by CLI (e.g. a_bucket_cli, not a_bucket) like below.

$ cat hpc-cluster-small/primary/main.tf
...
terraform {
  backend "gcs" {
    bucket                      = "a_bucket_cli"
    impersonate_service_account = "a_bucket_reader_cli@project.iam.gserviceaccount.com"
    prefix                      = "hpc-cluster-small/hpc-small/primary"
  }
}
...

Expand command

Tested ./ghpc expand similar to ./ghpc create with the test cases above.

If no Terraform backend config is specified, terraform_backend_defaults doesn't have any values in expanded.yaml like below:

$ cat expanded.yaml
...
terraform_backend_defaults:
  type: ""
  configuration: {}

If Terraform backend config is specified by Yaml or CLI, terraform_backend_defaults looks like below:

terraform_backend_defaults:
  type: gcs
  configuration:
    bucket: a_bucket
    impersonate_service_account: a_bucket_reader@project.iam.gserviceaccount.com

If Terraform backend config is specified by Yaml and CLI, terraform_backend_defaults has the values set by CLI:

terraform_backend_defaults:
  type: gcs
  configuration:
    bucket: a_bucket_cli
    impersonate_service_account: a_bucket_reader_cli@project.iam.gserviceaccount.com

Appendix

Run with parameters at minimum:

$ ./ghpc create --vars "project_id=${PROJECT_ID}" examples/hpc-cluster-small.yaml --backend-config "bucket=a_bucket_cli"
Terraform group was successfully created in directory hpc-cluster-small/primary
To deploy, run the following commands:
  terraform -chdir=hpc-cluster-small/primary init
  terraform -chdir=hpc-cluster-small/primary validate
  terraform -chdir=hpc-cluster-small/primary apply

$ cat hpc-cluster-small/primary/main.tf
...
terraform {
  backend "gcs" {
    bucket = "a_bucket_cli"
    prefix = "hpc-cluster-small/hpc-small/primary"
  }
}
...

Submission Checklist

• Have you installed and run this change against pre-commit? pre-commit install
• Are all tests passing? make tests
• If applicable, have you written additional unit tests to cover this
  change?
• Is unit test coverage still above 80%?
• Have you updated any application documentation such as READMEs and user
  guides?
• Have you followed the guidelines in our Contributing document?