Datalake V1

[ShadeWyrm]

Adds the stuff for the datalake. SUPER WIP.

[craigzour]

Had a first look at it

[craigzour]

We already had an existing S3 module. Is there a reason why you created a new buckets one?

[ShadeWyrm]

I was just trying to keep things independent for now, so that I could cut out things as necessary and not get lost in pre-existing scripts. I'd look at moving them into the s3 module prior maybe - but I also like the idea of not tightly coupling so if we need to nuke or change it can be quick.

Lemme know thoughts!~ :)

[craigzour]

I also like the idea of keeping things independent. What we could do is to just put it in the S3 module but have separate .tf files to identify the new buckets as being part of the Datalake work.

[github-actions]

⚠ Terrform update available

Terraform: 1.10.3 (using 1.9.8)
Terragrunt: 0.70.4 (using 0.69.2)

[github-actions]

Staging: load_testing

✅   Terraform Init: success
✅   Terraform Validate: success
✅   Terraform Format: success
✅   Terraform Plan: success
✅   Conftest: success

Plan: 0 to add, 4 to change, 0 to destroy

Show summary

CHANGE	NAME
update	aws_iam_policy.load_test_lambda
	aws_ssm_parameter.load_testing_form_id
	aws_ssm_parameter.load_testing_form_private_key
	aws_ssm_parameter.load_testing_zitadel_app_private_key

Show plan

Resource actions are indicated with the following symbols:
  ~ update in-place
 <= read (data resources)

Terraform will perform the following actions:

  # data.aws_iam_policy_document.load_test_lambda will be read during apply
  # (depends on a resource or a module with changes pending)
 <= data "aws_iam_policy_document" "load_test_lambda" {
      + id            = (known after apply)
      + json          = (known after apply)
      + minified_json = (known after apply)

      + statement {
          + actions   = [
              + "ssm:GetParameters",
            ]
          + effect    = "Allow"
          + resources = [
              + "arn:aws:ssm:ca-central-1:687401027353:parameter/load-testing/form-id",
              + "arn:aws:ssm:ca-central-1:687401027353:parameter/load-testing/form-private-key",
              + "arn:aws:ssm:ca-central-1:687401027353:parameter/load-testing/zitadel-app-private-key",
            ]
          + sid       = "GetSSMParameters"
        }
      + statement {
          + actions   = [
              + "lambda:InvokeFunction",
            ]
          + effect    = "Allow"
          + resources = [
              + "arn:aws:lambda:ca-central-1:687401027353:function:Submission",
            ]
          + sid       = "InvokeSubmissionLambda"
        }
    }

  # aws_iam_policy.load_test_lambda will be updated in-place
  ~ resource "aws_iam_policy" "load_test_lambda" {
        id               = "arn:aws:iam::687401027353:policy/LoadTestLambda"
        name             = "LoadTestLambda"
      ~ policy           = jsonencode(
            {
              - Statement = [
                  - {
                      - Action   = "ssm:GetParameters"
                      - Effect   = "Allow"
                      - Resource = [
                          - "arn:aws:ssm:ca-central-1:687401027353:parameter/load-testing/zitadel-app-private-key",
                          - "arn:aws:ssm:ca-central-1:687401027353:parameter/load-testing/form-private-key",
                          - "arn:aws:ssm:ca-central-1:687401027353:parameter/load-testing/form-id",
                        ]
                      - Sid      = "GetSSMParameters"
                    },
                  - {
                      - Action   = "lambda:InvokeFunction"
                      - Effect   = "Allow"
                      - Resource = "arn:aws:lambda:ca-central-1:687401027353:function:Submission"
                      - Sid      = "InvokeSubmissionLambda"
                    },
                ]
              - Version   = "2012-10-17"
            }
        ) -> (known after apply)
        tags             = {}
        # (7 unchanged attributes hidden)
    }

  # aws_ssm_parameter.load_testing_form_id will be updated in-place
  ~ resource "aws_ssm_parameter" "load_testing_form_id" {
        id              = "/load-testing/form-id"
      + insecure_value  = (known after apply)
        name            = "/load-testing/form-id"
        tags            = {}
      ~ version         = 1 -> (known after apply)
        # (9 unchanged attributes hidden)
    }

  # aws_ssm_parameter.load_testing_form_private_key will be updated in-place
  ~ resource "aws_ssm_parameter" "load_testing_form_private_key" {
        id              = "/load-testing/form-private-key"
      + insecure_value  = (known after apply)
        name            = "/load-testing/form-private-key"
        tags            = {}
      ~ version         = 3 -> (known after apply)
        # (9 unchanged attributes hidden)
    }

  # aws_ssm_parameter.load_testing_zitadel_app_private_key will be updated in-place
  ~ resource "aws_ssm_parameter" "load_testing_zitadel_app_private_key" {
        id              = "/load-testing/zitadel-app-private-key"
      + insecure_value  = (known after apply)
        name            = "/load-testing/zitadel-app-private-key"
        tags            = {}
      ~ version         = 3 -> (known after apply)
        # (9 unchanged attributes hidden)
    }

Plan: 0 to add, 4 to change, 0 to destroy.

─────────────────────────────────────────────────────────────────────────────

Saved the plan to: plan.tfplan

To perform exactly these actions, run the following command to apply:
    terraform apply "plan.tfplan"

Show Conftest results

WARN - plan.json - main - Missing Common Tags: ["aws_iam_policy.load_test_lambda"]
WARN - plan.json - main - Missing Common Tags: ["aws_iam_role.load_test_lambda"]
WARN - plan.json - main - Missing Common Tags: ["aws_lambda_function.load_testing"]
WARN - plan.json - main - Missing Common Tags: ["aws_ssm_parameter.load_testing_form_id"]
WARN - plan.json - main - Missing Common Tags: ["aws_ssm_parameter.load_testing_form_private_key"]
WARN - plan.json - main - Missing Common Tags: ["aws_ssm_parameter.load_testing_zitadel_app_private_key"]

25 tests, 19 passed, 6 warnings, 0 failures, 0 exceptions