fix: rework response archiver lambda

[craigzour]

Summary | Résumé

• Reworked response archiver lambda.
  • Increased lambda timeout from 3 seconds to 5 minutes
  • Code now processes chunks of 100 responses at a time
  • Slightly changed archiving location for both the submission and potential attachments to group them so that it is easier to reassemble

Test instructions

• Create form with file inputs (you have to turn on experimental features in the feature flags). Having file inputs allows us to fully test the archiving lambda but it can work without it as well.
• Submit at least one response to that form you created.
• Manually edit the removalDate of your response(s) using the https://app.localstack.cloud/inst/default/resources tool. Changed it so that the timestamp is in the past.
• Manually trigger the response archiver lambda using awslocal lambda invoke --function-name Response_Archiver /dev/null.
• The response(s) should now be archived. You can check both the DynamoDB and S3 services to see the result. The response should have been removed from DynamoDB and saved in the forms-local-archive-storage S3 bucket.

[github-actions]

⚠ Terrform update available

Terragrunt: 0.54.17 (using 0.54.8)

[github-actions]

Staging: secrets

✅   Terraform Init: success
✅   Terraform Validate: success
✅   Terraform Format: success
✅   Terraform Plan: success
✅   Conftest: success

⚠️   Warning: resources will be destroyed by this change!

Plan: 1 to add, 0 to change, 1 to destroy

Show summary

CHANGE	NAME
recreate	aws_secretsmanager_secret_version.notify_callback_bearer_token

Show plan

Resource actions are indicated with the following symbols:
-/+ destroy and then create replacement

Terraform will perform the following actions:

  # aws_secretsmanager_secret_version.notify_callback_bearer_token must be replaced
-/+ resource "aws_secretsmanager_secret_version" "notify_callback_bearer_token" {
      ~ arn            = "arn:aws:secretsmanager:ca-central-1:687401027353:secret:notify_callback_bearer_token-aXJPLs" -> (known after apply)
      ~ id             = "arn:aws:secretsmanager:ca-central-1:687401027353:secret:notify_callback_bearer_token-aXJPLs|4A43D97C-2937-4E91-A924-ADF60955E459" -> (known after apply)
      ~ secret_string  = (sensitive value) # forces replacement
      ~ version_id     = "4A43D97C-2937-4E91-A924-ADF60955E459" -> (known after apply)
      ~ version_stages = [
          - "AWSPREVIOUS",
        ] -> (known after apply)
        # (1 unchanged attribute hidden)
    }

Plan: 1 to add, 0 to change, 1 to destroy.

─────────────────────────────────────────────────────────────────────────────

Saved the plan to: plan.tfplan

To perform exactly these actions, run the following command to apply:
    terraform apply "plan.tfplan"

Show Conftest results

WARN - plan.json - main - Missing Common Tags: ["aws_secretsmanager_secret.freshdesk_api_key"]
WARN - plan.json - main - Missing Common Tags: ["aws_secretsmanager_secret.notify_api_key"]
WARN - plan.json - main - Missing Common Tags: ["aws_secretsmanager_secret.notify_callback_bearer_token"]
WARN - plan.json - main - Missing Common Tags: ["aws_secretsmanager_secret.recaptcha_secret"]
WARN - plan.json - main - Missing Common Tags: ["aws_secretsmanager_secret.token_secret"]

24 tests, 19 passed, 5 warnings, 0 failures, 0 exceptions

[github-actions]

Staging: lambdas

✅   Terraform Init: success
✅   Terraform Validate: success
✅   Terraform Format: success
✅   Terraform Plan: success
✅   Conftest: success

Plan: 0 to add, 3 to change, 0 to destroy

Show summary

CHANGE	NAME
update	aws_lambda_function.response_archiver
	aws_lambda_function.vault_integrity
	aws_s3_object.response_archiver_code

Show plan

Resource actions are indicated with the following symbols:
  ~ update in-place

Terraform will perform the following actions:

  # aws_lambda_function.response_archiver will be updated in-place
  ~ resource "aws_lambda_function" "response_archiver" {
      ~ handler                        = "archiver.handler" -> "response_archiver.handler"
        id                             = "Response_Archiver"
      ~ last_modified                  = "2024-01-11T18:12:30.000+0000" -> (known after apply)
      ~ s3_object_version              = "lFW80OyC8w01i3y9eRqUjPPmQAXRtl0Q" -> (known after apply)
      ~ source_code_hash               = "HlOcaGXKLFMSZO3DF101vb5Af5YWWPGBw6Z16Zu9hVI=" -> "iCOPYEkuyPrXdcFRLr6qL6swUo/kXiUZTdHb0tWWuOE="
        tags                           = {}
      ~ timeout                        = 10 -> 300
        # (19 unchanged attributes hidden)

        # (3 unchanged blocks hidden)
    }

  # aws_lambda_function.vault_integrity will be updated in-place
  ~ resource "aws_lambda_function" "vault_integrity" {
        id                             = "Vault_Data_Integrity_Check"
      ~ last_modified                  = "2024-01-12T15:13:35.000+0000" -> (known after apply)
      ~ source_code_hash               = "PD5RmSEVjAYMZc1lvfSSn6O3RadR+J09hCPKVbQS/+g=" -> "TowbMcppnki+0a5fq50Oral3CqleiwGw7U1igvFz0Ws="
        tags                           = {}
        # (24 unchanged attributes hidden)

        # (3 unchanged blocks hidden)
    }

  # aws_s3_object.response_archiver_code will be updated in-place
  ~ resource "aws_s3_object" "response_archiver_code" {
        id                     = "response_archiver_code"
      ~ source_hash            = "HlOcaGXKLFMSZO3DF101vb5Af5YWWPGBw6Z16Zu9hVI=" -> "iCOPYEkuyPrXdcFRLr6qL6swUo/kXiUZTdHb0tWWuOE="
        tags                   = {}
      ~ version_id             = "lFW80OyC8w01i3y9eRqUjPPmQAXRtl0Q" -> (known after apply)
        # (11 unchanged attributes hidden)
    }

Plan: 0 to add, 3 to change, 0 to destroy.

─────────────────────────────────────────────────────────────────────────────

Saved the plan to: plan.tfplan

To perform exactly these actions, run the following command to apply:
    terraform apply "plan.tfplan"

Show Conftest results

WARN - plan.json - main - Missing Common Tags: ["aws_cloudwatch_event_rule.cron_2am_every_day"]
WARN - plan.json - main - Missing Common Tags: ["aws_cloudwatch_event_rule.cron_3am_every_day"]
WARN - plan.json - main - Missing Common Tags: ["aws_cloudwatch_event_rule.cron_4am_every_day"]
WARN - plan.json - main - Missing Common Tags: ["aws_cloudwatch_event_rule.cron_5am_every_business_day"]
WARN - plan.json - main - Missing Common Tags: ["aws_cloudwatch_log_group.archive_form_templates"]
WARN - plan.json - main - Missing Common Tags: ["aws_cloudwatch_log_group.audit_logs"]
WARN - plan.json - main - Missing Common Tags: ["aws_cloudwatch_log_group.dead_letter_queue_consumer"]
WARN - plan.json - main - Missing Common Tags: ["aws_cloudwatch_log_group.nagware"]
WARN - plan.json - main - Missing Common Tags: ["aws_cloudwatch_log_group.reliability"]
WARN - plan.json - main - Missing Common Tags: ["aws_cloudwatch_log_group.response_archiver"]
WARN - plan.json - main - Missing Common Tags: ["aws_cloudwatch_log_group.submission"]
WARN - plan.json - main - Missing Common Tags: ["aws_cloudwatch_log_group.vault_integrity"]
WARN - plan.json - main - Missing Common Tags: ["aws_iam_policy.lambda_dynamodb"]
WARN - plan.json - main - Missing Common Tags: ["aws_iam_policy.lambda_kms"]
WARN - plan.json - main - Missing Common Tags: ["aws_iam_policy.lambda_logging"]
WARN - plan.json - main - Missing Common Tags: ["aws_iam_policy.lambda_rds"]
WARN - plan.json - main - Missing Common Tags: ["aws_iam_policy.lambda_s3"]
WARN - plan.json - main - Missing Common Tags: ["aws_iam_policy.lambda_secrets"]
WARN - plan.json - main - Missing Common Tags: ["aws_iam_policy.lambda_sns"]
WARN - plan.json - main - Missing Common Tags: ["aws_iam_policy.lambda_sqs"]
WARN - plan.json - main - Missing Common Tags: ["aws_iam_role.lambda"]
WARN - plan.json - main - Missing Common Tags: ["aws_lambda_function.audit_logs"]
WARN - plan.json - main - Missing Common Tags: ["aws_lambda_function.form_archiver"]
WARN -...