From 20e0fc12be3789102a537b8ea8217c8dd64b99fb Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Cl=C3=A9ment=20JANIN?= Date: Tue, 9 Apr 2024 14:51:45 -0400 Subject: [PATCH] feature: deploy redis and postgresql in localstack (#620) * feature: implement Redis and PostgreSQL in Localstack * feature: make it possible to tweak Localstack persistence settings using environment variables * feature: reintroduce clean terraform state files option in localstack services script * chore: update README.md * Update env/cloud/secrets/terragrunt.hcl Co-authored-by: Bryan Robitaille --------- Co-authored-by: Bryan Robitaille --- .env.example | 15 +- README.md | 279 ++++-------------- aws/alarms/lambda.tf | 2 +- aws/alarms/lambda/notify_slack/utils.ts | 2 +- aws/ecr/ecr.tf | 2 - .../code/form_archiver/lib/templates.ts | 106 +------ aws/lambdas/code/form_archiver/package.json | 4 +- aws/lambdas/code/form_archiver/yarn.lock | 161 +--------- aws/lambdas/code/nagware/lib/templates.ts | 180 ++++------- aws/lambdas/code/nagware/nagware.ts | 34 ++- aws/lambdas/code/nagware/package.json | 4 +- aws/lambdas/code/nagware/yarn.lock | 161 +--------- .../code/reliability/lib/s3FileInput.ts | 3 +- aws/lambdas/code/reliability/lib/templates.ts | 226 ++++---------- aws/lambdas/code/reliability/package.json | 2 - aws/lambdas/code/reliability/reliability.ts | 23 +- aws/lambdas/code/reliability/yarn.lock | 161 +--------- aws/lambdas/form_archiver.tf | 14 +- aws/lambdas/nagware.tf | 4 - aws/lambdas/reliability.tf | 4 - aws/lambdas/vault_integrity.tf | 1 - aws/network/inputs.tf | 5 - aws/network/vpc_flow_log.tf | 3 +- aws/rds/rds.tf | 1 - aws/redis/redis.tf | 4 +- docker-compose.yml | 35 +-- env/cloud/app/terragrunt.hcl | 1 - env/cloud/lambdas/terragrunt.hcl | 7 +- env/cloud/network/.terraform.lock.hcl | 45 +++ env/cloud/network/terragrunt.hcl | 14 - env/cloud/rds/.terraform.lock.hcl | 45 +++ env/cloud/rds/terragrunt.hcl | 9 +- env/cloud/redis/.terraform.lock.hcl | 45 +++ env/cloud/secrets/terragrunt.hcl | 10 +- env/common/local-provider.tf | 2 +- localstack_services.sh | 45 ++- 36 files changed, 439 insertions(+), 1220 deletions(-) create mode 100644 env/cloud/network/.terraform.lock.hcl create mode 100644 env/cloud/rds/.terraform.lock.hcl create mode 100644 env/cloud/redis/.terraform.lock.hcl diff --git a/.env.example b/.env.example index 59e160540..d41f7ef10 100644 --- a/.env.example +++ b/.env.example @@ -1,3 +1,12 @@ -APP_ENV="local" -NOTIFY_API_KEY="" -LOCALSTACK_AUTH_TOKEN="your_pro_license_token" \ No newline at end of file +APP_ENV=local + +LOCALSTACK_AUTH_TOKEN= + +# Default values will be SNAPSHOT_LOAD_STRATEGY=ON_STARTUP and SNAPSHOT_SAVE_STRATEGY=SCHEDULED +# with a SNAPSHOT_FLUSH_INTERVAL=60 if you do not set anything below. The interval variable is only used if the save strategy is set to scheduled. +# See https://docs.localstack.cloud/user-guide/state-management/persistence/#save-strategies for more information. +SNAPSHOT_SAVE_STRATEGY= +SNAPSHOT_FLUSH_INTERVAL= +SNAPSHOT_LOAD_STRATEGY= + +NOTIFY_API_KEY= \ No newline at end of file diff --git a/README.md b/README.md index 063e8c4b4..9ca3e6621 100644 --- a/README.md +++ b/README.md @@ -6,11 +6,7 @@ Infrastructure as Code for the GC Forms environment. Pull Requests in this repository require all commits to be signed before they can be merged. Please see [this guide](https://docs.github.com/en/github/authenticating-to-github/managing-commit-signature-verification) for more information. -## Running Lambdas and DBs locally - -You will need to have the following installed on a macOS machine. - -### Prerequisites: +## Prerequisites: - [Docker Hub](https://docs.docker.com/desktop/mac/install/) or [Colima](https://github.com/abiosoft/colima) @@ -27,17 +23,6 @@ You will need to have the following installed on a macOS machine. 1. `brew install warrensbox/tap/tgswitch` 1. `tgswitch 0.54.8` -- Yarn: `brew install yarn` - -- AWS CLI: `brew install awscli` - -- AWS SAM CLI - - 1. If you previously had it installed with Brew then you should first uninstall this package using `brew uninstall aws-sam-cli` - 1. To install the AWS SAM CLI tool, follow the instructions on https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/install-sam-cli.html under "macOS" and "Command line - All users" (you will have to download the .pkg file and run a command to install it on your machine) - - Please note that the latest tested version of AWS SAM CLI with your infrastructure code was `1.99.0`. - ### If using Colima - Docker: `brew install docker docker-compose docker-credential-manager` @@ -56,8 +41,6 @@ nano ~/.docker/config.json - Colima: `brew insteall colima` -Ensure that apps like AWS SAM can connect to the correct docker.sock - ```shell # as /var/ is a protected directory, we will need sudo sudo ln ~/.colima/default/docker.sock /var/run @@ -69,249 +52,109 @@ ls /var/run Colima can be set as a service to start on login: `brew services start colima` -### Starting LocalStack and E2E testing from devcontainers +## Request Localstack Pro license -**# TODO Outdated #** +You will need to create a Localstack account using your CDS email address [here](https://app.localstack.cloud/sign-in) and then ask your supervisor to assign you a Pro license license. -For instructions on how to run without using dev containers please skip to the next section. +## Set your environment variables -1. forms-terraform: +Create an `.env` file at the root of the project and use the `.env.example` as a template. You can find some of the values in 1Password > Local Development .ENV secure note. +The `LOCALSTACK_AUTH_TOKEN` value will be accessible [here](https://app.localstack.cloud/workspace/auth-token) once you have been assigned a Pro license. -```sh -# Build the local infrastructure (run on first setup and when there are Terraform changes) -make terragrunt +## Start Localstack -# Start the lambda functions -make lambdas -``` - -2. platform-forms-client: - -```sh -# Install dependencies, run database migrations and start local server -yarn --cwd migrations install -yarn install -yarn dev +```shell +$ docker-compose up ``` -### Starting LocalStack and services without dev containers - -#### TLDR - -**Spare me the details and give me the commands** - -Only do once as part of setup: - -You will also have to supply the following environment variables and values to platform-forms-client to get everything working with the local lambdas and localstack. +
+See expected console output ```shell -# localstack only simulates a us-east-1 region -AWS_ACCESS_KEY_ID=test -AWS_SECRET_ACCESS_KEY=test -AWS_REGION=ca-central-1 -RELIABILITY_FILE_STORAGE=forms-local-reliability-file-storage -LOCAL_AWS_ENDPOINT=http://127.0.0.1:4566 +[+] Building 0.0s (0/0) +[+] Running 2/2 + ✔ Network forms-terraform_default Created 0.1s + ✔ Container GCForms_LocalStack Created 0.1s +Attaching to GCForms_LocalStack +GCForms_LocalStack | +GCForms_LocalStack | LocalStack version: 3.2.1.dev20240306170817 +GCForms_LocalStack | LocalStack Docker container id: 00e39dc6785e +GCForms_LocalStack | LocalStack build date: 2024-03-06 +GCForms_LocalStack | LocalStack build git hash: 93fc329 +GCForms_LocalStack | +GCForms_LocalStack | 2024-03-27T14:11:56.175 INFO --- [ MainThread] l.bootstrap.licensingv2 : Successfully requested and activated new license :pro 🔑✅ +GCForms_LocalStack | 2024-03-27T14:11:58.611 INFO --- [ MainThread] l.p.snapshot.plugins : registering ON_STARTUP load strategy +GCForms_LocalStack | 2024-03-27T14:11:59.649 INFO --- [ MainThread] l.p.snapshot.plugins : registering SCHEDULED save strategy +GCForms_LocalStack | 2024-03-27T14:11:59.713 INFO --- [ MainThread] l.extensions.platform : loaded 0 extensions +GCForms_LocalStack | 2024-03-27T14:12:00.097 INFO --- [-functhread4] hypercorn.error : Running on https://0.0.0.0:4566 (CTRL + C to quit) +GCForms_LocalStack | 2024-03-27T14:12:00.097 INFO --- [-functhread4] hypercorn.error : Running on https://0.0.0.0:4566 (CTRL + C to quit) +GCForms_LocalStack | 2024-03-27T14:12:00.098 INFO --- [-functhread4] hypercorn.error : Running on https://0.0.0.0:443 (CTRL + C to quit) +GCForms_LocalStack | 2024-03-27T14:12:00.098 INFO --- [-functhread4] hypercorn.error : Running on https://0.0.0.0:443 (CTRL + C to quit) +GCForms_LocalStack | 2024-03-27T14:12:00.316 INFO --- [ MainThread] localstack.utils.bootstrap : Execution of "start_runtime_components" took 602.48ms +GCForms_LocalStack | Ready. +GCForms_LocalStack | 2024-03-27T14:12:03.093 INFO --- [ MainThread] l.p.snapshot.plugins : restoring state of all services on startup ``` -#### Set your environment variables (optional) +
-Copy the `.env.example` file and rename it to `.env`. Some of the variables require values that can be found in the 1Password shared secrets note. +Once Localstack is ready to use you should be able to interact with local AWS services using the [Localstack web application](https://app.localstack.cloud/inst/default/resources). -#### Launch the infrastructure and the application +**If the Localstack web application is not able to connect to the instance you just started you may have to add `127.0.0.1 localhost.localstack.cloud` to your `/etc/hosts`.** -Everytime you want to run localstack and lambdas locally +## Deploy infrastructure -1. In one terminal run `docker-compose up` -2. In a second terminal run `./localstack_services.sh` -3. In a third terminal in the `platform-forms-client` repo run `yarn dev` +Now that we have localstack up and running it's time to deploy our local AWS services to mimic our cloud environments. -#### It didn't work...I need the details - -Once you have localstack installed you should be able to start the localstack container and services using the following command. +#### Deploy on fresh Localstack instance ```shell -$ docker-compose up +$ ./localstack_services.sh clean ``` -You should see the following output if localstack has successfully started +The `clean` argument will make sure all existing Terraform state files are being deleted first. + +#### Deploy on existing Localstack instance ```shell -❯ docker-compose up ─╯ -[+] Running 3/0 - ✔ Container GCForms_Redis Created - ✔ Container GCForms_DB Created - ✔ Container GCForms_LocalStack Created -Attaching to GCForms_DB, GCForms_LocalStack, GCForms_Redis -GCForms_DB | -GCForms_DB | -GCForms_DB | 2023-12-07 14:22:54.172 UTC [1] LOG: listening on IPv4 address "0.0.0.0", port 5432 -GCForms_DB | 2023-12-07 14:22:54.173 UTC [1] LOG: listening on IPv6 address "::", port 5432 -GCForms_DB | 2023-12-07 14:22:54.177 UTC [1] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432" -GCForms_DB | 2023-12-07 14:22:54.198 UTC [26] LOG: database system was shut down at 2023-12-07 14:13:14 UTC -GCForms_DB | 2023-12-07 14:22:54.204 UTC [1] LOG: database system is ready to accept connections -GCForms_Redis | 1:C 07 Dec 2023 14:22:54.239 # WARNING Memory overcommit must be enabled! Without it, a background save or replication may fail under low memory condition. Being disabled, it can also cause failures without low memory condition, see https://github.com/jemalloc/jemalloc/issues/1328. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect. -GCForms_Redis | 1:C 07 Dec 2023 14:22:54.239 * oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo -GCForms_Redis | 1:C 07 Dec 2023 14:22:54.239 * Redis version=7.2.3, bits=64, commit=00000000, modified=0, pid=1, just started -GCForms_Redis | 1:C 07 Dec 2023 14:22:54.239 # Warning: no config file specified, using the default config. In order to specify a config file use redis-server /path/to/redis.conf -GCForms_Redis | 1:M 07 Dec 2023 14:22:54.239 * monotonic clock: POSIX clock_gettime -GCForms_Redis | 1:M 07 Dec 2023 14:22:54.240 * Running mode=standalone, port=6379. -GCForms_Redis | 1:M 07 Dec 2023 14:22:54.240 * Server initialized -GCForms_Redis | 1:M 07 Dec 2023 14:22:54.240 * Loading RDB produced by version 7.2.3 -GCForms_Redis | 1:M 07 Dec 2023 14:22:54.240 * RDB age 579 seconds -GCForms_Redis | 1:M 07 Dec 2023 14:22:54.240 * RDB memory usage when created 0.83 Mb -GCForms_Redis | 1:M 07 Dec 2023 14:22:54.240 * Done loading RDB, keys loaded: 6, keys expired: 0. -GCForms_Redis | 1:M 07 Dec 2023 14:22:54.241 * DB loaded from disk: 0.000 seconds -GCForms_Redis | 1:M 07 Dec 2023 14:22:54.241 * Ready to accept connections tcp -GCForms_LocalStack | -GCForms_LocalStack | LocalStack version: 3.0.3.dev -GCForms_LocalStack | LocalStack Docker container id: 0211f3486795 -GCForms_LocalStack | LocalStack build date: 2023-12-05 -GCForms_LocalStack | LocalStack build git hash: c1dcbc50 -GCForms_LocalStack | -GCForms_LocalStack | 2023-12-07T14:22:56.069 INFO --- [-functhread4] hypercorn.error : Running on https://0.0.0.0:4566 (CTRL + C to quit) -GCForms_LocalStack | 2023-12-07T14:22:56.069 INFO --- [-functhread4] hypercorn.error : Running on https://0.0.0.0:4566 (CTRL + C to quit) -GCForms_LocalStack | Ready. +$ ./localstack_services.sh ``` -### Setting up local infrastructure +This is something you may have to do if you want to deploy an infrastructure update to your current Localstack instance. -Now that we have localstack up and running it's time to configure our local AWS services to mimic our cloud environments +**Please note that if you stop Localstack you don't need to run this script again.** +**Localstack Pro offers automatic persistence for all deployed services. This is enabled by default and can be tweaked through your `.env` file.** -**Please note if you stop localstack you will need to run this script again** -**Localstack does not persist states between restarts of the service** +Congratulations! You should now have all the necessary infrastructure configured on Localstack to support all the web applications functions completely locally without needing an AWS account. -run `./localstack_services.sh` +## How to manually invoke a Lambda function -Congratulations! You should now have all the necessary infrastructure configured on localstack to run lambda functions completely locally without needing an AWS account. +```shell +$ awslocal lambda invoke --function-name output.txt +``` -### Dynamo Database Table Schemas +In case you want to invoke a function that expects a specific payload you can pass it using the `--payload '{}'` argument. -#### Vault Table +## Dynamo Database Table Schemas -##### Table +### Vault Table + +#### Table ![Vault Table](./readme_images/Vault.png) -##### Archive Global Secondary Index +#### Archive Global Secondary Index This Index supports the archiving of Vault responses ![Archive GSI](./readme_images/GSI_Vault_Archive.png) -##### Status Global Secondary Index +#### Status Global Secondary Index This Index supports the future feature of the Retrieval API. Essentially the ability to retrieve responses without using the Application Interface. ![Status Index](./readme_images/GSI_Vault_Status.png) -##### Nagware Global Secondary Index +#### Nagware Global Secondary Index This Index supports the Nagware feature. It gives the ability to retrieve form submissions with a specific status and creation date. ![Nagware Index](./readme_images/GSI_Vault_Nagware.png) -### Invoking Lambdas manually - -**# TODO Update #** - -**Lambda's are now invoked automatically similar to the cloud environment** - -If you want to invoke a lambda specifically, here’s the example command: -`aws lambda invoke --function-name "Submission" --endpoint-url "http://127.0.0.1:3001" --no-verify-ssl --payload fileb://./file.json out.txt` -**NOTE:** _`fileb://` allows a JSON file that uses UTF-8 encoding for the payload._ - -Troubleshooting -db host env var: runs in docker - need to get localhost for host machine -if you get connection errors: postgresql.conf listen address “\*” - -Notes: -When running locally using AWS SAM, the env var `LOCALSTACK = true` is set automatically - so I hook into this for local testing - -#### Running the reliability lambda - -Unfortunately due to AWS SAM limitations it is not possible to automatically trigger the reliability lambda function whenever an event is pushed to the SQS queue via the Submission lambda. - -In order to run the reliability lambda. First spin up local lambdas using the `start_local_lambda` script and submit a form through the platform-forms-client. - -You should see in the console the submission ID after the Submission function has successfully been invoked - -```shell -START RequestId: 21a1df8b-ed4b-4fd5-8dad-e92ee76341ee Version: $LATEST -2022-01-19T09:16:08.084Z 21a1df8b-ed4b-4fd5-8dad-e92ee76341ee INFO {"status": "success", "sqsMessage": "aab25a19-e12e-5e72-fcde-b26ca07b2cda", "submissionID": "2dd4930d-cd77-41b3-a68e-9f44ef9e80f5"} -END RequestId: 21a1df8b-ed4b-4fd5-8dad-e92ee76341ee -REPORT RequestId: 21a1df8b-ed4b-4fd5-8dad-e92ee76341ee Init Duration: 0.48 ms Duration: 797.46 ms Billed Duration: 798 ms Memory Size: 128 MB Max Memory Used: 128 MB -2022-01-19 04:16:08 127.0.0.1 - - [19/Jan/2022 04:16:08] "POST /2015-03-31/functions/Submission/invocations HTTP/1.1" 200 - -``` - -You can then take this submission id and use the `invoke_reliability` script to simulate the reliability function being invoked by a SQS event. - -```shell -$ ./invoke_reliability.sh 2dd4930d-cd77-41b3-a68e-9f44ef9e80f5 -Reading invoke payload from stdin (you can also pass it from file with --event) -Invoking reliability.handler (nodejs14.x) -ReliabilityLayer is a local Layer in the template -Building image....................... -Skip pulling image and use local one: samcli/lambda:nodejs14.x-x86_64-2ab34c74bed6bccfbae4c6fc8. - -Mounting /Users/omarnasr/Documents/work/forms-staging-terraform/aws/app/lambda/reliability as /var/task:ro,delegated inside runtime container -START RequestId: ca3b4eed-9c78-46b0-ab55-62ccb8a93357 Version: $LATEST -2022-01-19T09:19:07.811Z ca3b4eed-9c78-46b0-ab55-62ccb8a93357 INFO Lambda Template Client successfully triggered -2022-01-19T09:19:09.199Z ca3b4eed-9c78-46b0-ab55-62ccb8a93357 INFO {"status": "success", "submissionID": "2dd4930d-cd77-41b3-a68e-9f44ef9e80f5", "sqsMessage":"aab25a19-e12e-5e72-fcde-b26ca07b2cda", "method":"notify"} -{"$metadata":{"httpStatusCode":200,"requestId":"44208450-2823-491c-8fbd-47248443e97a","attempts":1,"totalRetryDelay":0},"ConsumedCapacity":{"CapacityUnits":1,"TableName":"ReliabilityQueue"}}END RequestId: ca3b4eed-9c78-46b0-ab55-62ccb8a93357 -REPORT RequestId: ca3b4eed-9c78-46b0-ab55-62ccb8a93357 Init Duration: 0.17 ms Duration: 8051.83 ms Billed Duration: 8052 ms Memory Size: 128 MB Max Memory Used: 128 MB -``` - -Please note you must configure the `NOTIFY_API_KEY` in the `templates.yml` for this to work. If you have configured it correctly... you should successfully receive an email with the form response - -#### Running the archiver lambda - -Unfortunately due to AWS SAM limitations it is not possible to automatically trigger the archiver lambda function whenever a modification is made to the DynamoDB Vault table. - -In order to run the archiver lambda you should have the FormID and the SubmissionID in your possession so that you can pass them to the `invoke_archiver` script. - -```shell -$ ./invoke_archiver.sh 1 2dd4930d-cd77-41b3-a68e-9f44ef9e80f5 -Reading invoke payload from stdin (you can also pass it from file with --event) -Invoking archiver.handler (nodejs12.x) -ArchiverLayer is a local Layer in the template -Building image....................... -Skip pulling image and use local one: samcli/lambda:nodejs12.x-x86_64-201e8924bd486130d7628ec48. - -Mounting /Users/clementjanin/github/forms-terraform/aws/app/lambda/archive_form_responses as /var/task:ro,delegated inside runtime container -START RequestId: fc1f1509-63af-4a96-a798-81a295e6ca2f Version: $LATEST -END RequestId: fc1f1509-63af-4a96-a798-81a295e6ca2f -REPORT RequestId: fc1f1509-63af-4a96-a798-81a295e6ca2f Init Duration: 0.27 ms Duration: 627.46 ms Billed Duration: 628 ms Memory Size: 128 MB Max Memory Used: 128 MB -{"statusCode":"SUCCESS"} -``` - -#### Running the archive_form_templates lambda - -Unfortunately due to AWS SAM limitations it is not possible to automatically trigger the archive_form_templates lambda function on a daily basis. - -In order to run the archive_form_templates lambda you will have to call the `invoke_archive_form_templates.sh` script. - -```shell -$ ./invoke_archive_form_templates.sh - -Reading invoke payload from stdin (you can also pass it from file with --event) -Invoking archiver.handler (nodejs14.x) -ReliabilityLayer is a local Layer in the template -Building image....................... -Skip pulling image and use local one: samcli/lambda:nodejs14.x-x86_64-2ab34c74bed6bccfbae4c6fc8. - -Mounting /Users/clementjanin/github/forms-terraform/aws/app/lambda/archive_form_templates as /var/task:ro,delegated inside runtime container -START RequestId: 9ce70de4-77c1-4a39-9d4e-e46bd73d1091 Version: $LATEST -END RequestId: 9ce70de4-77c1-4a39-9d4e-e46bd73d1091 -REPORT RequestId: 9ce70de4-77c1-4a39-9d4e-e46bd73d1091 Init Duration: 0.05 ms Duration: 426.80 ms Billed Duration: 427 ms Memory Size: 128 MB Max Memory Used: 128 MB -{"statusCode":"SUCCESS"} -``` - -## Terraform secrets - -Terraform will require the following variables to plan and apply: - -```hcl -ecs_secret_token_secret # JSON Web Token signing secret -notify_api_key # Notify API key to send messages -freshdesk_api_key # FreshDesk API key to send messages -rds_db_password # Database password -slack_webhook # Slack webhook to send CloudWatch notifications -opsgenie_api_key # OpsGenie API key to create alert -``` +# Traduction en français à venir... \ No newline at end of file diff --git a/aws/alarms/lambda.tf b/aws/alarms/lambda.tf index 54caa2d0e..4cafa417e 100644 --- a/aws/alarms/lambda.tf +++ b/aws/alarms/lambda.tf @@ -28,7 +28,7 @@ resource "aws_lambda_function" "notify_slack" { environment { variables = { - ENVIRONMENT = title(var.env) + ENVIRONMENT = var.env SLACK_WEBHOOK = var.slack_webhook OPSGENIE_API_KEY = var.opsgenie_api_key } diff --git a/aws/alarms/lambda/notify_slack/utils.ts b/aws/alarms/lambda/notify_slack/utils.ts index 6c8165128..82878a3e4 100644 --- a/aws/alarms/lambda/notify_slack/utils.ts +++ b/aws/alarms/lambda/notify_slack/utils.ts @@ -43,7 +43,7 @@ export const ungzip = (input: Buffer) => { export const sendToSlack = async (logGroup: string, logMessage: string, logLevel: string) => { console.log("Sending to Slack..."); - const environment = process.env.ENVIRONMENT || "Staging"; + const environment = process.env.ENVIRONMENT || "staging"; const logLevelThemeForSlack = logLevelAsEmojiAndColor(logLevel); const postData = { channel: `#forms-${environment.toLowerCase()}-events`, diff --git a/aws/ecr/ecr.tf b/aws/ecr/ecr.tf index 09cc7a932..769dfb10c 100644 --- a/aws/ecr/ecr.tf +++ b/aws/ecr/ecr.tf @@ -45,8 +45,6 @@ resource "aws_ecr_repository" "load_test_repository" { image_scanning_configuration { scan_on_push = true } - - } resource "aws_ecr_lifecycle_policy" "load_test_policy" { diff --git a/aws/lambdas/code/form_archiver/lib/templates.ts b/aws/lambdas/code/form_archiver/lib/templates.ts index b3e6dd3a8..13dce2da6 100644 --- a/aws/lambdas/code/form_archiver/lib/templates.ts +++ b/aws/lambdas/code/form_archiver/lib/templates.ts @@ -1,16 +1,21 @@ import { RDSDataClient, ExecuteStatementCommand } from "@aws-sdk/client-rds-data"; -import pg from "pg"; - -const REGION = process.env.REGION; /** * Delete all form templates that have been marked as archived (has an TTL value that is not null) */ -const deleteFormTemplatesMarkedAsArchived = async () => { +export const deleteFormTemplatesMarkedAsArchived = async () => { try { - const request = `DELETE FROM "Template" WHERE ttl IS NOT NULL AND ttl < CURRENT_TIMESTAMP`; - const database = process.env.LOCALSTACK === "true" ? requestSAM : requestRDS; - await database(request, []); + const rdsDataClient = new RDSDataClient({ region: process.env.REGION }); + + const executeStatementCommand = new ExecuteStatementCommand({ + database: process.env.DB_NAME, + resourceArn: process.env.DB_ARN, + secretArn: process.env.DB_SECRET, + sql: `DELETE FROM "Template" WHERE ttl IS NOT NULL AND ttl < CURRENT_TIMESTAMP`, + includeResultMetadata: false, // set to true if we want metadata like column names + }); + + await rdsDataClient.send(executeStatementCommand); } catch (error) { // Warn Message will be sent to slack console.warn( @@ -22,90 +27,3 @@ const deleteFormTemplatesMarkedAsArchived = async () => { ); } }; - -/** - * Creates and processes request to LOCAL AWS SAM DB - * @param {string} SQL - * @param {string[]} parameters - * @returns PG Client return value - */ -const requestSAM = async (SQL: string, parameters: Array) => { - const dbClient = new pg.Client(); - - try { - if ( - process.env.PGHOST && - process.env.PGUSER && - process.env.PGDATABASE && - process.env.PGPASSWORD - ) { - dbClient.connect(); - } else { - throw new Error("Missing Environment Variables for DB config"); - } - - const data = await dbClient.query(SQL, parameters); - - return parseConfig(data.rows); - } catch (error) { - throw new Error( - `Error issuing command to Local SAM AWS DB. Reason: ${(error as Error).message}.` - ); - } finally { - dbClient.end(); - } -}; - -/** - * Creates and processes request to RDS - * @param {string} SQL - * @param {{name: string, value: {stringValue: string}[]}} parameters - * @returns RDS client return value - */ -const requestRDS = async ( - SQL: string, - parameters: { name: string; value: { stringValue: string } }[] -) => { - try { - const dbClient = new RDSDataClient({ region: REGION }); - - const params = { - database: process.env.DB_NAME, - resourceArn: process.env.DB_ARN, - secretArn: process.env.DB_SECRET, - sql: SQL, - includeResultMetadata: false, // set to true if we want metadata like column names - parameters: parameters, - }; - - const command = new ExecuteStatementCommand(params); - - const data = await dbClient.send(command); - - return parseConfig(data.records); - } catch (error) { - throw new Error(`Error issuing command to AWS RDS. Reason: ${(error as Error).message}.`); - } -}; - -const parseConfig = (records: any[] | undefined) => { - if (records) { - const parsedRecords = records.map((record) => { - let formConfig; - if (!(process.env.LOCALSTACK === "true")) { - formConfig = JSON.parse(record[0].stringValue.trim(1, -1)) || undefined; - } else { - formConfig = record.jsonConfig; - } - return { - formConfig, - }; - }); - - return { records: parsedRecords }; - } - - return { records: [] }; -}; - -export { deleteFormTemplatesMarkedAsArchived }; diff --git a/aws/lambdas/code/form_archiver/package.json b/aws/lambdas/code/form_archiver/package.json index 6bede6afe..414f6abf0 100644 --- a/aws/lambdas/code/form_archiver/package.json +++ b/aws/lambdas/code/form_archiver/package.json @@ -11,13 +11,11 @@ "postbuild": "cp package.json dist/package.json && cp yarn.lock dist/yarn.lock && cd ./dist && yarn install --production" }, "dependencies": { - "@aws-sdk/client-rds-data": "3.430.0", - "pg": "^8.7.3" + "@aws-sdk/client-rds-data": "3.430.0" }, "devDependencies": { "@types/aws-lambda": "^8.10.126", "@types/node": "^20.9.0", - "@types/pg": "^8.10.9", "typescript": "^5.2.2" } } diff --git a/aws/lambdas/code/form_archiver/yarn.lock b/aws/lambdas/code/form_archiver/yarn.lock index d33037b39..6a4d2f51c 100644 --- a/aws/lambdas/code/form_archiver/yarn.lock +++ b/aws/lambdas/code/form_archiver/yarn.lock @@ -858,32 +858,18 @@ resolved "https://registry.yarnpkg.com/@types/aws-lambda/-/aws-lambda-8.10.128.tgz#1c26e3e062e33961b250355cb4c9b6b3544b74a9" integrity sha512-bw4+ORfyywsj9Tq3WJA9j6HG4EbYy+dz+k8LN73ApmwjTdf82ZCSELz3b7BJQzeG5HE8IaTy9SXTHmruIK5ZlQ== -"@types/node@*", "@types/node@^20.9.0": +"@types/node@^20.9.0": version "20.9.4" resolved "https://registry.yarnpkg.com/@types/node/-/node-20.9.4.tgz#cc8f970e869c26834bdb7ed480b30ede622d74c7" integrity sha512-wmyg8HUhcn6ACjsn8oKYjkN/zUzQeNtMy44weTJSM6p4MMzEOuKbA3OjJ267uPCOW7Xex9dyrNTful8XTQYoDA== dependencies: undici-types "~5.26.4" -"@types/pg@^8.10.9": - version "8.10.9" - resolved "https://registry.yarnpkg.com/@types/pg/-/pg-8.10.9.tgz#d20bb948c6268c5bd847e2bf968f1194c5a2355a" - integrity sha512-UksbANNE/f8w0wOMxVKKIrLCbEMV+oM1uKejmwXr39olg4xqcfBDbXxObJAt6XxHbDa4XTKOlUEcEltXDX+XLQ== - dependencies: - "@types/node" "*" - pg-protocol "*" - pg-types "^4.0.1" - bowser@^2.11.0: version "2.11.0" resolved "https://registry.yarnpkg.com/bowser/-/bowser-2.11.0.tgz#5ca3c35757a7aa5771500c70a73a9f91ef420a8f" integrity sha512-AlcaJBi/pqqJBIQ8U9Mcpc9i8Aqxn88Skv5d+xBX006BY5u8N3mGLHa5Lgppa7L/HfwgwLgZ6NYs+Ag6uUmJRA== -buffer-writer@2.0.0: - version "2.0.0" - resolved "https://registry.yarnpkg.com/buffer-writer/-/buffer-writer-2.0.0.tgz#ce7eb81a38f7829db09c873f2fbb792c0c98ec04" - integrity sha512-a7ZpuTZU1TRtnwyCNW3I5dc0wWNC3VR9S++Ewyk2HHZdrO3CQJqSpd+95Us590V6AL7JqUAH2IwZ/398PmNFgw== - fast-xml-parser@4.2.5: version "4.2.5" resolved "https://registry.yarnpkg.com/fast-xml-parser/-/fast-xml-parser-4.2.5.tgz#a6747a09296a6cb34f2ae634019bf1738f3b421f" @@ -891,146 +877,6 @@ fast-xml-parser@4.2.5: dependencies: strnum "^1.0.5" -obuf@~1.1.2: - version "1.1.2" - resolved "https://registry.yarnpkg.com/obuf/-/obuf-1.1.2.tgz#09bea3343d41859ebd446292d11c9d4db619084e" - integrity sha512-PX1wu0AmAdPqOL1mWhqmlOd8kOIZQwGZw6rh7uby9fTc5lhaOWFLX3I6R1hrF9k3zUY40e6igsLGkDXK92LJNg== - -packet-reader@1.0.0: - version "1.0.0" - resolved "https://registry.yarnpkg.com/packet-reader/-/packet-reader-1.0.0.tgz#9238e5480dedabacfe1fe3f2771063f164157d74" - integrity sha512-HAKu/fG3HpHFO0AA8WE8q2g+gBJaZ9MG7fcKk+IJPLTGAD6Psw4443l+9DGRbOIh3/aXr7Phy0TjilYivJo5XQ== - -pg-cloudflare@^1.1.1: - version "1.1.1" - resolved "https://registry.yarnpkg.com/pg-cloudflare/-/pg-cloudflare-1.1.1.tgz#e6d5833015b170e23ae819e8c5d7eaedb472ca98" - integrity sha512-xWPagP/4B6BgFO+EKz3JONXv3YDgvkbVrGw2mTo3D6tVDQRh1e7cqVGvyR3BE+eQgAvx1XhW/iEASj4/jCWl3Q== - -pg-connection-string@^2.6.2: - version "2.6.2" - resolved "https://registry.yarnpkg.com/pg-connection-string/-/pg-connection-string-2.6.2.tgz#713d82053de4e2bd166fab70cd4f26ad36aab475" - integrity sha512-ch6OwaeaPYcova4kKZ15sbJ2hKb/VP48ZD2gE7i1J+L4MspCtBMAx8nMgz7bksc7IojCIIWuEhHibSMFH8m8oA== - -pg-int8@1.0.1: - version "1.0.1" - resolved "https://registry.yarnpkg.com/pg-int8/-/pg-int8-1.0.1.tgz#943bd463bf5b71b4170115f80f8efc9a0c0eb78c" - integrity sha512-WCtabS6t3c8SkpDBUlb1kjOs7l66xsGdKpIPZsg4wR+B3+u9UAum2odSsF9tnvxg80h4ZxLWMy4pRjOsFIqQpw== - -pg-numeric@1.0.2: - version "1.0.2" - resolved "https://registry.yarnpkg.com/pg-numeric/-/pg-numeric-1.0.2.tgz#816d9a44026086ae8ae74839acd6a09b0636aa3a" - integrity sha512-BM/Thnrw5jm2kKLE5uJkXqqExRUY/toLHda65XgFTBTFYZyopbKjBe29Ii3RbkvlsMoFwD+tHeGaCjjv0gHlyw== - -pg-pool@^3.6.1: - version "3.6.1" - resolved "https://registry.yarnpkg.com/pg-pool/-/pg-pool-3.6.1.tgz#5a902eda79a8d7e3c928b77abf776b3cb7d351f7" - integrity sha512-jizsIzhkIitxCGfPRzJn1ZdcosIt3pz9Sh3V01fm1vZnbnCMgmGl5wvGGdNN2EL9Rmb0EcFoCkixH4Pu+sP9Og== - -pg-protocol@*, pg-protocol@^1.6.0: - version "1.6.0" - resolved "https://registry.yarnpkg.com/pg-protocol/-/pg-protocol-1.6.0.tgz#4c91613c0315349363af2084608db843502f8833" - integrity sha512-M+PDm637OY5WM307051+bsDia5Xej6d9IR4GwJse1qA1DIhiKlksvrneZOYQq42OM+spubpcNYEo2FcKQrDk+Q== - -pg-types@^2.1.0: - version "2.2.0" - resolved "https://registry.yarnpkg.com/pg-types/-/pg-types-2.2.0.tgz#2d0250d636454f7cfa3b6ae0382fdfa8063254a3" - integrity sha512-qTAAlrEsl8s4OiEQY69wDvcMIdQN6wdz5ojQiOy6YRMuynxenON0O5oCpJI6lshc6scgAY8qvJ2On/p+CXY0GA== - dependencies: - pg-int8 "1.0.1" - postgres-array "~2.0.0" - postgres-bytea "~1.0.0" - postgres-date "~1.0.4" - postgres-interval "^1.1.0" - -pg-types@^4.0.1: - version "4.0.1" - resolved "https://registry.yarnpkg.com/pg-types/-/pg-types-4.0.1.tgz#31857e89d00a6c66b06a14e907c3deec03889542" - integrity sha512-hRCSDuLII9/LE3smys1hRHcu5QGcLs9ggT7I/TCs0IE+2Eesxi9+9RWAAwZ0yaGjxoWICF/YHLOEjydGujoJ+g== - dependencies: - pg-int8 "1.0.1" - pg-numeric "1.0.2" - postgres-array "~3.0.1" - postgres-bytea "~3.0.0" - postgres-date "~2.0.1" - postgres-interval "^3.0.0" - postgres-range "^1.1.1" - -pg@^8.7.3: - version "8.11.3" - resolved "https://registry.yarnpkg.com/pg/-/pg-8.11.3.tgz#d7db6e3fe268fcedd65b8e4599cda0b8b4bf76cb" - integrity sha512-+9iuvG8QfaaUrrph+kpF24cXkH1YOOUeArRNYIxq1viYHZagBxrTno7cecY1Fa44tJeZvaoG+Djpkc3JwehN5g== - dependencies: - buffer-writer "2.0.0" - packet-reader "1.0.0" - pg-connection-string "^2.6.2" - pg-pool "^3.6.1" - pg-protocol "^1.6.0" - pg-types "^2.1.0" - pgpass "1.x" - optionalDependencies: - pg-cloudflare "^1.1.1" - -pgpass@1.x: - version "1.0.5" - resolved "https://registry.yarnpkg.com/pgpass/-/pgpass-1.0.5.tgz#9b873e4a564bb10fa7a7dbd55312728d422a223d" - integrity sha512-FdW9r/jQZhSeohs1Z3sI1yxFQNFvMcnmfuj4WBMUTxOrAyLMaTcE1aAMBiTlbMNaXvBCQuVi0R7hd8udDSP7ug== - dependencies: - split2 "^4.1.0" - -postgres-array@~2.0.0: - version "2.0.0" - resolved "https://registry.yarnpkg.com/postgres-array/-/postgres-array-2.0.0.tgz#48f8fce054fbc69671999329b8834b772652d82e" - integrity sha512-VpZrUqU5A69eQyW2c5CA1jtLecCsN2U/bD6VilrFDWq5+5UIEVO7nazS3TEcHf1zuPYO/sqGvUvW62g86RXZuA== - -postgres-array@~3.0.1: - version "3.0.2" - resolved "https://registry.yarnpkg.com/postgres-array/-/postgres-array-3.0.2.tgz#68d6182cb0f7f152a7e60dc6a6889ed74b0a5f98" - integrity sha512-6faShkdFugNQCLwucjPcY5ARoW1SlbnrZjmGl0IrrqewpvxvhSLHimCVzqeuULCbG0fQv7Dtk1yDbG3xv7Veog== - -postgres-bytea@~1.0.0: - version "1.0.0" - resolved "https://registry.yarnpkg.com/postgres-bytea/-/postgres-bytea-1.0.0.tgz#027b533c0aa890e26d172d47cf9ccecc521acd35" - integrity sha512-xy3pmLuQqRBZBXDULy7KbaitYqLcmxigw14Q5sj8QBVLqEwXfeybIKVWiqAXTlcvdvb0+xkOtDbfQMOf4lST1w== - -postgres-bytea@~3.0.0: - version "3.0.0" - resolved "https://registry.yarnpkg.com/postgres-bytea/-/postgres-bytea-3.0.0.tgz#9048dc461ac7ba70a6a42d109221619ecd1cb089" - integrity sha512-CNd4jim9RFPkObHSjVHlVrxoVQXz7quwNFpz7RY1okNNme49+sVyiTvTRobiLV548Hx/hb1BG+iE7h9493WzFw== - dependencies: - obuf "~1.1.2" - -postgres-date@~1.0.4: - version "1.0.7" - resolved "https://registry.yarnpkg.com/postgres-date/-/postgres-date-1.0.7.tgz#51bc086006005e5061c591cee727f2531bf641a8" - integrity sha512-suDmjLVQg78nMK2UZ454hAG+OAW+HQPZ6n++TNDUX+L0+uUlLywnoxJKDou51Zm+zTCjrCl0Nq6J9C5hP9vK/Q== - -postgres-date@~2.0.1: - version "2.0.1" - resolved "https://registry.yarnpkg.com/postgres-date/-/postgres-date-2.0.1.tgz#638b62e5c33764c292d37b08f5257ecb09231457" - integrity sha512-YtMKdsDt5Ojv1wQRvUhnyDJNSr2dGIC96mQVKz7xufp07nfuFONzdaowrMHjlAzY6GDLd4f+LUHHAAM1h4MdUw== - -postgres-interval@^1.1.0: - version "1.2.0" - resolved "https://registry.yarnpkg.com/postgres-interval/-/postgres-interval-1.2.0.tgz#b460c82cb1587507788819a06aa0fffdb3544695" - integrity sha512-9ZhXKM/rw350N1ovuWHbGxnGh/SNJ4cnxHiM0rxE4VN41wsg8P8zWn9hv/buK00RP4WvlOyr/RBDiptyxVbkZQ== - dependencies: - xtend "^4.0.0" - -postgres-interval@^3.0.0: - version "3.0.0" - resolved "https://registry.yarnpkg.com/postgres-interval/-/postgres-interval-3.0.0.tgz#baf7a8b3ebab19b7f38f07566c7aab0962f0c86a" - integrity sha512-BSNDnbyZCXSxgA+1f5UU2GmwhoI0aU5yMxRGO8CdFEcY2BQF9xm/7MqKnYoM1nJDk8nONNWDk9WeSmePFhQdlw== - -postgres-range@^1.1.1: - version "1.1.3" - resolved "https://registry.yarnpkg.com/postgres-range/-/postgres-range-1.1.3.tgz#9ccd7b01ca2789eb3c2e0888b3184225fa859f76" - integrity sha512-VdlZoocy5lCP0c/t66xAfclglEapXPCIVhqqJRncYpvbCgImF0w67aPKfbqUMr72tO2k5q0TdTZwCLjPTI6C9g== - -split2@^4.1.0: - version "4.2.0" - resolved "https://registry.yarnpkg.com/split2/-/split2-4.2.0.tgz#c9c5920904d148bab0b9f67145f245a86aadbfa4" - integrity sha512-UcjcJOWknrNkF6PLX83qcHM6KHgVKNkV62Y8a5uYDVv9ydGQVwAHMKqHdJje1VTWpljG0WYpCDhrCdAOYH4TWg== - strnum@^1.0.5: version "1.0.5" resolved "https://registry.yarnpkg.com/strnum/-/strnum-1.0.5.tgz#5c4e829fe15ad4ff0d20c3db5ac97b73c9b072db" @@ -1060,8 +906,3 @@ uuid@^8.3.2: version "8.3.2" resolved "https://registry.yarnpkg.com/uuid/-/uuid-8.3.2.tgz#80d5b5ced271bb9af6c445f21a1a04c606cefbe2" integrity sha512-+NYs2QeMWy+GWFOEm9xnn6HCDp0l7QBD7ml8zLUmJ+93Q5NF0NocErnwkTkXVFNiX3/fpC6afS8Dhb/gz7R7eg== - -xtend@^4.0.0: - version "4.0.2" - resolved "https://registry.yarnpkg.com/xtend/-/xtend-4.0.2.tgz#bb72779f5fa465186b1f438f674fa347fdb5db54" - integrity sha512-LKYU1iAXJXUgAXn9URjiu+MWhyUXHsvfp7mcuYm9dSUKK0/CjtrUwFAxD82/mCWbtLsGjFIad0wIsod4zrTAEQ== diff --git a/aws/lambdas/code/nagware/lib/templates.ts b/aws/lambdas/code/nagware/lib/templates.ts index 63d8dcf62..5c1cf8f7a 100644 --- a/aws/lambdas/code/nagware/lib/templates.ts +++ b/aws/lambdas/code/nagware/lib/templates.ts @@ -1,34 +1,33 @@ import { RDSDataClient, ExecuteStatementCommand } from "@aws-sdk/client-rds-data"; -import pg from "pg"; -export async function getTemplateInfo(formID: string) { +export type TemplateInfo = { + formName: string; + owners: { + name: string | undefined; + email: string; + }[]; + isPublished: boolean; +}; + +export async function getTemplateInfo(formID: string): Promise { try { - const { SQL, parameters } = createSQLString(formID); - const requestResult = process.env.LOCALSTACK === "true" ? requestSAM : requestRDS; - const result = await requestResult(SQL, parameters as any); + const rdsDataClient = new RDSDataClient({ region: process.env.REGION }); - if (result) { - return result; - } else { - throw new Error(`Could not find any template with form identifier: ${formID}.`); - } - } catch (error) { - throw new Error( - `Failed to retrieve template information. Reason: ${(error as Error).message}.` - ); - } -} + // Due to Localstack limitations we have to define aliases for fields that have the same name + const sqlStatement = ` + SELECT usr."name" AS user_name, usr."email", tem."name" AS template_name, tem."jsonConfig", tem."isPublished" + FROM "User" usr + JOIN "_TemplateToUser" ttu ON usr."id" = ttu."B" + JOIN "Template" tem ON tem."id" = ttu."A" + WHERE ttu."A" = :formID + `; -const createSQLString = (formID: string) => { - const selectSQL = ` - SELECT usr."name", usr."email", tem."name", tem."jsonConfig", tem."isPublished" - FROM "User" usr - JOIN "_TemplateToUser" ttu ON usr."id" = ttu."B" - JOIN "Template" tem ON tem."id" = ttu."A" - `; - if (!(process.env.LOCALSTACK === "true")) { - return { - SQL: `${selectSQL} WHERE ttu."A" = :formID`, + const executeStatementCommand = new ExecuteStatementCommand({ + database: process.env.DB_NAME, + resourceArn: process.env.DB_ARN, + secretArn: process.env.DB_SECRET, + sql: sqlStatement, + includeResultMetadata: false, // set to true if we want metadata like column names parameters: [ { name: "formID", @@ -37,71 +36,45 @@ const createSQLString = (formID: string) => { }, }, ], - }; - } else { - return { - SQL: `${selectSQL} WHERE ttu."A" = $1`, - parameters: [formID], - }; - } -}; + }); -const parseQueryResponse = (records: any[]) => { - if (records.length === 0) return null; + const response = await rdsDataClient.send(executeStatementCommand); - let formName = ""; + if (response.records && response.records.length > 0) { + const firstRecord = response.records[0]; - const firstRecord = records[0]; - let isPublished = false; + if ( + firstRecord[2].stringValue === undefined || // template name + firstRecord[3].stringValue === undefined || // template jsonConfig + firstRecord[4].booleanValue === undefined // template isPublished + ) { + throw new Error( + `Missing required parameters: template name = ${firstRecord[2].stringValue} ; template jsonConfig = ${firstRecord[3].stringValue} ; template isPublished = ${firstRecord[4].stringValue}.` + ); + } - if (!(process.env.LOCALSTACK === "true")) { - const jsonConfig = JSON.parse(firstRecord[3].stringValue.trim(1, -1)) || undefined; - formName = - firstRecord[2].stringValue !== "" - ? firstRecord[2].stringValue - : `${jsonConfig.titleEn} - ${jsonConfig.titleFr}`; - isPublished = firstRecord[4].booleanValue; - } else { - formName = - firstRecord.name !== "" - ? firstRecord.name - : `${firstRecord.jsonConfig.titleEn} - ${firstRecord.jsonConfig.titleFr}`; - isPublished = firstRecord.isPublished; - } + const jsonConfig = JSON.parse(firstRecord[3].stringValue.trim()); - const owners = records.map((record) => { - if (!(process.env.LOCALSTACK === "true")) { - return { name: record[0].stringValue, email: record[1].stringValue }; - } else { - return { name: record.name, email: record.email }; - } - }); + const formName = + firstRecord[2].stringValue !== "" + ? firstRecord[2].stringValue + : `${jsonConfig.titleEn} - ${jsonConfig.titleFr}`; - return { formName, owners, isPublished }; -}; + const isPublished = firstRecord[4].booleanValue; -/** - * Creates and processes request to LOCAL AWS SAM DB - * @param {string} SQL - * @param {string[]} parameters - * @returns PG Client return value - */ -const requestSAM = async (SQL: string, parameters: string[]) => { - // Placed outside of try block to be referenced in finally - const dbClient = new pg.Client(); - try { - if ( - process.env.PGHOST && - process.env.PGUSER && - process.env.PGDATABASE && - process.env.PGPASSWORD - ) { - dbClient.connect(); + const owners = response.records.map((record) => { + // make sure owner email is defined + if (record[1].stringValue === undefined) { + throw new Error(`Missing required parameters: owner email.`); + } + + return { name: record[0].stringValue, email: record[1].stringValue }; + }); + + return { formName, owners, isPublished }; } else { - throw new Error("Missing Environment Variables for DB config"); + throw new Error(`Could not find any template with form identifier: ${formID}.`); } - const data = await dbClient.query(SQL, parameters); - return parseQueryResponse(data.rows); } catch (error) { console.error( JSON.stringify({ @@ -109,45 +82,8 @@ const requestSAM = async (SQL: string, parameters: string[]) => { error: (error as Error).message, }) ); - // Lift more generic error to be able to capture event info higher in scope - throw new Error("Error connecting to LOCAL AWS SAM DB"); - } finally { - dbClient.end(); - } -}; - -/** - * Creates and processes request to RDS - * @param {string} SQL - * @param {{name: string, value: {stringValue: string}[]}} parameters - * @returns RDS client return value - */ -const requestRDS = async ( - SQL: string, - parameters: { name: string; value: { stringValue: string } }[] -) => { - try { - const dbClient = new RDSDataClient({ region: process.env.REGION }); - const params = { - database: process.env.DB_NAME, - resourceArn: process.env.DB_ARN, - secretArn: process.env.DB_SECRET, - sql: SQL, - includeResultMetadata: false, // set to true if we want metadata like column names - parameters: parameters, - }; - const command = new ExecuteStatementCommand(params); - - const data = await dbClient.send(command); - return parseQueryResponse(data.records ?? []); - } catch (error) { - console.error( - JSON.stringify({ - status: "error", - error: (error as Error).message, - }) + throw new Error( + `Failed to retrieve template information. Reason: ${(error as Error).message}.` ); - // Lift more generic error to be able to capture event info higher in scope - throw new Error("Error connecting to RDS"); } -}; +} diff --git a/aws/lambdas/code/nagware/nagware.ts b/aws/lambdas/code/nagware/nagware.ts index 1b0438e19..faed08e38 100644 --- a/aws/lambdas/code/nagware/nagware.ts +++ b/aws/lambdas/code/nagware/nagware.ts @@ -15,7 +15,7 @@ type NagwareDetection = { formTimestamp: number; formId: string; formName: string; - owners: { name: string; email: string; }[] + owners: { name?: string; email: string }[]; }; export const handler: Handler = async () => { @@ -24,7 +24,10 @@ export const handler: Handler = async () => { const isSunday = new Date().getDay() == 0; // 0 is Sunday - await nagOrDelete(oldestFormResponseByFormID, { shouldSendEmail: isSunday == false, shouldSendSlackNotification: isSunday }); + await nagOrDelete(oldestFormResponseByFormID, { + shouldSendEmail: isSunday == false, + shouldSendSlackNotification: isSunday, + }); return { statusCode: "SUCCESS", @@ -66,7 +69,10 @@ async function findOldestFormResponseByFormID() { return Object.values(reduceResult); } -async function nagOrDelete(oldestFormResponseByFormID: { formID: string; createdAt: number }[], notificationSettings: NotificationSettings) { +async function nagOrDelete( + oldestFormResponseByFormID: { formID: string; createdAt: number }[], + notificationSettings: NotificationSettings +) { for (const formResponse of oldestFormResponseByFormID) { try { const templateInfo = await getTemplateInfo(formResponse.formID); @@ -78,12 +84,15 @@ async function nagOrDelete(oldestFormResponseByFormID: { formID: string; created } } - logNagwareDetection({ - formTimestamp: formResponse.createdAt, - formId: formResponse.formID, - formName: templateInfo.formName, - owners: templateInfo.owners, - }, notificationSettings.shouldSendSlackNotification); + logNagwareDetection( + { + formTimestamp: formResponse.createdAt, + formId: formResponse.formID, + formName: templateInfo.formName, + owners: templateInfo.owners, + }, + notificationSettings.shouldSendSlackNotification + ); } else { // Delete form response if form is not published and older than 28 days await deleteOldTestResponses(formResponse.formID); @@ -102,7 +111,10 @@ async function nagOrDelete(oldestFormResponseByFormID: { formID: string; created } } -function logNagwareDetection(nagwareDetection: NagwareDetection, shouldSendSlackNotification: boolean) { +function logNagwareDetection( + nagwareDetection: NagwareDetection, + shouldSendSlackNotification: boolean +) { const diffMs = Math.abs(Date.now() - nagwareDetection.formTimestamp); const diffDays = Math.ceil(diffMs / (1000 * 60 * 60 * 24)); @@ -122,4 +134,4 @@ ${diffDays} days since submission }) ); } -} \ No newline at end of file +} diff --git a/aws/lambdas/code/nagware/package.json b/aws/lambdas/code/nagware/package.json index e42fa08ef..ec3ba4352 100644 --- a/aws/lambdas/code/nagware/package.json +++ b/aws/lambdas/code/nagware/package.json @@ -15,13 +15,11 @@ "@aws-sdk/client-rds-data": "3.430.0", "@aws-sdk/client-secrets-manager": "^3.478.0", "axios": "^1.6.2", - "notifications-node-client": "5.2.3", - "pg": "^8.7.3" + "notifications-node-client": "5.2.3" }, "devDependencies": { "@types/aws-lambda": "^8.10.128", "@types/node": "^20.9.4", - "@types/pg": "^8.10.9", "typescript": "^5.3.2" } } diff --git a/aws/lambdas/code/nagware/yarn.lock b/aws/lambdas/code/nagware/yarn.lock index 872a3073c..4cd96f74d 100644 --- a/aws/lambdas/code/nagware/yarn.lock +++ b/aws/lambdas/code/nagware/yarn.lock @@ -1549,22 +1549,13 @@ resolved "https://registry.yarnpkg.com/@types/aws-lambda/-/aws-lambda-8.10.128.tgz#1c26e3e062e33961b250355cb4c9b6b3544b74a9" integrity sha512-bw4+ORfyywsj9Tq3WJA9j6HG4EbYy+dz+k8LN73ApmwjTdf82ZCSELz3b7BJQzeG5HE8IaTy9SXTHmruIK5ZlQ== -"@types/node@*", "@types/node@^20.9.4": +"@types/node@^20.9.4": version "20.9.4" resolved "https://registry.yarnpkg.com/@types/node/-/node-20.9.4.tgz#cc8f970e869c26834bdb7ed480b30ede622d74c7" integrity sha512-wmyg8HUhcn6ACjsn8oKYjkN/zUzQeNtMy44weTJSM6p4MMzEOuKbA3OjJ267uPCOW7Xex9dyrNTful8XTQYoDA== dependencies: undici-types "~5.26.4" -"@types/pg@^8.10.9": - version "8.10.9" - resolved "https://registry.yarnpkg.com/@types/pg/-/pg-8.10.9.tgz#d20bb948c6268c5bd847e2bf968f1194c5a2355a" - integrity sha512-UksbANNE/f8w0wOMxVKKIrLCbEMV+oM1uKejmwXr39olg4xqcfBDbXxObJAt6XxHbDa4XTKOlUEcEltXDX+XLQ== - dependencies: - "@types/node" "*" - pg-protocol "*" - pg-types "^4.0.1" - asynckit@^0.4.0: version "0.4.0" resolved "https://registry.yarnpkg.com/asynckit/-/asynckit-0.4.0.tgz#c79ed97f7f34cb8f2ba1bc9790bcc366474b4b79" @@ -1596,11 +1587,6 @@ buffer-equal-constant-time@1.0.1: resolved "https://registry.yarnpkg.com/buffer-equal-constant-time/-/buffer-equal-constant-time-1.0.1.tgz#f8e71132f7ffe6e01a5c9697a4c6f3e48d5cc819" integrity sha512-zRpUiDwd/xk6ADqPMATG8vc9VPrkck7T07OIx0gnjmJAnHnTVXNQG3vfvWNuiZIkwu9KrKdA1iJKfsfTVxE6NA== -buffer-writer@2.0.0: - version "2.0.0" - resolved "https://registry.yarnpkg.com/buffer-writer/-/buffer-writer-2.0.0.tgz#ce7eb81a38f7829db09c873f2fbb792c0c98ec04" - integrity sha512-a7ZpuTZU1TRtnwyCNW3I5dc0wWNC3VR9S++Ewyk2HHZdrO3CQJqSpd+95Us590V6AL7JqUAH2IwZ/398PmNFgw== - combined-stream@^1.0.8: version "1.0.8" resolved "https://registry.yarnpkg.com/combined-stream/-/combined-stream-1.0.8.tgz#c3d45a8b34fd730631a110a8a2520682b31d5a7f" @@ -1751,141 +1737,6 @@ obliterator@^1.6.1: resolved "https://registry.yarnpkg.com/obliterator/-/obliterator-1.6.1.tgz#dea03e8ab821f6c4d96a299e17aef6a3af994ef3" integrity sha512-9WXswnqINnnhOG/5SLimUlzuU1hFJUc8zkwyD59Sd+dPOMf05PmnYG/d6Q7HZ+KmgkZJa1PxRso6QdM3sTNHig== -obuf@~1.1.2: - version "1.1.2" - resolved "https://registry.yarnpkg.com/obuf/-/obuf-1.1.2.tgz#09bea3343d41859ebd446292d11c9d4db619084e" - integrity sha512-PX1wu0AmAdPqOL1mWhqmlOd8kOIZQwGZw6rh7uby9fTc5lhaOWFLX3I6R1hrF9k3zUY40e6igsLGkDXK92LJNg== - -packet-reader@1.0.0: - version "1.0.0" - resolved "https://registry.yarnpkg.com/packet-reader/-/packet-reader-1.0.0.tgz#9238e5480dedabacfe1fe3f2771063f164157d74" - integrity sha512-HAKu/fG3HpHFO0AA8WE8q2g+gBJaZ9MG7fcKk+IJPLTGAD6Psw4443l+9DGRbOIh3/aXr7Phy0TjilYivJo5XQ== - -pg-cloudflare@^1.1.1: - version "1.1.1" - resolved "https://registry.yarnpkg.com/pg-cloudflare/-/pg-cloudflare-1.1.1.tgz#e6d5833015b170e23ae819e8c5d7eaedb472ca98" - integrity sha512-xWPagP/4B6BgFO+EKz3JONXv3YDgvkbVrGw2mTo3D6tVDQRh1e7cqVGvyR3BE+eQgAvx1XhW/iEASj4/jCWl3Q== - -pg-connection-string@^2.6.2: - version "2.6.2" - resolved "https://registry.yarnpkg.com/pg-connection-string/-/pg-connection-string-2.6.2.tgz#713d82053de4e2bd166fab70cd4f26ad36aab475" - integrity sha512-ch6OwaeaPYcova4kKZ15sbJ2hKb/VP48ZD2gE7i1J+L4MspCtBMAx8nMgz7bksc7IojCIIWuEhHibSMFH8m8oA== - -pg-int8@1.0.1: - version "1.0.1" - resolved "https://registry.yarnpkg.com/pg-int8/-/pg-int8-1.0.1.tgz#943bd463bf5b71b4170115f80f8efc9a0c0eb78c" - integrity sha512-WCtabS6t3c8SkpDBUlb1kjOs7l66xsGdKpIPZsg4wR+B3+u9UAum2odSsF9tnvxg80h4ZxLWMy4pRjOsFIqQpw== - -pg-numeric@1.0.2: - version "1.0.2" - resolved "https://registry.yarnpkg.com/pg-numeric/-/pg-numeric-1.0.2.tgz#816d9a44026086ae8ae74839acd6a09b0636aa3a" - integrity sha512-BM/Thnrw5jm2kKLE5uJkXqqExRUY/toLHda65XgFTBTFYZyopbKjBe29Ii3RbkvlsMoFwD+tHeGaCjjv0gHlyw== - -pg-pool@^3.6.1: - version "3.6.1" - resolved "https://registry.yarnpkg.com/pg-pool/-/pg-pool-3.6.1.tgz#5a902eda79a8d7e3c928b77abf776b3cb7d351f7" - integrity sha512-jizsIzhkIitxCGfPRzJn1ZdcosIt3pz9Sh3V01fm1vZnbnCMgmGl5wvGGdNN2EL9Rmb0EcFoCkixH4Pu+sP9Og== - -pg-protocol@*, pg-protocol@^1.6.0: - version "1.6.0" - resolved "https://registry.yarnpkg.com/pg-protocol/-/pg-protocol-1.6.0.tgz#4c91613c0315349363af2084608db843502f8833" - integrity sha512-M+PDm637OY5WM307051+bsDia5Xej6d9IR4GwJse1qA1DIhiKlksvrneZOYQq42OM+spubpcNYEo2FcKQrDk+Q== - -pg-types@^2.1.0: - version "2.2.0" - resolved "https://registry.yarnpkg.com/pg-types/-/pg-types-2.2.0.tgz#2d0250d636454f7cfa3b6ae0382fdfa8063254a3" - integrity sha512-qTAAlrEsl8s4OiEQY69wDvcMIdQN6wdz5ojQiOy6YRMuynxenON0O5oCpJI6lshc6scgAY8qvJ2On/p+CXY0GA== - dependencies: - pg-int8 "1.0.1" - postgres-array "~2.0.0" - postgres-bytea "~1.0.0" - postgres-date "~1.0.4" - postgres-interval "^1.1.0" - -pg-types@^4.0.1: - version "4.0.1" - resolved "https://registry.yarnpkg.com/pg-types/-/pg-types-4.0.1.tgz#31857e89d00a6c66b06a14e907c3deec03889542" - integrity sha512-hRCSDuLII9/LE3smys1hRHcu5QGcLs9ggT7I/TCs0IE+2Eesxi9+9RWAAwZ0yaGjxoWICF/YHLOEjydGujoJ+g== - dependencies: - pg-int8 "1.0.1" - pg-numeric "1.0.2" - postgres-array "~3.0.1" - postgres-bytea "~3.0.0" - postgres-date "~2.0.1" - postgres-interval "^3.0.0" - postgres-range "^1.1.1" - -pg@^8.7.3: - version "8.11.3" - resolved "https://registry.yarnpkg.com/pg/-/pg-8.11.3.tgz#d7db6e3fe268fcedd65b8e4599cda0b8b4bf76cb" - integrity sha512-+9iuvG8QfaaUrrph+kpF24cXkH1YOOUeArRNYIxq1viYHZagBxrTno7cecY1Fa44tJeZvaoG+Djpkc3JwehN5g== - dependencies: - buffer-writer "2.0.0" - packet-reader "1.0.0" - pg-connection-string "^2.6.2" - pg-pool "^3.6.1" - pg-protocol "^1.6.0" - pg-types "^2.1.0" - pgpass "1.x" - optionalDependencies: - pg-cloudflare "^1.1.1" - -pgpass@1.x: - version "1.0.5" - resolved "https://registry.yarnpkg.com/pgpass/-/pgpass-1.0.5.tgz#9b873e4a564bb10fa7a7dbd55312728d422a223d" - integrity sha512-FdW9r/jQZhSeohs1Z3sI1yxFQNFvMcnmfuj4WBMUTxOrAyLMaTcE1aAMBiTlbMNaXvBCQuVi0R7hd8udDSP7ug== - dependencies: - split2 "^4.1.0" - -postgres-array@~2.0.0: - version "2.0.0" - resolved "https://registry.yarnpkg.com/postgres-array/-/postgres-array-2.0.0.tgz#48f8fce054fbc69671999329b8834b772652d82e" - integrity sha512-VpZrUqU5A69eQyW2c5CA1jtLecCsN2U/bD6VilrFDWq5+5UIEVO7nazS3TEcHf1zuPYO/sqGvUvW62g86RXZuA== - -postgres-array@~3.0.1: - version "3.0.2" - resolved "https://registry.yarnpkg.com/postgres-array/-/postgres-array-3.0.2.tgz#68d6182cb0f7f152a7e60dc6a6889ed74b0a5f98" - integrity sha512-6faShkdFugNQCLwucjPcY5ARoW1SlbnrZjmGl0IrrqewpvxvhSLHimCVzqeuULCbG0fQv7Dtk1yDbG3xv7Veog== - -postgres-bytea@~1.0.0: - version "1.0.0" - resolved "https://registry.yarnpkg.com/postgres-bytea/-/postgres-bytea-1.0.0.tgz#027b533c0aa890e26d172d47cf9ccecc521acd35" - integrity sha512-xy3pmLuQqRBZBXDULy7KbaitYqLcmxigw14Q5sj8QBVLqEwXfeybIKVWiqAXTlcvdvb0+xkOtDbfQMOf4lST1w== - -postgres-bytea@~3.0.0: - version "3.0.0" - resolved "https://registry.yarnpkg.com/postgres-bytea/-/postgres-bytea-3.0.0.tgz#9048dc461ac7ba70a6a42d109221619ecd1cb089" - integrity sha512-CNd4jim9RFPkObHSjVHlVrxoVQXz7quwNFpz7RY1okNNme49+sVyiTvTRobiLV548Hx/hb1BG+iE7h9493WzFw== - dependencies: - obuf "~1.1.2" - -postgres-date@~1.0.4: - version "1.0.7" - resolved "https://registry.yarnpkg.com/postgres-date/-/postgres-date-1.0.7.tgz#51bc086006005e5061c591cee727f2531bf641a8" - integrity sha512-suDmjLVQg78nMK2UZ454hAG+OAW+HQPZ6n++TNDUX+L0+uUlLywnoxJKDou51Zm+zTCjrCl0Nq6J9C5hP9vK/Q== - -postgres-date@~2.0.1: - version "2.0.1" - resolved "https://registry.yarnpkg.com/postgres-date/-/postgres-date-2.0.1.tgz#638b62e5c33764c292d37b08f5257ecb09231457" - integrity sha512-YtMKdsDt5Ojv1wQRvUhnyDJNSr2dGIC96mQVKz7xufp07nfuFONzdaowrMHjlAzY6GDLd4f+LUHHAAM1h4MdUw== - -postgres-interval@^1.1.0: - version "1.2.0" - resolved "https://registry.yarnpkg.com/postgres-interval/-/postgres-interval-1.2.0.tgz#b460c82cb1587507788819a06aa0fffdb3544695" - integrity sha512-9ZhXKM/rw350N1ovuWHbGxnGh/SNJ4cnxHiM0rxE4VN41wsg8P8zWn9hv/buK00RP4WvlOyr/RBDiptyxVbkZQ== - dependencies: - xtend "^4.0.0" - -postgres-interval@^3.0.0: - version "3.0.0" - resolved "https://registry.yarnpkg.com/postgres-interval/-/postgres-interval-3.0.0.tgz#baf7a8b3ebab19b7f38f07566c7aab0962f0c86a" - integrity sha512-BSNDnbyZCXSxgA+1f5UU2GmwhoI0aU5yMxRGO8CdFEcY2BQF9xm/7MqKnYoM1nJDk8nONNWDk9WeSmePFhQdlw== - -postgres-range@^1.1.1: - version "1.1.3" - resolved "https://registry.yarnpkg.com/postgres-range/-/postgres-range-1.1.3.tgz#9ccd7b01ca2789eb3c2e0888b3184225fa859f76" - integrity sha512-VdlZoocy5lCP0c/t66xAfclglEapXPCIVhqqJRncYpvbCgImF0w67aPKfbqUMr72tO2k5q0TdTZwCLjPTI6C9g== - proxy-from-env@^1.1.0: version "1.1.0" resolved "https://registry.yarnpkg.com/proxy-from-env/-/proxy-from-env-1.1.0.tgz#e102f16ca355424865755d2c9e8ea4f24d58c3e2" @@ -1901,11 +1752,6 @@ semver@^5.6.0: resolved "https://registry.yarnpkg.com/semver/-/semver-5.7.2.tgz#48d55db737c3287cd4835e17fa13feace1c41ef8" integrity sha512-cBznnQ9KjJqU67B52RMC65CMarK2600WFnbkcaiwWq3xy/5haFJlshgnpjovMVJ+Hff49d8GEn0b87C5pDQ10g== -split2@^4.1.0: - version "4.2.0" - resolved "https://registry.yarnpkg.com/split2/-/split2-4.2.0.tgz#c9c5920904d148bab0b9f67145f245a86aadbfa4" - integrity sha512-UcjcJOWknrNkF6PLX83qcHM6KHgVKNkV62Y8a5uYDVv9ydGQVwAHMKqHdJje1VTWpljG0WYpCDhrCdAOYH4TWg== - strnum@^1.0.5: version "1.0.5" resolved "https://registry.yarnpkg.com/strnum/-/strnum-1.0.5.tgz#5c4e829fe15ad4ff0d20c3db5ac97b73c9b072db" @@ -1935,8 +1781,3 @@ uuid@^8.3.2: version "8.3.2" resolved "https://registry.yarnpkg.com/uuid/-/uuid-8.3.2.tgz#80d5b5ced271bb9af6c445f21a1a04c606cefbe2" integrity sha512-+NYs2QeMWy+GWFOEm9xnn6HCDp0l7QBD7ml8zLUmJ+93Q5NF0NocErnwkTkXVFNiX3/fpC6afS8Dhb/gz7R7eg== - -xtend@^4.0.0: - version "4.0.2" - resolved "https://registry.yarnpkg.com/xtend/-/xtend-4.0.2.tgz#bb72779f5fa465186b1f438f674fa347fdb5db54" - integrity sha512-LKYU1iAXJXUgAXn9URjiu+MWhyUXHsvfp7mcuYm9dSUKK0/CjtrUwFAxD82/mCWbtLsGjFIad0wIsod4zrTAEQ== diff --git a/aws/lambdas/code/reliability/lib/s3FileInput.ts b/aws/lambdas/code/reliability/lib/s3FileInput.ts index a2d35f135..bc666d391 100644 --- a/aws/lambdas/code/reliability/lib/s3FileInput.ts +++ b/aws/lambdas/code/reliability/lib/s3FileInput.ts @@ -18,8 +18,7 @@ const s3Client = new S3Client({ forcePathStyle: true, }) as NodeJsClient; -const environment = - process.env.ENVIRONMENT || (process.env.LOCALSTACK === "true" ? "local" : "staging"); +const environment = process.env.ENVIRONMENT; const reliabilityBucketName = `forms-${environment}-reliability-file-storage`; const vaultBucketName = `forms-${environment}-vault-file-storage`; diff --git a/aws/lambdas/code/reliability/lib/templates.ts b/aws/lambdas/code/reliability/lib/templates.ts index fb7b0a026..96c9081b1 100644 --- a/aws/lambdas/code/reliability/lib/templates.ts +++ b/aws/lambdas/code/reliability/lib/templates.ts @@ -1,147 +1,32 @@ import { RDSDataClient, ExecuteStatementCommand } from "@aws-sdk/client-rds-data"; -import pg from "pg"; +import { FormProperties } from "./types.js"; -const REGION = process.env.REGION; - -/** - * Get's the Form property on the Form Configuration - * @param {string} formID - * @returns Form property of Form Configuration including Delivery option - */ -export const getTemplateFormConfig = async (formID: string) => { - try { - // Return early if require params not provided - if (formID === null || typeof formID === "undefined") { - console.warn( - JSON.stringify({ - msg: "Can not retrieve template form config because no form ID was provided", - }) - ); - return null; - } - - const { SQL, parameters } = createSQLString(formID); - - const getTemplateData = process.env.LOCALSTACK === "true" ? requestSAM : requestRDS; - - // I know it's cheating to use 'any' but we'll refactor later - const data = await getTemplateData(SQL, parameters as any); - - if (data.records.length === 1) { - return { ...data.records[0] }; - } else { - return null; - } - } catch (error) { - console.warn( - JSON.stringify({ - level: "warn", - msg: "Failed to retrieve template form config from DB", - error: (error as Error).message, - }) - ); - // Log full error to console, it will not be sent to Slack - console.warn( - JSON.stringify({ - msg: `Failed to retrieve template form config because of following error: ${ - (error as Error).message - }`, - }) - ); - // Return as if no template with ID was found. - // Handle error in calling function if template is not found. - return null; - } +export type TemplateInfo = { + formConfig: FormProperties; + deliveryOption?: { + emailAddress: string; + emailSubjectEn: string | undefined; + emailSubjectFr: string | undefined; + }; }; -/** - * Creates and processes request to LOCAL AWS SAM DB - * @param {string} SQL - * @param {string[]} parameters - * @returns PG Client return value - */ -const requestSAM = async (SQL: string, parameters: string[]) => { - // Placed outside of try block to be referenced in finally - const dbClient = new pg.Client(); +export async function getTemplateInfo(formID: string): Promise { try { - if ( - process.env.PGHOST && - process.env.PGUSER && - process.env.PGDATABASE && - process.env.PGPASSWORD - ) { - dbClient.connect(); - } else { - throw new Error("Missing Environment Variables for DB config"); - } - const data = await dbClient.query(SQL, parameters); - return parseConfig(data.rows); - } catch (error) { - console.error( - JSON.stringify({ - level: "error", - status: "error", - msg: "Error issuing command to Local SAM AWS DB", - error: (error as Error).message, - }) - ); - // Lift more generic error to be able to capture event info higher in scope - throw new Error("Error connecting to LOCAL AWS SAM DB"); - } finally { - dbClient.end(); - } -}; + const rdsDataClient = new RDSDataClient({ region: process.env.REGION }); -/** - * Creates and processes request to RDS - * @param {string} SQL - * @param {{name: string, value: {stringValue: string}[]}} parameters - * @returns RDS client return value - */ -const requestRDS = async ( - SQL: string, - parameters: { name: string; value: { stringValue: string } }[] -) => { - try { - const dbClient = new RDSDataClient({ region: REGION }); - const params = { + const sqlStatement = ` + SELECT t."jsonConfig", deli."emailAddress", deli."emailSubjectEn", deli."emailSubjectFr" + FROM "Template" t + LEFT JOIN "DeliveryOption" deli ON t.id = deli."templateId" + WHERE t.id = :formID + `; + + const executeStatementCommand = new ExecuteStatementCommand({ database: process.env.DB_NAME, resourceArn: process.env.DB_ARN, secretArn: process.env.DB_SECRET, - sql: SQL, + sql: sqlStatement, includeResultMetadata: false, // set to true if we want metadata like column names - parameters: parameters, - }; - const command = new ExecuteStatementCommand(params); - - const data = await dbClient.send(command); - return parseConfig(data.records); - } catch (error) { - console.error( - JSON.stringify({ - level: "error", - severity: 2, - status: "error", - msg: "Error issuing command to AWS RDS", - error: (error as Error).message, - }) - ); - // Lift more generic error to be able to capture event info higher in scope - throw new Error("Error connecting to RDS"); - } -}; - -/** - * Creates correct SQL string and params depending on environment - * @param {string} formID - */ -const createSQLString = (formID: string) => { - const selectSQL = `SELECT t."jsonConfig", deli."emailAddress", deli."emailSubjectEn", deli."emailSubjectFr" - FROM "Template" t - LEFT JOIN "DeliveryOption" deli ON t.id = deli."templateId"`; - if (!(process.env.LOCALSTACK === "true")) { - return { - SQL: `${selectSQL} WHERE t.id = :formID`, parameters: [ { name: "formID", @@ -150,47 +35,46 @@ const createSQLString = (formID: string) => { }, }, ], - }; - } else { - return { - SQL: `${selectSQL} WHERE t.id = $1`, - parameters: [formID], - }; - } -}; + }); -const parseConfig = (records: any[] | undefined) => { - if (records) { - const parsedRecords = records.map((record) => { - let formConfig; - let deliveryOption; - if (!(process.env.LOCALSTACK === "true")) { - formConfig = JSON.parse(record[0].stringValue.trim(1, -1)) || undefined; - deliveryOption = record[1].stringValue - ? { - emailAddress: record[1].stringValue, - emailSubjectEn: record[2].stringValue, - emailSubjectFr: record[3].stringValue, - } - : null; - } else { - formConfig = record.jsonConfig; - deliveryOption = record.emailAddress - ? { - emailAddress: record.emailAddress, - emailSubjectEn: record.emailSubjectEn, - emailSubjectFr: record.emailSubjectFr, - } - : null; + const response = await rdsDataClient.send(executeStatementCommand); + + if (response.records && response.records.length === 1) { + const firstRecord = response.records[0]; + + // make sure template jsonConfig is defined + if (firstRecord[0].stringValue === undefined) { + throw new Error(`Missing required parameters: template jsonConfig.`); } + const formConfig = JSON.parse(firstRecord[0].stringValue!.trim()) as FormProperties; + + const deliveryOption = firstRecord[1].stringValue + ? { + emailAddress: firstRecord[1].stringValue, + emailSubjectEn: firstRecord[2].stringValue, + emailSubjectFr: firstRecord[3].stringValue, + } + : null; + return { formConfig, - deliveryOption, + ...(deliveryOption && { deliveryOption }), }; - }); - return { records: parsedRecords }; - } + } else { + return null; + } + } catch (error) { + console.warn( + JSON.stringify({ + level: "warn", + msg: "Failed to retrieve template form config from DB", + error: (error as Error).message, + }) + ); - return { records: [] }; -}; + // Return as if no template with ID was found. + // Handle error in calling function if template is not found. + return null; + } +} diff --git a/aws/lambdas/code/reliability/package.json b/aws/lambdas/code/reliability/package.json index 2efcd9f6d..4e2097ee5 100644 --- a/aws/lambdas/code/reliability/package.json +++ b/aws/lambdas/code/reliability/package.json @@ -21,14 +21,12 @@ "axios": "^1.0.0", "json2md": "^1.10.0", "notifications-node-client": "^5.1.0", - "pg": "^8.7.3", "uuid": "^8.3.2" }, "devDependencies": { "@types/aws-lambda": "^8.10.126", "@types/json2md": "^1.5.4", "@types/node": "^20.9.0", - "@types/pg": "^8.10.9", "@types/uuid": "^9.0.7", "typescript": "^5.2.2" } diff --git a/aws/lambdas/code/reliability/reliability.ts b/aws/lambdas/code/reliability/reliability.ts index e64e25fae..81899fe6f 100644 --- a/aws/lambdas/code/reliability/reliability.ts +++ b/aws/lambdas/code/reliability/reliability.ts @@ -1,7 +1,7 @@ import { Handler, SQSEvent } from "aws-lambda"; import sendToNotify from "./lib/notifyProcessing.js"; import sendToVault from "./lib/vaultProcessing.js"; -import { getTemplateFormConfig } from "./lib/templates.js"; +import { getTemplateInfo } from "./lib/templates.js"; import { getSubmission } from "./lib/dataLayer.js"; export const handler: Handler = async (event: SQSEvent) => { @@ -13,7 +13,7 @@ export const handler: Handler = async (event: SQSEvent) => { const messageData = await getSubmission(message); const submissionID = messageData.Item?.SubmissionID ?? message.submissionID; const formID = - // dynamodb client could possibl return as a number due to early form identifiers being numeric + // dynamodb client could possibly return as a number due to early form identifiers being numeric typeof messageData.Item?.FormID === "string" ? messageData.Item?.FormID : messageData.Item?.FormID.toString() ?? null; @@ -43,13 +43,24 @@ export const handler: Handler = async (event: SQSEvent) => { return { status: true }; } - const configs = await getTemplateFormConfig(formID); + if (formID === null || typeof formID === "undefined") { + console.error( + JSON.stringify({ + level: "error", + severity: 2, + msg: `Can't process submission because of null or undefined formID.`, + }) + ); + throw new Error(`Required formID parameter is null or undefined.`); + } + + const templateInfo = await getTemplateInfo(formID); - if (configs !== null && configs.formConfig !== null) { + if (templateInfo !== null && templateInfo.formConfig !== null) { // Add form config back to submission to be processed - formSubmission.form = configs.formConfig; + formSubmission.form = templateInfo.formConfig; // add delivery option to formsubmission - formSubmission.deliveryOption = configs.deliveryOption; + formSubmission.deliveryOption = templateInfo.deliveryOption; } else { console.error( JSON.stringify({ diff --git a/aws/lambdas/code/reliability/yarn.lock b/aws/lambdas/code/reliability/yarn.lock index e33893d84..f4bbed826 100644 --- a/aws/lambdas/code/reliability/yarn.lock +++ b/aws/lambdas/code/reliability/yarn.lock @@ -1927,22 +1927,13 @@ resolved "https://registry.yarnpkg.com/@types/json2md/-/json2md-1.5.4.tgz#c418a77e40155451f4a3994bb5e4a1b93972e8d0" integrity sha512-OFTAYD7Nnyu7FZPGnDwYbGOTKqzDfX71uFSgTbGhcr0aHCi17QkSOY3wO5H4yv5h23Ly+7suvf2lzHcdeDIH2Q== -"@types/node@*", "@types/node@^20.9.0": +"@types/node@^20.9.0": version "20.9.0" resolved "https://registry.yarnpkg.com/@types/node/-/node-20.9.0.tgz#bfcdc230583aeb891cf51e73cfdaacdd8deae298" integrity sha512-nekiGu2NDb1BcVofVcEKMIwzlx4NjHlcjhoxxKBNLtz15Y1z7MYf549DFvkHSId02Ax6kGwWntIBPC3l/JZcmw== dependencies: undici-types "~5.26.4" -"@types/pg@^8.10.9": - version "8.10.9" - resolved "https://registry.yarnpkg.com/@types/pg/-/pg-8.10.9.tgz#d20bb948c6268c5bd847e2bf968f1194c5a2355a" - integrity sha512-UksbANNE/f8w0wOMxVKKIrLCbEMV+oM1uKejmwXr39olg4xqcfBDbXxObJAt6XxHbDa4XTKOlUEcEltXDX+XLQ== - dependencies: - "@types/node" "*" - pg-protocol "*" - pg-types "^4.0.1" - "@types/uuid@^9.0.7": version "9.0.7" resolved "https://registry.yarnpkg.com/@types/uuid/-/uuid-9.0.7.tgz#b14cebc75455eeeb160d5fe23c2fcc0c64f724d8" @@ -1979,11 +1970,6 @@ buffer-equal-constant-time@1.0.1: resolved "https://registry.yarnpkg.com/buffer-equal-constant-time/-/buffer-equal-constant-time-1.0.1.tgz#f8e71132f7ffe6e01a5c9697a4c6f3e48d5cc819" integrity sha512-zRpUiDwd/xk6ADqPMATG8vc9VPrkck7T07OIx0gnjmJAnHnTVXNQG3vfvWNuiZIkwu9KrKdA1iJKfsfTVxE6NA== -buffer-writer@2.0.0: - version "2.0.0" - resolved "https://registry.yarnpkg.com/buffer-writer/-/buffer-writer-2.0.0.tgz#ce7eb81a38f7829db09c873f2fbb792c0c98ec04" - integrity sha512-a7ZpuTZU1TRtnwyCNW3I5dc0wWNC3VR9S++Ewyk2HHZdrO3CQJqSpd+95Us590V6AL7JqUAH2IwZ/398PmNFgw== - combined-stream@^1.0.8: version "1.0.8" resolved "https://registry.yarnpkg.com/combined-stream/-/combined-stream-1.0.8.tgz#c3d45a8b34fd730631a110a8a2520682b31d5a7f" @@ -2146,141 +2132,6 @@ obliterator@^1.6.1: resolved "https://registry.yarnpkg.com/obliterator/-/obliterator-1.6.1.tgz#dea03e8ab821f6c4d96a299e17aef6a3af994ef3" integrity sha512-9WXswnqINnnhOG/5SLimUlzuU1hFJUc8zkwyD59Sd+dPOMf05PmnYG/d6Q7HZ+KmgkZJa1PxRso6QdM3sTNHig== -obuf@~1.1.2: - version "1.1.2" - resolved "https://registry.yarnpkg.com/obuf/-/obuf-1.1.2.tgz#09bea3343d41859ebd446292d11c9d4db619084e" - integrity sha512-PX1wu0AmAdPqOL1mWhqmlOd8kOIZQwGZw6rh7uby9fTc5lhaOWFLX3I6R1hrF9k3zUY40e6igsLGkDXK92LJNg== - -packet-reader@1.0.0: - version "1.0.0" - resolved "https://registry.yarnpkg.com/packet-reader/-/packet-reader-1.0.0.tgz#9238e5480dedabacfe1fe3f2771063f164157d74" - integrity sha512-HAKu/fG3HpHFO0AA8WE8q2g+gBJaZ9MG7fcKk+IJPLTGAD6Psw4443l+9DGRbOIh3/aXr7Phy0TjilYivJo5XQ== - -pg-cloudflare@^1.1.1: - version "1.1.1" - resolved "https://registry.yarnpkg.com/pg-cloudflare/-/pg-cloudflare-1.1.1.tgz#e6d5833015b170e23ae819e8c5d7eaedb472ca98" - integrity sha512-xWPagP/4B6BgFO+EKz3JONXv3YDgvkbVrGw2mTo3D6tVDQRh1e7cqVGvyR3BE+eQgAvx1XhW/iEASj4/jCWl3Q== - -pg-connection-string@^2.6.2: - version "2.6.2" - resolved "https://registry.yarnpkg.com/pg-connection-string/-/pg-connection-string-2.6.2.tgz#713d82053de4e2bd166fab70cd4f26ad36aab475" - integrity sha512-ch6OwaeaPYcova4kKZ15sbJ2hKb/VP48ZD2gE7i1J+L4MspCtBMAx8nMgz7bksc7IojCIIWuEhHibSMFH8m8oA== - -pg-int8@1.0.1: - version "1.0.1" - resolved "https://registry.yarnpkg.com/pg-int8/-/pg-int8-1.0.1.tgz#943bd463bf5b71b4170115f80f8efc9a0c0eb78c" - integrity sha512-WCtabS6t3c8SkpDBUlb1kjOs7l66xsGdKpIPZsg4wR+B3+u9UAum2odSsF9tnvxg80h4ZxLWMy4pRjOsFIqQpw== - -pg-numeric@1.0.2: - version "1.0.2" - resolved "https://registry.yarnpkg.com/pg-numeric/-/pg-numeric-1.0.2.tgz#816d9a44026086ae8ae74839acd6a09b0636aa3a" - integrity sha512-BM/Thnrw5jm2kKLE5uJkXqqExRUY/toLHda65XgFTBTFYZyopbKjBe29Ii3RbkvlsMoFwD+tHeGaCjjv0gHlyw== - -pg-pool@^3.6.1: - version "3.6.1" - resolved "https://registry.yarnpkg.com/pg-pool/-/pg-pool-3.6.1.tgz#5a902eda79a8d7e3c928b77abf776b3cb7d351f7" - integrity sha512-jizsIzhkIitxCGfPRzJn1ZdcosIt3pz9Sh3V01fm1vZnbnCMgmGl5wvGGdNN2EL9Rmb0EcFoCkixH4Pu+sP9Og== - -pg-protocol@*, pg-protocol@^1.6.0: - version "1.6.0" - resolved "https://registry.yarnpkg.com/pg-protocol/-/pg-protocol-1.6.0.tgz#4c91613c0315349363af2084608db843502f8833" - integrity sha512-M+PDm637OY5WM307051+bsDia5Xej6d9IR4GwJse1qA1DIhiKlksvrneZOYQq42OM+spubpcNYEo2FcKQrDk+Q== - -pg-types@^2.1.0: - version "2.2.0" - resolved "https://registry.yarnpkg.com/pg-types/-/pg-types-2.2.0.tgz#2d0250d636454f7cfa3b6ae0382fdfa8063254a3" - integrity sha512-qTAAlrEsl8s4OiEQY69wDvcMIdQN6wdz5ojQiOy6YRMuynxenON0O5oCpJI6lshc6scgAY8qvJ2On/p+CXY0GA== - dependencies: - pg-int8 "1.0.1" - postgres-array "~2.0.0" - postgres-bytea "~1.0.0" - postgres-date "~1.0.4" - postgres-interval "^1.1.0" - -pg-types@^4.0.1: - version "4.0.1" - resolved "https://registry.yarnpkg.com/pg-types/-/pg-types-4.0.1.tgz#31857e89d00a6c66b06a14e907c3deec03889542" - integrity sha512-hRCSDuLII9/LE3smys1hRHcu5QGcLs9ggT7I/TCs0IE+2Eesxi9+9RWAAwZ0yaGjxoWICF/YHLOEjydGujoJ+g== - dependencies: - pg-int8 "1.0.1" - pg-numeric "1.0.2" - postgres-array "~3.0.1" - postgres-bytea "~3.0.0" - postgres-date "~2.0.1" - postgres-interval "^3.0.0" - postgres-range "^1.1.1" - -pg@^8.7.3: - version "8.11.3" - resolved "https://registry.yarnpkg.com/pg/-/pg-8.11.3.tgz#d7db6e3fe268fcedd65b8e4599cda0b8b4bf76cb" - integrity sha512-+9iuvG8QfaaUrrph+kpF24cXkH1YOOUeArRNYIxq1viYHZagBxrTno7cecY1Fa44tJeZvaoG+Djpkc3JwehN5g== - dependencies: - buffer-writer "2.0.0" - packet-reader "1.0.0" - pg-connection-string "^2.6.2" - pg-pool "^3.6.1" - pg-protocol "^1.6.0" - pg-types "^2.1.0" - pgpass "1.x" - optionalDependencies: - pg-cloudflare "^1.1.1" - -pgpass@1.x: - version "1.0.5" - resolved "https://registry.yarnpkg.com/pgpass/-/pgpass-1.0.5.tgz#9b873e4a564bb10fa7a7dbd55312728d422a223d" - integrity sha512-FdW9r/jQZhSeohs1Z3sI1yxFQNFvMcnmfuj4WBMUTxOrAyLMaTcE1aAMBiTlbMNaXvBCQuVi0R7hd8udDSP7ug== - dependencies: - split2 "^4.1.0" - -postgres-array@~2.0.0: - version "2.0.0" - resolved "https://registry.yarnpkg.com/postgres-array/-/postgres-array-2.0.0.tgz#48f8fce054fbc69671999329b8834b772652d82e" - integrity sha512-VpZrUqU5A69eQyW2c5CA1jtLecCsN2U/bD6VilrFDWq5+5UIEVO7nazS3TEcHf1zuPYO/sqGvUvW62g86RXZuA== - -postgres-array@~3.0.1: - version "3.0.2" - resolved "https://registry.yarnpkg.com/postgres-array/-/postgres-array-3.0.2.tgz#68d6182cb0f7f152a7e60dc6a6889ed74b0a5f98" - integrity sha512-6faShkdFugNQCLwucjPcY5ARoW1SlbnrZjmGl0IrrqewpvxvhSLHimCVzqeuULCbG0fQv7Dtk1yDbG3xv7Veog== - -postgres-bytea@~1.0.0: - version "1.0.0" - resolved "https://registry.yarnpkg.com/postgres-bytea/-/postgres-bytea-1.0.0.tgz#027b533c0aa890e26d172d47cf9ccecc521acd35" - integrity sha512-xy3pmLuQqRBZBXDULy7KbaitYqLcmxigw14Q5sj8QBVLqEwXfeybIKVWiqAXTlcvdvb0+xkOtDbfQMOf4lST1w== - -postgres-bytea@~3.0.0: - version "3.0.0" - resolved "https://registry.yarnpkg.com/postgres-bytea/-/postgres-bytea-3.0.0.tgz#9048dc461ac7ba70a6a42d109221619ecd1cb089" - integrity sha512-CNd4jim9RFPkObHSjVHlVrxoVQXz7quwNFpz7RY1okNNme49+sVyiTvTRobiLV548Hx/hb1BG+iE7h9493WzFw== - dependencies: - obuf "~1.1.2" - -postgres-date@~1.0.4: - version "1.0.7" - resolved "https://registry.yarnpkg.com/postgres-date/-/postgres-date-1.0.7.tgz#51bc086006005e5061c591cee727f2531bf641a8" - integrity sha512-suDmjLVQg78nMK2UZ454hAG+OAW+HQPZ6n++TNDUX+L0+uUlLywnoxJKDou51Zm+zTCjrCl0Nq6J9C5hP9vK/Q== - -postgres-date@~2.0.1: - version "2.0.1" - resolved "https://registry.yarnpkg.com/postgres-date/-/postgres-date-2.0.1.tgz#638b62e5c33764c292d37b08f5257ecb09231457" - integrity sha512-YtMKdsDt5Ojv1wQRvUhnyDJNSr2dGIC96mQVKz7xufp07nfuFONzdaowrMHjlAzY6GDLd4f+LUHHAAM1h4MdUw== - -postgres-interval@^1.1.0: - version "1.2.0" - resolved "https://registry.yarnpkg.com/postgres-interval/-/postgres-interval-1.2.0.tgz#b460c82cb1587507788819a06aa0fffdb3544695" - integrity sha512-9ZhXKM/rw350N1ovuWHbGxnGh/SNJ4cnxHiM0rxE4VN41wsg8P8zWn9hv/buK00RP4WvlOyr/RBDiptyxVbkZQ== - dependencies: - xtend "^4.0.0" - -postgres-interval@^3.0.0: - version "3.0.0" - resolved "https://registry.yarnpkg.com/postgres-interval/-/postgres-interval-3.0.0.tgz#baf7a8b3ebab19b7f38f07566c7aab0962f0c86a" - integrity sha512-BSNDnbyZCXSxgA+1f5UU2GmwhoI0aU5yMxRGO8CdFEcY2BQF9xm/7MqKnYoM1nJDk8nONNWDk9WeSmePFhQdlw== - -postgres-range@^1.1.1: - version "1.1.3" - resolved "https://registry.yarnpkg.com/postgres-range/-/postgres-range-1.1.3.tgz#9ccd7b01ca2789eb3c2e0888b3184225fa859f76" - integrity sha512-VdlZoocy5lCP0c/t66xAfclglEapXPCIVhqqJRncYpvbCgImF0w67aPKfbqUMr72tO2k5q0TdTZwCLjPTI6C9g== - proxy-from-env@^1.1.0: version "1.1.0" resolved "https://registry.yarnpkg.com/proxy-from-env/-/proxy-from-env-1.1.0.tgz#e102f16ca355424865755d2c9e8ea4f24d58c3e2" @@ -2296,11 +2147,6 @@ semver@^5.6.0: resolved "https://registry.yarnpkg.com/semver/-/semver-5.7.2.tgz#48d55db737c3287cd4835e17fa13feace1c41ef8" integrity sha512-cBznnQ9KjJqU67B52RMC65CMarK2600WFnbkcaiwWq3xy/5haFJlshgnpjovMVJ+Hff49d8GEn0b87C5pDQ10g== -split2@^4.1.0: - version "4.2.0" - resolved "https://registry.yarnpkg.com/split2/-/split2-4.2.0.tgz#c9c5920904d148bab0b9f67145f245a86aadbfa4" - integrity sha512-UcjcJOWknrNkF6PLX83qcHM6KHgVKNkV62Y8a5uYDVv9ydGQVwAHMKqHdJje1VTWpljG0WYpCDhrCdAOYH4TWg== - strnum@^1.0.5: version "1.0.5" resolved "https://registry.yarnpkg.com/strnum/-/strnum-1.0.5.tgz#5c4e829fe15ad4ff0d20c3db5ac97b73c9b072db" @@ -2330,8 +2176,3 @@ uuid@^8.3.2: version "8.3.2" resolved "https://registry.yarnpkg.com/uuid/-/uuid-8.3.2.tgz#80d5b5ced271bb9af6c445f21a1a04c606cefbe2" integrity sha512-+NYs2QeMWy+GWFOEm9xnn6HCDp0l7QBD7ml8zLUmJ+93Q5NF0NocErnwkTkXVFNiX3/fpC6afS8Dhb/gz7R7eg== - -xtend@^4.0.0: - version "4.0.2" - resolved "https://registry.yarnpkg.com/xtend/-/xtend-4.0.2.tgz#bb72779f5fa465186b1f438f674fa347fdb5db54" - integrity sha512-LKYU1iAXJXUgAXn9URjiu+MWhyUXHsvfp7mcuYm9dSUKK0/CjtrUwFAxD82/mCWbtLsGjFIad0wIsod4zrTAEQ== diff --git a/aws/lambdas/form_archiver.tf b/aws/lambdas/form_archiver.tf index d767b2108..e9838d1ea 100644 --- a/aws/lambdas/form_archiver.tf +++ b/aws/lambdas/form_archiver.tf @@ -29,16 +29,10 @@ resource "aws_lambda_function" "form_archiver" { environment { variables = { - ENVIRONMENT = var.env - REGION = var.region - DB_ARN = var.rds_cluster_arn - DB_SECRET = var.database_secret_arn - DB_NAME = var.rds_db_name - LOCALSTACK = var.localstack_hosted - PGHOST = var.localstack_hosted ? "host.docker.internal" : null - PGUSER = var.localstack_hosted ? "postgres" : null - PGDATABASE = var.localstack_hosted ? "formsDB" : null - PGPASSWORD = var.localstack_hosted ? "chummy" : null + REGION = var.region + DB_ARN = var.rds_cluster_arn + DB_SECRET = var.database_secret_arn + DB_NAME = var.rds_db_name } } diff --git a/aws/lambdas/nagware.tf b/aws/lambdas/nagware.tf index 5f96af249..2bfd4f2e7 100644 --- a/aws/lambdas/nagware.tf +++ b/aws/lambdas/nagware.tf @@ -39,10 +39,6 @@ resource "aws_lambda_function" "nagware" { NOTIFY_API_KEY = var.notify_api_key_secret_arn TEMPLATE_ID = var.gc_template_id LOCALSTACK = var.localstack_hosted - PGHOST = var.localstack_hosted ? "host.docker.internal" : null - PGUSER = var.localstack_hosted ? "postgres" : null - PGDATABASE = var.localstack_hosted ? "formsDB" : null - PGPASSWORD = var.localstack_hosted ? "chummy" : null } } diff --git a/aws/lambdas/reliability.tf b/aws/lambdas/reliability.tf index e6127191b..745aff19a 100644 --- a/aws/lambdas/reliability.tf +++ b/aws/lambdas/reliability.tf @@ -34,10 +34,6 @@ resource "aws_lambda_function" "reliability" { DB_SECRET = var.database_secret_arn DB_NAME = var.rds_db_name LOCALSTACK = var.localstack_hosted - PGHOST = var.localstack_hosted ? "host.docker.internal" : null - PGUSER = var.localstack_hosted ? "postgres" : null - PGDATABASE = var.localstack_hosted ? "formsDB" : null - PGPASSWORD = var.localstack_hosted ? "chummy" : null } } diff --git a/aws/lambdas/vault_integrity.tf b/aws/lambdas/vault_integrity.tf index 3c98634f4..0b78e2293 100644 --- a/aws/lambdas/vault_integrity.tf +++ b/aws/lambdas/vault_integrity.tf @@ -9,7 +9,6 @@ data "archive_file" "vault_integrity_code" { } resource "aws_s3_object" "vault_integrity_code" { - bucket = var.lambda_code_id key = "vault_integrity_code" source = data.archive_file.vault_integrity_code.output_path diff --git a/aws/network/inputs.tf b/aws/network/inputs.tf index 32cde5c9a..76c84ccf0 100644 --- a/aws/network/inputs.tf +++ b/aws/network/inputs.tf @@ -1,8 +1,3 @@ -variable "kms_key_cloudwatch_arn" { - description = "CloudWatch KMS key ARN used to encrypt the VPC flow logs" - type = string -} - variable "vpc_cidr_block" { description = "IP CIDR block of the VPC" type = string diff --git a/aws/network/vpc_flow_log.tf b/aws/network/vpc_flow_log.tf index 781da59c6..2fc808b0f 100644 --- a/aws/network/vpc_flow_log.tf +++ b/aws/network/vpc_flow_log.tf @@ -3,11 +3,10 @@ # Capture network traffic over the VPC # resource "aws_flow_log" "vpc_flow_logs" { + count = var.env == "local" ? 0 : 1 log_destination = "arn:aws:s3:::${var.cbs_satellite_bucket_name}/vpc_flow_logs/" log_destination_type = "s3" traffic_type = "ALL" vpc_id = aws_vpc.forms.id log_format = "$${vpc-id} $${version} $${account-id} $${interface-id} $${srcaddr} $${dstaddr} $${srcport} $${dstport} $${protocol} $${packets} $${bytes} $${start} $${end} $${action} $${log-status} $${subnet-id} $${instance-id}" - - } diff --git a/aws/rds/rds.tf b/aws/rds/rds.tf index 95deb18da..4d48517d7 100644 --- a/aws/rds/rds.tf +++ b/aws/rds/rds.tf @@ -37,7 +37,6 @@ resource "aws_rds_cluster" "forms" { allow_major_version_upgrade = true copy_tags_to_snapshot = true - scaling_configuration { auto_pause = false max_capacity = 8 diff --git a/aws/redis/redis.tf b/aws/redis/redis.tf index fa706cab6..d88a7b4bb 100644 --- a/aws/redis/redis.tf +++ b/aws/redis/redis.tf @@ -9,8 +9,8 @@ resource "aws_elasticache_replication_group" "redis" { replication_group_id = "gcforms-redis-rep-group" description = "Redis cluster for GCForms" node_type = "cache.t2.micro" - num_cache_clusters = 2 - engine_version = "6.x" + num_cache_clusters = var.env == "local" ? 1 : 2 # In Localstack, if Elasticache is used to host a Redis service it cannot have more than one node + engine_version = var.env == "local" ? "6.2" : "6.x" # Localstack does not accept the use of latest minor version (`.x`) at creation time parameter_group_name = "default.redis6.x" port = 6379 multi_az_enabled = true diff --git a/docker-compose.yml b/docker-compose.yml index d884ec760..b9a943e62 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -8,8 +8,9 @@ services: ports: - "127.0.0.1:4566:4566" # LocalStack Gateway - "127.0.0.1:4510-4559:4510-4559" # external services port range + - "127.0.0.1:6379:6379" # Redis environment: - - SERVICES=cloudwatch,dynamodb,ec2,events,iam,kinesis,kms,lambda,logs,s3,secretsmanager,sns,sqs + - SERVICES=cloudwatch,dynamodb,ec2,events,iam,kinesis,kms,lambda,logs,s3,secretsmanager,sns,sqs,elasticache,rds - LAMBDA_RUNTIME_ENVIRONMENT_TIMEOUT=60 - LAMBDA_SYNCHRONOUS_CREATE=1 - LAMBDA_IGNORE_ARCHITECTURE=1 @@ -17,33 +18,9 @@ services: - DOCKER_HOST=unix:///var/run/docker.sock - PERSISTENCE=1 - LOCALSTACK_AUTH_TOKEN=${LOCALSTACK_AUTH_TOKEN} - - SNAPSHOT_LOAD_STRATEGY=ON_STARTUP - - SNAPSHOT_FLUSH_INTERVAL=60 + - SNAPSHOT_SAVE_STRATEGY=${SNAPSHOT_SAVE_STRATEGY:-SCHEDULED} + - SNAPSHOT_FLUSH_INTERVAL=${SNAPSHOT_FLUSH_INTERVAL:-60} + - SNAPSHOT_LOAD_STRATEGY=${SNAPSHOT_LOAD_STRATEGY:-ON_STARTUP} volumes: - /var/run/docker.sock:/var/run/docker.sock - - localstack:/var/lib/localstack" - - db: - container_name: "GCForms_DB" - image: postgres:13-alpine - environment: - - POSTGRES_USER=postgres - - POSTGRES_PASSWORD=chummy - - PGDATA=/var/lib/postgresql/data - ports: - - "5432:5432" - expose: - - "5432" - command: - - "postgres" - - "-c" - - "listen_addresses=*" - restart: always - redis: - container_name: "GCForms_Redis" - restart: always - image: redis:6-alpine - ports: - - "6379:6379" - expose: - - "6379" + - localstack:/var/lib/localstack" \ No newline at end of file diff --git a/env/cloud/app/terragrunt.hcl b/env/cloud/app/terragrunt.hcl index 8243e2bb6..93eabec6e 100644 --- a/env/cloud/app/terragrunt.hcl +++ b/env/cloud/app/terragrunt.hcl @@ -181,7 +181,6 @@ inputs = { vault_file_storage_id = dependency.s3.outputs.vault_file_storage_id reliability_file_storage_arn = dependency.s3.outputs.reliability_file_storage_arn reliability_file_storage_id = dependency.s3.outputs.reliability_file_storage_id - } include { diff --git a/env/cloud/lambdas/terragrunt.hcl b/env/cloud/lambdas/terragrunt.hcl index d355a377d..e10f6e57c 100644 --- a/env/cloud/lambdas/terragrunt.hcl +++ b/env/cloud/lambdas/terragrunt.hcl @@ -26,7 +26,6 @@ dependency "app" { } dependency "rds" { - enabled = local.env == "local" ? false : true config_path = "../rds" mock_outputs_merge_strategy_with_state = "shallow" mock_outputs_allowed_terraform_commands = ["init", "fmt", "validate", "plan", "show"] @@ -126,9 +125,9 @@ inputs = { kms_key_cloudwatch_arn = dependency.kms.outputs.kms_key_cloudwatch_arn kms_key_dynamodb_arn = dependency.kms.outputs.kms_key_dynamodb_arn - rds_cluster_arn = local.env == "local" ? null : dependency.rds.outputs.rds_cluster_arn - rds_db_name = local.env == "local" ? null : dependency.rds.outputs.rds_db_name - database_secret_arn = local.env == "local" ? "arn:aws:secretsmanager:ca-central-1:000000000000:secret:database_secret-VvMslX" : dependency.rds.outputs.database_secret_arn + rds_cluster_arn = dependency.rds.outputs.rds_cluster_arn + rds_db_name = dependency.rds.outputs.rds_db_name + database_secret_arn = dependency.rds.outputs.database_secret_arn sqs_reliability_queue_arn = dependency.sqs.outputs.sqs_reliability_queue_arn sqs_reliability_queue_id = dependency.sqs.outputs.sqs_reliability_queue_id diff --git a/env/cloud/network/.terraform.lock.hcl b/env/cloud/network/.terraform.lock.hcl new file mode 100644 index 000000000..effccc803 --- /dev/null +++ b/env/cloud/network/.terraform.lock.hcl @@ -0,0 +1,45 @@ +# This file is maintained automatically by "terraform init". +# Manual edits may be lost in future updates. + +provider "registry.terraform.io/hashicorp/aws" { + version = "5.31.0" + constraints = "5.31.0" + hashes = [ + "h1:2eauBmfftzGMpzFQn9aHSXiyaO3Ve5cnihmXcKGGpgU=", + "zh:0cdb9c2083bf0902442384f7309367791e4640581652dda456f2d6d7abf0de8d", + "zh:2fe4884cb9642f48a5889f8dff8f5f511418a18537a9dfa77ada3bcdad391e4e", + "zh:36d8bdd72fe61d816d0049c179f495bc6f1e54d8d7b07c45b62e5e1696882a89", + "zh:539dd156e3ec608818eb21191697b230117437a58587cbd02ce533202a4dd520", + "zh:6a53f4b57ac4eb3479fc0d8b6e301ca3a27efae4c55d9f8bd24071b12a03361c", + "zh:6faeb8ff6792ca7af1c025255755ad764667a300291cc10cea0c615479488c87", + "zh:7d9423149b323f6d0df5b90c4d9029e5455c670aea2a7eb6fef4684ba7eb2e0b", + "zh:8235badd8a5d0993421cacf5ead48fac73d3b5a25c8a68599706a404b1f70730", + "zh:860b4f60842b2879c5128b7e386c8b49adeda9287fed12c5cd74861bb659bbcd", + "zh:9b12af85486a96aedd8d7984b0ff811a4b42e3d88dad1a3fb4c0b580d04fa425", + "zh:b021fceaf9382c8fe3c6eb608c24d01dce3d11ba7e65bb443d51ca9b90e9b237", + "zh:b38b0bfc1c69e714e80cf1c9ea06e687ee86aa9f45694be28eb07adcebbe0489", + "zh:c972d155f6c01af9690a72adfb99cfc24ef5ef311ca92ce46b9b13c5c153f572", + "zh:e0dd29920ec84fdb6026acff44dcc1fb1a24a0caa093fa04cdbc713d384c651d", + "zh:e3127ebd2cb0374cd1808f911e6bffe2f4ac4d84317061381242353f3a7bc27d", + ] +} + +provider "registry.terraform.io/hashicorp/random" { + version = "3.6.0" + constraints = "3.6.0" + hashes = [ + "h1:p6WG1IPHnqx1fnJVKNjv733FBaArIugqy58HRZnpPCk=", + "zh:03360ed3ecd31e8c5dac9c95fe0858be50f3e9a0d0c654b5e504109c2159287d", + "zh:1c67ac51254ba2a2bb53a25e8ae7e4d076103483f55f39b426ec55e47d1fe211", + "zh:24a17bba7f6d679538ff51b3a2f378cedadede97af8a1db7dad4fd8d6d50f829", + "zh:30ffb297ffd1633175d6545d37c2217e2cef9545a6e03946e514c59c0859b77d", + "zh:454ce4b3dbc73e6775f2f6605d45cee6e16c3872a2e66a2c97993d6e5cbd7055", + "zh:78d5eefdd9e494defcb3c68d282b8f96630502cac21d1ea161f53cfe9bb483b3", + "zh:91df0a9fab329aff2ff4cf26797592eb7a3a90b4a0c04d64ce186654e0cc6e17", + "zh:aa57384b85622a9f7bfb5d4512ca88e61f22a9cea9f30febaa4c98c68ff0dc21", + "zh:c4a3e329ba786ffb6f2b694e1fd41d413a7010f3a53c20b432325a94fa71e839", + "zh:e2699bc9116447f96c53d55f2a00570f982e6f9935038c3810603572693712d0", + "zh:e747c0fd5d7684e5bfad8aa0ca441903f15ae7a98a737ff6aca24ba223207e2c", + "zh:f1ca75f417ce490368f047b63ec09fd003711ae48487fba90b4aba2ccf71920e", + ] +} diff --git a/env/cloud/network/terragrunt.hcl b/env/cloud/network/terragrunt.hcl index 0d8941c69..c1aabd7ba 100644 --- a/env/cloud/network/terragrunt.hcl +++ b/env/cloud/network/terragrunt.hcl @@ -2,21 +2,7 @@ terraform { source = "../../../aws//network" } -dependencies { - paths = ["../kms"] -} - -dependency "kms" { - config_path = "../kms" - mock_outputs_merge_strategy_with_state = "shallow" - mock_outputs_allowed_terraform_commands = ["init", "fmt", "validate", "plan", "show"] - mock_outputs = { - kms_key_cloudwatch_arn = null - } -} - inputs = { - kms_key_cloudwatch_arn = dependency.kms.outputs.kms_key_cloudwatch_arn vpc_cidr_block = "172.16.0.0/16" vpc_name = "forms" } diff --git a/env/cloud/rds/.terraform.lock.hcl b/env/cloud/rds/.terraform.lock.hcl new file mode 100644 index 000000000..effccc803 --- /dev/null +++ b/env/cloud/rds/.terraform.lock.hcl @@ -0,0 +1,45 @@ +# This file is maintained automatically by "terraform init". +# Manual edits may be lost in future updates. + +provider "registry.terraform.io/hashicorp/aws" { + version = "5.31.0" + constraints = "5.31.0" + hashes = [ + "h1:2eauBmfftzGMpzFQn9aHSXiyaO3Ve5cnihmXcKGGpgU=", + "zh:0cdb9c2083bf0902442384f7309367791e4640581652dda456f2d6d7abf0de8d", + "zh:2fe4884cb9642f48a5889f8dff8f5f511418a18537a9dfa77ada3bcdad391e4e", + "zh:36d8bdd72fe61d816d0049c179f495bc6f1e54d8d7b07c45b62e5e1696882a89", + "zh:539dd156e3ec608818eb21191697b230117437a58587cbd02ce533202a4dd520", + "zh:6a53f4b57ac4eb3479fc0d8b6e301ca3a27efae4c55d9f8bd24071b12a03361c", + "zh:6faeb8ff6792ca7af1c025255755ad764667a300291cc10cea0c615479488c87", + "zh:7d9423149b323f6d0df5b90c4d9029e5455c670aea2a7eb6fef4684ba7eb2e0b", + "zh:8235badd8a5d0993421cacf5ead48fac73d3b5a25c8a68599706a404b1f70730", + "zh:860b4f60842b2879c5128b7e386c8b49adeda9287fed12c5cd74861bb659bbcd", + "zh:9b12af85486a96aedd8d7984b0ff811a4b42e3d88dad1a3fb4c0b580d04fa425", + "zh:b021fceaf9382c8fe3c6eb608c24d01dce3d11ba7e65bb443d51ca9b90e9b237", + "zh:b38b0bfc1c69e714e80cf1c9ea06e687ee86aa9f45694be28eb07adcebbe0489", + "zh:c972d155f6c01af9690a72adfb99cfc24ef5ef311ca92ce46b9b13c5c153f572", + "zh:e0dd29920ec84fdb6026acff44dcc1fb1a24a0caa093fa04cdbc713d384c651d", + "zh:e3127ebd2cb0374cd1808f911e6bffe2f4ac4d84317061381242353f3a7bc27d", + ] +} + +provider "registry.terraform.io/hashicorp/random" { + version = "3.6.0" + constraints = "3.6.0" + hashes = [ + "h1:p6WG1IPHnqx1fnJVKNjv733FBaArIugqy58HRZnpPCk=", + "zh:03360ed3ecd31e8c5dac9c95fe0858be50f3e9a0d0c654b5e504109c2159287d", + "zh:1c67ac51254ba2a2bb53a25e8ae7e4d076103483f55f39b426ec55e47d1fe211", + "zh:24a17bba7f6d679538ff51b3a2f378cedadede97af8a1db7dad4fd8d6d50f829", + "zh:30ffb297ffd1633175d6545d37c2217e2cef9545a6e03946e514c59c0859b77d", + "zh:454ce4b3dbc73e6775f2f6605d45cee6e16c3872a2e66a2c97993d6e5cbd7055", + "zh:78d5eefdd9e494defcb3c68d282b8f96630502cac21d1ea161f53cfe9bb483b3", + "zh:91df0a9fab329aff2ff4cf26797592eb7a3a90b4a0c04d64ce186654e0cc6e17", + "zh:aa57384b85622a9f7bfb5d4512ca88e61f22a9cea9f30febaa4c98c68ff0dc21", + "zh:c4a3e329ba786ffb6f2b694e1fd41d413a7010f3a53c20b432325a94fa71e839", + "zh:e2699bc9116447f96c53d55f2a00570f982e6f9935038c3810603572693712d0", + "zh:e747c0fd5d7684e5bfad8aa0ca441903f15ae7a98a737ff6aca24ba223207e2c", + "zh:f1ca75f417ce490368f047b63ec09fd003711ae48487fba90b4aba2ccf71920e", + ] +} diff --git a/env/cloud/rds/terragrunt.hcl b/env/cloud/rds/terragrunt.hcl index d3697b30b..9257e350b 100644 --- a/env/cloud/rds/terragrunt.hcl +++ b/env/cloud/rds/terragrunt.hcl @@ -24,10 +24,13 @@ inputs = { private_subnet_ids = dependency.network.outputs.private_subnet_ids rds_security_group_id = dependency.network.outputs.rds_security_group_id + rds_db_user = local.env == "local" ? "localstack_postgres" : "postgres" # We cannot use `postgres` as a username in Localstack rds_db_name = "forms" - rds_db_subnet_group_name = local.env == "production" ? "forms-db" : "forms-staging-db" - rds_db_user = "postgres" - rds_name = local.env == "production" ? "forms-db" : "forms-staging-db" + rds_name = local.env == "staging" ? "forms-staging-db" : "forms-db" + rds_db_subnet_group_name = local.env == "staging" ? "forms-staging-db" : "forms-db" + + # Overwritten in GitHub Actions by TFVARS + rds_db_password = "chummy" # RDS database password used for local setup } include { diff --git a/env/cloud/redis/.terraform.lock.hcl b/env/cloud/redis/.terraform.lock.hcl new file mode 100644 index 000000000..effccc803 --- /dev/null +++ b/env/cloud/redis/.terraform.lock.hcl @@ -0,0 +1,45 @@ +# This file is maintained automatically by "terraform init". +# Manual edits may be lost in future updates. + +provider "registry.terraform.io/hashicorp/aws" { + version = "5.31.0" + constraints = "5.31.0" + hashes = [ + "h1:2eauBmfftzGMpzFQn9aHSXiyaO3Ve5cnihmXcKGGpgU=", + "zh:0cdb9c2083bf0902442384f7309367791e4640581652dda456f2d6d7abf0de8d", + "zh:2fe4884cb9642f48a5889f8dff8f5f511418a18537a9dfa77ada3bcdad391e4e", + "zh:36d8bdd72fe61d816d0049c179f495bc6f1e54d8d7b07c45b62e5e1696882a89", + "zh:539dd156e3ec608818eb21191697b230117437a58587cbd02ce533202a4dd520", + "zh:6a53f4b57ac4eb3479fc0d8b6e301ca3a27efae4c55d9f8bd24071b12a03361c", + "zh:6faeb8ff6792ca7af1c025255755ad764667a300291cc10cea0c615479488c87", + "zh:7d9423149b323f6d0df5b90c4d9029e5455c670aea2a7eb6fef4684ba7eb2e0b", + "zh:8235badd8a5d0993421cacf5ead48fac73d3b5a25c8a68599706a404b1f70730", + "zh:860b4f60842b2879c5128b7e386c8b49adeda9287fed12c5cd74861bb659bbcd", + "zh:9b12af85486a96aedd8d7984b0ff811a4b42e3d88dad1a3fb4c0b580d04fa425", + "zh:b021fceaf9382c8fe3c6eb608c24d01dce3d11ba7e65bb443d51ca9b90e9b237", + "zh:b38b0bfc1c69e714e80cf1c9ea06e687ee86aa9f45694be28eb07adcebbe0489", + "zh:c972d155f6c01af9690a72adfb99cfc24ef5ef311ca92ce46b9b13c5c153f572", + "zh:e0dd29920ec84fdb6026acff44dcc1fb1a24a0caa093fa04cdbc713d384c651d", + "zh:e3127ebd2cb0374cd1808f911e6bffe2f4ac4d84317061381242353f3a7bc27d", + ] +} + +provider "registry.terraform.io/hashicorp/random" { + version = "3.6.0" + constraints = "3.6.0" + hashes = [ + "h1:p6WG1IPHnqx1fnJVKNjv733FBaArIugqy58HRZnpPCk=", + "zh:03360ed3ecd31e8c5dac9c95fe0858be50f3e9a0d0c654b5e504109c2159287d", + "zh:1c67ac51254ba2a2bb53a25e8ae7e4d076103483f55f39b426ec55e47d1fe211", + "zh:24a17bba7f6d679538ff51b3a2f378cedadede97af8a1db7dad4fd8d6d50f829", + "zh:30ffb297ffd1633175d6545d37c2217e2cef9545a6e03946e514c59c0859b77d", + "zh:454ce4b3dbc73e6775f2f6605d45cee6e16c3872a2e66a2c97993d6e5cbd7055", + "zh:78d5eefdd9e494defcb3c68d282b8f96630502cac21d1ea161f53cfe9bb483b3", + "zh:91df0a9fab329aff2ff4cf26797592eb7a3a90b4a0c04d64ce186654e0cc6e17", + "zh:aa57384b85622a9f7bfb5d4512ca88e61f22a9cea9f30febaa4c98c68ff0dc21", + "zh:c4a3e329ba786ffb6f2b694e1fd41d413a7010f3a53c20b432325a94fa71e839", + "zh:e2699bc9116447f96c53d55f2a00570f982e6f9935038c3810603572693712d0", + "zh:e747c0fd5d7684e5bfad8aa0ca441903f15ae7a98a737ff6aca24ba223207e2c", + "zh:f1ca75f417ce490368f047b63ec09fd003711ae48487fba90b4aba2ccf71920e", + ] +} diff --git a/env/cloud/secrets/terragrunt.hcl b/env/cloud/secrets/terragrunt.hcl index ac9e87951..597f498ab 100644 --- a/env/cloud/secrets/terragrunt.hcl +++ b/env/cloud/secrets/terragrunt.hcl @@ -6,7 +6,6 @@ include { path = find_in_parent_folders() } - locals { env = get_env("APP_ENV", "local") ecs_secret_token = get_env("ECS_SECRET_TOKEN", "I_am_not_a_secret_token") @@ -14,16 +13,15 @@ locals { notify_callback_bearer_token = get_env("NOTIFY_CALLBACK_BEARER_TOKEN", "I_am_not_a_secret_token") notify_api_key = get_env("NOTIFY_API_KEY", "I_am_not_a_secret_token") freshdesk_api_key = get_env("FRESHDESK_API_KEY", "I_am_not_a_secret_token") + rds_db_password = "chummy" } - inputs = { ecs_secret_token = local.ecs_secret_token recaptcha_secret = local.recaptcha_secret notify_callback_bearer_token = local.notify_callback_bearer_token notify_api_key = local.notify_api_key freshdesk_api_key = local.freshdesk_api_key -} - - - + # Overwritten in GitHub Actions by TFVARS + rds_db_password = local.rds_db_password +} \ No newline at end of file diff --git a/env/common/local-provider.tf b/env/common/local-provider.tf index 19e298dff..4570cbab9 100644 --- a/env/common/local-provider.tf +++ b/env/common/local-provider.tf @@ -21,7 +21,7 @@ provider "aws" { access_key = "test" secret_key = "test" region = "ca-central-1" - s3_use_path_style = true + s3_use_path_style = true skip_credentials_validation = true skip_metadata_api_check = true skip_requesting_account_id = true diff --git a/localstack_services.sh b/localstack_services.sh index e76c51982..d96c8f2a2 100755 --- a/localstack_services.sh +++ b/localstack_services.sh @@ -25,20 +25,25 @@ tfswitch 1.6.6 basedir=$(pwd) -printf "${color}=> Cleaning up previous caches, terraform state, and lambda dependencies${reset}\n" +ACTION=$1 -printf "${color}...Purging stale localstack related files${reset}\n" -find $basedir/env/cloud -type d -name .terragrunt-cache -prune -print -exec rm -rf {} \; +if [[ "${ACTION}" == "clean" ]] +then + printf "${color}=> Cleaning up previous caches, terraform state, and lambda dependencies${reset}\n" -printf "${color}...Purging stale terraform state files${reset}\n" -find $basedir/env -type f -name terraform.tfstate -prune -exec rm -fv {} \; + printf "${color}...Purging stale localstack related files${reset}\n" + find $basedir/env/cloud -type d -name .terragrunt-cache -prune -print -exec rm -rf {} \; -printf "${color}...Clearing old lambda_code archive files${reset}\n" -rm -v /tmp/*.zip || true + printf "${color}...Purging stale terraform state files${reset}\n" + find $basedir/env -type f -name terraform.tfstate -prune -exec rm -fv {} \; -printf "${color}...Removing old lambda dependencies${reset}\n" -cd $basedir/aws/lambdas/code -./deps.sh delete + printf "${color}...Clearing old lambda_code archive files${reset}\n" + rm -v /tmp/*.zip || true + + printf "${color}...Removing old lambda dependencies${reset}\n" + cd $basedir/aws/lambdas/code + ./deps.sh delete +fi printf "${color}=> Creating AWS services in Localstack${reset}\n" @@ -46,12 +51,12 @@ printf "${color}...Setting up KMS${reset}\n" cd $basedir/env/cloud/kms terragrunt apply --terragrunt-non-interactive -auto-approve --terragrunt-log-level warn -printf "${color}...Setting up Secrets Manager${reset}\n" -cd $basedir/env/cloud/secrets +printf "${color}...Setting up Network${reset}\n" +cd $basedir/env/cloud/network terragrunt apply --terragrunt-non-interactive -auto-approve --terragrunt-log-level warn -printf "${color}...Setting up S3${reset}\n" -cd $basedir/env/cloud/s3 +printf "${color}...Setting up Secrets Manager${reset}\n" +cd $basedir/env/cloud/secrets terragrunt apply --terragrunt-non-interactive -auto-approve --terragrunt-log-level warn printf "${color}...Creating SQS queue${reset}\n" @@ -62,6 +67,18 @@ printf "${color}...Creating SNS queue${reset}\n" cd $basedir/env/cloud/sns terragrunt apply --terragrunt-non-interactive -auto-approve --terragrunt-log-level warn +printf "${color}...Setting up Redis${reset}\n" +cd $basedir/env/cloud/redis +terragrunt apply --terragrunt-non-interactive -auto-approve --terragrunt-log-level warn + +printf "${color}...Setting up RDS${reset}\n" +cd $basedir/env/cloud/rds +terragrunt apply --terragrunt-non-interactive -auto-approve --terragrunt-log-level warn + +printf "${color}...Setting up S3${reset}\n" +cd $basedir/env/cloud/s3 +terragrunt apply --terragrunt-non-interactive -auto-approve --terragrunt-log-level warn + printf "${color}...Creating the DynamoDB database${reset}\n" cd $basedir/env/cloud/dynamodb terragrunt apply --terragrunt-non-interactive -auto-approve --terragrunt-log-level warn