diff --git a/README.md b/README.md index 1b026cd..cf7b872 100644 --- a/README.md +++ b/README.md @@ -1,3 +1,6 @@ +[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](./LICENSE) +![version](https://img.shields.io/badge/version-1.0.0-blue) + The Open Source project has one clear but distinctive focus - Enabling AWS customers to automatically onboard their IoT Devices into the AWS IoT Core (device-onboarding-as-a-Service) following a self-managed approach. Customers with the "1NCE Connect" product can map their IoT devices via SIM cards to certificates for the AWS IoT Core. The certificates allow publishing, subscription, and connection to AWS IoT Core MQTT broker. Certificates and other credentials for MQTT subscription can be retrieved from HTTP GET endpoint by using the SIM-as-an-Identity service. Service allows using a single endpoint for all devices. Individual certificates are returned to each device depending on the IP address, that is being used in the 1NCE network. @@ -28,6 +31,8 @@ At least one 1NCE SIM card and access to the 1NCE.com portal. Access to an AWS a A user with an API role should be created. Credentials will be used to retrieve all customers' 1NCE SIM cards via [Get ALL SIMs](https://help.1nce.com/dev-hub/reference/getsimsusingget) Endpoint. ##### OpenvpnOnboardingUsername, OpenvpnOnboardingPassword +> :warning: This project uses an OpenVPN connection and may be dropped if you use the same credentials for another purpose. + [1NCE OpenVPN credentials](https://portal.1nce.com/portal/customer/configuration/credentials) can be downloaded: 1NCE portal > Configuration > OpenVPN Configuration > Download credentials.txt @@ -52,6 +57,15 @@ Example: `https://fka8ojq6lh.execute-api.us-west-1.amazonaws.com/APIGatewayStage Breakout region configured in the 1NCE Portal > Configuration > Breakout settings Default: eu-central-1 + +##### SNSSuccessTopicSubscriptionEmail: +E-mail for [SNS Success Topic](#success-topic) subscription. Accepts empty string for no subscription or valid e-mail address. If an e-mail address is provided, please try to approve the approval request immediately to not miss any events about the stack rollout process. < br/> +> :warning: Verbose! The e-mail notifications can be a very large number. Expect at least 1 e-mail for each SIM. + +##### SNSFailureTopicSubscriptionEmail: +E-mail for [SNS Failure Topic](#failure-topic) subscription. Accepts empty string for no subscription or valid e-mail address. If an e-mail address is provided, please try to approve the approval request immediately to not miss any events about the stack rollout process. < br/> + +


# Low-level docs @@ -161,6 +175,14 @@ Lambda step-by-step flow: 2. If SIM that exists in DB is not returned by 1NCE API, then SIM details are being sent to [SIMs disable SQS](#sims-disablefifo) 5. If any error occurs - failure details are sent to [SNS Failure Topic](#failure-topic) +Cron job: +This lambda is triggered by a cron job defined in the CloudFormation parameters (LambdaCron), by default it will be triggered every day at 1 am to identify SIM changes + +Memory size: +This lambda has 3GB of RAM memory to be prepared to fetch and compare thousands of SIMs between the API and Dynamo database. + +Execution time: +Depending on your number of SIMs this lambda can take up to 15 minutes because it is fetching all SIMs from the API over the HTTPS protocol and comparing each SIM change with the database ### Create SIM lambda @@ -245,7 +267,34 @@ EC2 instance flow: - Download the Nginx config template from the public S3 bucket. Fill Nginx config. Run the Nginx server. - Reboot EC2 instance. +
+EC2 instance will send failure message to [SNS Failure Topic](#failure-topic) in such cases: +- Failure to get an SSM parameter +- Failure to put an SSM parameter +- Failure to download configuration file from S3 bucket +- Failure to get a secret from secrets manager +- Failure to get API keys from API Gateway +- If openVPN service will not be able to get tun0 interface within 120s +- If nginx service will fail to be started +Example message: +``` +{ + "timestamp":1680526955876, + "message": "EC2 Nginx server failure" +} +``` + +
+EC2 instance will send success message to [SNS Success Topic](#success-topic) if EC2 configuration will be successfull. + +Example message: +``` +{ + "timestamp":1680526955876, + "message": "EC2 instance for onboarding service configured correctly" +} +``` ### Connect to EC2 machine. 1. AWS > EC2 > Instances @@ -263,6 +312,11 @@ EC2 instance flow: `client credentials.txt openvpn-1nce-client.conf server update-resolv-conf` credentials.txt should contain the same content as 1NCE portal > Configuration > OpenVPN Configuration > Download credentials.txt +
+In order to validate if openvpn can be started manually the following command can be used: +
+`sudo openvpn --config /etc/openvpn/openvpn-1nce-client.conf` + More details https://help.1nce.com/dev-hub/docs/network-services-vpn-service @@ -378,6 +432,8 @@ Deployment values is a file located in the root of the repository and has the re - `breakoutRegionSSMParamName` is the SSM parameter name where the VPN breakout region will be placed - `openVPNCredentialsSecretName` is the AWS Secrets name where the OpenVPN credentials will be placed. - `onboardingApiKeyName` is the name of the API Key used in the Onboarding API Gateway. +- `snsSuccessTopicName` is the SNS topic name where success messages will be published +- `snsFailureTopicName` is the SNS topic name where failure messages will be published ### The script @@ -412,6 +468,178 @@ Script input parameters: | 2 | version-number | No | Non mandatory parameter, default value will be taken from `version` value in `deploymentValues.yaml` file. All files are uploaded to folder with the name of this parameter on S3 bucket | | 3 | latest | No | Non mandatory parameter, if provided uploads `device-onboarding-main.yaml` file to the latest folder in the bucket | +# HTTPS support +If you need to add https support to your stack, follow the next steps: + +1. Create AWS Route53 DNS record with type `A` or `AAAA` and the value must be the EC2 server ip. + - Go to Route 53 > Hosted zones and select your hosted zone + - Click on "Create record" + - Fill the `subdomain` value + - Fill the `ip` value with the EC2 server ip inside of the VPN. This ip value can be found in `Systems Manager > Parameter Store > openvpn-onboarding-proxy-server` + +2. Create IAM user and policy to access Route 53. + > :warning: **If Route 53 is hosted by another AWS account**: User and policy must be created there + + 1. Create IAM policy + - Go to AWS IAM > Policies + - Click on "Create Policy" + - Select `JSON` tab + - Copy and paste this document policy and replace the `REPLACE_WITH_HOSTED_ZONE_ID` with hosted zone id from Reoute 53 + ``` + { + "Version": "2012-10-17", + "Statement": [ + { + "Effect": "Allow", + "Action": [ + "route53:ListHostedZones", + "route53:GetChange" + ], + "Resource": [ + "*" + ] + }, + { + "Effect" : "Allow", + "Action" : [ + "route53:ChangeResourceRecordSets" + ], + "Resource" : [ + "arn:aws:route53:::hostedzone/REPLACE_WITH_HOSTED_ZONE_ID" + ] + } + ] + } + ``` + - Click on "Next: Tags" + - Click on "Next: Review" + - Fill the policy name + - Click on "Create policy" + + 2. Create IAM user and attach created policy + - Go to AWS IAM > Users + - Click on "Add Users" + - Fill the username field and click on "Next" + - Select "Attach policies directly" + - Search and select the created policy + - Click on "Next" + - Click on "Create user" + + 3. Generate access keys + - Open create IAM user + - Go to "Security credentials" tab + - Click on "Create access key" + - Select "Command Line Interface (CLI)" + - Click on "Next" and "Create access key" + - Save the access key and secret access key + - Click on "Done" + +3. Connect to the EC2 instance and install [certbot](https://certbot.eff.org/) and [certbot-dns-route53](https://certbot-dns-route53.readthedocs.io/en/stable/) + ``` + sudo snap install core + sudo snap refresh core + sudo snap install --classic certbot + sudo ln -s /snap/bin/certbot /usr/bin/certbot + sudo snap set certbot trust-plugin-with-root=ok + sudo snap install certbot-dns-route53 + ``` + +4. Add these lines at the end of the nginx configuration file to listen on port 80 and this server will be used to generate the ssl certificate. + Nginx configuration file location: `/etc/nginx/sites-available/default`: + ``` + server { + listen 80 default_server; + listen [::]:80 default_server; + root /var/www/html; + server_name _; + } + ``` + +5. Restart ngnix server + ``` + sudo systemctl restart nginx + ``` + +6. Generate [Let's Encrypt](https://letsencrypt.org/) ssl certificate + + 1. Set AWS credentials + - Create AWS config file + ``` + cd ~ + mkdir .aws + sudo nano .aws/config + ``` + + - Fill in and save this file with the IAM user information created in section 2 + ``` + [default] + aws_access_key_id=REPLACE_WITH_AWS_ACCESS_KEY + aws_secret_access_key=REPLACE_WITH_AWS_SECRET_ACCESS_KEY + ``` + + 2. Generate [Let's Encrypt](https://letsencrypt.org/) ssl certificate. + Values to be replaced: + - REPLACE_WITH_DOMAIN: Domain used to generate the SSL certificate (example: domain.com) + - REPLACE_WITH_EMAIL: Email to subscribe [Let's Encrypt expiry notifications](https://letsencrypt.org/docs/expiration-emails/) + + ``` + cd ~ + certbot certonly --dns-route53 -d REPLACE_WITH_DOMAIN --logs-dir ~/letsencrypt/log/ --config-dir ~/letsencrypt/config/ --work-dir ~/letsencrypt/work/ -m REPLACE_WITH_EMAIL --agree-tos --non-interactive --post-hook "sudo systemctl restart nginx" + ``` + +7. Update nginx configuration (`/etc/nginx/sites-available/default`) to add ssl certificate + 1. Update the listen port to 443 (default https port) and add the `ssl` tag to the server listen ports. + Note: If you want, you can set another https listen port + ``` + listen 443 default_server ssl; + listen [::]:443 default_server ssl; + ``` + + 2. Add certificate and key to the NGINX configuration inside of the `server` object at the same level as `root` and `server_name`. + ``` + ssl_certificate /home/ssm-user/letsencrypt/config/live/REPLACE_WITH_DOMAIN/fullchain.pem; + ssl_certificate_key /home/ssm-user/letsencrypt/config/live/REPLACE_WITH_DOMAIN/privkey.pem; + ``` + + 3. After updating the nginx configuration file should look like this: + ``` + server { + listen 443 default_server ssl; + listen [::]:443 default_server ssl; + + root /var/www/html; + + ssl_certificate /home/ssm-user/letsencrypt/config/live/REPLACE_WITH_DOMAIN/fullchain.pem; + ssl_certificate_key /home/ssm-user/letsencrypt/config/live/REPLACE_WITH_DOMAIN/privkey.pem; + + server_name _; + location / { + proxy_pass REPLACE_WITH_API_GATEWAY_URL; + proxy_set_header onboarding-ip $proxy_add_x_forwarded_for; + proxy_set_header x-api-key REPLACE_WITH_API_KEY; + proxy_ssl_server_name on; + } + } + + server { + listen 80 default_server; + listen [::]:80 default_server; + root /var/www/html; + server_name _; + } + ``` + +8) Restart ngnix server + ``` + sudo systemctl restart nginx + ``` + +9) Test automatic renewal + ``` + cd ~ + certbot renew --dns-route53 --logs-dir ~/letsencrypt/log/ --config-dir ~/letsencrypt/config/ --work-dir ~/letsencrypt/work/ --non-interactive --post-hook "sudo systemctl restart nginx" --dry-run + ``` + # Delete and cleanup process If the stack is being deleted, then the majority of resources will be deleted by the CFN stack. Some of the resources still need to be cleaned/deleted manually: @@ -420,11 +648,19 @@ If the stack is being deleted, then the majority of resources will be deleted by - IoT Core Certs - IoT Core Policy +if HTTPS support has been implemented: +- IAM policy +- IAM user +- Route 53 `A` or `AAAA` record + # FAQ Q: My SIMs were migrated to a different 1NCE organization and stopped working.
A: If SIMs were migrated to a different 1NCE organization, this means that SIM will not be returned via [Get ALL SIMs](https://help.1nce.com/dev-hub/reference/getsimsusingget) Endpoint. Therefore [SIM retrieval lambda](#sim-retrieval-lambda) will send an SQS message to disable the SIM. SIM is being disabled by setting the certificate in DynamoDB as `a: false`. When onboarding will be called for such SIM card a message `{"message":"Device with the IP=1.2.3.4 is not active"}$` with 404 status code will be returned. +Q: My HTTPS setup was done with success but it stopped to work. +A: It's a manual process and if the EC2 server shuts down for any reason, autoscaling will replace the old instance with a new one. Once this new instance is live, the HTTPS process must be repeated. + # Asking for help The most effective communication with our team is through GitHub. Simply create a [new issue](https://github.com/1NCE-GmbH/1nce-iot-device-onboarding/issues/new/choose) and select from a range of templates covering bug reports, feature requests, documentation issues, or General Questions. diff --git a/applications/src/create-sim/create-sim.spec.ts b/applications/src/create-sim/create-sim.spec.ts index 9b6e1af..306f987 100644 --- a/applications/src/create-sim/create-sim.spec.ts +++ b/applications/src/create-sim/create-sim.spec.ts @@ -54,7 +54,7 @@ describe("Create SIM", () => { beforeEach(() => { jest.resetAllMocks(); - iotCertificate = new IoTCoreCertificate({ id: "id", arn: "arn", pem: "pem", privateKey: "private-key" }); + iotCertificate = new IoTCoreCertificate({ id: "cert-id", arn: "arn", pem: "pem", privateKey: "private-key" }); iotThing = new IoTCoreThing({ name: "123456789" }); }); @@ -62,6 +62,7 @@ describe("Create SIM", () => { PK: "IP#10.0.0.0", SK: "P#MQTT", crt: "pem", + crtid: "cert-id", ct: mockDate.toISOString(), ut: mockDate.toISOString(), i: "123456789", @@ -132,6 +133,7 @@ describe("Create SIM", () => { iccid: "123456789", ip: "10.0.0.0", certificate: "pem", + certificateId: "cert-id", privateKey: "private-key", active: true, createdTime: mockDate, @@ -142,6 +144,7 @@ describe("Create SIM", () => { iccid: "123456789", ip: "10.0.0.0", certificate: "pem", + certificateId: "cert-id", privateKey: "private-key", active: true, createdTime: mockDate, @@ -170,6 +173,7 @@ describe("Create SIM", () => { iccid: "123456789", ip: "10.0.0.0", certificate: "pem", + certificateId: "cert-id", privateKey: "private-key", active: true, createdTime: mockDate, @@ -181,6 +185,7 @@ describe("Create SIM", () => { iccid: "123456789", ip: "10.0.0.0", certificate: "pem", + certificateId: "cert-id", privateKey: "private-key", active: true, createdTime: mockDate, @@ -213,6 +218,7 @@ describe("Create SIM", () => { iccid: "123456789", ip: "10.0.0.0", certificate: "pem", + certificateId: "cert-id", privateKey: "private-key", active: true, createdTime: mockDate, @@ -225,6 +231,7 @@ describe("Create SIM", () => { iccid: "123456789", ip: "10.0.0.0", certificate: "pem", + certificateId: "cert-id", privateKey: "private-key", active: true, createdTime: mockDate, @@ -259,6 +266,7 @@ describe("Create SIM", () => { iccid: "123456789", ip: "10.0.0.0", certificate: "pem", + certificateId: "cert-id", privateKey: "private-key", active: true, createdTime: mockDate, @@ -271,6 +279,7 @@ describe("Create SIM", () => { iccid: "123456789", ip: "10.0.0.0", certificate: "pem", + certificateId: "cert-id", privateKey: "private-key", active: true, createdTime: mockDate, @@ -303,6 +312,7 @@ describe("Create SIM", () => { iccid: "123456789", ip: "10.0.0.0", certificate: "pem", + certificateId: "cert-id", privateKey: "private-key", active: true, createdTime: mockDate, @@ -312,6 +322,7 @@ describe("Create SIM", () => { iccid: "123456789", ip: "10.0.0.0", certificate: "pem", + certificateId: "cert-id", privateKey: "private-key", active: true, createdTime: mockDate, @@ -325,6 +336,7 @@ describe("Create SIM", () => { iccid: "123456789", ip: "10.0.0.0", certificate: "pem", + certificateId: "cert-id", privateKey: "private-key", active: true, createdTime: mockDate, @@ -337,6 +349,7 @@ describe("Create SIM", () => { function mockCertificateInstance(): void { const mockIoTCoreCertInstance = mockIoTCoreCert.mock.instances[0]; mockIoTCoreCertInstance.attachPolicy = mockIoTCoreCertAttachPolicy; + mockIoTCoreCertInstance.id = "cert-id"; mockIoTCoreCertInstance.arn = "arn"; mockIoTCoreCertInstance.certificate = "pem"; mockIoTCoreCertInstance.privateKey = "private-key"; diff --git a/applications/src/create-sim/create-sim.ts b/applications/src/create-sim/create-sim.ts index a818556..32baaf1 100644 --- a/applications/src/create-sim/create-sim.ts +++ b/applications/src/create-sim/create-sim.ts @@ -39,6 +39,7 @@ async function handleSQSRecord(record: SQSRecord): Promise { ip: body.ip, active: true, certificate: iotCoreCertificate.certificate, + certificateId: iotCoreCertificate.id, privateKey: iotCoreCertificate.privateKey, }); diff --git a/applications/src/device-onboarding/device-onboarding.spec.ts b/applications/src/device-onboarding/device-onboarding.spec.ts index a6a40e0..94ec689 100644 --- a/applications/src/device-onboarding/device-onboarding.spec.ts +++ b/applications/src/device-onboarding/device-onboarding.spec.ts @@ -120,6 +120,7 @@ describe("Device Onboarding", () => { ip: "10.0.0.1", active: false, certificate: "aa", + certificateId: "cert-id", privateKey: "PK", })); const result = await handler(sampleEvent, sampleContext); @@ -146,6 +147,7 @@ describe("Device Onboarding", () => { ip: "10.0.0.1", active: true, certificate: "aa", + certificateId: "cert-id", privateKey: "PK", })); const result = await handler(sampleEvent, sampleContext); @@ -178,6 +180,7 @@ describe("Device Onboarding", () => { ip: "10.0.0.1", active: true, certificate: "aa", + certificateId: "cert-id", privateKey: "PK", })); const result = await handler(sampleEvent, sampleContext); diff --git a/applications/src/disable-sim/disable-sim.spec.ts b/applications/src/disable-sim/disable-sim.spec.ts index a9d31b1..3f84007 100644 --- a/applications/src/disable-sim/disable-sim.spec.ts +++ b/applications/src/disable-sim/disable-sim.spec.ts @@ -45,6 +45,7 @@ describe("Disable SIM", () => { updatedTime: mockDate.toISOString(), active: false, certificate: "pem", + certificateId: "cert-id", privateKey: "private-key", }); @@ -117,6 +118,7 @@ describe("Disable SIM", () => { iccid: "123456789", ip: "10.0.0.0", certificate: "pem", + certificateId: "cert-id", privateKey: "private-key", active: false, createdTime: mockDate, diff --git a/applications/src/shared/services/errorHandlingService.spec.ts b/applications/src/shared/services/errorHandlingService.spec.ts index b5d856d..1e94c4a 100644 --- a/applications/src/shared/services/errorHandlingService.spec.ts +++ b/applications/src/shared/services/errorHandlingService.spec.ts @@ -93,6 +93,7 @@ describe("Error service", () => { iccid: "123456789", active: true, certificate: "pem", + certificateId: "cert-id", privateKey: "private-key", })); diff --git a/applications/src/shared/services/managementApiService.spec.ts b/applications/src/shared/services/managementApiService.spec.ts index 7cd0dce..f032c96 100644 --- a/applications/src/shared/services/managementApiService.spec.ts +++ b/applications/src/shared/services/managementApiService.spec.ts @@ -144,11 +144,11 @@ describe("Management API Service", () => { const result = await getAllSims("JWT_TOKEN"); expect(result).toStrictEqual([ - new SIM({ iccid: "1111111111", ip: "10.0.0.1", active: true, certificate: "", privateKey: "" }), - new SIM({ iccid: "2222222222", ip: "10.0.0.2", active: true, certificate: "", privateKey: "" }), - new SIM({ iccid: "3333333333", ip: "10.0.0.3", active: true, certificate: "", privateKey: "" }), - new SIM({ iccid: "4444444444", ip: "10.0.0.4", active: true, certificate: "", privateKey: "" }), - new SIM({ iccid: "5555555555", ip: "10.0.0.5", active: true, certificate: "", privateKey: "" }), + new SIM({ iccid: "1111111111", ip: "10.0.0.1", active: true, certificate: "", certificateId: "", privateKey: "" }), + new SIM({ iccid: "2222222222", ip: "10.0.0.2", active: true, certificate: "", certificateId: "", privateKey: "" }), + new SIM({ iccid: "3333333333", ip: "10.0.0.3", active: true, certificate: "", certificateId: "", privateKey: "" }), + new SIM({ iccid: "4444444444", ip: "10.0.0.4", active: true, certificate: "", certificateId: "", privateKey: "" }), + new SIM({ iccid: "5555555555", ip: "10.0.0.5", active: true, certificate: "", certificateId: "", privateKey: "" }), ]); expect(mockGetAxios).toHaveBeenCalledTimes(2); expect(console.log).toHaveBeenNthCalledWith(1, "Retrieving SIMs. Page: 1"); diff --git a/applications/src/shared/services/managementApiService.ts b/applications/src/shared/services/managementApiService.ts index 204e64b..17e7834 100644 --- a/applications/src/shared/services/managementApiService.ts +++ b/applications/src/shared/services/managementApiService.ts @@ -72,6 +72,7 @@ async function getSimsPerPage(authToken: string, page: number, pageSize: number) ip: simData.ip_address, active: true, certificate: "", + certificateId: "", privateKey: "", })), totalPages: parseInt(response.headers["x-total-pages"]), diff --git a/applications/src/shared/services/simService.spec.ts b/applications/src/shared/services/simService.spec.ts index 9e0e4d9..9bdbdb7 100644 --- a/applications/src/shared/services/simService.spec.ts +++ b/applications/src/shared/services/simService.spec.ts @@ -5,14 +5,14 @@ import { marshall } from "@aws-sdk/util-dynamodb"; import { mocked } from "jest-mock"; import { NotFoundError } from "../types/error"; import { SIM } from "../types/sim"; -import { getItem, query, updateItem } from "../utils/dynamoHelper"; +import { getItem, scan, updateItem } from "../utils/dynamoHelper"; import { disableSim, getDbSimByIp, getDbSims } from "./simService"; jest.mock("../utils/dynamoHelper"); jest.mock("../utils/awsIotCoreHelper"); jest.mock("../utils/snsHelper"); -const mockQuery = mocked(query); +const mockScan = mocked(scan); const mockGetItem = mocked(getItem); const mockUpdateItem = mocked(updateItem); console.error = jest.fn(); @@ -36,7 +36,7 @@ describe("SIM Service", () => { describe("getDbSims", () => { it("should return SIM instances array", async () => { - mockQuery.mockResolvedValueOnce([ + mockScan.mockResolvedValueOnce([ { PK: { S: "" }, SK: { S: "" }, @@ -46,6 +46,7 @@ describe("SIM Service", () => { ut: { S: "2023-02-01T00:00:00.000Z" }, a: { BOOL: true }, crt: { S: "certificate" }, + crtid: { S: "cert-id" }, prk: { S: "private_key" }, }, { @@ -57,6 +58,7 @@ describe("SIM Service", () => { ut: { S: "2023-02-01T00:00:00.000Z" }, a: { BOOL: true }, crt: { S: "certificate" }, + crtid: { S: "cert-id" }, prk: { S: "private_key" }, }, ]); @@ -71,6 +73,7 @@ describe("SIM Service", () => { updatedTime: "2023-02-01T00:00:00.000Z", active: true, certificate: "certificate", + certificateId: "cert-id", privateKey: "private_key", }), new SIM({ @@ -80,14 +83,15 @@ describe("SIM Service", () => { updatedTime: "2023-02-01T00:00:00.000Z", active: true, certificate: "certificate", + certificateId: "cert-id", privateKey: "private_key", }), ]); - expect(mockQuery).toHaveBeenCalledWith({ TableName: "SIMS_TABLE" }); + expect(mockScan).toHaveBeenCalledWith({ TableName: "SIMS_TABLE" }); }); it("should return empty array when query result is null", async () => { - mockQuery.mockResolvedValueOnce(undefined); + mockScan.mockResolvedValueOnce(undefined); const result = await getDbSims(); @@ -95,7 +99,7 @@ describe("SIM Service", () => { }); it("should throw error when the database query fails", async () => { - mockQuery.mockRejectedValueOnce("Database error"); + mockScan.mockRejectedValueOnce("Database error"); try { await getDbSims(); @@ -119,6 +123,7 @@ describe("SIM Service", () => { ut: { S: "2023-02-01T00:00:00.000Z" }, a: { B: true }, crt: { S: "certificate" }, + crtid: { S: "cert-id" }, prk: { S: "private_key" }, }); @@ -132,6 +137,7 @@ describe("SIM Service", () => { updatedTime: "2023-02-01T00:00:00.000Z", active: true, certificate: "certificate", + certificateId: "cert-id", privateKey: "private_key", }), ); @@ -264,6 +270,7 @@ describe("SIM Service", () => { PK: "IP#10.0.0.0", SK: "P#MQTT", crt: "pem", + crtid: "cert-id", ct: mockDate.toISOString(), ut: mockDate.toISOString(), i: "123456789", @@ -278,6 +285,7 @@ describe("SIM Service", () => { expect(sim).toStrictEqual(new SIM({ active: false, certificate: "pem", + certificateId: "cert-id", privateKey: "private-key", iccid: "123456789", ip: "10.0.0.0", diff --git a/applications/src/shared/services/simService.ts b/applications/src/shared/services/simService.ts index 27d5d05..1b21635 100644 --- a/applications/src/shared/services/simService.ts +++ b/applications/src/shared/services/simService.ts @@ -1,6 +1,6 @@ import { marshall, unmarshall } from "@aws-sdk/util-dynamodb"; import { fromItem, keyFromIp, type IDynamoSim, type SIM } from "../types/sim"; -import { getItem, query, updateItem } from "../utils/dynamoHelper"; +import { getItem, scan, updateItem } from "../utils/dynamoHelper"; import { ConditionalCheckFailedException, type UpdateItemCommandInput, type GetItemCommandInput } from "@aws-sdk/client-dynamodb"; import { NotFoundError } from "../types/error"; @@ -8,7 +8,7 @@ const SIMS_TABLE = process.env.SIMS_TABLE as string; export async function getDbSims(): Promise { try { - const dbSims = await query({ TableName: SIMS_TABLE }); + const dbSims = await scan({ TableName: SIMS_TABLE }); return (dbSims ?? []).map(sim => fromItem(unmarshall(sim) as IDynamoSim)); } catch (error) { diff --git a/applications/src/shared/services/successMessageService.spec.ts b/applications/src/shared/services/successMessageService.spec.ts index 2f28fbc..cdd84c4 100644 --- a/applications/src/shared/services/successMessageService.spec.ts +++ b/applications/src/shared/services/successMessageService.spec.ts @@ -44,6 +44,7 @@ describe("Success Message Service", () => { ip: "10.0.0.0", active: true, certificate: "pem", + certificateId: "cert-id", privateKey: "private-key", }), "enabled"); @@ -68,6 +69,7 @@ describe("Success Message Service", () => { ip: "10.0.0.0", active: true, certificate: "pem", + certificateId: "cert-id", privateKey: "private-key", }), "enabled"); @@ -95,6 +97,7 @@ describe("Success Message Service", () => { ip: "10.0.0.0", active: true, certificate: "pem", + certificateId: "cert-id", privateKey: "private-key", }), "enabled"); @@ -119,6 +122,7 @@ describe("Success Message Service", () => { ip: "10.0.0.0", active: true, certificate: "pem", + certificateId: "cert-id", privateKey: "private-key", }), "enabled"); diff --git a/applications/src/shared/types/sim.spec.ts b/applications/src/shared/types/sim.spec.ts index cf14216..727aea6 100644 --- a/applications/src/shared/types/sim.spec.ts +++ b/applications/src/shared/types/sim.spec.ts @@ -12,6 +12,7 @@ describe("SIM", () => { updatedTime: mockDate.toISOString(), active: true, certificate: "pem", + certificateId: "cert-id", privateKey: "private-key", }); @@ -19,6 +20,7 @@ describe("SIM", () => { PK: "IP#10.0.0.0", SK: "P#MQTT", crt: "pem", + crtid: "cert-id", ct: "2023-02-01T00:00:00.000Z", ut: "2023-02-01T00:00:00.000Z", i: "123456789", @@ -48,6 +50,7 @@ describe("SIM", () => { ip: "10.0.0.0", active: true, certificate: "certificate", + certificateId: "cert-id", privateKey: "private_key", }); @@ -65,6 +68,7 @@ describe("SIM", () => { ip: "10.0.0.0", active: true, certificate: "certificate", + certificateId: "cert-id", privateKey: "private_key", }); @@ -100,6 +104,7 @@ describe("SIM", () => { SK: "P#MQTT", a: true, crt: "pem", + crtid: "cert-id", ct: mockDate.toISOString(), ut: mockDate.toISOString(), i: "123456789", diff --git a/applications/src/shared/types/sim.ts b/applications/src/shared/types/sim.ts index ec31c2f..55c4b2d 100644 --- a/applications/src/shared/types/sim.ts +++ b/applications/src/shared/types/sim.ts @@ -11,6 +11,7 @@ export class SIM { updatedTime: Date; active: boolean; certificate: string; + certificateId: string; privateKey: string; constructor(sim: ISim) { @@ -20,6 +21,7 @@ export class SIM { this.updatedTime = sim.updatedTime ? new Date(sim.updatedTime) : new Date(); this.active = sim.active; this.certificate = sim.certificate; + this.certificateId = sim.certificateId; this.privateKey = sim.privateKey; } @@ -40,6 +42,7 @@ export class SIM { PK: `IP#${this.ip}`, SK: "P#MQTT", crt: this.certificate, + crtid: this.certificateId, prk: this.privateKey, ct: this.createdTime.toISOString(), ut: this.updatedTime.toISOString(), @@ -69,6 +72,7 @@ export function fromItem(item: IDynamoSim): SIM { updatedTime: item.ut, active: item.a, certificate: item.crt, + certificateId: item.crtid, privateKey: item.prk, }); } @@ -80,6 +84,7 @@ export interface ISim { updatedTime?: string; active: boolean; certificate: string; + certificateId: string; privateKey: string; } @@ -111,6 +116,8 @@ export interface SimPerPageResults { export interface IDynamoSim extends IDynamoItem { // certificate crt: string; + // certificate id + crtid: string; // private key prk: string; // active diff --git a/applications/src/shared/utils/dynamoHelper.spec.ts b/applications/src/shared/utils/dynamoHelper.spec.ts index a9a8fb3..8872cde 100644 --- a/applications/src/shared/utils/dynamoHelper.spec.ts +++ b/applications/src/shared/utils/dynamoHelper.spec.ts @@ -1,7 +1,7 @@ import { DynamoDB, ScanCommand, PutItemCommand, GetItemCommand, UpdateItemCommand } from "@aws-sdk/client-dynamodb"; import { marshall } from "@aws-sdk/util-dynamodb"; import { mockClient } from "aws-sdk-client-mock"; -import { query, putItem, getItem, updateItem } from "../utils/dynamoHelper"; +import { scan, putItem, getItem, updateItem } from "../utils/dynamoHelper"; console.log = jest.fn(); @@ -13,23 +13,24 @@ describe("dynamoHelper", () => { jest.resetAllMocks(); }); - describe("query", () => { - it("should query and return items", async () => { + describe("scan", () => { + it("should scan and return items with pagination", async () => { const params = { TableName: "test", Key: { PK: "PK", }, + Limit: 1, }; DynamoDbClientMock.on( ScanCommand, params, - ).resolves({ Items: [{ test: 1 } as any] }); + ).resolves({ Items: [{ test: 1, item: 2 } as any] }); - const res = await query(params); - expect(res).toStrictEqual([{ test: 1 }]); - expect(console.log).toHaveBeenCalledWith("Querying dynamo items", params); + const res = await scan(params); + expect(res).toStrictEqual([{ test: 1, item: 2 }]); + expect(console.log).toHaveBeenCalledWith("Scanning dynamo items", params); }); }); diff --git a/applications/src/shared/utils/dynamoHelper.ts b/applications/src/shared/utils/dynamoHelper.ts index 73b7e80..6bb5c17 100644 --- a/applications/src/shared/utils/dynamoHelper.ts +++ b/applications/src/shared/utils/dynamoHelper.ts @@ -10,18 +10,32 @@ import { type GetItemCommandInput, type UpdateItemCommandInput, type UpdateItemCommandOutput, + type AttributeValue, } from "@aws-sdk/client-dynamodb"; const DYNAMO_DB_VERSION = "2012-08-10"; const dynamoDb = new DynamoDB({ apiVersion: DYNAMO_DB_VERSION }); -export async function query(params: QueryInput): Promise { - console.log("Querying dynamo items", params); +export async function scan(params: QueryInput): Promise { + console.log("Scanning dynamo items", params); + let result, exclusiveStartKey; + const accumulated = new Array>(); - const command = new ScanCommand(params); - const result = await dynamoDb.send(command); - return result.Items; + do { + params.ExclusiveStartKey = exclusiveStartKey; + + const command = new ScanCommand(params); + result = await dynamoDb.send(command); + + exclusiveStartKey = result.LastEvaluatedKey; + if (result.Items) { + accumulated.push(...result.Items); + } + console.log(`Scanned ${accumulated.length} DB items`); + } while (result.LastEvaluatedKey); + + return accumulated; } export async function getItem(params: GetItemCommandInput): Promise { diff --git a/applications/src/sim-retrieval/sim-retrieval.spec.ts b/applications/src/sim-retrieval/sim-retrieval.spec.ts index 89c3680..baa3ff2 100644 --- a/applications/src/sim-retrieval/sim-retrieval.spec.ts +++ b/applications/src/sim-retrieval/sim-retrieval.spec.ts @@ -63,6 +63,7 @@ describe("SIM Retrieval", () => { ip: "10.0.0.1", active: true, certificate: "certificate", + certificateId: "cert-id", privateKey: "private_key", }), ]); @@ -87,6 +88,7 @@ describe("SIM Retrieval", () => { ip: "10.0.0.1", active: true, certificate: "certificate", + certificateId: "cert-id", privateKey: "private_key", }), ]); @@ -96,6 +98,7 @@ describe("SIM Retrieval", () => { ip: "10.0.0.1", active: true, certificate: "certificate", + certificateId: "cert-id", privateKey: "private_key", }), ]); @@ -119,6 +122,7 @@ describe("SIM Retrieval", () => { ip: "10.0.0.1", active: true, certificate: "certificate", + certificateId: "cert-id", privateKey: "private_key", }), new SIM({ @@ -126,6 +130,7 @@ describe("SIM Retrieval", () => { ip: "10.0.0.2", active: true, certificate: "certificate", + certificateId: "cert-id", privateKey: "private_key", }), ]); @@ -135,6 +140,7 @@ describe("SIM Retrieval", () => { ip: "10.0.0.1", active: true, certificate: "certificate", + certificateId: "cert-id", privateKey: "private_key", }), ]); @@ -148,6 +154,7 @@ describe("SIM Retrieval", () => { ip: "10.0.0.2", active: true, certificate: "certificate", + certificateId: "cert-id", privateKey: "private_key", }).buildSqsMessageEntry(), ]); @@ -167,6 +174,7 @@ describe("SIM Retrieval", () => { ip: "10.0.0.1", active: true, certificate: "certificate", + certificateId: "cert-id", privateKey: "private_key", }), new SIM({ @@ -174,6 +182,7 @@ describe("SIM Retrieval", () => { ip: "10.0.0.2", active: true, certificate: "certificate", + certificateId: "cert-id", privateKey: "private_key", }), ]); @@ -183,6 +192,7 @@ describe("SIM Retrieval", () => { ip: "10.0.0.1", active: true, certificate: "certificate", + certificateId: "cert-id", privateKey: "private_key", }), new SIM({ @@ -190,6 +200,7 @@ describe("SIM Retrieval", () => { ip: "10.0.0.2", active: true, certificate: "certificate", + certificateId: "cert-id", privateKey: "private_key", }), new SIM({ @@ -197,6 +208,7 @@ describe("SIM Retrieval", () => { ip: "10.0.0.3", active: true, certificate: "certificate", + certificateId: "cert-id", privateKey: "private_key", }), ]); @@ -210,6 +222,7 @@ describe("SIM Retrieval", () => { ip: "10.0.0.3", active: true, certificate: "certificate", + certificateId: "cert-id", privateKey: "private_key", }).buildSqsMessageEntry(), ]); @@ -229,6 +242,7 @@ describe("SIM Retrieval", () => { ip: "10.0.0.1", active: true, certificate: "certificate", + certificateId: "cert-id", privateKey: "private_key", }), new SIM({ @@ -236,6 +250,7 @@ describe("SIM Retrieval", () => { ip: "10.0.0.2", active: true, certificate: "certificate", + certificateId: "cert-id", privateKey: "private_key", }), ]); @@ -245,6 +260,7 @@ describe("SIM Retrieval", () => { ip: "10.0.0.1", active: true, certificate: "certificate", + certificateId: "cert-id", privateKey: "private_key", }), new SIM({ @@ -252,6 +268,7 @@ describe("SIM Retrieval", () => { ip: "10.0.0.2", active: true, certificate: "certificate", + certificateId: "cert-id", privateKey: "private_key", }), new SIM({ @@ -259,6 +276,7 @@ describe("SIM Retrieval", () => { ip: "10.0.0.3", active: false, certificate: "certificate", + certificateId: "cert-id", privateKey: "private_key", }), ]); @@ -282,6 +300,7 @@ describe("SIM Retrieval", () => { ip: "10.0.0.1", active: true, certificate: "certificate", + certificateId: "cert-id", privateKey: "private_key", }), new SIM({ @@ -289,6 +308,7 @@ describe("SIM Retrieval", () => { ip: "10.0.0.2", active: true, certificate: "certificate", + certificateId: "cert-id", privateKey: "private_key", }), ]); @@ -298,6 +318,7 @@ describe("SIM Retrieval", () => { ip: "10.0.0.1", active: false, certificate: "certificate", + certificateId: "cert-id", privateKey: "private_key", }), new SIM({ @@ -305,6 +326,7 @@ describe("SIM Retrieval", () => { ip: "10.0.0.2", active: true, certificate: "certificate", + certificateId: "cert-id", privateKey: "private_key", }), ]); @@ -318,6 +340,7 @@ describe("SIM Retrieval", () => { ip: "10.0.0.1", active: true, certificate: "certificate", + certificateId: "cert-id", privateKey: "private_key", }).buildSqsMessageEntry(), ]); @@ -337,6 +360,7 @@ describe("SIM Retrieval", () => { ip: "10.0.0.1", active: true, certificate: "certificate", + certificateId: "cert-id", privateKey: "private_key", }), new SIM({ @@ -344,6 +368,7 @@ describe("SIM Retrieval", () => { ip: "10.0.0.2", active: true, certificate: "certificate", + certificateId: "cert-id", privateKey: "private_key", }), new SIM({ @@ -351,6 +376,7 @@ describe("SIM Retrieval", () => { ip: "10.0.0.4", active: true, certificate: "certificate", + certificateId: "cert-id", privateKey: "private_key", }), ]); @@ -360,6 +386,7 @@ describe("SIM Retrieval", () => { ip: "10.0.0.1", active: true, certificate: "certificate", + certificateId: "cert-id", privateKey: "private_key", }), new SIM({ @@ -367,6 +394,7 @@ describe("SIM Retrieval", () => { ip: "10.0.0.2", active: true, certificate: "certificate", + certificateId: "cert-id", privateKey: "private_key", }), new SIM({ @@ -374,6 +402,7 @@ describe("SIM Retrieval", () => { ip: "10.0.0.3", active: true, certificate: "certificate", + certificateId: "cert-id", privateKey: "private_key", }), ]); @@ -387,6 +416,7 @@ describe("SIM Retrieval", () => { ip: "10.0.0.4", active: true, certificate: "certificate", + certificateId: "cert-id", privateKey: "private_key", }).buildSqsMessageEntry(), ]); @@ -396,6 +426,7 @@ describe("SIM Retrieval", () => { ip: "10.0.0.3", active: true, certificate: "certificate", + certificateId: "cert-id", privateKey: "private_key", }).buildSqsMessageEntry(), ]); @@ -415,6 +446,7 @@ describe("SIM Retrieval", () => { ip: "10.0.0.1", active: true, certificate: "certificate", + certificateId: "cert-id", privateKey: "private_key", }), ]); @@ -431,6 +463,7 @@ describe("SIM Retrieval", () => { ip: "10.0.0.1", active: true, certificate: "certificate", + certificateId: "cert-id", privateKey: "private_key", }).buildSqsMessageEntry(), ]); diff --git a/deploymentValues.yaml b/deploymentValues.yaml index 74dd7f3..01b1b8e 100644 --- a/deploymentValues.yaml +++ b/deploymentValues.yaml @@ -5,6 +5,8 @@ proxyServerSSMParamName: openvpn-onboarding-proxy-server breakoutRegionSSMParamName: breakout-region openVPNCredentialsSecretName: open-source-device-onboarding-openvpn-credentials onboardingApiKeyName: device-onboarding-key +snsSuccessTopicName: onboarding-success +snsFailureTopicName: onboarding-failure dev: codeBaseBucket: device-onboarding-dev-cloudformation-templates diff --git a/scripts/build.sh b/scripts/build.sh index b167056..fdeecaa 100644 --- a/scripts/build.sh +++ b/scripts/build.sh @@ -40,6 +40,8 @@ proxy_server_ssm_param_name=$(yq '.proxyServerSSMParamName' deploymentValues.yam breakout_region_ssm_param_name=$(yq '.breakoutRegionSSMParamName' deploymentValues.yaml) openvpn_creds_secret_name=$(yq '.openVPNCredentialsSecretName' deploymentValues.yaml) onboarding_api_key_name=$(yq '.onboardingApiKeyName' deploymentValues.yaml) +sns_failure_topic_name=$(yq '.snsFailureTopicName' deploymentValues.yaml) +sns_success_topic_name=$(yq '.snsSuccessTopicName' deploymentValues.yaml) nginx_port=8080 cfn_codebase_bucket=$(myenv=$1 yq '.[env(myenv)].codeBaseBucket' deploymentValues.yaml) if [ -z "$cfn_codebase_bucket" ]; then @@ -81,9 +83,10 @@ sed -i s,replace_with_ssm_param_name_to_breakout_region,"$breakout_region_ssm_pa sed -i s,replace_with_secret_name_to_openvpn_creds,"$openvpn_creds_secret_name",g build/ec2-user-data.bash sed -i s,replace_with_onboarding_api_key_name,"$onboarding_api_key_name",g build/ec2-user-data.bash sed -i s,replace_with_nginx_port,"$nginx_port",g build/ec2-user-data.bash +sed -i s,replace_with_sns_success_topic_name,"$sns_success_topic_name",g build/ec2-user-data.bash +sed -i s,replace_with_sns_failure_topic_name,"$sns_failure_topic_name",g build/ec2-user-data.bash echo "Replacing values in the device-onboarding-main.yaml template..." -encoded_user_data=$(base64 -w0 build/ec2-user-data.bash) sed -i s,replace_with_version,"$version",g build/device-onboarding-main.yaml sed -i s,replace_with_code_bucket_name,"$cfn_codebase_bucket",g build/device-onboarding-main.yaml sed -i s,replace_with_code_bucket_region_name,"$cfn_codebase_bucket_region",g build/device-onboarding-main.yaml @@ -93,7 +96,12 @@ sed -i s,replace_with_ssm_param_name_to_proxy_server,"$proxy_server_ssm_param_na sed -i s,replace_with_ssm_param_name_to_breakout_region,"$breakout_region_ssm_param_name",g build/device-onboarding-main.yaml sed -i s,replace_with_secret_name_to_openvpn_creds,"$openvpn_creds_secret_name",g build/device-onboarding-main.yaml sed -i s,replace_with_onboarding_api_key_name,"$onboarding_api_key_name",g build/device-onboarding-main.yaml -sed -i s,replace_with_user_data_base_64_script,"$encoded_user_data",g build/device-onboarding-main.yaml +sed -i s,replace_with_sns_success_topic_name,"$sns_success_topic_name",g build/device-onboarding-main.yaml +sed -i s,replace_with_sns_failure_topic_name,"$sns_failure_topic_name",g build/device-onboarding-main.yaml + +echo "Replacing values in the autoscaling.yaml template..." +encoded_user_data=$(base64 -w0 build/ec2-user-data.bash) +sed -i s,replace_with_user_data_base_64_script,"$encoded_user_data",g build/autoscaling.yaml echo "Cleaning the clutter..." rm -rf build/ec2-user-data.bash diff --git a/scripts/ec2-user-data.bash b/scripts/ec2-user-data.bash index 99434d3..7713697 100755 --- a/scripts/ec2-user-data.bash +++ b/scripts/ec2-user-data.bash @@ -1,22 +1,50 @@ #!/bin/bash apt-get update -y apt-get install -y openvpn net-tools unzip curl nginx jq +# Save SNS topic names +region=`curl http://169.254.169.254/latest/meta-data/placement/region` +aws_account=`curl -s http://169.254.169.254/latest/dynamic/instance-identity/document | jq -r .accountId` + +SNS_SUCCESS_TOPIC_ARN=arn:aws:sns:$region:$aws_account:replace_with_sns_success_topic_name +SNS_FAILURE_TOPIC_ARN=arn:aws:sns:$region:$aws_account:replace_with_sns_failure_topic_name +# Function for publishing to SNS regarding failures +FAILURE_FOUND=false +send_sns_failure_on_error () { + res=$? + if [ $res -ne 0 ]; then + echo "Sending SNS Failure message" + FAILURE_FOUND=true + aws sns publish --topic-arn $SNS_FAILURE_TOPIC_ARN --message "{\"message\": \"$1\", \"timestamp\": $EPOCHSECONDS}" + fi +} +# Function For Getting value from SSM parametrs +GLOBAL_SSM_PARAMETER='' +get_ssm_parameter () { + GLOBAL_SSM_PARAMETER=`aws ssm get-parameter --name $1` + send_sns_failure_on_error "EC2 Get SSM parameter $1 failure" +} # Download and install AWS curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip" unzip awscliv2.zip ./aws/install # Get breakout region -breakout_region=$(aws ssm get-parameter --name replace_with_ssm_param_name_to_breakout_region | jq -r '.Parameter.Value') +get_ssm_parameter "replace_with_ssm_param_name_to_breakout_region" +breakout_region=$(echo $GLOBAL_SSM_PARAMETER | jq -r '.Parameter.Value') # Download and save OpenVPN configuration according to breakout region -curl https://replace_with_code_bucket_name.s3.replace_with_code_bucket_region_name.amazonaws.com/replace_with_version/openVpnConfig/$breakout_region.conf -o /etc/openvpn/openvpn-1nce-client.conf +aws s3api get-object --bucket "replace_with_code_bucket_name" --key "replace_with_version/openVpnConfig/$breakout_region.conf" "/etc/openvpn/openvpn-1nce-client.conf" +send_sns_failure_on_error "EC2 Get Breakout Region Config from S3 bucket Failure" # Get and Save OpenVPN credentials file -openvpn_credentials=$(aws secretsmanager get-secret-value --secret-id replace_with_secret_name_to_openvpn_creds | jq -r '.SecretString') +openvpn_credentials_secret=`aws secretsmanager get-secret-value --secret-id replace_with_secret_name_to_openvpn_creds` +send_sns_failure_on_error "EC2 Get Secretsmanager secret replace_with_secret_name_to_openvpn_credsFailure" +openvpn_credentials=$(echo $openvpn_credentials_secret | jq -r '.SecretString') echo -e "$(echo $openvpn_credentials | jq -r '.username')\n$(echo $openvpn_credentials | jq -r '.password')" > /etc/openvpn/credentials.txt # Start OpenVPN as a service sudo systemctl start openvpn@openvpn-1nce-client # Wait for tun0 interface to be present ifc=tun0 -while true; do +loop_iteration=0 +iteration_count=40 +while [ $loop_iteration -le $iteration_count ] ; do ifconfig $ifc res=$? if [ $res -eq 0 ]; then @@ -25,26 +53,53 @@ while true; do fi echo Waiting for interface $ifc sleep 3 + (( loop_iteration++ )) done + +if [ $loop_iteration -gt $iteration_count ]; then + echo "Sending SNS Failure message" + FAILURE_FOUND=true + aws sns publish --topic-arn $SNS_FAILURE_TOPIC_ARN --message "{\"message\": \"EC2 openVpn service failed to init $ifc interface\", \"timestamp\": $EPOCHSECONDS}" +fi + # Get OpenVPN tunnel ip address ip_uncut=`ifconfig $ifc | grep -i netmask` ip_endcut=${ip_uncut#*inet} ip=${ip_endcut% netmask*} # Store OpenVPN tunnel ip address in SSM -onboarding_path=$(aws ssm get-parameter --name replace_with_ssm_param_name_to_onboarding_path | jq -r '.Parameter.Value') +get_ssm_parameter "replace_with_ssm_param_name_to_onboarding_path" +onboarding_path=$(echo $GLOBAL_SSM_PARAMETER | jq -r '.Parameter.Value') + aws ssm put-parameter --name replace_with_ssm_param_name_to_proxy_server --value "$ip:replace_with_nginx_port/$onboarding_path" --type String --overwrite +send_sns_failure_on_error "EC2 put SSM parameter replace_with_ssm_param_name_to_proxy_server failure" # Get and export Nginx config as env variables -export ONBOARDING_ENDPOINT=$(aws ssm get-parameter --name replace_with_ssm_param_name_to_api_gateway_url | jq -r '.Parameter.Value') -export ONBOARDING_X_API_KEY=$(aws apigateway get-api-keys --name-query replace_with_onboarding_api_key_name --include-values | jq -r '.items[0].value') +get_ssm_parameter "replace_with_ssm_param_name_to_api_gateway_url" +export ONBOARDING_ENDPOINT=$(echo $GLOBAL_SSM_PARAMETER | jq -r '.Parameter.Value') + +apigateway_onbarding_x_api_key_response=`aws apigateway get-api-keys --name-query "replace_with_onboarding_api_key_name" --include-values` +send_sns_failure_on_error "EC2 Get apigateway keys replace_with_onboarding_api_key_name failure" +export ONBOARDING_X_API_KEY=$(echo $apigateway_onbarding_x_api_key_response | jq -r '.items[0].value') + export NGINX_PORT=replace_with_nginx_port export DOLLAR="$" # needed to escape nginx built-in env variables # Download Nginx template and set the server -curl https://replace_with_code_bucket_name.s3.replace_with_code_bucket_region_name.amazonaws.com/replace_with_version/nginxConfig/nginx.conf -o /etc/nginx/conf.d/my-template.conf.template +aws s3api get-object --bucket "replace_with_code_bucket_name" --key "replace_with_version/nginxConfig/nginx.conf" "/etc/nginx/conf.d/my-template.conf.template" +send_sns_failure_on_error "EC2 Get Nginx Config from S3 bucket Failure" envsubst < /etc/nginx/conf.d/my-template.conf.template > /etc/nginx/sites-available/default sudo chmod 644 /etc/nginx/sites-available/default # Run Nginx server as a service sudo systemctl stop nginx sudo systemctl start nginx sudo systemctl enable nginx + +sleep 5 # Give some time to start the service +cat "/var/run/nginx.pid" +send_sns_failure_on_error "EC2 Nginx server failure" + +# Send Success Message to SNS +if [ $FAILURE_FOUND = false ]; then + echo "Sending SNS Success message" + aws sns publish --topic-arn $SNS_SUCCESS_TOPIC_ARN --message "{\"message\": \"EC2 instance for onboarding service configured correctly\", \"timestamp\": $EPOCHSECONDS}" +fi # Reboot machine reboot diff --git a/templates/autoscaling.yaml b/templates/autoscaling.yaml index ef49adb..ccdac53 100644 --- a/templates/autoscaling.yaml +++ b/templates/autoscaling.yaml @@ -23,9 +23,16 @@ Parameters: VPCPrivateSubnetAvailabilityZone: Type: String Description: VPC Private Subnet Availability zone - EC2UserData: + SnsFailureSummaryTopicARN: + Type: String + Description: SNS topic to post summary message on failure case + SnsSuccessSummaryTopicARN: + Type: String + Description: SNS topic to post summary message on success case + S3CodeOriginBucket: + Description: Public S3 bucket. + Default: open-source-device-onboarding Type: String - Description: User Data script Resources: #======================================================= # @@ -72,6 +79,18 @@ Resources: - apigateway:GET Resource: - Fn::Sub: arn:aws:apigateway:${AWS::Region}::/apikeys + - Effect: Allow + Action: + - sns:Publish + Resource: + - Ref: SnsSuccessSummaryTopicARN + - Ref: SnsFailureSummaryTopicARN + - Effect: Allow + Action: + - s3:GetObject + Resource: + - Fn::Sub: arn:aws:s3:::${S3CodeOriginBucket} + - Fn::Sub: arn:aws:s3:::${S3CodeOriginBucket}/* IAMInstanceProfile: Type: AWS::IAM::InstanceProfile @@ -112,8 +131,7 @@ Resources: Properties: LaunchTemplateName: openvpn-onboarding-launch-template LaunchTemplateData: - UserData: - Ref: EC2UserData + UserData: replace_with_user_data_base_64_script IamInstanceProfile: Arn: Fn::GetAtt: diff --git a/templates/device-onboarding-main.yaml b/templates/device-onboarding-main.yaml index 1fbaefc..a98a5a5 100644 --- a/templates/device-onboarding-main.yaml +++ b/templates/device-onboarding-main.yaml @@ -44,6 +44,14 @@ Parameters: - us-east-1 - us-west-1 Description: Breakout region configured in the 1NCE Portal + SNSFailureTopicSubscriptionEmail: + Description: Email for Failure Topic Subscription. Empty string or valid e-mail. + Type: String + AllowedPattern: ^$|[^\s@]+@[^\s@]+\.[^\s@]+ + SNSSuccessTopicSubscriptionEmail: + Description: Email for Success Topic Subscription. Empty string or valid e-mail. Verbose ! + Type: String + AllowedPattern: ^$|[^\s@]+@[^\s@]+\.[^\s@]+ Mappings: #======================================================= # @@ -65,7 +73,8 @@ Mappings: ProxyServerSSMParamName: replace_with_ssm_param_name_to_proxy_server BreakoutRegionSSMParamName: replace_with_ssm_param_name_to_breakout_region OpenVPNCredentialsSecretName: replace_with_secret_name_to_openvpn_creds - UserDataBase64Script: replace_with_user_data_base_64_script + SNSFailureTopicName: replace_with_sns_failure_topic_name + SNSSuccessTopicName: replace_with_sns_success_topic_name Resources: #======================================================= @@ -531,6 +540,21 @@ Resources: SNSResourcesStack: Type: AWS::CloudFormation::Stack Properties: + Parameters: + SNSSuccessTopicName: + Fn::FindInMap: + - Configuration + - BaseConfiguration + - SNSSuccessTopicName + SNSFailureTopicName: + Fn::FindInMap: + - Configuration + - BaseConfiguration + - SNSFailureTopicName + SNSFailureTopicSubscriptionEmail: + Ref: SNSFailureTopicSubscriptionEmail + SNSSuccessTopicSubscriptionEmail: + Ref: SNSSuccessTopicSubscriptionEmail TemplateURL: Fn::Join: - "" @@ -590,11 +614,19 @@ Resources: Fn::GetAtt: - NetworkResourcesStack - Outputs.VPCPrivateSubnetAvailabilityZone - EC2UserData: + SnsFailureSummaryTopicARN: + Fn::GetAtt: + - SNSResourcesStack + - Outputs.SNSFailureTopicArn + SnsSuccessSummaryTopicARN: + Fn::GetAtt: + - SNSResourcesStack + - Outputs.SNSSuccessTopicArn + S3CodeOriginBucket: Fn::FindInMap: - Configuration - BaseConfiguration - - UserDataBase64Script + - CodebaseBucket TemplateURL: Fn::Join: - "" diff --git a/templates/s3-lambda-code.yaml b/templates/s3-lambda-code.yaml index 4934170..23a1b5f 100644 --- a/templates/s3-lambda-code.yaml +++ b/templates/s3-lambda-code.yaml @@ -8,7 +8,6 @@ Parameters: #======================================================= S3CodeOriginBucket: Description: Public S3 bucket. - Default: open-source-device-onboarding Type: String SimRetrievalLambdaZipPath: Description: Zip path for the compressed folder with the sim retrieval lambda code. diff --git a/templates/sim-retrieval.yaml b/templates/sim-retrieval.yaml index eb212af..3e64bc0 100644 --- a/templates/sim-retrieval.yaml +++ b/templates/sim-retrieval.yaml @@ -197,7 +197,7 @@ Resources: Ref: S3LocalBucketName S3Key: Ref: SimRetrievalLambdaZipPath - MemorySize: 128 + MemorySize: 3072 Role: Fn::GetAtt: - LambdaSimRetrievalIAMRole diff --git a/templates/sns.yaml b/templates/sns.yaml index fb7b64d..df6ab00 100644 --- a/templates/sns.yaml +++ b/templates/sns.yaml @@ -1,5 +1,42 @@ AWSTemplateFormatVersion: "2010-09-09" Description: SNS topics for openvpn onboarding notifications +Parameters: + #======================================================= + # + # CloudFormation Parameters + # + #======================================================= + SNSFailureTopicName: + Description: Name for SNS where the onboarding failures will be published + Type: String + SNSSuccessTopicName: + Description: Name for SNS where the onboarding success events will be published + Type: String + SNSFailureTopicSubscriptionEmail: + Description: Email for Failure Topic Subscription. Empty string or valid e-mail. + Type: String + AllowedPattern: ^$|[^\s@]+@[^\s@]+\.[^\s@]+ + SNSSuccessTopicSubscriptionEmail: + Description: Email for Success Topic Subscription. Empty string or valid e-mail. Verbose! + Type: String + AllowedPattern: ^$|[^\s@]+@[^\s@]+\.[^\s@]+ + +Conditions: + #======================================================= + # + # Conditions + # + #======================================================= + SNSFailureTopicSubscriptionEmailExists: + Fn::Not: + - Fn::Equals: + - Ref: SNSFailureTopicSubscriptionEmail + - "" + SNSSuccessTopicSubscriptionEmailExists: + Fn::Not: + - Fn::Equals: + - Ref: SNSSuccessTopicSubscriptionEmail + - "" Resources: #======================================================= # @@ -9,12 +46,34 @@ Resources: SNSFailureTopic: Type: AWS::SNS::Topic Properties: - TopicName: onboarding-failure + TopicName: + Ref: SNSFailureTopicName + + SNSFailureTopicSubscription: + Type: AWS::SNS::Subscription + Properties: + Endpoint: + Ref: SNSFailureTopicSubscriptionEmail + Protocol: email + TopicArn: + Ref: SNSFailureTopic + Condition: SNSFailureTopicSubscriptionEmailExists SNSSuccessTopic: Type: AWS::SNS::Topic Properties: - TopicName: onboarding-success + TopicName: + Ref: SNSSuccessTopicName + + SNSSuccessTopicSubscription: + Type: AWS::SNS::Subscription + Properties: + Endpoint: + Ref: SNSSuccessTopicSubscriptionEmail + Protocol: email + TopicArn: + Ref: SNSSuccessTopic + Condition: SNSSuccessTopicSubscriptionEmailExists SNSTopicPolicy: Type: AWS::SNS::TopicPolicy @@ -83,15 +142,3 @@ Outputs: Fn::GetAtt: - SNSSuccessTopic - TopicArn - SNSFailureTopicName: - Description: Name for SNS where the onboarding failures will be published - Value: - Fn::GetAtt: - - SNSFailureTopic - - TopicName - SNSSuccessTopicName: - Description: Name for SNS where the onboarding success events will be published - Value: - Fn::GetAtt: - - SNSSuccessTopic - - TopicName