Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: OB-36691 add lambda and s3 bucket to record push metrics config #338

Merged
merged 2 commits into from
Oct 17, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion DEVELOPER.md
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@ and the most important variables, run `make help`, e.g:
VARIABLES:
APPS = config configsubscription firehose forwarder logwriter metricstream stack
AWS_REGION = us-west-2
GO_BINS = forwarder subscriber
GO_BINS = forwarder subscriber metricsconfigurator
GO_BUILD_DIRS = bin/linux_arm64 .go/bin/linux_arm64 .go/cache .go/pkg
TF_TESTS = config configsubscription firehose forwarder forwarder_s3 logwriter metricstream simple stack
VERSION = v1.19.2-4-gb1238b5-dirty
Expand Down
Binary file added apps/.DS_Store
Binary file not shown.
146 changes: 140 additions & 6 deletions apps/metricstream/template.yaml
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
---
AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: 'Stream CloudWatch Metrics to S3.'
Metadata:
AWS::ServerlessRepo::Application:
Expand All @@ -16,7 +17,7 @@ Parameters:
BucketArn:
Type: String
Description: >-
S3 Bucket ARN to write log records to.
S3 Bucket ARN to write metrics.
AllowedPattern: "^arn:.*$"
Prefix:
Type: String
Expand All @@ -27,8 +28,8 @@ Parameters:
Type: String
Description: >-
A file hosted in S3 containing list of metrics to stream.
Default: 's3://observeinc/cloudwatchmetrics/filters/recommended.yaml'
AllowedPattern: "^s3:\/\/.*"
Default: 's3://observeinc/cloudwatchmetrics/filters/empty.yaml'
AllowedPattern: "^(s3:\/\/.*)?$"
obs-gh-virjramakrishnan marked this conversation as resolved.
Show resolved Hide resolved
OutputFormat:
Type: String
Description: >-
Expand Down Expand Up @@ -61,11 +62,41 @@ Parameters:
Description: |
Buffer incoming data to the specified size, in MiBs, before delivering it
to the destination.

ObserveAccountID:
obs-gh-virjramakrishnan marked this conversation as resolved.
Show resolved Hide resolved
Type: String
Description: Observe Account ID
AllowedPattern: '\d*'
Default: ''
ObserveDomainName:
Type: String
Description: >-
The domain name we will retrieve metric configuration from.
Default: ''
UpdateTimestamp:
Type: String
Description: Timestamp when the metric stream was created or updated.
Default: ''
AllowedPattern: '^[0-9]*$'
DatasourceID:
Type: String
Description: >-
The datasource for this metric stream.
Default: ''
AllowedPattern: '\d*'
GQLToken:
Type: String
NoEcho: true
Description: >-
The token used to retrieve metric configuration.
Default: ''
Conditions:
UseStackName: !Equals
- !Ref NameOverride
- ''
DeployLambda: !Not
- !Equals
- !Ref DatasourceID
obs-gh-virjramakrishnan marked this conversation as resolved.
Show resolved Hide resolved
- ""

Resources:
DeliveryStreamRole:
Expand Down Expand Up @@ -106,6 +137,27 @@ Resources:
Resource:
- !Ref BucketArn
- !Sub '${BucketArn}/${Prefix}*'
LambdaLogGroup:
Type: 'AWS::Logs::LogGroup'
Properties:
LogGroupName: !Join
- ''
- - /aws/lambda/
- !If
- UseStackName
- !Ref AWS::StackName
- !Ref NameOverride
- '-'
- !Select [2, !Split ["/", !Ref "AWS::StackId"]]
# This grabs the unique id part of the stack id.
# This part is unique every time the stack is created.
# the reason we have this is to avoid an error
# if the stack is destroyed and created again.
# If the log group name is the same on recreation,
# an error will be thrown.
RetentionInDays: 365
DeletionPolicy: Retain
UpdateReplacePolicy: Retain
LogGroup:
Type: 'AWS::Logs::LogGroup'
Properties:
Expand Down Expand Up @@ -186,9 +238,91 @@ Resources:
Name: "AWS::Include"
Parameters:
Location: !Ref FilterUri
# end of processable content for include
# End of processable content for include.
# AWS::Include will be replaced with the file FilterUri references.
# This macro always runs, so we must always provide it with a valid
# S3 file, even if we overwrite it with a lambda.
# filterURI is decided in the parent stack becauseAWS does not
# allow an !If function inside an AWS::Include.
OutputFormat: !Ref OutputFormat

MetricsConfiguratorRole:
Type: AWS::IAM::Role
Condition: DeployLambda
Properties:
AssumeRolePolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Principal:
Service: lambda.amazonaws.com
Action: sts:AssumeRole
Policies:
- PolicyName: MetricsConfiguratorPolicy
PolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Action:
- cloudwatch:PutMetricStream
Resource: !GetAtt MetricStream.Arn
- Effect: Allow
Action:
- iam:PassRole
Resource: !GetAtt MetricStreamRole.Arn
- Effect: Allow
Action:
- logs:CreateLogGroup
- logs:CreateLogStream
- logs:PutLogEvents
Resource: !GetAtt LambdaLogGroup.Arn
- Effect: Allow
Action:
- secretsmanager:GetSecretValue
Resource: !Ref GQLTokenSecret
MetricsConfigurator:
Type: AWS::Serverless::Function
Condition: DeployLambda
Metadata:
BuildMethod: makefile
Properties:
FunctionName: !If
- UseStackName
- !Ref AWS::StackName
- !Ref NameOverride
Role: !GetAtt MetricsConfiguratorRole.Arn
CodeUri: ../../bin/linux_arm64
Handler: bootstrap
Runtime: provided.al2
Architectures:
- arm64
LoggingConfig:
LogGroup: !Ref LambdaLogGroup
Environment:
Variables:
VERBOSITY: 6
ACCOUNT_ID: !Ref ObserveAccountID
OBSERVE_DOMAIN_NAME: !Ref ObserveDomainName
DATASOURCE_ID: !Ref DatasourceID
SECRET_NAME: !Ref GQLTokenSecret
obs-gh-virjramakrishnan marked this conversation as resolved.
Show resolved Hide resolved
# fields are necessary to update the metric stream
METRIC_STREAM_NAME: !Ref MetricStream
FIREHOSE_ARN: !GetAtt DeliveryStream.Arn
ROLE_ARN: !GetAtt MetricStreamRole.Arn
OUTPUT_FORMAT: !Ref OutputFormat
GQLTokenSecret:
Type: AWS::SecretsManager::Secret
Condition: DeployLambda
Properties:
Description: GQL Token Secret
Name: !Sub "observe-gql-token-${AWS::StackName}"
SecretString: !Ref GQLToken
StackCreationUpdateCustomResource:
Type: Custom::StackCreationUpdateTrigger
Condition: DeployLambda
Properties:
ServiceToken: !GetAtt MetricsConfigurator.Arn
StackName: !Ref AWS::StackName
UpdateTimestamp: !Ref UpdateTimestamp
Outputs:
FirehoseArn:
Description: >-
Expand Down
75 changes: 69 additions & 6 deletions apps/stack/template.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -107,13 +107,53 @@ Parameters:
Type: String
Description: >-
S3 URI containing filters for metrics to be collected by CloudWatch
Metrics Stream. If empty, no metrics will be collected.
Default: 's3://observeinc/cloudwatchmetrics/filters/recommended.yaml'
Metrics Stream. If neither this nor DatasourceID is provided, no metrics
will be collected.
Default: 's3://observeinc/cloudwatchmetrics/filters/empty.yaml'
# the empty.yaml file functions as a sentinel value: it lists a namespace
# that does not exist, and this causes no metrics to be collected.
AllowedPattern: "^(s3:\/\/.*)?$"
ObserveAccountID:
# this and the field below are necessary for us to be able to make a
# request to Observe's GraphQL API - we use them to form the URL we
# make the request to.
Type: String
Description: Observe Account ID
AllowedPattern: '\d*'
Default: ''
ObserveDomainName:
Type: String
Description: >-
The domain name we will retrieve metric configuration from.
Default: ''
UpdateTimestamp:
# Used to trigger a stack update when the lambda is used to update the
# metric stream. The change in timestamp is used to trigger an update
# of the custom resource. Without a change, the custom resource will
# not be updated, and the lambda will not run.
Type: String
Description: Unix timestamp when metric stream was created or updated.
Default: ''
AllowedPattern: "^[0-9]*$"
DatasourceID:
Type: String
Description: >-
The datasource for this metric stream.
AllowedPattern: '\d*'
Default: ''
GQLToken:
# The token for making a request to Observe's
# GraphQL API. It is stored in Secrets Manager.
# noEcho prevents it from being displayed in the console.
Type: String
NoEcho: true
Description: >-
The token used to retrieve metric configuration.
Default: ''
SourceBucketNames:
Type: CommaDelimitedList
Description: >-
A list of bucket names which the forwarder is allowed to read from.
A list of bucket names which the forwarder will read from.
Default: ""
AllowedPattern: "^[a-z0-9-.]*(\\*)?$"
ContentTypeOverrides:
Expand Down Expand Up @@ -161,9 +201,18 @@ Conditions:
UseStackName: !Equals
- !Ref NameOverride
- ""
EnableMetricStream: !Not
EnableMetricStream: !Or
- !Not
- !Equals
- !Ref MetricStreamFilterUri
- ""
- !Not
- !Equals
- !Ref DatasourceID
- ""
DeployLambda: !Not
- !Equals
- !Ref MetricStreamFilterUri
- !Ref DatasourceID
- ""

Resources:
Expand Down Expand Up @@ -310,7 +359,21 @@ Resources:
- !Ref Topic
Parameters:
BucketArn: !GetAtt Bucket.Arn
FilterUri: !Ref MetricStreamFilterUri
FilterUri: !If
- DeployLambda
- 's3://observeinc/cloudwatchmetrics/filters/default.yaml'
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

does the lambda ever use this file? if not does this even need to be a real s3 file?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

also we should write some comment here to explain why this is necessary

- !Ref MetricStreamFilterUri
# We provide a yaml file URI here where the contents are a sentinel
# value that the user would never enter. Then the lambda modifies
# the metricstream to reflect the actual config provided. We need
# this sentinel value here initially so that if we switch back to the
# yaml based approach, the cloudformation stack will recognize that
# the value from the yaml file has changed and will update.
ObserveAccountID: !Ref ObserveAccountID
ObserveDomainName: !Ref ObserveDomainName
UpdateTimestamp: !Ref UpdateTimestamp
GQLToken: !Ref GQLToken
DatasourceID: !Ref DatasourceID
NameOverride: !If
- UseStackName
- !Sub "${AWS::StackName}-MetricStream"
Expand Down
35 changes: 35 additions & 0 deletions cmd/metricsconfigurator/main.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
package main

import (
"context"
"fmt"

awslambda "github.com/aws/aws-lambda-go/lambda"

handler "github.com/observeinc/aws-sam-apps/pkg/handler/metricsconfigurator"
"github.com/observeinc/aws-sam-apps/pkg/lambda"
"github.com/observeinc/aws-sam-apps/pkg/lambda/metricsconfigurator"
)

var (
rec *metricsconfigurator.Lambda
)

func init() {
ctx := context.Background()

var config handler.Config
err := lambda.ProcessEnv(ctx, &config)
if err != nil {
panic(fmt.Errorf("failed to initialize config: %w", err))
}

rec, err = metricsconfigurator.New(ctx, &config)
if err != nil {
panic(fmt.Errorf("failed to configure entrypoint: %w", err))
}
}

func main() {
awslambda.StartWithOptions(rec.Entrypoint, awslambda.WithEnableSIGTERM(rec.Shutdown))
}
10 changes: 6 additions & 4 deletions go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -6,13 +6,15 @@ toolchain go1.22.2

require (
github.com/aws/aws-lambda-go v1.47.0
github.com/aws/aws-sdk-go-v2 v1.30.3
github.com/aws/aws-sdk-go-v2 v1.31.0
github.com/aws/aws-sdk-go-v2/config v1.27.27
github.com/aws/aws-sdk-go-v2/feature/s3/manager v1.17.10
github.com/aws/aws-sdk-go-v2/service/cloudwatch v1.41.2
github.com/aws/aws-sdk-go-v2/service/cloudwatchlogs v1.37.3
github.com/aws/aws-sdk-go-v2/service/s3 v1.58.3
github.com/aws/aws-sdk-go-v2/service/secretsmanager v1.33.4
github.com/aws/aws-sdk-go-v2/service/sqs v1.34.3
github.com/aws/smithy-go v1.20.3
github.com/aws/smithy-go v1.21.0
github.com/go-logr/logr v1.4.2
github.com/google/go-cmp v0.6.0
github.com/hashicorp/go-retryablehttp v0.7.7
Expand All @@ -38,8 +40,8 @@ require (
github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.6.3 // indirect
github.com/aws/aws-sdk-go-v2/credentials v1.17.27 // indirect
github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.16.11 // indirect
github.com/aws/aws-sdk-go-v2/internal/configsources v1.3.15 // indirect
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.6.15 // indirect
github.com/aws/aws-sdk-go-v2/internal/configsources v1.3.18 // indirect
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.6.18 // indirect
github.com/aws/aws-sdk-go-v2/internal/ini v1.8.0 // indirect
github.com/aws/aws-sdk-go-v2/internal/v4a v1.3.15 // indirect
github.com/aws/aws-sdk-go-v2/service/dynamodb v1.34.1 // indirect
Expand Down
Loading