Skip to content

Commit

Permalink
fix(logwriter): trigger on update (#319)
Browse files Browse the repository at this point in the history
It is common practice to iteratively adjust parameters in the logwriter stack and
refine filters through trial and error. Previously, log discovery and
subscription would only run once our scheduler fired. This commit
additionally ensures we manually trigger the lambda on any parameter
change to our CloudFormation stack.

The commit adds:
- a custom resource to our CloudFormation logwriter stack which embeds
  all parameters. Any change to our input parameters will generate a
  CloudFormation event which triggers the subscriber Lambda function.
- a handler to the subscriber Lambda function which processes
  CloudFormation events. On stack updates, we will initiate a
  discovery request. We do not need to do any handling on install, since
  the scheduler will fire on first time install.
- a step to our integration tests which updates the stack to exercise
  the new functionality. IAM permissions were added to account for
  new behavior.

CloudFormation triggers are notoriously fussy. Our lambda must respond
promptly, or risk holding up the CloudFormation process for an
inordinately long time. To avoid this fate, this commit:
- is liberal in what it accepts as a CloudFormation event. Any payload
  that has sufficient metadata to generate a response will be
  interpreted as being valid. This ensures we are future proof to any
  schema extensions.
- immediately acknowledges the CloudFormation event before initiating
  discovery in order to not hold up install / update / delete process.
  • Loading branch information
jta authored Jul 12, 2024
1 parent f560523 commit 2007579
Show file tree
Hide file tree
Showing 11 changed files with 401 additions and 3 deletions.
28 changes: 27 additions & 1 deletion apps/logwriter/template.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,7 @@ Metadata:
Parameters:
- LogGroupNamePatterns
- LogGroupNamePrefixes
- ExcludeLogGroupNamePatterns
- DiscoveryRate
- FilterName
- FilterPattern
Expand Down Expand Up @@ -76,7 +77,7 @@ Parameters:
ExcludeLogGroupNamePatterns:
Type: CommaDelimitedList
Description: >-
Comma separated list of patterns. This paramter is used to filter out log
Comma separated list of patterns. This parameter is used to filter out log
groups from subscription, and supports the use of regular expressions.
Default: ''
DiscoveryRate:
Expand Down Expand Up @@ -545,6 +546,31 @@ Resources:
- HasLogGroupNamePrefixes
- !Ref LogGroupNamePrefixes
- []
Trigger:
Type: Custom::Trigger
Condition: HasDiscoveryRate
DependsOn:
- DiscoverySchedule
Properties:
ServiceTimeout: 60
ServiceToken: !GetAtt Subscriber.Arn
# List all parameters here, any change will trigger update
BucketArn: !Ref BucketArn
Prefix: !Ref Prefix
LogGroupNamePatterns: !Ref LogGroupNamePatterns
LogGroupNamePrefixes: !Ref LogGroupNamePrefixes
ExcludeLogGroupNamePatterns: !Ref ExcludeLogGroupNamePatterns
DiscoveryRate: !Ref DiscoveryRate
FilterName: !Ref FilterName
FilterPattern: !Ref FilterPattern
NameOverride: !Ref NameOverride
BufferingInterval: !Ref BufferingInterval
BufferingSize: !Ref BufferingSize
NumWorkers: !Ref NumWorkers
MemorySize: !Ref MemorySize
Timeout: !Ref Timeout
DebugEndpoint: !Ref DebugEndpoint
Verbosity: !Ref Verbosity
Outputs:
FirehoseArn:
Description: >-
Expand Down
2 changes: 1 addition & 1 deletion docs/logwriter.md
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,7 @@ You can optionally enable automatic log group subscription. Configuring either `
| `Prefix` | String | Optional prefix to write log records to. |
| `LogGroupNamePatterns` | CommaDelimitedList | Comma separated list of patterns. We will only subscribe to log groups that have names matching any of the provided strings based on a case-sensitive substring search. See the AWS `DescribeLogGroups` action for more information. To subscribe to all log groups, use the wildcard operator *. |
| `LogGroupNamePrefixes` | CommaDelimitedList | Comma separated list of prefixes. The lambda function will only apply to log groups that start with a provided string. To subscribe to all log groups, use the wildcard operator *. |
| `ExcludeLogGroupNamePatterns` | CommaDelimitedList | Comma separated list of patterns. This paramter is used to filter out log groups from subscription, and supports the use of regular expressions. |
| `ExcludeLogGroupNamePatterns` | CommaDelimitedList | Comma separated list of patterns. This parameter is used to filter out log groups from subscription, and supports the use of regular expressions. |
| `DiscoveryRate` | String | EventBridge rate expression for periodically triggering discovery. If not set, no eventbridge rules are configured. |
| `FilterName` | String | Subscription filter name. Existing filters that have this name as a prefix will be removed. |
| `FilterPattern` | String | Subscription filter pattern. |
Expand Down
22 changes: 22 additions & 0 deletions integration/tests/logwriter.tftest.hcl
Original file line number Diff line number Diff line change
Expand Up @@ -45,10 +45,14 @@ variables {
"lambda:DeleteFunction",
"lambda:GetEventSourceMapping",
"lambda:GetFunction",
"lambda:InvokeFunction",
"lambda:ListEventSourceMappings",
"lambda:ListTags",
"lambda:TagResource",
"lambda:UntagResource",
"lambda:UpdateEventSourceMapping",
"lambda:UpdateFunctionCode",
"lambda:UpdateFunctionConfiguration",
"logs:CreateLogGroup",
"logs:CreateLogStream",
"logs:DeleteLogGroup",
Expand Down Expand Up @@ -130,3 +134,21 @@ run "check_eventbridge_invoked" {
error_message = "Failed to verify subscriber invocation"
}
}

run "update" {
variables {
setup = run.setup
app = "logwriter"
parameters = {
BucketArn = run.create_bucket.arn
LogGroupNamePatterns = "*"
DiscoveryRate = "24 hours"
NameOverride = run.setup.id
Verbosity = 4
}
capabilities = [
"CAPABILITY_IAM",
"CAPABILITY_AUTO_EXPAND",
]
}
}
1 change: 1 addition & 0 deletions integration/tests/stack.tftest.hcl
Original file line number Diff line number Diff line change
Expand Up @@ -64,6 +64,7 @@ variables {
"lambda:GetFunction",
"lambda:GetFunctionCodeSigningConfig",
"lambda:GetRuntimeManagementConfig",
"lambda:InvokeFunction",
"lambda:ListEventSourceMappings",
"lambda:ListTags",
"lambda:TagResource",
Expand Down
91 changes: 91 additions & 0 deletions pkg/handler/subscriber/cloudformation.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,91 @@
package subscriber

import (
"context"
"encoding/json"
"errors"
"fmt"

"github.com/aws/aws-lambda-go/cfn"
"github.com/aws/aws-lambda-go/lambdacontext"
"github.com/go-logr/logr"
)

var (
ErrMalformedEvent = errors.New("malformed cloudformation event")
)

// CloudFormationEvent adds a field which is not declared in cloudformation package
type CloudFormationEvent struct {
*cfn.Event
}

// UnmarshalJSON provides a custom unmarshaller that allows unknown fields.
// It is critical that we respond to any CloudFormation event, otherwise stack
// install, updates and deletes will stall. In order to protect ourselves
// against unexpected fields, we succeed unmarshalling so long as we have the
// necessary elements to form a response.
func (c *CloudFormationEvent) UnmarshalJSON(data []byte) error {
if err := json.Unmarshal(data, &c.Event); err != nil {
return err
}

switch {
case c.RequestID == "":
case c.ResponseURL == "":
case c.LogicalResourceID == "":
case c.StackID == "":
default:
return nil
}

return fmt.Errorf("not a cloudformation event")
}

func makeStrSlice(item any) ([]*string, error) {
vs, ok := item.([]any)
if !ok {
return nil, fmt.Errorf("failed to cast %v to slice", item)
}
var ret []*string
for _, v := range vs {
s, ok := v.(string)
if !ok {
return nil, fmt.Errorf("failed to cast %v to string", v)
}
ret = append(ret, &s)
}
return ret, nil
}

// HandleCloudFormation triggers discovery on CloudFormation stack updates
func (h *Handler) HandleCloudFormation(ctx context.Context, ev *CloudFormationEvent) (*Response, error) {
logger := logr.FromContextOrDiscard(ctx)

if ev == nil {
return &Response{}, nil
}

logger.V(3).Info("handling cloudformation event", "requestType", ev.RequestType)

response := cfn.NewResponse(ev.Event)
response.PhysicalResourceID = lambdacontext.LogStreamName
response.Status = cfn.StatusSuccess
err := response.Send()
if err != nil {
return nil, fmt.Errorf("failed to send cloudformation response: %w", err)
}

if ev.RequestType == cfn.RequestUpdate {
var req DiscoveryRequest
if req.LogGroupNamePatterns, err = makeStrSlice(ev.ResourceProperties["LogGroupNamePatterns"]); err != nil {
return nil, fmt.Errorf("failed to extract logGroupNamePatterns: %w", err)
}
if req.LogGroupNamePrefixes, err = makeStrSlice(ev.ResourceProperties["LogGroupNamePrefixes"]); err != nil {
return nil, fmt.Errorf("failed to extract logGroupNamePrefixes: %w", err)
}
return h.HandleDiscoveryRequest(ctx, &req)
}

return &Response{}, nil
}
2 changes: 1 addition & 1 deletion pkg/lambda/subscriber/lambda.go
Original file line number Diff line number Diff line change
Expand Up @@ -126,7 +126,7 @@ func New(ctx context.Context, cfg *Config) (*Lambda, error) {
Logger: logger,
}

if err := mux.Register(is.HandleRequest, is.HandleSQS); err != nil {
if err := mux.Register(is.HandleRequest, is.HandleSQS, is.HandleCloudFormation); err != nil {
return nil, fmt.Errorf("failed to register functions: %w", err)
}

Expand Down
36 changes: 36 additions & 0 deletions vendor/github.com/aws/aws-lambda-go/cfn/README.md

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

27 changes: 27 additions & 0 deletions vendor/github.com/aws/aws-lambda-go/cfn/event.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

89 changes: 89 additions & 0 deletions vendor/github.com/aws/aws-lambda-go/cfn/response.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading

0 comments on commit 2007579

Please sign in to comment.