-
Notifications
You must be signed in to change notification settings - Fork 4.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Filebeat] Refactor AWS S3 input with workers #27199
Merged
andrewkroh
merged 10 commits into
elastic:master
from
andrewkroh:feature/fb/aws-s3-refactor
Aug 12, 2021
Merged
Changes from 6 commits
Commits
Show all changes
10 commits
Select commit
Hold shift + click to select a range
d9e2eb3
Refactor AWS S3 input with workers
andrewkroh c789583
Use InitializeAWSConfig
andrewkroh 8981559
Add godocs to the internal interfaces between components.
andrewkroh 1059a52
Add s3Lister interface for mocking pagination of S3 ListObjects calls
andrewkroh d04c0f8
Add comment to acker Wait
andrewkroh 652f534
Remove superfluous sync.WaitGroup Wait()
andrewkroh 1317fac
Merge remote-tracking branch 'elastic/master' into feature/fb/aws-s3-…
andrewkroh 8818991
Add new config parameters to reference.yml
andrewkroh b508531
Optimize uploading b/c it was slow in aws v2 sdk
andrewkroh d246c04
Merge branch 'feature/fb/aws-s3-refactor' of github.com:andrewkroh/be…
andrewkroh File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Large diffs are not rendered by default.
Oops, something went wrong.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file was deleted.
Oops, something went wrong.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,3 @@ | ||
| terraform/ | ||
| outputs.yml | ||
| *.tfstate* |
57 changes: 57 additions & 0 deletions
57
x-pack/filebeat/input/awss3/_meta/terraform/.terraform.lock.hcl
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Oops, something went wrong.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,43 @@ | ||
| # Terraform setup for AWS S3 Input Integration Tests | ||
|
|
||
| This directory contains a Terrafrom module that creates the AWS resources needed | ||
| for executing the integration tests for the `aws-s3` Filebeat input. It creates | ||
| an S3 bucket and SQS queue and configures S3 `ObjectCreated:*` notifications to | ||
| be delivered to SQS. | ||
|
|
||
| It outputs configuration information that is consumed by the tests to | ||
| `outputs.yml`. The AWS resources are randomly named to prevent name collisions | ||
| between multiple users. | ||
|
|
||
| ### Usage | ||
|
|
||
| You must have the appropriate AWS environment variables for authentication set | ||
| before running Terraform or the integration tests. The AWS key must be | ||
| authorized to create and destroy S3 buckets and SQS queues. | ||
|
|
||
| 1. Execute terraform in this directory to create the resources. This will also | ||
| write the `outputs.yml`. | ||
|
|
||
| `terraform apply` | ||
|
|
||
| 2. (Optional) View the output configuration. | ||
|
|
||
| ```yaml | ||
| "aws_region": "us-east-1" | ||
| "bucket_name": "filebeat-s3-integtest-8iok1h" | ||
| "queue_url": "https://sqs.us-east-1.amazonaws.com/144492464627/filebeat-s3-integtest-8iok1h" | ||
| ``` | ||
|
|
||
| 2. Execute the integration test. | ||
|
|
||
| ``` | ||
| cd x-pack/filebeat/inputs/awss3 | ||
| go test -tags aws,integration -run TestInputRun -v . | ||
| ``` | ||
|
|
||
| 3. Cleanup AWS resources. Execute terraform to remove the SQS queue and delete | ||
| the S3 bucket and its contents. | ||
|
|
||
| `terraform destroy` | ||
|
|
||
|
|
||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,62 @@ | ||
| terraform { | ||
| required_providers { | ||
| aws = { | ||
| source = "hashicorp/aws" | ||
| version = "~> 3.52" | ||
| } | ||
| } | ||
| } | ||
|
|
||
| provider "aws" { | ||
| region = var.aws_region | ||
| } | ||
|
|
||
| resource "random_string" "random" { | ||
| length = 6 | ||
| special = false | ||
| upper = false | ||
| } | ||
|
|
||
| resource "aws_s3_bucket" "filebeat-integtest" { | ||
| bucket = "filebeat-s3-integtest-${random_string.random.result}" | ||
| force_destroy = true | ||
| } | ||
|
|
||
| resource "aws_sqs_queue" "filebeat-integtest" { | ||
| name = "filebeat-s3-integtest-${random_string.random.result}" | ||
| policy = <<POLICY | ||
| { | ||
| "Version": "2012-10-17", | ||
| "Id": "sqspolicy", | ||
| "Statement": [ | ||
| { | ||
| "Effect": "Allow", | ||
| "Principal": "*", | ||
| "Action": "sqs:SendMessage", | ||
| "Resource": "arn:aws:sqs:*:*:filebeat-s3-integtest-${random_string.random.result}", | ||
| "Condition": { | ||
| "ArnEquals": { "aws:SourceArn": "${aws_s3_bucket.filebeat-integtest.arn}" } | ||
| } | ||
| } | ||
| ] | ||
| } | ||
| POLICY | ||
|
|
||
| depends_on = [ | ||
| aws_s3_bucket.filebeat-integtest, | ||
| ] | ||
| } | ||
|
|
||
| resource "aws_s3_bucket_notification" "bucket_notification" { | ||
| bucket = aws_s3_bucket.filebeat-integtest.id | ||
|
|
||
| queue { | ||
| queue_arn = aws_sqs_queue.filebeat-integtest.arn | ||
| events = ["s3:ObjectCreated:*"] | ||
| } | ||
|
|
||
| depends_on = [ | ||
| aws_s3_bucket.filebeat-integtest, | ||
| aws_sqs_queue.filebeat-integtest, | ||
| ] | ||
| } |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@kaiyan-sheng would this work with our existing AWS ci integration? that would be great.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think we have the aws tests run weekly right now in the CI and this would work. @v1v Do you know if this is easy to add?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The AWS tests run on a weekly basis, as you said, and it runs the commands defined in:
beats/x-pack/metricbeat/Jenkinsfile.yml
Lines 35 to 47 in d898533
mage build testruns in the folderx-pack/metricbeatso, if the above-mentioned command is part of the mage then it should work.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
the test requires applying a terraform file (and destroying it at the end):
https://github.com/elastic/beats/blob/master/x-pack/filebeat/input/awss3/_meta/terraform/README.md#usage
will this happen?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
btw, shouldn't be the file https://github.com/elastic/beats/blob/master/x-pack/filebeat/Jenkinsfile.yml?