Functionbeat initial PR (#8485) #8678

ph · 2018-10-22T12:32:05Z

This PR provides the following

Plugin infra for developing providers and functions
A local stdin provider only used for testing, I will remove it in the
final version.
AWS provider and function types for:

Cloudwatch logs
SQS
Kinesis
Api web gateway proxy

License checker
Packaging of artifact
Runners
CLI infrastructure
CLI to push a cloudwatch logs function.
CLI to delete any function
Processors support.
Types to validate value from the users and the lambda function.

What it doesn't provides

ECS and full event extraction. (NOT for v1)
Specifying the AWS credentials in the configuration
CLI for SQS, Kinesis, API
Robust CLI interaction with the API, rollback on failure / versioning.
Removal of not supported outputs
Removal of seccomp check
Integration tests
Updated build task to produce containing the user executable beat and
the linux beats.
Concurrency / memory settings

x-pack/functionbeat/provider/aws/cli_manager.go

x-pack/functionbeat/functionbeat.yml

x-pack/functionbeat/provider/aws/op_wait_cloud_formation.go

urso · 2018-10-23T17:58:59Z

Yay for final push.

With testing and other findings from @exekias, I'd prefer if we'd add some more checks to functionbeat so to error out early on invalid names.

ph · 2018-10-23T20:03:03Z

@exekias I've created the following PR to address the issue that you have found in this PR.

#8713

ph · 2018-10-24T17:18:49Z

Blocked by #8729, the PR fixes the make check and also running the tests.

* Beatless initial PR Add a new beat inside the x-pack folder under the Elastic License, minimal requirement changes to have a build and a test running. Main makefile exclude ASL2 for x-pack but check for Elastic. Beats can override the license in their Makefile.

In beat each data collector need to initialize his own beat.Client to have access to the pipeline. The current pipeline implementation is completely asynchronous, meaning when you publish something to the queue, you don't know if it will be send or when it will be send. Some system like aws lambda requires to be in sync, when the method return we expect the events to be send. This PR allow to change the behavior to have a sync publish that leverage the pipeline callbacks. Notes: it also changes the client interface, since publish and publishAll can return an error. Usage: ``` sc, err := NewSyncClient(pipeline, beat.ClientConfig{}) if !assert.NoError(t, err) { return } err := sc.PublishAll() if err != nil { ... } sc.Wait() // block until the publish is done. defer sc.Close() // this call will also block ```

* License manager Implements a License manager inside beats, as we development more features that depends on the licensing and the capabilities of a remote cluster we need a unique way to access that information. This commit implements the following: Add a License manager that can be started at the beginning of the beats instance initialization. The manager takes a fetcher, currently we only support Elasticsearch as the license backend but we could add support for an Logstash endpoint that could proxy the license. Notes: - By default when the manager is started, no license is available, calling `Get()` on the manager will return a license not found. - The manager will periodically retrieve the license from the fetcher. - When an error occurs on the periodic check, the license wont be invalidated right away but will enter a grace period, after this period the license will be invalidated and will replaced by the OSS license. - License and capabilities and be retrieved by calling `Get()` or registering a type implementing the `Watcher` interface.

* Vendor AWS SDK for beatless and the dependencies Vendor: https://github.com/aws/aws-lambda-go/ https://github.com/aws/aws-sdk-go-v2 Adding LICENSE-SUMMARY to the skip files this explain that some part of the code is APLv2 and the other MIT-0 but this is not a license.

Remove aws/lambda Remove aws/cloudwatchlogs add aws/cloudformation add go-jmespath add yaml dependenceis for goformation

* This add beatless/functionbeat to the build processes and do the following changes: Use a custom packages.yml to only build some of the artifact. Each distribution also include a linux binary to be run on AWS lambda Some refactoring is done in the mage.go file to allow to specify a specs file more easily. Make sure that beatless is added to build/distributions/beatless * fix the issue with license on the fields.go * Adding minimal docs to satisfy DTD requirements and not fail the build

## This PR provides the following 1. Plugin infra for developing providers and functions 2. A local stdin provider only used for testing, I will remove it in the final version. 3. AWS provider and function types for: - Cloudwatch logs - SQS - Kinesis - Api web gateway proxy 4. License checker 5. Packaging of artifact 6. Runners 7. CLI infrastructure 8. CLI to push a cloudwatch logs function. 9. CLI to delete any function 10. Processors support. 11. Types to validate value from the users and the lambda function. ## What it doesn't provides - ECS and full event extraction. (NOT for v1) - Specifying the AWS credentials in the configuration - CLI for SQS, Kinesis, API - Robust CLI interaction with the API, rollback on failure / versioning. - Removal of not supported outputs - Removal of seccomp check - Integration tests - Updated build task to produce containing the user executable beat and the linux beats. - Concurrency / memory settings

Rename beatless for functionbeat, the new name align more to what this beat does and also as the same naming scheme as the other beat by terminating by `beat`. --- ## Tested - Mage package created the right artifact. - Content of packages are now functionbeat - Configuration was updated to reflect functionbeat top level keys. - Deploying a function works with the change.

When you are pushing a cloudformation template and no resources has changed but the artifact with the configuration is different, cloudformation will not publish the new method. To go around that, each time that the package is pushed to S3 when change the URL of the template, since the stack point to a new URL it will trigger an update to the lambda function.

- Allow users to define the S3 bucket used for artifact - Replace the log group name with valid string - Enforce function name to only contains some chars - Enforce log group name validation - Add better code handling when waiting for cloudformation status.

* Add support for the CLI to push a lambda to monitor SQS queues. This all support to configure triggers in the configuration file to allow a lambda function to monitor one or more queue. Note: we assume that the body of an SQS message is a string, if we receive a JSON encoded string we can use the `decode_json_fields`.

…n generating the fields.go (#8729) * Fix an issue where we were trying to get use a non existing directory when generating the fields.go Functionbeat doesn't have a OSS implementation. * Add an XPACK_ONLY flag so we can skip some commands in libbeat

ph · 2018-10-24T18:30:33Z

Rebuild to includes the latest fields changes from master.

1. Plugin infra for developing providers and functions 2. A local stdin provider only used for testing, I will remove it in the final version. 3. AWS provider and function types for: - Cloudwatch logs - SQS - Kinesis - Api web gateway proxy 4. License checker 5. Packaging of artifact 6. Runners 7. CLI infrastructure 8. CLI to push a cloudwatch logs/sqs function. 9. CLI to delete any function 10. Processors support. 11. Types to validate value from the users and the lambda function. - ECS and full event extraction. (NOT for v1) - Specifying the AWS credentials in the configuration - CLI for SQS, Kinesis, API - Robust CLI interaction with the API, rollback on failure / versioning. - Removal of not supported outputs - Removal of seccomp check - Integration tests - Updated build task to produce containing the user executable beat and the linux beats. - Concurrency / memory settings (cherry picked from commit 3ef0b4e)

ph added review needs_backport PR is waiting to be backported to other branches. Functionbeat labels Oct 22, 2018

ph changed the title ~~Function initial PR (#8485)~~ Functionbeat initial PR (#8485) Oct 22, 2018

ph requested a review from urso October 22, 2018 12:42