[Meta] Add FunctionBeat to run beat on serverless infrastructure

[ph]

Beatless is a new beat designed to beat lightweight and be able to run on existing cloud infrastructure. The first version will focus on AWS. The development of the feature is currently under x-pack sub folder. And the work is merged into a feature branch https://github.com/elastic/beats/tree/feature/beatless

Features:

• First providers AWS and Local
• Accepts new events from:
  • STDIN (mostly used for easier testing)
  • Cloudwatch Logs
  • Kinesis
  • SQS
  • API Gateway
• The architecture should support running multiple functions concurrently but provider implementation can restrict that.
• A provider is a singleton in the configuration.

Tasks:

• Xpack binaries X-Pack binaries #7783
• Basic beats skeleton Beatless initial PR #7837
• Functionbeat special dual binary package (NO DMG, NO RPM, NO DEB) Add functionbeat to the build process #8639
• Lookup Elasticsearch available feature via license check. Query a remote Elasticsearch for License and features #7946
• Create a global registry Refactor: Introduce a Global Registry #7392
• Wrap the global registry into a beat specific registry handling types assertion. [WIP] Beatless initial beat (local only) #8170
• Local provider [WIP] Beatless initial beat (local only) #8170
  • STDIN function type [WIP] Beatless initial beat (local only) #8170
• Add support for a sync publisher, most function SDK expect the ack when the function return. PR: Add a sync client #7934
• AWS lambda provider. [WIP] Beatless initial beat (local only) #8170
  • Api Gateway [WIP] Beatless initial beat (local only) #8170
    • integration tests
  • Cloudwatch [WIP] Beatless initial beat (local only) #8170
    • integration tests Functionbeat: Add integration tests for cloudwatch #8850
  • Kinesis [WIP] Beatless initial beat (local only) #8170
    • integration tests
  • SQS [WIP] Beatless initial beat (local only) #8170
    • integration tests
  • Make the integration test run on Travis (assert on cloud?)
• Support modules Functionbeat module support #8266
• Support processors
• Add metadata from the provider.
• Only has the ES output.

Monitoring / Telemetry

• Report monitoring events as part of the logs
• Limit the rate of reporting
• Allow a function to monitor monitoring events (it should be a module)
• Make it easy to push a function to acts as a monitoring collector (part of the CLI)

Medium priority

• CLI command to push the config + artifact to the provider.
  • Cloudwatchlogs CLI Beatless initial PR #8485
  • SQS CLI Add support for the CLI to push a lambda to monitor SQS queues. #8649
  • API gateway CLI
  • Kinesis CLI

Low priority

• Disable features for faster startup
  • Seccomp
  • Logging to file
  • Remove unsupported spooling to disk
  • Remove unsupported

Nice to have:

• Wrap some heartbeat monitor into lambda beat.
• better local validation
• GCP PoC
• add a post compile check for the package size. (Some provider impose limits on size)
• Allow to read the DLQ https://docs.aws.amazon.com/lambda/latest/dg/dlq.html

Config Architecture:

beatless.provider.aws:
  enabled: true
  # .... Provider configuration
  # Credentials for pushing the function.
  functions:
    - type: cloudwatch

      stream: messages
      fields:
        logs_type: cloudwatch-messages
      limits:
        timeout: 5s
        concurency: 20
    - type: kinesys
      name: kinesis-ingress
      stream: firehos
      processors:  #....
    - type: s3
      bucket: logs.elastic.co

output.elasticsearch:
  host: abc
  username: elastic
  password: abc123

[ph]

Functions can embed customs processors.

[ph]

Added a notes concerning monitory of beatless, it actually make sense that we use a module which will be preconfigured to work on event of data.

Probably a custom processor that will drop events that we don't need and just extract the monitoring data or drop + Dissect.

[ph]

Before working on the CLI integration I'd like to have some integration tests done, in #8170 we support the following trigger on AWS: SQS, Cloudwatch Logs, Kinesis and the API Gateway.

The following are the steps required to be able to run and asserts that a function is actually doing the right thing.

• build beatless
• push beatless to AWS
• Create data for the specific trigger (SQS, Kinesis, ApiGateway)
• teardown the function
• Assert the data in ES

The above will required:

• AWS credentials available in the test environment
• An ES cluster available.

If possible I want something close to how testing modules works in Filebeat.

[ph]

@tsg @monicasarbu The todo items in this issue are up to date.

[ph]

I am Currently working on integration tests starting with cloudwatch logs see #8850 for the testing plan.

[ph]

To better focus, I've created another meta for next phase of functionbeat at #9013