Lambdabeat allows you to ship AWS Lambda metrics (by way of the Cloudwatch API) to Elasticsearch.
Here is an example of a document produced by Lambdabeat:
POST http://localhost:9200/lambdabeat-2016.06.14/_search?pretty
{
"size": 1,
"query": {
"match_all": {}
}
}
{ "took" : 3, "timed_out" : false, "_shards" : { "total" : 5, "successful" : 5, "failed" : 0 }, "hits" : { "total" : 79, "max_score" : 1.0, "hits" : [ { "_index" : "lambdabeat-2016.06.14", "_type" : "metric", "_id" : "AVVQbTk8fZl27BsUqkOv", "_score" : 1.0, "_source" : { "@timestamp" : "2016-06-14T01:03:00.000Z", "beat" : { "hostname" : "baamonde-ThinkPad-T460s", "name" : "baamonde-ThinkPad-T460s" }, "code-size" : 3564954, "description" : "my lambda function", "duration-average" : 2718.6, "duration-maximum" : 2718.6, "duration-minimum" : 2718.6, "duration-sample-count" : 1, "duration-sum" : 2718.6, "duration-unit" : "Milliseconds", "errors-average" : 0, "errors-maximum" : 0, "errors-minimum" : 0, "errors-sample-count" : 1, "errors-sum" : 0, "errors-unit" : "Count", "function" : "my-lambda-function", "handler" : "my-lambda-function.handler", "invocations-average" : 1, "invocations-maximum" : 1, "invocations-minimum" : 1, "invocations-sample-count" : 1, "invocations-sum" : 1, "invocations-unit" : "Count", "last-modified" : "2016-04-04T02:46:48.557+0000", "memory-size" : 128, "runtime" : "python2.7", "throttles-average" : 0, "throttles-maximum" : 0, "throttles-minimum" : 0, "throttles-sample-count" : 1, "throttles-sum" : 0, "throttles-unit" : "Count", "timeout" : 60, "type" : "metric", "version" : "$LATEST" } } ] } }
Binaries are available here.
Released packages contain:
- The
lambdabeat
binary. - An example =lambdabeat.yml” configuration file.
- The
lambdabeat.template.json
mapping template.
Lambdabeat fetches data from the Cloudwatch API. As such, it does not need to be deployed anywhere in particular, but it does require valid AWS credentials to be specified either:
- In an AWS configuration file located at
~/.aws/credentials.
- As environment variables (
AWS_ACCESS_KEY_ID
andAWS_SECRET_ACCESS_KEY
.)
See here for further details regarding credentials.
Additionally, Lambdabeat requires that a default AWS region be set in the
lambdabeat.yml
configuration file.
Simply specify an array of Lambda functions for which you would like to retrieve
metrics in lambdabeat.yml
like so:
lambdabeat: functions: ["foo", "bar", "baz", "qux"]
Cloudwatch maintains data for these metrics with respect to Lambda:
Invocations
Errors
Duration
Throttles
These are largely self-explanatory, but see here for details and subtleties.
Lambdabeat fetches and indexes all of these.
Unlike many (really, most) other Beats, Lambdabeat’s notion of a period
does
not necessarily determine the timestamps of the events being indexed. Rather, the
Cloudwatch API provides data between two points in time at a given,
user-specified interval. This is configured via the interval
key in
lambdabeat.yml
.
Separately, the period
key specifies how often Lambdabeat itself should run.
As such, the value of the @timestamp
field for each event is not the time at
which Lambdabeat fetched the data; it’s the time for which Cloudwatch has
returned data. At query time, this is an unimportant implementation detail. What
matters is that the events indexed into Elasticsearch are timestamped in a
meaningful matter, which they are. It is important, however, to understand the
distinction at configuration time.
Note that the value of interval
must be a multiple of 60. Note also that it
only really makes sense for period
to be greater than or equal to interval
.
If you would like to index data that predates the first invocation of
Lambdabeat, you may do so by passing a the option -backfill-date
when invoking
lambdabeat. This will fetch and index all data between the given backfill-date
and the time at which Lambdabeat begins running.
For example:
lambdabeat -e -v -backfill-date="2016-06-08T00:00:00.000Z"
Lambdabeat currently supports Elasticsearch as an output. Simply provide an
array of hosts in lambdabeat.yml
like so:
output: elasticsearch: hosts: ["http://localhost:9200"]
Lambdabeat introduces no novel infrastructure with respect to outputs.
To build the binary for Lambdabeat run the command below. This will generate a binary in the same directory with the name lambdabeat.
make
To run Lambdabeat with debugging output enabled, run:
./lambdabeat -c lambdabeat.yml -e -d "*"
To produce a versioned tarball containing the lambdabeat binary, lambdabeat.yml, and lambdabeat index template, run:
make release
To clean up the build directory and generated artifacts, run:
make clean
To clone Lambdabeat from the git repository, run the following commands:
mkdir -p ${GOPATH}/github.com/michaelbaamonde cd ${GOPATH}/github.com/michaelbaamonde git clone https://github.com/michaelbaamonde/lambdabeat
For further development, check out the beat developer guide.