diff --git a/generate/beat/README.asciidoc b/generate/beat/README.asciidoc new file mode 100644 index 00000000000..6af210dcfae --- /dev/null +++ b/generate/beat/README.asciidoc @@ -0,0 +1,90 @@ +[[creating-your-beat]] +== Creating your own Beat + +The Beat generator enables you to create your own Beat in a few steps. + +[float] +=== Requirements + +To create your own Beat, you must have Golang 1.6.2 or later installed, and the `$GOPATH` +must be set up correctly. In addition, the following tools are quired: + +* https://www.python.org/downloads/[python] +* https://virtualenv.pypa.io/en/stable/[virtualenv] +* https://github.com/audreyr/cookiecutter[cookiecutter] + +Virtualenv and Cookiecutter are easiest installed with your package manager or https://pip.pypa.io/en/stable/[pip]. For more details on how to +install cookiecutter, see the http://cookiecutter.readthedocs.io/en/latest/installation.html[cookiecutter installation docs]. + + +[float] +=== Step 1 - Get the libbeat source code + +The first step is to get the libbeat source code: + +[source,bash] +---- +go get github.com/elastic/beats/libbeat +---- + +This will clone the beats repository into `GOPATH`. Now change to the directory where you want to create the beat. +This directory is normally located under `$GOPATH/src/github.com/{your-github-name}`. Inside this directory, run the command to create the beat. + + +[float] +=== Step 2 - Create the Beat + +Run the command: + +[source,bash] +---- +cookiecutter $GOPATH/src/github.com/elastic/beats/generate/beat +---- + +When prompted, enter the Beat name and path. + + +[float] +=== Step 3 - Init and create the metricset + +After creating the Beat, change the directory to `$GOPATH/src/github.com/{your-github-name}/{beat}` and run: + +[source,bash] +---- +make setup +---- + +This will do the initial setup for your Beat. + +For more details about creating a beat, see the https://www.elastic.co/guide/en/beats/libbeat/current/new-beat.html[beat developer docs]. + + +[float] +=== Step 4 - Build & Run + +To create a binary run the `make` command. This will create the binary in your beats directory. + +To run it, execute the binary. This will automatically load the default configuration which was generated by `make update`. + +[source,bash] +---- +./beatname -e -d "*" +---- + +This will run the beat with debug output enabled to the console to directly see what is happening. Stop the beat with `CTRL-C`. + +[float] +=== Step 5 - Package + +To create packages and binaries for different plaforms, https://www.docker.com/[docker] is required. +The first step is to get the most recent packaging tools into your beat: + +[source,bash] +---- +make package +---- + +This will fetch the most recent packaging tools and start the packaging of your beat. This can take a little bit longer. + + +For more details about creating a beat, see the https://www.elastic.co/guide/en/beats/libbeat/current/new-beat.html[beat developer docs]. diff --git a/generate/beat/README.md b/generate/beat/README.md deleted file mode 100644 index 6fb829dde92..00000000000 --- a/generate/beat/README.md +++ /dev/null @@ -1,34 +0,0 @@ -This generator makes it possible to generate your own beat in seconds and build up on it. - - -# Create your own beat - -To create your own beat based on this template, run inside your GOPATH where you want to create the beat. It is important that your [GOPATH](https://golang.org/doc/code.html#GOPATH) is setup up correctly. - -``` -git clone https://github.com/elastic/beats $GOPATH/src/github.com/elastic/beats -cookiecutter $GOPATH/src/github.com/elastic/beats/generate/beat -``` - -This requires [python](https://www.python.org/downloads/) and [cookiecutter](https://github.com/audreyr/cookiecutter) to be installed. More details on how to install cookiecutter can be found [here](http://cookiecutter.readthedocs.io/en/latest/installation.html). - - -# Goals - -This beat generator has several goals: - -* Create a running in beat in very few steps -* Have an environment for unit, integration and system testing ready to make testing a beat simple -* Ensure easy maintainable and standardised structure of a beat -* Provide all files needed to build up and grow a community around a beat -* Allow release management of a beat -* Make a beat easy to update to the most recent version of libbeat - -# Dependencies - -In order to build your beat, you need the following : - * [go](https://golang.org/doc/install) version >= 1.6 - * [glide](https://github.com/Masterminds/glide) version >= 0.10.0 - * [python](https://www.python.org/) version 2.x - * [virtualenv](https://virtualenv.pypa.io/en/stable/) - * [docker](https://www.docker.com/) diff --git a/generate/beat/{{cookiecutter.beat}}/Makefile b/generate/beat/{{cookiecutter.beat}}/Makefile index 27d4aa82e05..cefee942c3d 100644 --- a/generate/beat/{{cookiecutter.beat}}/Makefile +++ b/generate/beat/{{cookiecutter.beat}}/Makefile @@ -9,14 +9,14 @@ PREFIX?=. # Path to the libbeat Makefile -include $(ES_BEATS)/libbeat/scripts/Makefile -.PHONY: init -init: - glide update --no-recursive +# Initial beat setup +.PHONY: setup +setup: make update - git init -.PHONY: commit -commit: +.PHONY: git-init +git-init: + git init git add README.md CONTRIBUTING.md git commit -m "Initial commit" git add LICENSE @@ -29,14 +29,6 @@ commit: git add .travis.yml git commit -m "Add Travis CI" -.PHONY: update-deps -update-deps: - glide update --no-recursive --strip-vcs - # This is called by the beats packer before building starts .PHONY: before-build before-build: - -# Create binary packages for the beat -pack: create-packer - cd dev-tools/packer; make deps images {{cookiecutter.beat}} diff --git a/generate/beat/{{cookiecutter.beat}}/README.md b/generate/beat/{{cookiecutter.beat}}/README.md index b41033e722f..c63d794a010 100644 --- a/generate/beat/{{cookiecutter.beat}}/README.md +++ b/generate/beat/{{cookiecutter.beat}}/README.md @@ -9,8 +9,7 @@ Ensure that this folder is at the following location: ### Requirements -* [Golang](https://golang.org/dl/) 1.6 -* [Glide](https://github.com/Masterminds/glide) >= 0.10.0 +* [Golang](https://golang.org/dl/) 1.6.2 ### Init Project To get running with {{cookiecutter.beat|capitalize}}, run the following command: @@ -73,23 +72,6 @@ make coverage-report The test coverage is reported in the folder `./build/coverage/` - -### Package - -To be able to package {{cookiecutter.beat|capitalize}} the requirements are as follows: - - * [Docker Environment](https://docs.docker.com/engine/installation/) >= 1.10 - * $GOPATH/bin must be part of $PATH: `export PATH=${PATH}:${GOPATH}/bin` - -To cross-compile and package {{cookiecutter.beat|capitalize}} for all supported platforms, run the following commands: - -``` -cd dev-tools/packer -make deps -make images -make -``` - ### Update Each beat has a template for the mapping in elasticsearch and a documentation for the fields @@ -129,3 +111,14 @@ git clone https://{{cookiecutter.beat_path}}/{{cookiecutter.beat}} For further development, check out the [beat developer guide](https://www.elastic.co/guide/en/beats/libbeat/current/new-beat.html). + + +## Packaging + +The beat frameworks provides tools to crosscompile and package your beat for different platforms. This requires [docker](https://www.docker.com/) and vendoring as described above. To build packages of your beat, run the following command: + +``` +make package +``` + +This will fetch and create all images required for the build process. The hole process to finish can take several minutes. diff --git a/generate/beat/{{cookiecutter.beat}}/glide.yaml b/generate/beat/{{cookiecutter.beat}}/glide.yaml deleted file mode 100644 index ff1df4c39c9..00000000000 --- a/generate/beat/{{cookiecutter.beat}}/glide.yaml +++ /dev/null @@ -1,8 +0,0 @@ -package: {{cookiecutter.beat_path}}/{{cookiecutter.beat}} -import: -- package: github.com/elastic/beats - subpackages: - - libbeat/beat - - libbeat/common - - libbeat/logp - - libbeat/publisher diff --git a/generate/metricbeat/metricset/README.asciidoc b/generate/metricbeat/metricset/README.asciidoc index ad6d7881610..63260a208c3 100644 --- a/generate/metricbeat/metricset/README.asciidoc +++ b/generate/metricbeat/metricset/README.asciidoc @@ -8,21 +8,27 @@ own metricsets. === Requirements To create your own Beat, you must have Golang 1.6.2 or later installed, and the `$GOPATH` -must be set up correctly. You must also have https://www.python.org/downloads/[python] and https://github.com/audreyr/cookiecutter[cookiecutter] installed. For more details on how to install cookiecutter, -see the http://cookiecutter.readthedocs.io/en/latest/installation.html[cookiecutter installation docs]. +must be set up correctly. In addition, the following tools are quired: + +* https://www.python.org/downloads/[python] +* https://virtualenv.pypa.io/en/stable/[virtualenv] +* https://github.com/audreyr/cookiecutter[cookiecutter] + +Virtualenv and Cookiecutter are easiest installed with your package manager or https://pip.pypa.io/en/stable/[pip]. For more details on how to +install cookiecutter, see the http://cookiecutter.readthedocs.io/en/latest/installation.html[cookiecutter installation docs]. [float] -=== Step 1 - Clone the beats repository +=== Step 1 - Get the metricbeat source code -The first step is to clone the beats repository: +The first step is to get the metricbeat source code: [source,bash] ---- -git clone https://github.com/elastic/beats $GOPATH/src/github.com/elastic/beats +go get github.com/elastic/beats/metricbeat ---- -After cloning the beats repository into the `GOPATH`, change to the directory where you want to create the beat. -This directory is normally located under `$GOPATH/src/github.com/{your-github-com}`. Inside this directory, run the command to create the beat. +This will clone the beats repository into `GOPATH`. Now change to the directory where you want to create the beat. +This directory is normally located under `$GOPATH/src/github.com/{your-github-name}`. Inside this directory, run the command to create the beat. [float] @@ -41,7 +47,7 @@ When prompted, enter the Beat name and path. [float] === Step 3 - Init and create the metricset -After creating the Beat, change the directory to `$GOPATH/src/github.com/{your-github-com}/{beat}` and run: +After creating the Beat, change the directory to `$GOPATH/src/github.com/{your-github-name}/{beat}` and run: [source,bash] ---- @@ -52,3 +58,33 @@ This will do the initial setup for your Beat and also run `make create-metricset module name and metricset name of your Beat. For more details about creating a metricset, see the docs about https://www.elastic.co/guide/en/beats/metricbeat/current/creating-metricsets.html[creating a metricset]. + + +[float] +=== Step 4 - Build & Run + +To create a binary run the `make` command. This will create the binary in your beats directory. + +To run it, execute the binary. This will automatically load the default configuration which was generated by `make update`. + +[source,bash] +---- +./beatname -e -d "*" +---- + +This will run the beat with debug output enabled to the console to directly see what is happening. Stop the beat with `CTRL-C`. + +[float] +=== Step 5 - Package + +To create packages and binaries for different plaforms, https://www.docker.com/[docker] is required. +The first step is to get the most recent packaging tools into your beat: + +[source,bash] +---- +make package +---- + +This will fetch the most recent packaging tools and start the packaging of your beat. This can take a little bit longer. + +For more details about creating a metricset, see the docs about https://www.elastic.co/guide/en/beats/metricbeat/current/creating-metricsets.html[creating a metricset]. diff --git a/generate/metricbeat/metricset/{{cookiecutter.beat}}/Makefile b/generate/metricbeat/metricset/{{cookiecutter.beat}}/Makefile index 8999ba620fd..fddd60a35c0 100644 --- a/generate/metricbeat/metricset/{{cookiecutter.beat}}/Makefile +++ b/generate/metricbeat/metricset/{{cookiecutter.beat}}/Makefile @@ -22,7 +22,3 @@ update-deps: # This is called by the beats packer before building starts .PHONY: before-build before-build: - -# Create binary packages for the beat -pack: create-packer - cd dev-tools/packer; make deps images {{cookiecutter.beat}} diff --git a/generate/metricbeat/metricset/{{cookiecutter.beat}}/README.md b/generate/metricbeat/metricset/{{cookiecutter.beat}}/README.md index d6928422528..8a1393aea14 100644 --- a/generate/metricbeat/metricset/{{cookiecutter.beat}}/README.md +++ b/generate/metricbeat/metricset/{{cookiecutter.beat}}/README.md @@ -73,10 +73,10 @@ git commit -m "Add {{cookiecutter.beat}}" ## Packaging -The beat frameworks provides tools to crosscompile and package your beat for different platforms. This requires docker and vendoring as described above. To build packages of your beat, run the following command: +The beat frameworks provides tools to crosscompile and package your beat for different platforms. This requires [docker](https://www.docker.com/) and vendoring as described above. To build packages of your beat, run the following command: ``` -make pack +make package ``` -This will fetch and create all images required for the build process. The hole process to finish can take several minutes. After you will find your packages +This will fetch and create all images required for the build process. The hole process to finish can take several minutes. diff --git a/libbeat/scripts/Makefile b/libbeat/scripts/Makefile index d637f3c6dca..cd96cd95552 100755 --- a/libbeat/scripts/Makefile +++ b/libbeat/scripts/Makefile @@ -309,6 +309,13 @@ install-home: install -d -m 755 ${HOME_PREFIX}/scripts/ install -m 755 ${ES_BEATS}/libbeat/scripts/migrate_beat_config_1_x_to_5_0.py ${HOME_PREFIX}/scripts/ +# Creates packer files in a beat .PHONY: create-packer create-packer: python ${ES_BEATS}/libbeat/scripts/create_packer.py --es_beats=${ES_BEATS} --beat_path=${BEAT_DIR} --beat=${BEATNAME} + +# Create binary packages for the beat +package: create-packer + cd dev-tools/packer; \ + export ES_BEATS=${ES_BEATS} ; \ + make deps images ${BEATNAME} diff --git a/libbeat/scripts/dev-tools/packer/Makefile b/libbeat/scripts/dev-tools/packer/Makefile index 78c4ba93deb..0b924a548b1 100644 --- a/libbeat/scripts/dev-tools/packer/Makefile +++ b/libbeat/scripts/dev-tools/packer/Makefile @@ -4,7 +4,7 @@ BUILDID=$(shell git rev-parse HEAD) all: {beat}/deb {beat}/rpm {beat}/darwin {beat}/win {beat}/bin \ build/upload/build_id.txt -ES_BEATS=../../vendor/github.com/elastic/beats +ES_BEATS?=../../vendor/github.com/elastic/beats include $(ES_BEATS)/dev-tools/packer/scripts/Makefile