Unify beat generator and metricset generator (#2015)

* Replace git clone by go get command. This shows if golang was setup correctly
* Update the docs with the new changes
* Enhance and simplify packaging

generate/beat/README.asciidoc

@@ -0,0 +1,90 @@
+[[creating-your-beat]]
+== Creating your own Beat
+
+The Beat generator enables you to create your own Beat in a few steps.
+
+[float]
+=== Requirements
+
+To create your own Beat, you must have Golang 1.6.2 or later installed, and the `$GOPATH`
+must be set up correctly. In addition, the following tools are quired:
+
+* https://www.python.org/downloads/[python]
+* https://virtualenv.pypa.io/en/stable/[virtualenv]
+* https://github.com/audreyr/cookiecutter[cookiecutter]
+
+Virtualenv and Cookiecutter are easiest installed with your package manager or https://pip.pypa.io/en/stable/[pip]. For more details on how to
+install cookiecutter, see the http://cookiecutter.readthedocs.io/en/latest/installation.html[cookiecutter installation docs].
+
+
+[float]
+=== Step 1 - Get the libbeat source code
+
+The first step is to get the libbeat source code:
+
+[source,bash]
+----
+go get github.com/elastic/beats/libbeat
+----
+
+This will clone the beats repository into `GOPATH`. Now change to the directory where you want to create the beat.
+This directory is normally located under `$GOPATH/src/github.com/{your-github-name}`. Inside this directory, run the command to create the beat.
+
+
+[float]
+=== Step 2 - Create the Beat
+
+Run the command:
+
+[source,bash]
+----
+cookiecutter $GOPATH/src/github.com/elastic/beats/generate/beat
+----
+
+When prompted, enter the Beat name and path.
+
+
+[float]
+=== Step 3 - Init and create the metricset
+
+After creating the Beat, change the directory to `$GOPATH/src/github.com/{your-github-name}/{beat}` and run:
+
+[source,bash]
+----
+make setup
+----
+
+This will do the initial setup for your Beat.
+
+For more details about creating a beat, see the https://www.elastic.co/guide/en/beats/libbeat/current/new-beat.html[beat developer docs].
+
+
+[float]
+=== Step 4 - Build & Run
+
+To create a binary run the `make` command. This will create the binary in your beats directory.
+
+To run it, execute the binary. This will automatically load the default configuration which was generated by `make update`.
+
+[source,bash]
+----
+./beatname -e -d "*"
+----
+
+This will run the beat with debug output enabled to the console to directly see what is happening. Stop the beat with `CTRL-C`.
+
+[float]
+=== Step 5 - Package
+
+To create packages and binaries for different plaforms, https://www.docker.com/[docker] is required.
+The first step is to get the most recent packaging tools into your beat:
+
+[source,bash]
+----
+make package
+----
+
+This will fetch the most recent packaging tools and start the packaging of your beat. This can take a little bit longer.
+
+
+For more details about creating a beat, see the https://www.elastic.co/guide/en/beats/libbeat/current/new-beat.html[beat developer docs].

generate/beat/{{cookiecutter.beat}}/Makefile

@@ -9,14 +9,14 @@ PREFIX?=.
 # Path to the libbeat Makefile
 -include $(ES_BEATS)/libbeat/scripts/Makefile
 
-.PHONY: init
-init:
-	glide update --no-recursive
+# Initial beat setup
+.PHONY: setup
+setup:
 	make update
-	git init
 
-.PHONY: commit
-commit:
+.PHONY: git-init
+git-init:
+	git init
 	git add README.md CONTRIBUTING.md
 	git commit -m "Initial commit"
 	git add LICENSE
@@ -29,14 +29,6 @@ commit:
 	git add .travis.yml
 	git commit -m "Add Travis CI"
 
-.PHONY: update-deps
-update-deps:
-	glide update --no-recursive --strip-vcs
-
 # This is called by the beats packer before building starts
 .PHONY: before-build
 before-build:
-
-# Create binary packages for the beat
-pack: create-packer
-	cd dev-tools/packer; make deps images {{cookiecutter.beat}}

generate/beat/{{cookiecutter.beat}}/README.md

@@ -9,8 +9,7 @@ Ensure that this folder is at the following location:
 
 ### Requirements
 
-* [Golang](https://golang.org/dl/) 1.6
-* [Glide](https://github.com/Masterminds/glide) >= 0.10.0
+* [Golang](https://golang.org/dl/) 1.6.2
 
 ### Init Project
 To get running with {{cookiecutter.beat|capitalize}}, run the following command:
@@ -73,23 +72,6 @@ make coverage-report
 
 The test coverage is reported in the folder `./build/coverage/`
 
-
-### Package
-
-To be able to package {{cookiecutter.beat|capitalize}} the requirements are as follows:
-
- * [Docker Environment](https://docs.docker.com/engine/installation/) >= 1.10
- * $GOPATH/bin must be part of $PATH: `export PATH=${PATH}:${GOPATH}/bin`
-
-To cross-compile and package {{cookiecutter.beat|capitalize}} for all supported platforms, run the following commands:
-
-```
-cd dev-tools/packer
-make deps
-make images
-make
-```
-
 ### Update
 
 Each beat has a template for the mapping in elasticsearch and a documentation for the fields
@@ -129,3 +111,14 @@ git clone https://{{cookiecutter.beat_path}}/{{cookiecutter.beat}}
 
 
 For further development, check out the [beat developer guide](https://www.elastic.co/guide/en/beats/libbeat/current/new-beat.html).
+
+
+## Packaging
+
+The beat frameworks provides tools to crosscompile and package your beat for different platforms. This requires [docker](https://www.docker.com/) and vendoring as described above. To build packages of your beat, run the following command:
+
+```
+make package
+```
+
+This will fetch and create all images required for the build process. The hole process to finish can take several minutes.

generate/metricbeat/metricset/README.asciidoc

@@ -8,21 +8,27 @@ own metricsets.
 === Requirements
 
 To create your own Beat, you must have Golang 1.6.2 or later installed, and the `$GOPATH`
-must be set up correctly. You must also have https://www.python.org/downloads/[python] and https://github.com/audreyr/cookiecutter[cookiecutter] installed. For more details on how to install cookiecutter,
-see the http://cookiecutter.readthedocs.io/en/latest/installation.html[cookiecutter installation docs].
+must be set up correctly. In addition, the following tools are quired:
+
+* https://www.python.org/downloads/[python]
+* https://virtualenv.pypa.io/en/stable/[virtualenv]
+* https://github.com/audreyr/cookiecutter[cookiecutter]
+
+Virtualenv and Cookiecutter are easiest installed with your package manager or https://pip.pypa.io/en/stable/[pip]. For more details on how to
+install cookiecutter, see the http://cookiecutter.readthedocs.io/en/latest/installation.html[cookiecutter installation docs].
 
 [float]
-=== Step 1 - Clone the beats repository
+=== Step 1 - Get the metricbeat source code
 
-The first step is to clone the beats repository:
+The first step is to get the metricbeat source code:
 
 [source,bash]
 ----
-git clone https://github.com/elastic/beats $GOPATH/src/github.com/elastic/beats
+go get github.com/elastic/beats/metricbeat
 ----
 
-After cloning the beats repository into the `GOPATH`, change to the directory where you want to create the beat.
-This directory is normally located under `$GOPATH/src/github.com/{your-github-com}`. Inside this directory, run the command to create the beat.
+This will clone the beats repository into `GOPATH`. Now change to the directory where you want to create the beat.
+This directory is normally located under `$GOPATH/src/github.com/{your-github-name}`. Inside this directory, run the command to create the beat.
 
 
 [float]
@@ -41,7 +47,7 @@ When prompted, enter the Beat name and path.
 [float]
 === Step 3 - Init and create the metricset
 
-After creating the Beat, change the directory to `$GOPATH/src/github.com/{your-github-com}/{beat}` and run:
+After creating the Beat, change the directory to `$GOPATH/src/github.com/{your-github-name}/{beat}` and run:
 
 [source,bash]
 ----
@@ -52,3 +58,33 @@ This will do the initial setup for your Beat and also run `make create-metricset
 module name and metricset name of your Beat.
 
 For more details about creating a metricset, see the docs about https://www.elastic.co/guide/en/beats/metricbeat/current/creating-metricsets.html[creating a metricset].
+
+
+[float]
+=== Step 4 - Build & Run
+
+To create a binary run the `make` command. This will create the binary in your beats directory.
+
+To run it, execute the binary. This will automatically load the default configuration which was generated by `make update`.
+
+[source,bash]
+----
+./beatname -e -d "*"
+----
+
+This will run the beat with debug output enabled to the console to directly see what is happening. Stop the beat with `CTRL-C`.
+
+[float]
+=== Step 5 - Package
+
+To create packages and binaries for different plaforms, https://www.docker.com/[docker] is required.
+The first step is to get the most recent packaging tools into your beat:
+
+[source,bash]
+----
+make package
+----
+
+This will fetch the most recent packaging tools and start the packaging of your beat. This can take a little bit longer.
+
+For more details about creating a metricset, see the docs about https://www.elastic.co/guide/en/beats/metricbeat/current/creating-metricsets.html[creating a metricset].

generate/metricbeat/metricset/{{cookiecutter.beat}}/Makefile

@@ -22,7 +22,3 @@ update-deps:
 # This is called by the beats packer before building starts
 .PHONY: before-build
 before-build:
-
-# Create binary packages for the beat
-pack: create-packer
-	cd dev-tools/packer; make deps images {{cookiecutter.beat}}

generate/metricbeat/metricset/{{cookiecutter.beat}}/README.md

@@ -73,10 +73,10 @@ git commit -m "Add {{cookiecutter.beat}}"
 
 ## Packaging
 
-The beat frameworks provides tools to crosscompile and package your beat for different platforms. This requires docker and vendoring as described above. To build packages of your beat, run the following command:
+The beat frameworks provides tools to crosscompile and package your beat for different platforms. This requires [docker](https://www.docker.com/) and vendoring as described above. To build packages of your beat, run the following command:
 
 ```
-make pack
+make package
 ```
 
-This will fetch and create all images required for the build process. The hole process to finish can take several minutes. After you will find your packages
+This will fetch and create all images required for the build process. The hole process to finish can take several minutes.

libbeat/scripts/Makefile

@@ -309,6 +309,13 @@ install-home:
 	install -d -m 755 ${HOME_PREFIX}/scripts/
 	install -m 755 ${ES_BEATS}/libbeat/scripts/migrate_beat_config_1_x_to_5_0.py ${HOME_PREFIX}/scripts/
 
+# Creates packer files in a beat
 .PHONY: create-packer
 create-packer:
 	python ${ES_BEATS}/libbeat/scripts/create_packer.py --es_beats=${ES_BEATS} --beat_path=${BEAT_DIR} --beat=${BEATNAME}
+
+# Create binary packages for the beat
+package: create-packer
+	cd dev-tools/packer; \
+	export ES_BEATS=${ES_BEATS} ; \
+	make deps images ${BEATNAME}

libbeat/scripts/dev-tools/packer/Makefile

@@ -4,7 +4,7 @@ BUILDID=$(shell git rev-parse HEAD)
 all: {beat}/deb {beat}/rpm {beat}/darwin {beat}/win {beat}/bin \
 	build/upload/build_id.txt
 
-ES_BEATS=../../vendor/github.com/elastic/beats
+ES_BEATS?=../../vendor/github.com/elastic/beats
 include $(ES_BEATS)/dev-tools/packer/scripts/Makefile