System test for kibana module, xpack code path

[ycombinator]

Related: #12256.

This PR introduces a system test to exercise the xpack code path (xpack.enabled: true) of the Kibana Metricbeat module.

Testing this PR

1. Activate the virtual env for system tests.

   cd metricbeat
   . build/python-env/bin/activate
   

2. Build the test binary.

   make metricbeat.test
   

3. Run the integration tests

   INTEGRATION_TESTS=1  nosetests -v --nocapture tests/system/test_kibana.py
   

4. Assert that there are no errors.

[elasticmachine]

Pinging @elastic/stack-monitoring

[ycombinator]

jenkins, test this

[cachedout]

Unfortunately, I get errors though it is unclear to me if they are due to this change or a missing configuration step on my end:

======================================================================
ERROR: kibana-xpack module tests
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/mp/devel/go/src/github.com/elastic/beats/metricbeat/tests/system/test_kibana.py", line 66, in test_xpack
    proc = self.start_beat()
  File "/Users/mp/devel/go/src/github.com/elastic/beats/libbeat/tests/system/beat/beat.py", line 224, in start_beat
    proc.start()
  File "/Users/mp/devel/go/src/github.com/elastic/beats/libbeat/tests/system/beat/beat.py", line 68, in start
    env=variables)
  File "/usr/local/Cellar/python@2/2.7.16/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 394, in __init__
    errread, errwrite)
  File "/usr/local/Cellar/python@2/2.7.16/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 1047, in _execute_child
    raise child_exception
OSError: [Errno 2] No such file or directory
-------------------- >> begin captured logging << --------------------
compose.config.config: DEBUG: Using configuration files: ./docker-compose.yml
docker.utils.config: DEBUG: Trying paths: ['/Users/mp/.docker/config.json', '/Users/mp/.dockercfg']
docker.utils.config: DEBUG: Found file at path: /Users/mp/.docker/config.json
docker.auth: DEBUG: Found 'auths' section
docker.auth: DEBUG: Auth data for https://index.docker.io/v1/ is absent. Client might be using a credentials store instead.
--------------------- >> end captured logging << ---------------------

----------------------------------------------------------------------
Ran 2 tests in 516.205s

FAILED (errors=2)```

[ycombinator]

Sorry @cachedout, this is my fault. I missed a step in the instructions for testing this PR. I've added it now. Please try again. Thanks!

[cachedout]

I now only receive one error:

(python-env) ✔ ~/devel/go/src/github.com/elastic/beats/metricbeat [pr/12506 L|…1]                                                                                                                                                    [59/7918]
09:45 $ INTEGRATION_TESTS=1  nosetests -v --nocapture tests/system/test_kibana.py
Step 1/2 : FROM docker.elastic.co/elasticsearch/elasticsearch:7.0.0
 ---> 8f46db60ddd6
Step 2/2 : HEALTHCHECK --interval=1s --retries=300 CMD curl -f http://localhost:9200/_license
 ---> Using cache
 ---> 204bb7839d0f

Successfully built 204bb7839d0f
Successfully tagged metricbeat_elasticsearch:latest
Step 1/2 : FROM docker.elastic.co/kibana/kibana:7.0.0
 ---> 7f92ab934206
Step 2/2 : HEALTHCHECK --interval=1s --retries=300 --start-period=60s CMD python -c 'import urllib, json; response = urllib.urlopen("http://myelastic:changeme@localhost:5601/api/status"); data = json.loads(response.read()); exit(1) if dat
a["status"]["overall"]["state"] != "green" else exit(0);'
 ---> Using cache
 ---> 0fc811024ecc

Successfully built 0fc811024ecc
Successfully tagged metricbeat_kibana:latest
Recreating metricbeat_elasticsearch_1 ... done
Recreating metricbeat_kibana_1        ... done
kibana status metricset test ... {u'service': {u'version': u'7.0.0', u'type': u'kibana', u'name': u'kibana', u'address': u'localhost:32769', u'id': u'95c3d32e-d9bb-41bf-862f-2c993d166874'}, u'@timestamp': u'2019-06-18T07:46:19.035Z', u'ki
bana': {u'status': {u'status': {u'overall': {u'state': u'green'}}, u'metrics': {u'requests': {u'disconnects': 0, u'total': 0}, u'concurrent_connections': 0}, u'name': u'kibana'}}, u'event': {u'duration': 32548545, u'module': u'kibana', u'
dataset': u'kibana.status'}, u'host': {u'name': u'Mikes-MacBook-Pro.local'}, u'ecs': {u'version': u'1.0.1'}, u'agent': {u'ephemeral_id': u'8611ebe5-3933-42a5-b17b-39f205502696', u'type': u'metricbeat', u'hostname': u'Mikes-MacBook-Pro.loc
al', u'version': u'8.0.0', u'id': u'6f06621d-9015-4158-b178-a991a0b9ad04'}, u'metricset': {u'name': u'status'}}
ERROR
kibana-xpack module tests ... ok
Killing metricbeat_kibana_1           ... done
Killing metricbeat_elasticsearch_1    ... done

======================================================================
ERROR: kibana status metricset test
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/mp/devel/go/src/github.com/elastic/beats/metricbeat/tests/system/test_kibana.py", line 48, in test_status
    self.assert_fields_are_documented(evt)
  File "/Users/mp/devel/go/src/github.com/elastic/beats/libbeat/tests/system/beat/beat.py", line 661, in assert_fields_are_documented
    expected_fields, dict_fields, aliases = self.load_fields()
  File "/Users/mp/devel/go/src/github.com/elastic/beats/libbeat/tests/system/beat/beat.py", line 572, in load_fields
    with open(fields_doc, "r") as f:
IOError: [Errno 2] No such file or directory: '/Users/mp/devel/go/src/github.com/elastic/beats/metricbeat/fields.yml'

[cachedout]

After running mage fields, all tests pass. This looks good now. :)