Cherry-pick various test fixes to 5.4 (#4404)

* Ignore permission errors in Metricbeat’s TestFileSystemList (#3562)

The test can fail if some calls to statfs fail due to permission errors. For example:

`stat("/var/lib/docker/aufs/mnt/50d0d5f599f0f19450e7649f73a0e23da1f172048e555df2b1cb78b3fefa355b", 0x7ffd2e5b8ed0) = -1 EACCES (Permission denied)`

* Less strict error matching in Winlogbeat config_test

Error string testing is brittle. The PR makes the test less stringent by not checking the full error message that includes the Golang stdlib error.

* Miscellaneous test fixes

- Fix and enable the python smoke test for heartbeat
- Remove fmt.Printf from metricbeat ceph module
- Fix Windows path issues in libbeat/paths tests
- Fix ioutil.TempDir usage in Packetbeat tests (it broke windows)

* Using single quotes around Windows paths

The thrift test config used double quotes around Windows path separators and this was interpreted incorrectly in YAML parsing.

* Rename TestBadCondition to TestConditions

This test doesn’t actually test any bad conditions. Plus there is another test in the same directory with the name TestBadCondition.

* Use logp.Beta or logp.Experimental in metricsets

And in system tests, centralize the logic for asserting that there are no ERR or WARN in logs.

Filter out errors about “The service process could not connect to the service controller” that occur when testing on Jenkins where Jenkins itself is running as a service. This confuses the Beat because it thinks that it is running as service, but it’s not.

* Remove OS specific error message check from mockbeat

The error message “no such file or directory” is an OS specific error message. There is a different error message on Windows. Simply checking for “error loading config file” should be sufficient.

* Use shorter filename in Filebeat test for Windows

The test was failing on Windows when `os.rename` failed with `[Error 3] The system cannot find the path specified`. The root cause of the failure was that the path was ~260 characters on Jenkins which is greater than the `MAX_PATH` value in Windows. This PR shortens the test log’s name to resolve the issue.

The other changes to normalize the filepath are nice to have for Windows, but not strictly required.

* Add filesystem name to test error message

Errors that are logged by the system/filesystem test case don’t have enough context to debug them. This adds the filesystem that caused the error to the message.

* Clean geoip.paths before using the path (#4306)

Use filepath.Clean on the configured paths to fix any invalid OS path separators.

Skip the geoip test with symlinks on Windows (`os.symlink` isn’t supported on Windows).

* Improve winlogbeat checkpoint test (#4371)

Check for errors and exit to stop panics from occurring when the test fails.
Increase the sleep to 750ms to give the checkpoint more time to flush states to disk.

* Fix Winlogbeat test by checking full hostname (#3942)

The `computer_name` field in events is the full hostname, but the `win32api.GetComputerName` was returning the shortened netbios name. So the test fail on machines with longer hostnames.

* Shorten test name to shorten path for windows (#4360)

On our CI system we get the following error:
```
WindowsError: [Error 206] The filename or extension is too long: 'C:\\Users\\jenkins\\workspace\\elastic+beats+pull-request+multijob-windows\\beat\\filebeat\\label\\windows\\src\\github.com\\elastic\\beats\\filebeat\\build\\system-tests\\run\\test_prospector.Test.test_close_inactive_file_rotation_and_removal_while_new_file_created'
```

The problem is that this path is 267 characters long. According to https://msdn.microsoft.com/en-us/library/windows/desktop/aa365247(v=vs.85).aspx the MAX_PATH length is 260. In general it seems we could hit this limitation also in lots of other ways where we cannot just reduce the length of the test name. There seems to be some possibility to increase this limit:

  To specify an extended-length path, use the "\\?\" prefix. For example, "\\?\D:\very long path".

We should test if this could also solve the problem as it raises the limit to 32767.

* Filter relative mounts in system filesystem metricset (#4370)

Add filtering to system filesystem metricset to remove relative mountpoints like those from Linux network namespaces.

CHANGELOG.asciidoc

@@ -33,9 +33,16 @@ https://github.com/elastic/beats/compare/v5.4.1...master[Check the HEAD diff]
 *Heartbeat*
 
 *Metricbeat*
+- Fix a debug statement that said a module wrapper had stopped when it hadn't. {pull}4264[4264]
+- Use MemAvailable value from /proc/meminfo on Linux 3.14. {pull}4316[4316]
+- Fix panic when events were dropped by filters. {issue}4327[4327]
+- Add filtering to system filesystem metricset to remove relative mountpoints like those
+  from Linux network namespaces. {pull}4370[4370]
 
 *Packetbeat*
 
+- Clean configured geoip.paths before attempting to open the database. {pull}4306[4306]
+
 *Winlogbeat*
 
 

filebeat/tests/system/test_prospector.py

@@ -485,22 +485,23 @@ def test_close_inactive_file_rotation_and_removal(self):
 
         filebeat.check_kill_and_wait()
 
-    def test_close_inactive_file_rotation_and_removal_while_new_file_created(self):
+    def test_close_inactive_file_rotation_and_removal2(self):
         """
         Test that close_inactive still applies also if file was rotated,
         new file created, and rotated file removed.
         """
+        log_path = os.path.abspath(os.path.join(self.working_dir, "log"))
+        os.mkdir(log_path)
+        testfile = os.path.join(log_path, "a.log")
+        renamed_file = os.path.join(log_path, "b.log")
+
         self.render_config_template(
-            path=os.path.abspath(self.working_dir) + "/log/test.log",
+            path=testfile,
             ignore_older="1h",
             close_inactive="3s",
             scan_frequency="0.1s",
         )
 
-        os.mkdir(self.working_dir + "/log/")
-        testfile = self.working_dir + "/log/test.log"
-        renamed_file = self.working_dir + "/log/test_renamed.log"
-
         filebeat = self.start_beat()
 
         # wait for first  "Start next scan" log message

heartbeat/Makefile

@@ -1,6 +1,6 @@
 BEAT_NAME=heartbeat
 BEAT_DESCRIPTION?=Ping remote services for availability and log results to Elasticsearch or send to Logstash.
-SYSTEM_TESTS=false
+SYSTEM_TESTS=true
 TEST_ENVIRONMENT=false
 
 # Path to the libbeat Makefile

heartbeat/tests/system/config/heartbeat.yml.j2

@@ -1,78 +1,8 @@
-################### Beat Configuration #########################
-
-
-
-############################# Output ##########################################
-
-# Configure what outputs to use when sending the data collected by the beat.
-# You can enable one or multiple outputs by setting enabled option to true.
-output:
-
-  ### File as output
-  file:
-    # Enabling file output
-    enabled: true
-
-    # Path to the directory where to save the generated files. The option is mandatory.
-    path: {{ output_file_path|default(beat.working_dir + "/output") }}
-
-
-    # Name of the generated files. The default is `heartbeat` and it generates
-    # files: `heartbeat`, `heartbeat.1`, `heartbeat.2`, etc.
-    filename: "{{ output_file_filename|default("heartbeat") }}"
-
-    # Maximum size in kilobytes of each file. When this size is reached, the files are
-    # rotated. The default value is 10 MB.
-    #rotate_every_kb: 10000
-
-    # Maximum number of files under path. When this number of files is reached, the
-    # oldest file is deleted and the rest are shifted from last to first. The default
-    # is 7 files.
-    #number_of_files: 7
-
-
-
-############################# Beat #########################################
-
-# The name of the shipper that publishes the network data. It can be used to group
-# all the transactions sent by a single shipper in the web interface.
-# If this options is not defined, the hostname is used.
-#name:
-
-# The tags of the shipper are included in their own field with each
-# transaction published. Tags make it easy to group servers by different
-# logical properties.
-#tags: ["service-X", "web-tier"]
-
-
-
-############################# Logging #########################################
-
-#logging:
-  # Send all logging output to syslog. On Windows default is false, otherwise
-  # default is true.
-  #to_syslog: true
-
-  # Write all logging output to files. Beats automatically rotate files if configurable
-  # limit is reached.
-  #to_files: false
-
-  # Enable debug output for selected components.
-  #selectors: []
-
-  # Set log level
-  #level: error
-
-  #files:
-    # The directory where the log files will written to.
-    #path: /var/log/heartbeat
-
-    # The name of the files where the logs are written to.
-    #name: heartbeat
-
-    # Configure log file size limit. If limit is reached, log file will be
-    # automatically rotated
-    #rotateeverybytes: 10485760 # = 10MB
-
-    # Number of rotated log files to keep. Oldest files will be deleted first.
-    #keepfiles: 7
+heartbeat.monitors:
+- type: icmp
+  hosts: ["localhost"]
+  schedule: '@every 10s'
+
+output.file:
+  path: {{ output_file_path|default(beat.working_dir + "/output") }}
+  filename: "{{ output_file_filename|default("heartbeat") }}"

heartbeat/tests/system/heartbeat.py

@@ -1,12 +1,13 @@
+import os
 import sys
-sys.path.append('../../vendor/github.com/elastic/beats/libbeat/tests/system')
+
+sys.path.append(os.path.join(os.path.dirname(__file__), '../../../libbeat/tests/system'))
+
 from beat.beat import TestCase
 
 
 class BaseTest(TestCase):
-
     @classmethod
     def setUpClass(self):
         self.beat_name = "heartbeat"
-        self.build_path = "../../build/system-tests/"
-        self.beat_path = "../../heartbeat.test"
+        super(BaseTest, self).setUpClass()

heartbeat/tests/system/test_base.py

@@ -1,10 +1,9 @@
-from heartbeat import BaseTest
-
 import os
 
+from heartbeat import BaseTest
 
-class Test(BaseTest):
 
+class Test(BaseTest):
     def test_base(self):
         """
         Basic test with exiting Heartbeat normally
@@ -15,5 +14,4 @@ def test_base(self):
 
         heartbeat_proc = self.start_beat()
         self.wait_until(lambda: self.log_contains("heartbeat is running"))
-        exit_code = heartbeat_proc.kill_and_wait()
-        assert exit_code == 0
+        heartbeat_proc.check_kill_and_wait()

libbeat/common/geolite.go

@@ -30,6 +30,7 @@ func LoadGeoIPData(config Geoip) *libgeo.GeoIP {
 	// look for the first existing path
 	var geoipPath string
 	for _, path := range geoipPaths {
+		path = filepath.Clean(path)
 		fi, err := os.Lstat(path)
 		if err != nil {
 			logp.Err("GeoIP path could not be loaded: %s", path)

libbeat/paths/paths_test.go

@@ -3,6 +3,7 @@ package paths
 import (
 	"os"
 	"path/filepath"
+	"runtime"
 	"testing"
 
 	"github.com/stretchr/testify/assert"
@@ -12,48 +13,51 @@ func TestHomePath(t *testing.T) {
 	type io struct {
 		Home       string // cli flag home setting
 		Path       string // requested path
-		Result     string // expected result
+		ResultHome string // expected home path
 		ResultData string // expected data path
 	}
 
 	binDir, err := filepath.Abs(filepath.Dir(os.Args[0]))
-	assert.NoError(t, err)
+	if err != nil {
+		t.Fatal(err)
+	}
 
 	tests := []io{
 		{
 			Home:       binDir,
 			Path:       "test",
-			Result:     filepath.Join(binDir, "test"),
+			ResultHome: filepath.Join(binDir, "test"),
 			ResultData: filepath.Join(binDir, "data", "test"),
 		},
 		{
-			Home:       "/tmp",
+			Home:       rootDir("/tmp"),
 			Path:       "test",
-			Result:     "/tmp/test",
-			ResultData: "/tmp/data/test",
+			ResultHome: rootDir("/tmp/test"),
+			ResultData: rootDir("/tmp/data/test"),
 		},
 		{
-			Home:       "/home/",
-			Path:       "/abc/test",
-			Result:     "/abc/test",
-			ResultData: "/abc/test",
+			Home:       rootDir("/home"),
+			Path:       rootDir("/abc/test"),
+			ResultHome: rootDir("/abc/test"),
+			ResultData: rootDir("/abc/test"),
 		},
 	}
 
 	for _, test := range tests {
-		t.Log("Executing test", test)
 		cfg := Path{Home: test.Home}
-		assert.NoError(t, Paths.initPaths(&cfg))
+		if err := Paths.initPaths(&cfg); err != nil {
+			t.Errorf("error on %+v: %v", test, err)
+			continue
+		}
 
-		assert.Equal(t, test.Result, Resolve(Home, test.Path))
+		assert.Equal(t, test.ResultHome, Resolve(Home, test.Path), "failed on %+v", test)
 
 		// config path same as home path
-		assert.Equal(t, test.Result, Resolve(Config, test.Path))
+		assert.Equal(t, test.ResultHome, Resolve(Config, test.Path), "failed on %+v", test)
 
 		// data path under home path
-		assert.Equal(t, test.ResultData, Resolve(Data, test.Path))
+		assert.Equal(t, test.ResultData, Resolve(Data, test.Path), "failed on %+v", test)
 	}
-
 }
 
 func TestDataPath(t *testing.T) {
@@ -65,7 +69,9 @@ func TestDataPath(t *testing.T) {
 	}
 
 	binDir, err := filepath.Abs(filepath.Dir(os.Args[0]))
-	assert.NoError(t, err)
+	if err != nil {
+		t.Fatal(err)
+	}
 
 	tests := []io{
 		{
@@ -75,25 +81,27 @@ func TestDataPath(t *testing.T) {
 			ResultData: filepath.Join(binDir, "data", "test"),
 		},
 		{
-			Home:       "/tmp/",
-			Data:       "/root/",
+			Home:       rootDir("/tmp"),
+			Data:       rootDir("/root"),
 			Path:       "test",
-			ResultData: "/root/test",
+			ResultData: rootDir("/root/test"),
 		},
 		{
-			Home:       "/tmp/",
-			Data:       "/root/",
-			Path:       "/var/data",
-			ResultData: "/var/data",
+			Home:       rootDir("/tmp"),
+			Data:       rootDir("root"),
+			Path:       rootDir("/var/data"),
+			ResultData: rootDir("/var/data"),
 		},
 	}
 
 	for _, test := range tests {
-		t.Log("Executing test", test)
 		cfg := Path{Home: test.Home, Data: test.Data}
-		assert.NoError(t, Paths.initPaths(&cfg))
+		if err := Paths.initPaths(&cfg); err != nil {
+			t.Errorf("error on %+v: %v", test, err)
+			continue
+		}
 
-		assert.Equal(t, test.ResultData, Resolve(Data, test.Path))
+		assert.Equal(t, test.ResultData, Resolve(Data, test.Path), "failed on %+v", test)
 	}
 
 }
@@ -107,7 +115,9 @@ func TestLogsPath(t *testing.T) {
 	}
 
 	binDir, err := filepath.Abs(filepath.Dir(os.Args[0]))
-	assert.NoError(t, err)
+	if err != nil {
+		t.Fatal(err)
+	}
 
 	tests := []io{
 		{
@@ -117,25 +127,35 @@ func TestLogsPath(t *testing.T) {
 			ResultLogs: filepath.Join(binDir, "logs", "test"),
 		},
 		{
-			Home:       "/tmp/",
-			Logs:       "/var/",
+			Home:       rootDir("/tmp"),
+			Logs:       rootDir("/var"),
 			Path:       "log",
-			ResultLogs: "/var/log",
+			ResultLogs: rootDir("/var/log"),
 		},
 		{
-			Home:       "/tmp/",
-			Logs:       "/root/",
-			Path:       "/var/log",
-			ResultLogs: "/var/log",
+			Home:       rootDir("tmp"),
+			Logs:       rootDir("root"),
+			Path:       rootDir("/var/log"),
+			ResultLogs: rootDir("/var/log"),
 		},
 	}
 
 	for _, test := range tests {
-		t.Log("Executing test", test)
 		cfg := Path{Home: test.Home, Logs: test.Logs}
-		assert.NoError(t, Paths.initPaths(&cfg))
+		if err := Paths.initPaths(&cfg); err != nil {
+			t.Errorf("error on %+v: %v", test, err)
+			continue
+		}
 
 		assert.Equal(t, test.ResultLogs, Resolve(Logs, test.Path))
 	}
 
 }
+
+// rootDir builds an OS specific absolute root directory.
+func rootDir(path string) string {
+	if runtime.GOOS == "windows" {
+		return filepath.Join(`c:\`, path)
+	}
+	return filepath.Join("/", path)
+}

libbeat/processors/condition_test.go

@@ -20,7 +20,7 @@ func (c *countFilter) Run(e common.MapStr) (common.MapStr, error) {
 
 func (c *countFilter) String() string { return "count" }
 
-func TestBadCondition(t *testing.T) {
+func TestConditions(t *testing.T) {
 
 	if testing.Verbose() {
 		logp.LogInit(logp.LOG_DEBUG, "", false, true, []string{"*"})

libbeat/tests/system/test_base.py

@@ -26,7 +26,6 @@ def test_no_config(self):
 
         assert exit_code == 1
         assert self.log_contains("error loading config file") is True
-        assert self.log_contains("no such file or directory") is True
 
     def test_invalid_config(self):
         """