Add support for reading from UNIX datagram sockets #22699

kvch · 2020-11-20T20:50:05Z

What does this PR do?

This PR adds support for reading from UNIX datagram sockets both from the unix input and the syslog input. A new option is added to select the type of the socket named socket_type. Available options are: stream and datagram.

Why is it important?

A few applications which send logs over Unix sockets, use datagrams not streams. From now on, Filebeat can accept input from these applications as well.

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
~~- [ ] I have made corresponding change to the default configuration files~~
I have added tests that prove my fix is effective or that my feature works
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Related issues

Closes #18632

elasticmachine · 2020-11-20T21:03:02Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Build Cause: kvch commented: /test filebeat
Start Time: 2020-11-30T12:35:18.296+0000
Duration: 46 min 28 sec

Test stats 🧪

Test	Results
Failed	0
Passed	4523
Skipped	565
Total	5088

💚 Flaky test report

Tests succeeded.

Expand to view the summary

Test stats 🧪

Test	Results
Failed	0
Passed	4523
Skipped	565
Total	5088

elasticmachine · 2020-11-23T14:26:57Z

Pinging @elastic/integrations-services (Team:Services)

filebeat/inputsource/unix/config.go

filebeat/inputsource/unix/server.go

filebeat/input/unix/input.go

filebeat/inputsource/common/dgram/server.go

urso · 2020-11-24T14:34:50Z

filebeat/inputsource/common/dgram/server.go

+	return strings.Contains(err.Error(), windowErrBuffer)
+}
+
+func (l *Listener) Run(ctx context.Context) error {


If Run returns on error the input stops hard. Are there some errors we can handle gracefully? e.g.

unix file can not be opened by current user => retry with backoff (user can adjust permissions)

another filebeat instance is listening on unix socket => retry with backoff (other FB instance might go down => HA setup?)

UDP port is currently take. Similar to unix socket, if it is taken by another filebeat instance we might want to retry.

Do want to allow the handler to take down the server? e.g. distinguish between critical or non-critical errors? Or do we mandate that the handler MUST ONLY return an error if it is not Recoverable from within the handler (IO error). On IO error, can we recover by trying to reopen the connection and listen?

All in all, the knowledge if an error is 'fatal' for the input or not lives in the actual inputsource implementation, not here. It looks like you just moved the logic from the current UDP server (refactoring). Maybe let us create an issue instead of fixing/improving error handling here. I would presume that we can improve error handling in the TCP/unix-streaming inputs as well.

urso · 2020-11-24T14:36:55Z

filebeat/inputsource/common/dgram/server.go

+	}
+
+	l.tg.Go(func(ctx unison.Canceler) error {
+		connCtx, connCancel := ctxtool.WithFunc(ctxtool.FromCanceller(ctx), func() {


This looks similar to run. Do we want 'Start' to call Run or have a common func (l *Listener) readLoop(...)?

What about error handling in Start. If listener fails would it make sense to retry similar to Run and only fail initially if the error is really non-recoverable?

What about error handling in Start. If listener fails would it make sense to retry similar to Run and only fail initially if the error is really non-recoverable?

To me Start sounds like a function you would expect it try to start something and return if it cannot. Run is more like constantly retrying to do something.

This is why "non-recoverable". If the error can be recovered from I would expect the internal runLoop to continue trying to connect and collect data if possible. Otherwise I will be forced to monitor the health of each input (not really supported yet) and restart filebeat until it eventually went healthy.

If some component implements Start and Run, I would not expect them to behave differently depending on which method I use (beside Run should block and Start return immediately).

filebeat/inputsource/tcp/server_test.go

filebeat/tests/system/test_unix.py

urso · 2020-11-24T18:23:02Z

filebeat/tests/system/test_syslog.py

-                sock.send(m.encode("utf-8"))
+            sock = send_over_socket(path,
+                                    "<13>Oct 11 22:14:15 wopr.mymachine.co postfix/smtpd[2000]:"
+                                    " 'su root' failed for lonvick on /dev/pts/8 {}\n")


Maybe we want to send more than one event in this test?

Every test sends three events with the same message ending with the index of that message. You mean that we should send a different message every time we send an event?

Yes. But also consider to send very big messages that might not fit the read buffer. Test the edge cases. Plus if something fails it should be more obvious which exact message failed or was missing.

urso · 2020-11-24T18:31:51Z

The test failure looks like the Listener blocks in the waitgroup. A full stack trace for all go-routines is missing, but the streaming listener loop seems to not unblock which blocks 'Stop' during shutdown.

kvch · 2020-11-26T18:59:54Z

jenkins run tests

kvch · 2020-11-26T19:08:26Z

/test filebeat

kvch · 2020-11-30T09:32:32Z

/test filebeat

kvch · 2020-11-30T12:34:47Z

/test filebeat

## What does this PR do? This PR adds support for reading from UNIX datagram sockets both from the `unix` input and the `syslog` input. A new option is added to select the type of the socket named `socket_type`. Available options are: `stream` and `datagram`. ## Why is it important? A few applications which send logs over Unix sockets, use datagrams not streams. From now on, Filebeat can accept input from these applications as well. Closes elastic#18632 (cherry picked from commit 3a1d1ae)

## What does this PR do? This PR adds support for reading from UNIX datagram sockets both from the `unix` input and the `syslog` input. A new option is added to select the type of the socket named `socket_type`. Available options are: `stream` and `datagram`. ## Why is it important? A few applications which send logs over Unix sockets, use datagrams not streams. From now on, Filebeat can accept input from these applications as well. Closes #18632 (cherry picked from commit 3a1d1ae)

…-issues * upstream/master: (41 commits) Fix version parser regex for packaging (elastic#22581) Fix local_dynamic documentation and add providers inline doc. (elastic#22657) fix: use proper param name for e2e tests (elastic#22836) [Heartbeat] Fix exit on disabled monitor (elastic#22829) Update Golang to 1.14.12 (elastic#22790) docs: fix setup.template.overwrite typos (elastic#22804) Add docs section for ECS EC2 monitoring (elastic#22784) Fixing logic to keep list of unique cluster UUIDs (elastic#22808) Skip somewhat flaky UDP system test on Windows (elastic#22810) Fix polling node when it is not ready and monitor by hostname (elastic#22666) Skip Filebeat test_shutdown on windows 7 (elastic#22797) Make monitoring Namespace thread-safe (elastic#22640) Drop pkt_dstaddr and pkt_srcaddr when equals to "-" (elastic#22721) Add support for reading from UNIX datagram sockets (elastic#22699) Fix export dashboard command from Elastic Cloud (elastic#22746) Skip flaky winlogbeat test on Windows-7 (elastic#22754) Missing `>` (elastic#22763) (elastic#22766) Fix k8s watcher issue when node access to list nodes and ns (elastic#22714) [Metricbeat/Kibana/stats] Enforce `exclude_usage=true` (elastic#22732) Avoid sending non-numeric floats in cloud foundry integrations (elastic#22634) ...

…dows-7 * upstream/master: (41 commits) Fix version parser regex for packaging (elastic#22581) Fix local_dynamic documentation and add providers inline doc. (elastic#22657) fix: use proper param name for e2e tests (elastic#22836) [Heartbeat] Fix exit on disabled monitor (elastic#22829) Update Golang to 1.14.12 (elastic#22790) docs: fix setup.template.overwrite typos (elastic#22804) Add docs section for ECS EC2 monitoring (elastic#22784) Fixing logic to keep list of unique cluster UUIDs (elastic#22808) Skip somewhat flaky UDP system test on Windows (elastic#22810) Fix polling node when it is not ready and monitor by hostname (elastic#22666) Skip Filebeat test_shutdown on windows 7 (elastic#22797) Make monitoring Namespace thread-safe (elastic#22640) Drop pkt_dstaddr and pkt_srcaddr when equals to "-" (elastic#22721) Add support for reading from UNIX datagram sockets (elastic#22699) Fix export dashboard command from Elastic Cloud (elastic#22746) Skip flaky winlogbeat test on Windows-7 (elastic#22754) Missing `>` (elastic#22763) (elastic#22766) Fix k8s watcher issue when node access to list nodes and ns (elastic#22714) [Metricbeat/Kibana/stats] Enforce `exclude_usage=true` (elastic#22732) Avoid sending non-numeric floats in cloud foundry integrations (elastic#22634) ...

botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Nov 20, 2020

andresrc assigned kvch Nov 21, 2020

andresrc added the Team:Services (Deprecated) Label for the former Integrations-Services team label Nov 21, 2020

botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Nov 21, 2020

kvch requested review from urso and andrewstucki November 23, 2020 14:24

kvch changed the title ~~Feature filebeat unix datagram socket~~ Add support for reading from UNIX datagram sockets Nov 23, 2020

kvch marked this pull request as ready for review November 23, 2020 14:26

urso reviewed Nov 23, 2020

View reviewed changes

filebeat/inputsource/unix/config.go Show resolved Hide resolved

filebeat/inputsource/unix/config.go Outdated Show resolved Hide resolved

filebeat/inputsource/unix/server.go Show resolved Hide resolved

filebeat/inputsource/unix/server.go Outdated Show resolved Hide resolved

urso reviewed Nov 24, 2020

View reviewed changes

kvch force-pushed the feature-filebeat-unix-datagram-socket branch 2 times, most recently from 7904cc7 to 29f0f1f Compare November 24, 2020 17:22

Add support for UNIX datagram sockets in unix input

8a9f0a8

kvch force-pushed the feature-filebeat-unix-datagram-socket branch from 29f0f1f to 8a9f0a8 Compare November 24, 2020 17:24

urso reviewed Nov 24, 2020

View reviewed changes

kvch added 12 commits November 25, 2020 11:39

minor fixes to tests

db8b674

add dgram test

f719381

increase test coverage to 80%

f389291

more moves

9728568

add debug prints to check if events arrive

62afc31

tmp: one worker

13bf1c5

more debug

0a334f8

do not close connection until end of test

55d88f3

logging

64fb71a

use dial unix

4c80d2f

fix make check

560f8d2

tmp

413cd27

kvch force-pushed the feature-filebeat-unix-datagram-socket branch from 0a3f972 to 413cd27 Compare November 26, 2020 13:47

skip test on darwin

3d60fac

urso approved these changes Nov 26, 2020

View reviewed changes

fix syslog python tests

98f4cd8

kvch merged commit 3a1d1ae into elastic:master Nov 30, 2020

kvch added the needs_backport PR is waiting to be backported to other branches. label Nov 30, 2020

kvch mentioned this pull request Nov 30, 2020

Cherry-pick #22699 to 7.x: Add support for reading from UNIX datagram sockets #22791

Merged

5 tasks

kvch added v7.11.0 and removed needs_backport PR is waiting to be backported to other branches. labels Nov 30, 2020

kvch mentioned this pull request Nov 30, 2020

Separate recoverable and non-recoverable errors in UNIX datagram socket input #22792

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support for reading from UNIX datagram sockets #22699

Add support for reading from UNIX datagram sockets #22699

kvch commented Nov 20, 2020 •

edited

Loading

elasticmachine commented Nov 20, 2020 •

edited by jenkins-beats-ci bot

Loading

Build stats

Test stats 🧪

Test stats 🧪

elasticmachine commented Nov 23, 2020

urso Nov 24, 2020

urso Nov 24, 2020

kvch Nov 24, 2020

urso Nov 24, 2020

urso Nov 24, 2020

kvch Nov 25, 2020

urso Nov 25, 2020

urso commented Nov 24, 2020

kvch commented Nov 26, 2020

kvch commented Nov 26, 2020

kvch commented Nov 30, 2020

kvch commented Nov 30, 2020

Add support for reading from UNIX datagram sockets #22699

Add support for reading from UNIX datagram sockets #22699

Conversation

kvch commented Nov 20, 2020 • edited Loading

What does this PR do?

Why is it important?

Checklist

Related issues

elasticmachine commented Nov 20, 2020 • edited by jenkins-beats-ci bot Loading

💚 Build Succeeded

Build stats

Test stats 🧪

💚 Flaky test report

Test stats 🧪

elasticmachine commented Nov 23, 2020

urso Nov 24, 2020

Choose a reason for hiding this comment

urso Nov 24, 2020

Choose a reason for hiding this comment

kvch Nov 24, 2020

Choose a reason for hiding this comment

urso Nov 24, 2020

Choose a reason for hiding this comment

urso Nov 24, 2020

Choose a reason for hiding this comment

kvch Nov 25, 2020

Choose a reason for hiding this comment

urso Nov 25, 2020

Choose a reason for hiding this comment

urso commented Nov 24, 2020

kvch commented Nov 26, 2020

kvch commented Nov 26, 2020

kvch commented Nov 30, 2020

kvch commented Nov 30, 2020

kvch commented Nov 20, 2020 •

edited

Loading

elasticmachine commented Nov 20, 2020 •

edited by jenkins-beats-ci bot

Loading