[Journald] Restart journalctl if it exits unexpectedly

[belimawr]

Proposed commit message

If journalctl exits unexpectedly the journald input will restart it and set the cursor to the last know position. Any error/non zero return code is logged at level error.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Disruptive User Impact

Author's Checklist

• Test if this PR fixes Filbeat stops shipping journald logs when encountered "failed to read message field: bad message" error #32782

How to test this PR locally

1. Build Filebeat from this PR and start it with the following configuration file:

filebeat.yml

filebeat.inputs:
  - type: journald
    id: my-unique-input-id
    enabled: true
    syslog_identifiers:
      - "potato"

output:
  console:
    pretty: true

2. Publish some events with the syslog identfier potato

echo "$(date '+%Y-%m-%d %T') - foo"| systemd-cat -t potato

3. Wait to see the output in the console
4. Kill journalctl

killall journalctl

5. Wait journalctl to restart, look for a log entry like this:

{
  "log.level": "info",
  "@timestamp": "2024-08-20T12:41:29.682-0400",
  "log.logger": "input.journald.reader.journalctl-runner",
  "log.origin": {
    "function": "github.com/elastic/beats/v7/filebeat/input/journald/pkg/journalctl.Factory",
    "file.name": "journalctl/journalctl.go",
    "file.line": 120
  },
  "message": "journalctl started with PID 3119456",
  "service.name": "filebeat",
  "id": "filestream-input-id",
  "input_source": "LOCAL_SYSTEM_JOURNAL",
  "path": "LOCAL_SYSTEM_JOURNAL",
  "ecs.version": "1.6.0"
}

6. Publish a few more messages and ensure they appear in the console

echo "$(date '+%Y-%m-%d %T') - foo"| systemd-cat -t potato

Related issues

• Closes [Journald] input crashes with "failed to read message field: cannot allocate memory" #39352
• Closes Filbeat stops shipping journald logs when encountered "failed to read message field: bad message" error #32782

Tests

I run a test overnight where I used a mock to simulate constant failures and restarts of journalclt and monitored the host and Filebeat process to ensure there weren't any goroutine/resouce leaks. The screenshot below shows the CPU and memory usage from Filebeat as well as counters for the log entries stating journalctl crashed and was restarted.

## Use cases
## Screenshots
## Logs

[mergify]

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b restart-journalctl upstream/restart-journalctl
git merge upstream/main
git push upstream restart-journalctl

[mergify]

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b restart-journalctl upstream/restart-journalctl
git merge upstream/main
git push upstream restart-journalctl

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

[leehinman]

I'm wondering if we should also watch for the dataChan and errChan to close. For example, if we failed to read stdout for some reason, that go routine loop would exit

[belimawr]

This goroutine is here only to log any error returned by cmd.Wait, if the process exits unexpectedly its stderr and stdout will be closed and the reader gorourines will get an EOF or error. At the moment I'm doing the best to read and ship/log all data without over complicating the code and risking getting into a dead lock.

[leehinman]

if cancel.Done(), do we have to kill the process?

[belimawr]

This input.Canceler is in the context of the Next call, so I don't think we should kill the journalctl process. If the journald input is stopped, it will call Close on the Reader:

https://github.com/belimawr/beats/blob/e7d226030d8a90477949a4cd3a013e09c28db519/filebeat/input/journald/input.go#L236-L238

and the reader will kill the journalctl process

https://github.com/belimawr/beats/blob/e7d226030d8a90477949a4cd3a013e09c28db519/filebeat/input/journald/pkg/journalctl/reader.go#L192-L200.

I added some comments explaining kill needs to be called.

[mergify]

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b restart-journalctl upstream/restart-journalctl
git merge upstream/main
git push upstream restart-journalctl

[mergify]

backport-8.x has been added to help with the transition to the new branch 8.x.