[8.x](backport #41762) Use fingerprint file identity by default and migrate file state from native or path

[mergify]

Proposed commit message

This commit changes the default file_identity from native to
fingerprint, any previous state from native (or path) is
automatically migrated to fingerprint whe Filestream is starting.

The Filestream input has always had the ability to update file identifiers,
however it never worked as expected, leading to full data duplication
when changing the file identity. This commit fixes it to allow
changing the file identity from native (inode + device ID) and
path to fingerprint without any data duplication.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Disruptive User Impact

Because the fingerprint is the new default file identity, files are now only ingested when they reach at least 1024 bytes. The old default behaviour can be enabled by setting the file identity to native and disabling the fingerprint in the scanner.

filebeat.inputs:
  - type: filestream
    id: "8.x-default-behaviour"
    paths:
      - /tmp/flog.log
    file_identity.native: ~
    prospector:
      scanner:
        fingerprint.enabled: false

Author's Checklist

• Test with dynamic config reload
• Test with Kubernetes
• Test with Elastic-Agent
• Fix all the tests that break with the new behaviour
• Investigate which integration tests are going to break in the Elastic-Agent repo

Regarding the Elastic-Agent integration tests, most tests actually use the log input because when they were written, Filestream was not available as an integration package. The very few other test that use Filestrem either generate a log file large enough or are skipped as flaky.

How to test this PR locally

1. Create a log file with at least a few log lines and more than 1kb (e.g: /tmp/flog.log, 15 log lines), you can use flog with Docker:

   docker run -it --rm mingrammer/flog -n 15 > /tmp/flog.log
   

2. Start Filebeat with the following configuration

   filebeat.yml (native)

   filebeat.inputs:
     - type: filestream
       id: "test-migrate-ID"
       paths:
         - /tmp/flog.log
       file_identity.native: ~
       prospector:
         scanner:
           check_interval: 0.1s
           fingerprint.enabled: false
   
   queue.mem:
     flush.timeout: 0s
   
   output.file:
     path: ${path.home}
     filename: "output-file"
     rotate_on_startup: false
   
   logging:
     level: debug
     selectors:
       - input
       - input.filestream
       - input.filestream.prospector
     metrics:
       enabled: false

3. Wait until the file is fully ingested (wait for End of file reached: /tmp/flog.log; Backoff now. in the logs)

4. Ensure all events have been published to the output (wc -l ./output-file* should return 15)

5. Stop Filebeat

6. Change the file identity to fingerprint. It's the new default, hence it's not explicitly set.

   filebeat.yml (fingerprint)

   filebeat.inputs:
     - type: filestream
       id: "test-migrate-ID"
       paths:
         - /tmp/flog.log
       prospector:
         scanner:
           check_interval: 0.1s
   
   queue.mem:
     flush.timeout: 0s
   
   output.file:
     path: ${path.home}
     filename: "output-file"
     rotate_on_startup: false
   
   logging:
     level: debug
     selectors:
       - input
       - input.filestream
       - input.filestream.prospector
     metrics:
       enabled: false

7. Start Filebeat

8. Wait until the Filebeat "finds the end of the file" (wait for End of file reached: /tmp/flog.log; Backoff now. in the logs)

9. Ensure no extra event was published ((wc -l ./output-file* should still return 15)

10. Add 10 more lines to the file:

    docker run -it --rm mingrammer/flog -n 10 >> /tmp/flog.log
    

11. Wait until the new lines are ingested (wait for End of file reached: /tmp/flog.log; Backoff now. in the logs)

12. Ensure all events have been published to the output with no duplication (wc -l ./output-file* should return 25)

Related issues

• Closes Use fingerprint file identity by default and migrate all existing filestream inputs to it #40197

Use cases

Dealing with identity reuse (e.g: inode reuse) without facing re-ingestion of data with Filestream input

## Screenshots

Logs

---

This is an automatic backport of pull request #41762 done by [Mergify](https://mergify.com).

[mergify]

Cherry-pick of 78fe7a5 has failed:

On branch mergify/bp/8.x/pr-41762
Your branch is up to date with 'origin/8.x'.

You are currently cherry-picking commit 78fe7a5b7.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Changes to be committed:
	modified:   CHANGELOG.next.asciidoc
	modified:   filebeat/_meta/config/filebeat.global.reference.yml.tmpl
	modified:   filebeat/_meta/config/filebeat.inputs.reference.yml.tmpl
	modified:   filebeat/docs/faq.asciidoc
	modified:   filebeat/docs/inputs/input-filestream-file-options.asciidoc
	modified:   filebeat/docs/inputs/input-filestream.asciidoc
	modified:   filebeat/filebeat.reference.yml
	modified:   filebeat/include/list.go
	modified:   filebeat/input/filestream/environment_test.go
	modified:   filebeat/input/filestream/fswatch.go
	modified:   filebeat/input/filestream/fswatch_test.go
	modified:   filebeat/input/filestream/identifier.go
	modified:   filebeat/input/filestream/identifier_test.go
	modified:   filebeat/input/filestream/input_integration_test.go
	modified:   filebeat/input/filestream/input_test.go
	modified:   filebeat/input/filestream/internal/input-logfile/prospector.go
	modified:   filebeat/input/filestream/internal/input-logfile/store.go
	modified:   filebeat/input/filestream/internal/input-logfile/store_test.go
	modified:   filebeat/input/filestream/legacy_metrics_integration_test.go
	modified:   filebeat/input/filestream/metrics_integration_test.go
	modified:   filebeat/input/filestream/parsers_integration_test.go
	modified:   filebeat/input/filestream/prospector.go
	modified:   filebeat/input/filestream/prospector_creator.go
	modified:   filebeat/input/filestream/prospector_test.go
	new file:   filebeat/input/filestream/testdata/log.log
	modified:   filebeat/tests/integration/event_log_file_test.go
	modified:   filebeat/tests/integration/filestream_truncation_test.go
	modified:   filebeat/tests/integration/store_test.go
	modified:   filebeat/tests/integration/translate_ldap_attribute_test.go
	modified:   filebeat/tests/system/config/filestream-fixup-id.yml.j2
	modified:   filebeat/tests/system/test_reload_inputs.py
	modified:   libbeat/tests/integration/framework.go
	modified:   x-pack/filebeat/filebeat.reference.yml

Unmerged paths:
  (use "git add <file>..." to mark resolution)
	both modified:   filebeat/tests/integration/filestream_test.go

To fix up this pull request, you can check it out locally. See documentation: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally

[botelastic]

This pull request doesn't have a Team:<team> label.