Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Filebeat panic at startup #6145

Closed
andrewkroh opened this issue Jan 23, 2018 · 2 comments
Closed

Filebeat panic at startup #6145

andrewkroh opened this issue Jan 23, 2018 · 2 comments
Labels
Filebeat Filebeat

Comments

@andrewkroh
Copy link
Member

While running beats-tester there is a panic at startup for Filebeat.

For confirmed bugs, please report:

  • Version: 6.2.0 rc1
  • Operating System: Window 2012 R2
  • Steps to Reproduce: Run Beats-Tester
2018-01-22T16:55:39.473-0800    INFO    prospector/prospector.go:138    Stopping Prospector: 15026298159737792444
panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xc0000005 code=0x0 addr=0x0 pc=0x40131c]

goroutine 29 [running]:
sync/atomic.AddUint64(0x13b043b4, 0x1, 0x0, 0x8f6be5, 0x2e)
        /usr/local/go/src/sync/atomic/asm_386.s:112 +0xc
github.com/elastic/beats/libbeat/common/atomic.(*Uint64).Add(0x13b043b4, 0x1, 0x0, 0x0, 0x145e940)
        /go/src/github.com/elastic/beats/libbeat/common/atomic/atomic.go:66 +0x33
github.com/elastic/beats/libbeat/common/atomic.(*Uint64).Inc(0x13b043b4, 0x184ac8, 0x0)
        /go/src/github.com/elastic/beats/libbeat/common/atomic/atomic.go:68 +0x31
github.com/elastic/beats/filebeat/prospector/log.(*Prospector).startHarvester(0x13b042d0, 0x0, 0x0, 0x0, 0x13a15c0, 0x13
af23c0, 0x13ac1340, 0x10, 0x0, 0x0, ...)
        /go/src/github.com/elastic/beats/filebeat/prospector/log/prospector.go:594 +0x3e
github.com/elastic/beats/filebeat/prospector/log.(*Prospector).scan(0x13b042d0)
        /go/src/github.com/elastic/beats/filebeat/prospector/log/prospector.go:435 +0x4e4
github.com/elastic/beats/filebeat/prospector/log.(*Prospector).Run(0x13b042d0)
        /go/src/github.com/elastic/beats/filebeat/prospector/log/prospector.go:162 +0x8c
github.com/elastic/beats/filebeat/prospector.(*Prospector).Run(0x13af9140)
        /go/src/github.com/elastic/beats/filebeat/prospector/prospector.go:111 +0x2b
github.com/elastic/beats/filebeat/prospector.(*Prospector).Start.func1(0x13ac4c60, 0x13af9140)
        /go/src/github.com/elastic/beats/filebeat/prospector/prospector.go:104 +0x4b
created by github.com/elastic/beats/filebeat/prospector.(*Prospector).Start
        /go/src/github.com/elastic/beats/filebeat/prospector/prospector.go:97 +0x13a
PS C:\Users\vagrant\filebeat-6.2.0-windows-x86>

filebeat.log.txt
filebeat.yml.txt

@andrewkroh andrewkroh added the Filebeat Filebeat label Jan 23, 2018
andrewkroh added a commit to andrewkroh/beats-tester that referenced this issue Jan 23, 2018
- Add virtualenv for running Ansible in a reproducible manner.
- Change regex for matching Metricbeat output from the system-cpu metricset

  The order of the JSON keys in the `metricset` object is different in the output than before (I haven't determined why.)

- Update the win_copy dest argument to include the file name and not just the destination directory.
- Refactor the Makefile and update the README.
- Temporarily disable Filebeat testing on Windows while elastic/beats#6145 is resolved.
andrewkroh added a commit to andrewkroh/beats-tester that referenced this issue Jan 23, 2018
- Add virtualenv for running Ansible in a reproducible manner.
- Change regex for matching Metricbeat output from the system-cpu metricset

  The order of the JSON keys in the `metricset` object is different in the output than before (I haven't determined why.)

- Update the win_copy dest argument to include the file name and not just the destination directory.
- Refactor the Makefile and update the README.
- Temporarily disable Filebeat testing on Windows while elastic/beats#6145 is resolved.
@andrewkroh andrewkroh changed the title Filebeat panic on Windows at startup Filebeat panic at startup Jan 23, 2018
@andrewkroh
Copy link
Member Author

This is also happening on ubuntu-precise-32 with config:

filebeat:
  prospectors:
    -
      paths:
        - /var/log/test.log

output:
  file:
    path: /root/output
    filename: filebeat

Log:

root@vagrant-ubuntu-precise-32:~# filebeat -e -d "*" 2018-01-23T05:00:58.498Z INFO instance/beat.go:468 Home path: [/usr/share/filebeat] Config path: [/etc/filebeat] Data path: [/var/lib/filebeat] Logs path: [/var/log/filebeat] 2018-01-23T05:00:58.499Z DEBUG [beat] instance/beat.go:495 Beat metadata path: /var/lib/filebeat/meta.json 2018-01-23T05:00:58.501Z INFO instance/beat.go:475 Beat UUID: 1d097697-d99a-4352-b82a-2c3b3795782a 2018-01-23T05:00:58.502Z INFO instance/beat.go:213 Setup Beat: filebeat; Version: 6.2.0 2018-01-23T05:00:58.503Z DEBUG [beat] instance/beat.go:230 Initializing output plugins 2018-01-23T05:00:58.503Z DEBUG [processors] processors/processor.go:49 Processors: 2018-01-23T05:00:58.503Z INFO fileout/file.go:76 Initialized file output. path=/root/output/filebeat max_size_bytes=10485760 max_backups=7 permissions=-rw------- 2018-01-23T05:00:58.504Z INFO pipeline/module.go:76 Beat name: vagrant-ubuntu-precise-32 2018-01-23T05:00:58.504Z INFO instance/beat.go:301 filebeat start running. 2018-01-23T05:00:58.504Z DEBUG [registrar] registrar/registrar.go:88 Registry file set to: /var/lib/filebeat/registry 2018-01-23T05:00:58.505Z INFO registrar/registrar.go:108 Loading registrar data from /var/lib/filebeat/registry 2018-01-23T05:00:58.505Z INFO registrar/registrar.go:119 States Loaded from registrar: 0 2018-01-23T05:00:58.505Z WARN beater/filebeat.go:261 Filebeat is unable to load the Ingest Node pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the Ingest Node pipelines or are using Logstash pipelines, you can ignore this warning. 2018-01-23T05:00:58.505Z INFO crawler/crawler.go:48 Loading Prospectors: 1 2018-01-23T05:00:58.505Z DEBUG [processors] processors/processor.go:49 Processors: 2018-01-23T05:00:58.505Z DEBUG [prospector] log/config.go:178 recursive glob enabled 2018-01-23T05:00:58.504Z INFO [monitoring] log/log.go:95 Starting metrics logging every 30s 2018-01-23T05:00:58.506Z DEBUG [registrar] registrar/registrar.go:150 Starting Registrar 2018-01-23T05:00:58.506Z DEBUG [prospector] log/prospector.go:120 exclude_files: []. Number of stats: 0 2018-01-23T05:00:58.506Z DEBUG [prospector] log/prospector.go:141 Prospector with previous states loaded: 0 2018-01-23T05:00:58.506Z INFO log/prospector.go:111 Configured paths: [/var/log/test.log] 2018-01-23T05:00:58.506Z DEBUG [prospector] prospector/prospector.go:87 Starting prospector of type: log; ID: 14439312048545787300 2018-01-23T05:00:58.506Z INFO crawler/crawler.go:82 Loading and starting Prospectors completed. Enabled prospectors: 1 2018-01-23T05:00:58.506Z DEBUG [prospector] log/prospector.go:147 Start next scan 2018-01-23T05:00:58.506Z DEBUG [prospector] log/prospector.go:361 Check file for harvesting: /var/log/test.log 2018-01-23T05:00:58.506Z DEBUG [prospector] log/prospector.go:434 Start harvester for new file: /var/log/test.log 2018-01-23T05:00:58.506Z INFO prospector/prospector.go:138 Stopping Prospector: 14439312048545787300 panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x8662b0c]

goroutine 33 [running]:
sync/atomic.AddUint64(0x1a382594, 0x1, 0x0, 0x0, 0x1)
/usr/local/go/src/sync/atomic/asm_386.s:112 +0xc
github.com/elastic/beats/libbeat/common/atomic.(*Uint64).Add(0x1a382594, 0x1, 0x0, 0x1a33b9c8, 0x1a54f7a0)
/go/src/github.com/elastic/beats/libbeat/common/atomic/atomic.go:66 +0x33
github.com/elastic/beats/libbeat/common/atomic.(*Uint64).Inc(0x1a382594, 0xff, 0x9e34a79a)
/go/src/github.com/elastic/beats/libbeat/common/atomic/atomic.go:68 +0x31
github.com/elastic/beats/filebeat/prospector/log.(*Prospector).startHarvester(0x1a3824b0, 0x0, 0x0, 0x0, 0x95dcae0, 0x1a5a0e10, 0x1a4e8fa0, 0x11, 0x0, 0x0, ...)
/go/src/github.com/elastic/beats/filebeat/prospector/log/prospector.go:594 +0x3b
github.com/elastic/beats/filebeat/prospector/log.(*Prospector).scan(0x1a3824b0)
/go/src/github.com/elastic/beats/filebeat/prospector/log/prospector.go:435 +0x4e4
github.com/elastic/beats/filebeat/prospector/log.(*Prospector).Run(0x1a3824b0)
/go/src/github.com/elastic/beats/filebeat/prospector/log/prospector.go:162 +0x8c
github.com/elastic/beats/filebeat/prospector.(*Prospector).Run(0x1a576ac0)
/go/src/github.com/elastic/beats/filebeat/prospector/prospector.go:111 +0x2b
github.com/elastic/beats/filebeat/prospector.(*Prospector).Start.func1(0x1a527dc0, 0x1a576ac0)
/go/src/github.com/elastic/beats/filebeat/prospector/prospector.go:104 +0x4b
created by github.com/elastic/beats/filebeat/prospector.(*Prospector).Start
/go/src/github.com/elastic/beats/filebeat/prospector/prospector.go:97 +0x13a

@ruflin
Copy link
Contributor

ruflin commented Jan 23, 2018

I think the issue is on this line here:

if p.numHarvesters.Inc() > p.config.HarvesterLimit && p.config.HarvesterLimit > 0 {

It seems for whatever reason p.numHarvesters.Inc() does not exist or does not exist anymore. The strange part in the log is that based prospector/prospector.go:138 Stopping Prospector: 14439312048545787300 it seems like the prospector is directly stopped again after starting? On Windows I was thinking it could be because of an invalid path but @andrewkroh also found the issue on Ubuntu.

@urso could you have a look if there can be any reason that the atomic value above does not exist?

@andrewkroh Did you test on any other systems?
@adriansr It would be great if you could have a look as I assume the error was introduced with #5490

ruflin pushed a commit to elastic/beats-tester that referenced this issue Jan 23, 2018
- Add virtualenv for running Ansible in a reproducible manner.
- Change regex for matching Metricbeat output from the system-cpu metricset

  The order of the JSON keys in the `metricset` object is different in the output than before (I haven't determined why.)

- Update the win_copy dest argument to include the file name and not just the destination directory.
- Refactor the Makefile and update the README.
- Temporarily disable Filebeat testing on Windows while elastic/beats#6145 is resolved.
adriansr added a commit to adriansr/beats that referenced this issue Jan 23, 2018
There was a bug impacting 32-bit platforms where an atomic.Uint64
was not aligned to a 64 bit boundary, causing the log prospector
to panic on startup.

Instead of reordering the field to the start of the struct to guarantee
alignment, which is brittle, this patch just changes the harvester
limit to be a 32-bit unsigned integer.

Closes elastic#6145
exekias pushed a commit that referenced this issue Jan 23, 2018
There was a bug impacting 32-bit platforms where an atomic.Uint64
was not aligned to a 64 bit boundary, causing the log prospector
to panic on startup.

Instead of reordering the field to the start of the struct to guarantee
alignment, which is brittle, this patch just changes the harvester
limit to be a 32-bit unsigned integer.

Closes #6145
tsg pushed a commit to tsg/beats that referenced this issue Jan 23, 2018
There was a bug impacting 32-bit platforms where an atomic.Uint64
was not aligned to a 64 bit boundary, causing the log prospector
to panic on startup.

Instead of reordering the field to the start of the struct to guarantee
alignment, which is brittle, this patch just changes the harvester
limit to be a 32-bit unsigned integer.

Closes elastic#6145

(cherry picked from commit 9039cf7)
andrewkroh pushed a commit that referenced this issue Jan 23, 2018
There was a bug impacting 32-bit platforms where an atomic.Uint64
was not aligned to a 64 bit boundary, causing the log prospector
to panic on startup.

Instead of reordering the field to the start of the struct to guarantee
alignment, which is brittle, this patch just changes the harvester
limit to be a 32-bit unsigned integer.

Closes #6145

(cherry picked from commit 9039cf7)
leweafan pushed a commit to leweafan/beats that referenced this issue Apr 28, 2023
…ic#6155)

There was a bug impacting 32-bit platforms where an atomic.Uint64
was not aligned to a 64 bit boundary, causing the log prospector
to panic on startup.

Instead of reordering the field to the start of the struct to guarantee
alignment, which is brittle, this patch just changes the harvester
limit to be a 32-bit unsigned integer.

Closes elastic#6145

(cherry picked from commit 40489ff)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Filebeat Filebeat
Projects
None yet
Development

No branches or pull requests

2 participants