Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Filebeat] Flaky test_clean_removed_with_clean_inactive test #7842

Closed
ruflin opened this issue Aug 2, 2018 · 1 comment
Closed

[Filebeat] Flaky test_clean_removed_with_clean_inactive test #7842

ruflin opened this issue Aug 2, 2018 · 1 comment
Labels
Filebeat Filebeat flaky-test Unstable or unreliable test cases.

Comments

@ruflin
Copy link
Contributor

ruflin commented Aug 2, 2018

Error:

10:51:00 ======================================================================
10:51:00 FAIL: Checks that files which were removed, the state is removed
10:51:00 ----------------------------------------------------------------------
10:51:00 Traceback (most recent call last):
10:51:00   File "/private/var/lib/jenkins/workspace/elastic+beats+master+multijob-darwin/beat/filebeat/label/macosx/src/github.com/elastic/beats/filebeat/tests/system/test_registrar.py", line 969, in test_clean_removed_with_clean_inactive
10:51:00     assert len(data) == 1
10:51:00 AssertionError
10:51:00 
10:51:00 ----------------------------------------------------------------------
10:51:00 XML: /private/var/lib/jenkins/workspace/elastic+beats+master+multijob-darwin/beat/filebeat/label/macosx/src/github.com/elastic/beats/filebeat/build/TEST-system.xml

Log

2018-07-31T03:48:54.817-0700	INFO	instance/beat.go:544	Home path: [/private/var/lib/jenkins/workspace/elastic+beats+master+multijob-darwin/beat/filebeat/label/macosx/src/github.com/elastic/beats/filebeat/build/system-tests/run/test_registrar.Test.test_clean_removed_with_clean_inactive] Config path: [/private/var/lib/jenkins/workspace/elastic+beats+master+multijob-darwin/beat/filebeat/label/macosx/src/github.com/elastic/beats/filebeat/build/system-tests/run/test_registrar.Test.test_clean_removed_with_clean_inactive] Data path: [/private/var/lib/jenkins/workspace/elastic+beats+master+multijob-darwin/beat/filebeat/label/macosx/src/github.com/elastic/beats/filebeat/build/system-tests/run/test_registrar.Test.test_clean_removed_with_clean_inactive/data] Logs path: [/private/var/lib/jenkins/workspace/elastic+beats+master+multijob-darwin/beat/filebeat/label/macosx/src/github.com/elastic/beats/filebeat/build/system-tests/run/test_registrar.Test.test_clean_removed_with_clean_inactive/logs]
2018-07-31T03:48:54.818-0700	DEBUG	[beat]	instance/beat.go:571	Beat metadata path: /private/var/lib/jenkins/workspace/elastic+beats+master+multijob-darwin/beat/filebeat/label/macosx/src/github.com/elastic/beats/filebeat/build/system-tests/run/test_registrar.Test.test_clean_removed_with_clean_inactive/data/meta.json
2018-07-31T03:48:54.818-0700	INFO	instance/beat.go:551	Beat UUID: b288e142-593b-4b5e-8ea7-2e87a5c8404d
2018-07-31T03:48:54.818-0700	DEBUG	[seccomp]	seccomp/seccomp.go:88	Syscall filtering is only supported on Linux
2018-07-31T03:48:54.819-0700	INFO	[beat]	instance/beat.go:768	Beat info	{"system_info": {"beat": {"path": {"config": "/private/var/lib/jenkins/workspace/elastic+beats+master+multijob-darwin/beat/filebeat/label/macosx/src/github.com/elastic/beats/filebeat/build/system-tests/run/test_registrar.Test.test_clean_removed_with_clean_inactive", "data": "/private/var/lib/jenkins/workspace/elastic+beats+master+multijob-darwin/beat/filebeat/label/macosx/src/github.com/elastic/beats/filebeat/build/system-tests/run/test_registrar.Test.test_clean_removed_with_clean_inactive/data", "home": "/private/var/lib/jenkins/workspace/elastic+beats+master+multijob-darwin/beat/filebeat/label/macosx/src/github.com/elastic/beats/filebeat/build/system-tests/run/test_registrar.Test.test_clean_removed_with_clean_inactive", "logs": "/private/var/lib/jenkins/workspace/elastic+beats+master+multijob-darwin/beat/filebeat/label/macosx/src/github.com/elastic/beats/filebeat/build/system-tests/run/test_registrar.Test.test_clean_removed_with_clean_inactive/logs"}, "type": "filebeat", "uuid": "b288e142-593b-4b5e-8ea7-2e87a5c8404d"}}}
2018-07-31T03:48:54.819-0700	INFO	[beat]	instance/beat.go:777	Build info	{"system_info": {"build": {"commit": "unknown", "libbeat": "7.0.0-alpha1", "time": "1754-08-30T22:43:41.128Z", "version": "7.0.0-alpha1"}}}
2018-07-31T03:48:54.819-0700	INFO	[beat]	instance/beat.go:780	Go runtime info	{"system_info": {"go": {"os":"darwin","arch":"amd64","max_procs":4,"version":"go1.10.3"}}}
2018-07-31T03:48:54.821-0700	INFO	[beat]	instance/beat.go:784	Host info	{"system_info": {"host": {"architecture":"x86_64","boot_time":"2018-06-05T02:32:05.492851-07:00","hostname":"slave-c07ll940dwyl.local","ips":["127.0.0.1/8","::1/128","fe80::1/64","fe80::1856:4442:f1aa:20c9/64","207.254.41.61/23","192.168.99.1/24"],"kernel_version":"16.7.0","mac_addresses":["68:5b:35:99:c0:e7","a8:86:dd:a1:0e:cf","44:fb:42:ff:fe:c3:48:30","32:00:1c:34:83:00","0a:86:dd:a1:0e:cf","36:70:34:59:aa:95","32:00:1c:34:83:00","0a:00:27:00:00:00","0a:00:27:00:00:01"],"os":{"family":"darwin","platform":"darwin","name":"Mac OS X","version":"10.12.6","major":10,"minor":12,"patch":6,"build":"16G1212"},"timezone":"PDT","timezone_offset_sec":-25200}}}
2018-07-31T03:48:54.823-0700	INFO	[beat]	instance/beat.go:813	Process info	{"system_info": {"process": {"cwd": "/private/var/lib/jenkins/workspace/elastic+beats+master+multijob-darwin/beat/filebeat/label/macosx/src/github.com/elastic/beats/filebeat", "exe": "/private/var/lib/jenkins/workspace/elastic+beats+master+multijob-darwin/beat/filebeat/label/macosx/src/github.com/elastic/beats/filebeat/filebeat.test", "name": "filebeat.test", "pid": 26009, "ppid": 25873, "start_time": "2018-07-31T03:48:54.718-0700"}}}
2018-07-31T03:48:54.823-0700	INFO	instance/beat.go:273	Setup Beat: filebeat; Version: 7.0.0-alpha1
2018-07-31T03:48:54.823-0700	DEBUG	[beat]	instance/beat.go:290	Initializing output plugins
2018-07-31T03:48:54.823-0700	DEBUG	[processors]	processors/processor.go:66	Processors: 
2018-07-31T03:48:54.823-0700	DEBUG	[rotator]	file/rotator.go:135	Initialized file rotator	{"rotator": {"filename": "/private/var/lib/jenkins/workspace/elastic+beats+master+multijob-darwin/beat/filebeat/label/macosx/src/github.com/elastic/beats/filebeat/build/system-tests/run/test_registrar.Test.test_clean_removed_with_clean_inactive/output/filebeat", "max_size_bytes": 1024000, "max_backups": 7, "permissions": "-rw-------"}}
2018-07-31T03:48:54.824-0700	INFO	fileout/file.go:94	Initialized file output. path=/private/var/lib/jenkins/workspace/elastic+beats+master+multijob-darwin/beat/filebeat/label/macosx/src/github.com/elastic/beats/filebeat/build/system-tests/run/test_registrar.Test.test_clean_removed_with_clean_inactive/output/filebeat max_size_bytes=1024000 max_backups=7 permissions=-rw-------
2018-07-31T03:48:54.825-0700	DEBUG	[publish]	pipeline/consumer.go:137	start pipeline event consumer
2018-07-31T03:48:54.825-0700	INFO	pipeline/module.go:98	Beat name: slave-c07ll940dwyl.local
2018-07-31T03:48:54.826-0700	ERROR	fileset/modules.go:118	Not loading modules. Module directory not found: /private/var/lib/jenkins/workspace/elastic+beats+master+multijob-darwin/beat/filebeat/label/macosx/src/github.com/elastic/beats/filebeat/build/system-tests/run/test_registrar.Test.test_clean_removed_with_clean_inactive/module
2018-07-31T03:48:54.826-0700	INFO	[monitoring]	log/log.go:114	Starting metrics logging every 30s
2018-07-31T03:48:54.827-0700	INFO	instance/beat.go:367	filebeat start running.
2018-07-31T03:48:54.827-0700	INFO	registrar/registrar.go:97	No registry file found under: /private/var/lib/jenkins/workspace/elastic+beats+master+multijob-darwin/beat/filebeat/label/macosx/src/github.com/elastic/beats/filebeat/build/system-tests/run/test_registrar.Test.test_clean_removed_with_clean_inactive/registry. Creating a new registry file.
2018-07-31T03:48:54.827-0700	DEBUG	[registrar]	registrar/registrar.go:400	Write registry file: /private/var/lib/jenkins/workspace/elastic+beats+master+multijob-darwin/beat/filebeat/label/macosx/src/github.com/elastic/beats/filebeat/build/system-tests/run/test_registrar.Test.test_clean_removed_with_clean_inactive/registry
2018-07-31T03:48:54.828-0700	DEBUG	[registrar]	registrar/registrar.go:393	Registry file updated. 0 states written.
2018-07-31T03:48:54.828-0700	INFO	registrar/registrar.go:134	Loading registrar data from /private/var/lib/jenkins/workspace/elastic+beats+master+multijob-darwin/beat/filebeat/label/macosx/src/github.com/elastic/beats/filebeat/build/system-tests/run/test_registrar.Test.test_clean_removed_with_clean_inactive/registry
2018-07-31T03:48:54.828-0700	INFO	registrar/registrar.go:141	States Loaded from registrar: 0
2018-07-31T03:48:54.829-0700	WARN	beater/filebeat.go:371	Filebeat is unable to load the Ingest Node pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the Ingest Node pipelines or are using Logstash pipelines, you can ignore this warning.
2018-07-31T03:48:54.829-0700	DEBUG	[registrar]	registrar/registrar.go:267	Starting Registrar
2018-07-31T03:48:54.829-0700	INFO	crawler/crawler.go:72	Loading Inputs: 1
2018-07-31T03:48:54.830-0700	DEBUG	[processors]	processors/processor.go:66	Processors: 
2018-07-31T03:48:54.832-0700	DEBUG	[input]	log/config.go:201	recursive glob enabled
2018-07-31T03:48:54.834-0700	DEBUG	[input]	log/input.go:147	exclude_files: []. Number of stats: 0
2018-07-31T03:48:54.834-0700	DEBUG	[input]	log/input.go:168	input with previous states loaded: 0
2018-07-31T03:48:54.834-0700	INFO	log/input.go:138	Configured paths: [/private/var/lib/jenkins/workspace/elastic+beats+master+multijob-darwin/beat/filebeat/label/macosx/src/github.com/elastic/beats/filebeat/build/system-tests/run/test_registrar.Test.test_clean_removed_with_clean_inactive/log/input*]
2018-07-31T03:48:54.834-0700	INFO	input/input.go:114	Starting input of type: log; ID: 8456096243672335408 
2018-07-31T03:48:54.835-0700	INFO	crawler/crawler.go:106	Loading and starting Inputs completed. Enabled inputs: 1
2018-07-31T03:48:54.835-0700	DEBUG	[input]	log/input.go:174	Start next scan
2018-07-31T03:48:54.835-0700	DEBUG	[input]	log/input.go:404	Check file for harvesting: /private/var/lib/jenkins/workspace/elastic+beats+master+multijob-darwin/beat/filebeat/label/macosx/src/github.com/elastic/beats/filebeat/build/system-tests/run/test_registrar.Test.test_clean_removed_with_clean_inactive/log/input1
2018-07-31T03:48:54.835-0700	DEBUG	[input]	log/input.go:477	Start harvester for new file: /private/var/lib/jenkins/workspace/elastic+beats+master+multijob-darwin/beat/filebeat/label/macosx/src/github.com/elastic/beats/filebeat/build/system-tests/run/test_registrar.Test.test_clean_removed_with_clean_inactive/log/input1
2018-07-31T03:48:54.837-0700	DEBUG	[harvester]	log/harvester.go:486	Setting offset for file based on seek: /private/var/lib/jenkins/workspace/elastic+beats+master+multijob-darwin/beat/filebeat/label/macosx/src/github.com/elastic/beats/filebeat/build/system-tests/run/test_registrar.Test.test_clean_removed_with_clean_inactive/log/input1
2018-07-31T03:48:54.837-0700	DEBUG	[harvester]	log/harvester.go:472	Setting offset for file: /private/var/lib/jenkins/workspace/elastic+beats+master+multijob-darwin/beat/filebeat/label/macosx/src/github.com/elastic/beats/filebeat/build/system-tests/run/test_registrar.Test.test_clean_removed_with_clean_inactive/log/input1. Offset: 0 
2018-07-31T03:48:54.837-0700	DEBUG	[harvester]	log/harvester.go:387	Update state: /private/var/lib/jenkins/workspace/elastic+beats+master+multijob-darwin/beat/filebeat/label/macosx/src/github.com/elastic/beats/filebeat/build/system-tests/run/test_registrar.Test.test_clean_removed_with_clean_inactive/log/input1, offset: 0
2018-07-31T03:48:54.837-0700	DEBUG	[input]	file/states.go:68	New state added for /private/var/lib/jenkins/workspace/elastic+beats+master+multijob-darwin/beat/filebeat/label/macosx/src/github.com/elastic/beats/filebeat/build/system-tests/run/test_registrar.Test.test_clean_removed_with_clean_inactive/log/input1
2018-07-31T03:48:54.837-0700	DEBUG	[acker]	beater/acker.go:64	stateful ack	{"count": 1}
2018-07-31T03:48:54.838-0700	DEBUG	[registrar]	registrar/registrar.go:345	Processing 1 events
2018-07-31T03:48:54.838-0700	DEBUG	[input]	log/input.go:404	Check file for harvesting: /private/var/lib/jenkins/workspace/elastic+beats+master+multijob-darwin/beat/filebeat/label/macosx/src/github.com/elastic/beats/filebeat/build/system-tests/run/test_registrar.Test.test_clean_removed_with_clean_inactive/log/input2
2018-07-31T03:48:54.838-0700	DEBUG	[input]	file/states.go:68	New state added for /private/var/lib/jenkins/workspace/elastic+beats+master+multijob-darwin/beat/filebeat/label/macosx/src/github.com/elastic/beats/filebeat/build/system-tests/run/test_registrar.Test.test_clean_removed_with_clean_inactive/log/input1
2018-07-31T03:48:54.838-0700	DEBUG	[input]	log/input.go:477	Start harvester for new file: /private/var/lib/jenkins/workspace/elastic+beats+master+multijob-darwin/beat/filebeat/label/macosx/src/github.com/elastic/beats/filebeat/build/system-tests/run/test_registrar.Test.test_clean_removed_with_clean_inactive/log/input2
2018-07-31T03:48:54.838-0700	DEBUG	[registrar]	registrar/registrar.go:315	Registrar state updates processed. Count: 1
2018-07-31T03:48:54.838-0700	DEBUG	[registrar]	registrar/registrar.go:335	Registrar states cleaned up. Before: 1, After: 1, Pending: 1
2018-07-31T03:48:54.838-0700	DEBUG	[registrar]	registrar/registrar.go:400	Write registry file: /private/var/lib/jenkins/workspace/elastic+beats+master+multijob-darwin/beat/filebeat/label/macosx/src/github.com/elastic/beats/filebeat/build/system-tests/run/test_registrar.Test.test_clean_removed_with_clean_inactive/registry
2018-07-31T03:48:54.839-0700	DEBUG	[harvester]	log/harvester.go:486	Setting offset for file based on seek: /private/var/lib/jenkins/workspace/elastic+beats+master+multijob-darwin/beat/filebeat/label/macosx/src/github.com/elastic/beats/filebeat/build/system-tests/run/test_registrar.Test.test_clean_removed_with_clean_inactive/log/input2
2018-07-31T03:48:54.839-0700	DEBUG	[registrar]	registrar/registrar.go:393	Registry file updated. 1 states written.
2018-07-31T03:48:54.839-0700	DEBUG	[harvester]	log/harvester.go:472	Setting offset for file: /private/var/lib/jenkins/workspace/elastic+beats+master+multijob-darwin/beat/filebeat/label/macosx/src/github.com/elastic/beats/filebeat/build/system-tests/run/test_registrar.Test.test_clean_removed_with_clean_inactive/log/input2. Offset: 0 
2018-07-31T03:48:54.839-0700	DEBUG	[harvester]	log/harvester.go:387	Update state: /private/var/lib/jenkins/workspace/elastic+beats+master+multijob-darwin/beat/filebeat/label/macosx/src/github.com/elastic/beats/filebeat/build/system-tests/run/test_registrar.Test.test_clean_removed_with_clean_inactive/log/input2, offset: 0
2018-07-31T03:48:54.839-0700	DEBUG	[input]	file/states.go:68	New state added for /private/var/lib/jenkins/workspace/elastic+beats+master+multijob-darwin/beat/filebeat/label/macosx/src/github.com/elastic/beats/filebeat/build/system-tests/run/test_registrar.Test.test_clean_removed_with_clean_inactive/log/input2
2018-07-31T03:48:54.840-0700	DEBUG	[acker]	beater/acker.go:64	stateful ack	{"count": 1}
2018-07-31T03:48:54.840-0700	DEBUG	[registrar]	registrar/registrar.go:345	Processing 1 events
2018-07-31T03:48:54.840-0700	DEBUG	[input]	file/states.go:68	New state added for /private/var/lib/jenkins/workspace/elastic+beats+master+multijob-darwin/beat/filebeat/label/macosx/src/github.com/elastic/beats/filebeat/build/system-tests/run/test_registrar.Test.test_clean_removed_with_clean_inactive/log/input2
2018-07-31T03:48:54.840-0700	INFO	log/harvester.go:251	Harvester started for file: /private/var/lib/jenkins/workspace/elastic+beats+master+multijob-darwin/beat/filebeat/label/macosx/src/github.com/elastic/beats/filebeat/build/system-tests/run/test_registrar.Test.test_clean_removed_with_clean_inactive/log/input1
2018-07-31T03:48:54.840-0700	DEBUG	[registrar]	registrar/registrar.go:315	Registrar state updates processed. Count: 1
2018-07-31T03:48:54.840-0700	DEBUG	[input]	log/input.go:195	input states cleaned up. Before: 2, After: 2, Pending: 2
2018-07-31T03:48:54.841-0700	DEBUG	[publish]	pipeline/processor.go:308	Publish event: {
  "@timestamp": "2018-07-31T10:48:54.840Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "doc",
    "version": "7.0.0-alpha1"
  },
  "source": "/private/var/lib/jenkins/workspace/elastic+beats+master+multijob-darwin/beat/filebeat/label/macosx/src/github.com/elastic/beats/filebeat/build/system-tests/run/test_registrar.Test.test_clean_removed_with_clean_inactive/log/input1",
  "offset": 0,
  "message": "file to be removed",
  "prospector": {
    "type": "log"
  },
  "input": {
    "type": "log"
  },
  "beat": {
    "name": "slave-c07ll940dwyl.local",
    "hostname": "slave-c07ll940dwyl.local",
    "version": "7.0.0-alpha1"
  },
  "host": {
    "name": "slave-c07ll940dwyl.local"
  }
}
2018-07-31T03:48:54.841-0700	INFO	log/harvester.go:251	Harvester started for file: /private/var/lib/jenkins/workspace/elastic+beats+master+multijob-darwin/beat/filebeat/label/macosx/src/github.com/elastic/beats/filebeat/build/system-tests/run/test_registrar.Test.test_clean_removed_with_clean_inactive/log/input2
2018-07-31T03:48:54.840-0700	DEBUG	[registrar]	registrar/registrar.go:335	Registrar states cleaned up. Before: 2, After: 2, Pending: 2
2018-07-31T03:48:54.842-0700	DEBUG	[registrar]	registrar/registrar.go:400	Write registry file: /private/var/lib/jenkins/workspace/elastic+beats+master+multijob-darwin/beat/filebeat/label/macosx/src/github.com/elastic/beats/filebeat/build/system-tests/run/test_registrar.Test.test_clean_removed_with_clean_inactive/registry
2018-07-31T03:48:54.842-0700	DEBUG	[harvester]	log/log.go:102	End of file reached: /private/var/lib/jenkins/workspace/elastic+beats+master+multijob-darwin/beat/filebeat/label/macosx/src/github.com/elastic/beats/filebeat/build/system-tests/run/test_registrar.Test.test_clean_removed_with_clean_inactive/log/input1; Backoff now.
2018-07-31T03:48:54.843-0700	DEBUG	[registrar]	registrar/registrar.go:393	Registry file updated. 2 states written.
2018-07-31T03:48:54.843-0700	DEBUG	[publish]	pipeline/processor.go:308	Publish event: {
  "@timestamp": "2018-07-31T10:48:54.842Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "doc",
    "version": "7.0.0-alpha1"
  },
  "beat": {
    "name": "slave-c07ll940dwyl.local",
    "hostname": "slave-c07ll940dwyl.local",
    "version": "7.0.0-alpha1"
  },
  "host": {
    "name": "slave-c07ll940dwyl.local"
  },
  "source": "/private/var/lib/jenkins/workspace/elastic+beats+master+multijob-darwin/beat/filebeat/label/macosx/src/github.com/elastic/beats/filebeat/build/system-tests/run/test_registrar.Test.test_clean_removed_with_clean_inactive/log/input2",
  "offset": 0,
  "message": "2",
  "input": {
    "type": "log"
  },
  "prospector": {
    "type": "log"
  }
}
2018-07-31T03:48:54.843-0700	DEBUG	[harvester]	log/log.go:102	End of file reached: /private/var/lib/jenkins/workspace/elastic+beats+master+multijob-darwin/beat/filebeat/label/macosx/src/github.com/elastic/beats/filebeat/build/system-tests/run/test_registrar.Test.test_clean_removed_with_clean_inactive/log/input2; Backoff now.
2018-07-31T03:48:55.012-0700	DEBUG	[input]	input/input.go:152	Run input
2018-07-31T03:48:55.013-0700	DEBUG	[input]	log/input.go:174	Start next scan
2018-07-31T03:48:55.013-0700	DEBUG	[input]	log/input.go:404	Check file for harvesting: /private/var/lib/jenkins/workspace/elastic+beats+master+multijob-darwin/beat/filebeat/label/macosx/src/github.com/elastic/beats/filebeat/build/system-tests/run/test_registrar.Test.test_clean_removed_with_clean_inactive/log/input2
2018-07-31T03:48:55.013-0700	DEBUG	[input]	log/input.go:494	Update existing file for harvesting: /private/var/lib/jenkins/workspace/elastic+beats+master+multijob-darwin/beat/filebeat/label/macosx/src/github.com/elastic/beats/filebeat/build/system-tests/run/test_registrar.Test.test_clean_removed_with_clean_inactive/log/input2, offset: 2
2018-07-31T03:48:55.013-0700	DEBUG	[input]	log/input.go:546	Harvester for file is still running: /private/var/lib/jenkins/workspace/elastic+beats+master+multijob-darwin/beat/filebeat/label/macosx/src/github.com/elastic/beats/filebeat/build/system-tests/run/test_registrar.Test.test_clean_removed_with_clean_inactive/log/input2
2018-07-31T03:48:55.013-0700	DEBUG	[input]	log/input.go:404	Check file for harvesting: /private/var/lib/jenkins/workspace/elastic+beats+master+multijob-darwin/beat/filebeat/label/macosx/src/github.com/elastic/beats/filebeat/build/system-tests/run/test_registrar.Test.test_clean_removed_with_clean_inactive/log/input1
2018-07-31T03:48:55.014-0700	DEBUG	[input]	log/input.go:494	Update existing file for harvesting: /private/var/lib/jenkins/workspace/elastic+beats+master+multijob-darwin/beat/filebeat/label/macosx/src/github.com/elastic/beats/filebeat/build/system-tests/run/test_registrar.Test.test_clean_removed_with_clean_inactive/log/input1, offset: 19
2018-07-31T03:48:55.014-0700	DEBUG	[input]	log/input.go:546	Harvester for file is still running: /private/var/lib/jenkins/workspace/elastic+beats+master+multijob-darwin/beat/filebeat/label/macosx/src/github.com/elastic/beats/filebeat/build/system-tests/run/test_registrar.Test.test_clean_removed_with_clean_inactive/log/input1
2018-07-31T03:48:55.014-0700	DEBUG	[memqueue]	memqueue/ackloop.go:160	ackloop: receive ack [0: 0, 2]
2018-07-31T03:48:55.014-0700	DEBUG	[input]	log/input.go:195	input states cleaned up. Before: 2, After: 2, Pending: 2
2018-07-31T03:48:55.014-0700	DEBUG	[harvester]	log/log.go:102	End of file reached: /private/var/lib/jenkins/workspace/elastic+beats+master+multijob-darwin/beat/filebeat/label/macosx/src/github.com/elastic/beats/filebeat/build/system-tests/run/test_registrar.Test.test_clean_removed_with_clean_inactive/log/input1; Backoff now.
2018-07-31T03:48:55.014-0700	DEBUG	[memqueue]	memqueue/eventloop.go:535	broker ACK events: count=2, start-seq=1, end-seq=2

2018-07-31T03:48:55.014-0700	DEBUG	[memqueue]	memqueue/ackloop.go:128	ackloop: return ack to broker loop:2
2018-07-31T03:48:55.014-0700	DEBUG	[acker]	beater/acker.go:64	stateful ack	{"count": 2}
2018-07-31T03:48:55.014-0700	DEBUG	[memqueue]	memqueue/ackloop.go:131	ackloop:  done send ack
2018-07-31T03:48:55.014-0700	DEBUG	[registrar]	registrar/registrar.go:345	Processing 2 events
2018-07-31T03:48:55.014-0700	DEBUG	[registrar]	registrar/registrar.go:315	Registrar state updates processed. Count: 2
2018-07-31T03:48:55.015-0700	DEBUG	[registrar]	registrar/registrar.go:335	Registrar states cleaned up. Before: 2, After: 2, Pending: 2
2018-07-31T03:48:55.015-0700	DEBUG	[registrar]	registrar/registrar.go:400	Write registry file: /private/var/lib/jenkins/workspace/elastic+beats+master+multijob-darwin/beat/filebeat/label/macosx/src/github.com/elastic/beats/filebeat/build/system-tests/run/test_registrar.Test.test_clean_removed_with_clean_inactive/registry
2018-07-31T03:48:55.016-0700	DEBUG	[registrar]	registrar/registrar.go:393	Registry file updated. 2 states written.
2018-07-31T03:48:55.018-0700	DEBUG	[harvester]	log/log.go:102	End of file reached: /private/var/lib/jenkins/workspace/elastic+beats+master+multijob-darwin/beat/filebeat/label/macosx/src/github.com/elastic/beats/filebeat/build/system-tests/run/test_registrar.Test.test_clean_removed_with_clean_inactive/log/input2; Backoff now.
2018-07-31T03:48:55.179-0700	DEBUG	[input]	input/input.go:152	Run input
2018-07-31T03:48:55.180-0700	DEBUG	[input]	log/input.go:174	Start next scan
2018-07-31T03:48:55.180-0700	INFO	log/harvester.go:268	File was removed: /private/var/lib/jenkins/workspace/elastic+beats+master+multijob-darwin/beat/filebeat/label/macosx/src/github.com/elastic/beats/filebeat/build/system-tests/run/test_registrar.Test.test_clean_removed_with_clean_inactive/log/input1. Closing because close_removed is enabled.
2018-07-31T03:48:55.180-0700	DEBUG	[input]	log/input.go:404	Check file for harvesting: /private/var/lib/jenkins/workspace/elastic+beats+master+multijob-darwin/beat/filebeat/label/macosx/src/github.com/elastic/beats/filebeat/build/system-tests/run/test_registrar.Test.test_clean_removed_with_clean_inactive/log/input2
2018-07-31T03:48:55.180-0700	DEBUG	[input]	log/input.go:494	Update existing file for harvesting: /private/var/lib/jenkins/workspace/elastic+beats+master+multijob-darwin/beat/filebeat/label/macosx/src/github.com/elastic/beats/filebeat/build/system-tests/run/test_registrar.Test.test_clean_removed_with_clean_inactive/log/input2, offset: 2
2018-07-31T03:48:55.180-0700	DEBUG	[harvester]	log/harvester.go:507	Stopping harvester for file: /private/var/lib/jenkins/workspace/elastic+beats+master+multijob-darwin/beat/filebeat/label/macosx/src/github.com/elastic/beats/filebeat/build/system-tests/run/test_registrar.Test.test_clean_removed_with_clean_inactive/log/input1
2018-07-31T03:48:55.180-0700	DEBUG	[input]	log/input.go:546	Harvester for file is still running: /private/var/lib/jenkins/workspace/elastic+beats+master+multijob-darwin/beat/filebeat/label/macosx/src/github.com/elastic/beats/filebeat/build/system-tests/run/test_registrar.Test.test_clean_removed_with_clean_inactive/log/input2
2018-07-31T03:48:55.180-0700	DEBUG	[harvester]	log/harvester.go:517	Closing file: /private/var/lib/jenkins/workspace/elastic+beats+master+multijob-darwin/beat/filebeat/label/macosx/src/github.com/elastic/beats/filebeat/build/system-tests/run/test_registrar.Test.test_clean_removed_with_clean_inactive/log/input1
2018-07-31T03:48:55.180-0700	DEBUG	[harvester]	log/harvester.go:387	Update state: /private/var/lib/jenkins/workspace/elastic+beats+master+multijob-darwin/beat/filebeat/label/macosx/src/github.com/elastic/beats/filebeat/build/system-tests/run/test_registrar.Test.test_clean_removed_with_clean_inactive/log/input1, offset: 19
2018-07-31T03:48:55.180-0700	DEBUG	[input]	log/input.go:195	input states cleaned up. Before: 2, After: 2, Pending: 2
2018-07-31T03:48:55.181-0700	DEBUG	[input]	log/input.go:207	Remove state for file as file removed: /private/var/lib/jenkins/workspace/elastic+beats+master+multijob-darwin/beat/filebeat/label/macosx/src/github.com/elastic/beats/filebeat/build/system-tests/run/test_registrar.Test.test_clean_removed_with_clean_inactive/log/input1
2018-07-31T03:48:55.181-0700	DEBUG	[acker]	beater/acker.go:64	stateful ack	{"count": 1}
2018-07-31T03:48:55.181-0700	DEBUG	[registrar]	registrar/registrar.go:345	Processing 1 events
2018-07-31T03:48:55.181-0700	DEBUG	[registrar]	registrar/registrar.go:315	Registrar state updates processed. Count: 1
2018-07-31T03:48:55.181-0700	DEBUG	[state]	file/states.go:118	State removed for /private/var/lib/jenkins/workspace/elastic+beats+master+multijob-darwin/beat/filebeat/label/macosx/src/github.com/elastic/beats/filebeat/build/system-tests/run/test_registrar.Test.test_clean_removed_with_clean_inactive/log/input1 because of older: 0s
2018-07-31T03:48:55.181-0700	DEBUG	[registrar]	registrar/registrar.go:335	Registrar states cleaned up. Before: 2, After: 1, Pending: 1
2018-07-31T03:48:55.181-0700	DEBUG	[registrar]	registrar/registrar.go:400	Write registry file: /private/var/lib/jenkins/workspace/elastic+beats+master+multijob-darwin/beat/filebeat/label/macosx/src/github.com/elastic/beats/filebeat/build/system-tests/run/test_registrar.Test.test_clean_removed_with_clean_inactive/registry
2018-07-31T03:48:55.182-0700	DEBUG	[acker]	beater/acker.go:64	stateful ack	{"count": 1}
2018-07-31T03:48:55.182-0700	DEBUG	[harvester]	log/harvester.go:528	harvester cleanup finished for file: /private/var/lib/jenkins/workspace/elastic+beats+master+multijob-darwin/beat/filebeat/label/macosx/src/github.com/elastic/beats/filebeat/build/system-tests/run/test_registrar.Test.test_clean_removed_with_clean_inactive/log/input1
2018-07-31T03:48:55.182-0700	DEBUG	[registrar]	registrar/registrar.go:393	Registry file updated. 1 states written.
2018-07-31T03:48:55.183-0700	DEBUG	[registrar]	registrar/registrar.go:345	Processing 1 events
2018-07-31T03:48:55.183-0700	DEBUG	[input]	file/states.go:68	New state added for /private/var/lib/jenkins/workspace/elastic+beats+master+multijob-darwin/beat/filebeat/label/macosx/src/github.com/elastic/beats/filebeat/build/system-tests/run/test_registrar.Test.test_clean_removed_with_clean_inactive/log/input1
2018-07-31T03:48:55.183-0700	DEBUG	[registrar]	registrar/registrar.go:315	Registrar state updates processed. Count: 1
2018-07-31T03:48:55.183-0700	DEBUG	[registrar]	registrar/registrar.go:335	Registrar states cleaned up. Before: 2, After: 2, Pending: 2
2018-07-31T03:48:55.183-0700	DEBUG	[registrar]	registrar/registrar.go:400	Write registry file: /private/var/lib/jenkins/workspace/elastic+beats+master+multijob-darwin/beat/filebeat/label/macosx/src/github.com/elastic/beats/filebeat/build/system-tests/run/test_registrar.Test.test_clean_removed_with_clean_inactive/registry
2018-07-31T03:48:55.184-0700	DEBUG	[registrar]	registrar/registrar.go:393	Registry file updated. 2 states written.
2018-07-31T03:48:55.193-0700	DEBUG	[harvester]	log/log.go:102	End of file reached: /private/var/lib/jenkins/workspace/elastic+beats+master+multijob-darwin/beat/filebeat/label/macosx/src/github.com/elastic/beats/filebeat/build/system-tests/run/test_registrar.Test.test_clean_removed_with_clean_inactive/log/input2; Backoff now.
2018-07-31T03:48:55.352-0700	DEBUG	[input]	input/input.go:152	Run input
2018-07-31T03:48:55.352-0700	DEBUG	[input]	log/input.go:174	Start next scan
2018-07-31T03:48:55.352-0700	DEBUG	[input]	log/input.go:404	Check file for harvesting: /private/var/lib/jenkins/workspace/elastic+beats+master+multijob-darwin/beat/filebeat/label/macosx/src/github.com/elastic/beats/filebeat/build/system-tests/run/test_registrar.Test.test_clean_removed_with_clean_inactive/log/input2
2018-07-31T03:48:55.353-0700	DEBUG	[input]	log/input.go:494	Update existing file for harvesting: /private/var/lib/jenkins/workspace/elastic+beats+master+multijob-darwin/beat/filebeat/label/macosx/src/github.com/elastic/beats/filebeat/build/system-tests/run/test_registrar.Test.test_clean_removed_with_clean_inactive/log/input2, offset: 2
2018-07-31T03:48:55.353-0700	DEBUG	[input]	log/input.go:546	Harvester for file is still running: /private/var/lib/jenkins/workspace/elastic+beats+master+multijob-darwin/beat/filebeat/label/macosx/src/github.com/elastic/beats/filebeat/build/system-tests/run/test_registrar.Test.test_clean_removed_with_clean_inactive/log/input2
2018-07-31T03:48:55.353-0700	DEBUG	[state]	file/states.go:118	State removed for /private/var/lib/jenkins/workspace/elastic+beats+master+multijob-darwin/beat/filebeat/label/macosx/src/github.com/elastic/beats/filebeat/build/system-tests/run/test_registrar.Test.test_clean_removed_with_clean_inactive/log/input1 because of older: 0s
2018-07-31T03:48:55.353-0700	DEBUG	[input]	log/input.go:195	input states cleaned up. Before: 2, After: 1, Pending: 1
2018-07-31T03:48:55.364-0700	DEBUG	[publish]	pipeline/processor.go:308	Publish event: {
  "@timestamp": "2018-07-31T10:48:55.363Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "doc",
    "version": "7.0.0-alpha1"
  },
  "prospector": {
    "type": "log"
  },
  "input": {
    "type": "log"
  },
  "beat": {
    "name": "slave-c07ll940dwyl.local",
    "hostname": "slave-c07ll940dwyl.local",
    "version": "7.0.0-alpha1"
  },
  "host": {
    "name": "slave-c07ll940dwyl.local"
  },
  "offset": 2,
  "message": "make sure registry is written",
  "source": "/private/var/lib/jenkins/workspace/elastic+beats+master+multijob-darwin/beat/filebeat/label/macosx/src/github.com/elastic/beats/filebeat/build/system-tests/run/test_registrar.Test.test_clean_removed_with_clean_inactive/log/input2"
}
2018-07-31T03:48:55.364-0700	DEBUG	[harvester]	log/log.go:102	End of file reached: /private/var/lib/jenkins/workspace/elastic+beats+master+multijob-darwin/beat/filebeat/label/macosx/src/github.com/elastic/beats/filebeat/build/system-tests/run/test_registrar.Test.test_clean_removed_with_clean_inactive/log/input2; Backoff now.
2018-07-31T03:48:55.520-0700	DEBUG	[input]	input/input.go:152	Run input
2018-07-31T03:48:55.520-0700	DEBUG	[input]	log/input.go:174	Start next scan
2018-07-31T03:48:55.521-0700	DEBUG	[input]	log/input.go:404	Check file for harvesting: /private/var/lib/jenkins/workspace/elastic+beats+master+multijob-darwin/beat/filebeat/label/macosx/src/github.com/elastic/beats/filebeat/build/system-tests/run/test_registrar.Test.test_clean_removed_with_clean_inactive/log/input2
2018-07-31T03:48:55.521-0700	DEBUG	[input]	log/input.go:494	Update existing file for harvesting: /private/var/lib/jenkins/workspace/elastic+beats+master+multijob-darwin/beat/filebeat/label/macosx/src/github.com/elastic/beats/filebeat/build/system-tests/run/test_registrar.Test.test_clean_removed_with_clean_inactive/log/input2, offset: 32
2018-07-31T03:48:55.521-0700	DEBUG	[input]	log/input.go:546	Harvester for file is still running: /private/var/lib/jenkins/workspace/elastic+beats+master+multijob-darwin/beat/filebeat/label/macosx/src/github.com/elastic/beats/filebeat/build/system-tests/run/test_registrar.Test.test_clean_removed_with_clean_inactive/log/input2
2018-07-31T03:48:55.521-0700	DEBUG	[input]	log/input.go:195	input states cleaned up. Before: 1, After: 1, Pending: 1
2018-07-31T03:48:55.532-0700	DEBUG	[harvester]	log/log.go:102	End of file reached: /private/var/lib/jenkins/workspace/elastic+beats+master+multijob-darwin/beat/filebeat/label/macosx/src/github.com/elastic/beats/filebeat/build/system-tests/run/test_registrar.Test.test_clean_removed_with_clean_inactive/log/input2; Backoff now.
2018-07-31T03:48:55.533-0700	DEBUG	[memqueue]	memqueue/ackloop.go:160	ackloop: receive ack [1: 0, 1]
2018-07-31T03:48:55.533-0700	DEBUG	[memqueue]	memqueue/eventloop.go:535	broker ACK events: count=1, start-seq=3, end-seq=3

2018-07-31T03:48:55.533-0700	DEBUG	[memqueue]	memqueue/ackloop.go:128	ackloop: return ack to broker loop:1
2018-07-31T03:48:55.533-0700	DEBUG	[acker]	beater/acker.go:64	stateful ack	{"count": 1}
2018-07-31T03:48:55.533-0700	DEBUG	[memqueue]	memqueue/ackloop.go:131	ackloop:  done send ack
2018-07-31T03:48:55.533-0700	DEBUG	[registrar]	registrar/registrar.go:345	Processing 1 events
2018-07-31T03:48:55.533-0700	DEBUG	[registrar]	registrar/registrar.go:315	Registrar state updates processed. Count: 1
2018-07-31T03:48:55.533-0700	DEBUG	[registrar]	registrar/registrar.go:335	Registrar states cleaned up. Before: 2, After: 2, Pending: 2
2018-07-31T03:48:55.533-0700	DEBUG	[registrar]	registrar/registrar.go:400	Write registry file: /private/var/lib/jenkins/workspace/elastic+beats+master+multijob-darwin/beat/filebeat/label/macosx/src/github.com/elastic/beats/filebeat/build/system-tests/run/test_registrar.Test.test_clean_removed_with_clean_inactive/registry
2018-07-31T03:48:55.534-0700	DEBUG	[registrar]	registrar/registrar.go:393	Registry file updated. 2 states written.
2018-07-31T03:48:55.696-0700	DEBUG	[harvester]	log/log.go:102	End of file reached: /private/var/lib/jenkins/workspace/elastic+beats+master+multijob-darwin/beat/filebeat/label/macosx/src/github.com/elastic/beats/filebeat/build/system-tests/run/test_registrar.Test.test_clean_removed_with_clean_inactive/log/input2; Backoff now.
2018-07-31T03:48:55.696-0700	DEBUG	[input]	input/input.go:152	Run input
2018-07-31T03:48:55.696-0700	DEBUG	[input]	log/input.go:174	Start next scan
2018-07-31T03:48:55.697-0700	DEBUG	[input]	log/input.go:404	Check file for harvesting: /private/var/lib/jenkins/workspace/elastic+beats+master+multijob-darwin/beat/filebeat/label/macosx/src/github.com/elastic/beats/filebeat/build/system-tests/run/test_registrar.Test.test_clean_removed_with_clean_inactive/log/input2
2018-07-31T03:48:55.697-0700	DEBUG	[input]	log/input.go:494	Update existing file for harvesting: /private/var/lib/jenkins/workspace/elastic+beats+master+multijob-darwin/beat/filebeat/label/macosx/src/github.com/elastic/beats/filebeat/build/system-tests/run/test_registrar.Test.test_clean_removed_with_clean_inactive/log/input2, offset: 32
2018-07-31T03:48:55.697-0700	DEBUG	[input]	log/input.go:546	Harvester for file is still running: /private/var/lib/jenkins/workspace/elastic+beats+master+multijob-darwin/beat/filebeat/label/macosx/src/github.com/elastic/beats/filebeat/build/system-tests/run/test_registrar.Test.test_clean_removed_with_clean_inactive/log/input2
2018-07-31T03:48:55.697-0700	DEBUG	[input]	log/input.go:195	input states cleaned up. Before: 1, After: 1, Pending: 1
2018-07-31T03:48:55.869-0700	DEBUG	[harvester]	log/log.go:102	End of file reached: /private/var/lib/jenkins/workspace/elastic+beats+master+multijob-darwin/beat/filebeat/label/macosx/src/github.com/elastic/beats/filebeat/build/system-tests/run/test_registrar.Test.test_clean_removed_with_clean_inactive/log/input2; Backoff now.
2018-07-31T03:48:55.872-0700	DEBUG	[input]	input/input.go:152	Run input
2018-07-31T03:48:55.872-0700	DEBUG	[input]	log/input.go:174	Start next scan
2018-07-31T03:48:55.872-0700	DEBUG	[input]	log/input.go:404	Check file for harvesting: /private/var/lib/jenkins/workspace/elastic+beats+master+multijob-darwin/beat/filebeat/label/macosx/src/github.com/elastic/beats/filebeat/build/system-tests/run/test_registrar.Test.test_clean_removed_with_clean_inactive/log/input2
2018-07-31T03:48:55.873-0700	DEBUG	[input]	log/input.go:494	Update existing file for harvesting: /private/var/lib/jenkins/workspace/elastic+beats+master+multijob-darwin/beat/filebeat/label/macosx/src/github.com/elastic/beats/filebeat/build/system-tests/run/test_registrar.Test.test_clean_removed_with_clean_inactive/log/input2, offset: 32
2018-07-31T03:48:55.873-0700	DEBUG	[input]	log/input.go:546	Harvester for file is still running: /private/var/lib/jenkins/workspace/elastic+beats+master+multijob-darwin/beat/filebeat/label/macosx/src/github.com/elastic/beats/filebeat/build/system-tests/run/test_registrar.Test.test_clean_removed_with_clean_inactive/log/input2
2018-07-31T03:48:55.873-0700	DEBUG	[input]	log/input.go:195	input states cleaned up. Before: 1, After: 1, Pending: 1
2018-07-31T03:48:56.043-0700	DEBUG	[input]	input/input.go:152	Run input
2018-07-31T03:48:56.043-0700	DEBUG	[harvester]	log/log.go:102	End of file reached: /private/var/lib/jenkins/workspace/elastic+beats+master+multijob-darwin/beat/filebeat/label/macosx/src/github.com/elastic/beats/filebeat/build/system-tests/run/test_registrar.Test.test_clean_removed_with_clean_inactive/log/input2; Backoff now.
2018-07-31T03:48:56.043-0700	DEBUG	[input]	log/input.go:174	Start next scan
2018-07-31T03:48:56.043-0700	DEBUG	[input]	log/input.go:404	Check file for harvesting: /private/var/lib/jenkins/workspace/elastic+beats+master+multijob-darwin/beat/filebeat/label/macosx/src/github.com/elastic/beats/filebeat/build/system-tests/run/test_registrar.Test.test_clean_removed_with_clean_inactive/log/input2
2018-07-31T03:48:56.044-0700	DEBUG	[input]	log/input.go:494	Update existing file for harvesting: /private/var/lib/jenkins/workspace/elastic+beats+master+multijob-darwin/beat/filebeat/label/macosx/src/github.com/elastic/beats/filebeat/build/system-tests/run/test_registrar.Test.test_clean_removed_with_clean_inactive/log/input2, offset: 32
2018-07-31T03:48:56.044-0700	DEBUG	[input]	log/input.go:546	Harvester for file is still running: /private/var/lib/jenkins/workspace/elastic+beats+master+multijob-darwin/beat/filebeat/label/macosx/src/github.com/elastic/beats/filebeat/build/system-tests/run/test_registrar.Test.test_clean_removed_with_clean_inactive/log/input2
2018-07-31T03:48:56.044-0700	DEBUG	[input]	log/input.go:195	input states cleaned up. Before: 1, After: 1, Pending: 1
2018-07-31T03:48:56.212-0700	DEBUG	[input]	input/input.go:152	Run input
2018-07-31T03:48:56.212-0700	DEBUG	[harvester]	log/log.go:102	End of file reached: /private/var/lib/jenkins/workspace/elastic+beats+master+multijob-darwin/beat/filebeat/label/macosx/src/github.com/elastic/beats/filebeat/build/system-tests/run/test_registrar.Test.test_clean_removed_with_clean_inactive/log/input2; Backoff now.
2018-07-31T03:48:56.212-0700	DEBUG	[input]	log/input.go:174	Start next scan
2018-07-31T03:48:56.212-0700	DEBUG	[input]	log/input.go:404	Check file for harvesting: /private/var/lib/jenkins/workspace/elastic+beats+master+multijob-darwin/beat/filebeat/label/macosx/src/github.com/elastic/beats/filebeat/build/system-tests/run/test_registrar.Test.test_clean_removed_with_clean_inactive/log/input2
2018-07-31T03:48:56.212-0700	DEBUG	[input]	log/input.go:494	Update existing file for harvesting: /private/var/lib/jenkins/workspace/elastic+beats+master+multijob-darwin/beat/filebeat/label/macosx/src/github.com/elastic/beats/filebeat/build/system-tests/run/test_registrar.Test.test_clean_removed_with_clean_inactive/log/input2, offset: 32
2018-07-31T03:48:56.213-0700	DEBUG	[input]	log/input.go:546	Harvester for file is still running: /private/var/lib/jenkins/workspace/elastic+beats+master+multijob-darwin/beat/filebeat/label/macosx/src/github.com/elastic/beats/filebeat/build/system-tests/run/test_registrar.Test.test_clean_removed_with_clean_inactive/log/input2
2018-07-31T03:48:56.213-0700	DEBUG	[input]	log/input.go:195	input states cleaned up. Before: 1, After: 1, Pending: 1
2018-07-31T03:48:56.385-0700	DEBUG	[harvester]	log/log.go:102	End of file reached: /private/var/lib/jenkins/workspace/elastic+beats+master+multijob-darwin/beat/filebeat/label/macosx/src/github.com/elastic/beats/filebeat/build/system-tests/run/test_registrar.Test.test_clean_removed_with_clean_inactive/log/input2; Backoff now.
2018-07-31T03:48:56.387-0700	DEBUG	[input]	input/input.go:152	Run input
2018-07-31T03:48:56.387-0700	DEBUG	[input]	log/input.go:174	Start next scan
2018-07-31T03:48:56.388-0700	DEBUG	[input]	log/input.go:404	Check file for harvesting: /private/var/lib/jenkins/workspace/elastic+beats+master+multijob-darwin/beat/filebeat/label/macosx/src/github.com/elastic/beats/filebeat/build/system-tests/run/test_registrar.Test.test_clean_removed_with_clean_inactive/log/input2
2018-07-31T03:48:56.388-0700	DEBUG	[input]	log/input.go:494	Update existing file for harvesting: /private/var/lib/jenkins/workspace/elastic+beats+master+multijob-darwin/beat/filebeat/label/macosx/src/github.com/elastic/beats/filebeat/build/system-tests/run/test_registrar.Test.test_clean_removed_with_clean_inactive/log/input2, offset: 32
2018-07-31T03:48:56.388-0700	DEBUG	[input]	log/input.go:546	Harvester for file is still running: /private/var/lib/jenkins/workspace/elastic+beats+master+multijob-darwin/beat/filebeat/label/macosx/src/github.com/elastic/beats/filebeat/build/system-tests/run/test_registrar.Test.test_clean_removed_with_clean_inactive/log/input2
2018-07-31T03:48:56.388-0700	DEBUG	[input]	log/input.go:195	input states cleaned up. Before: 1, After: 1, Pending: 1
2018-07-31T03:48:56.542-0700	DEBUG	[input]	input/input.go:152	Run input
2018-07-31T03:48:56.542-0700	DEBUG	[harvester]	log/log.go:102	End of file reached: /private/var/lib/jenkins/workspace/elastic+beats+master+multijob-darwin/beat/filebeat/label/macosx/src/github.com/elastic/beats/filebeat/build/system-tests/run/test_registrar.Test.test_clean_removed_with_clean_inactive/log/input2; Backoff now.
2018-07-31T03:48:56.542-0700	DEBUG	[input]	log/input.go:174	Start next scan
2018-07-31T03:48:56.542-0700	DEBUG	[input]	log/input.go:404	Check file for harvesting: /private/var/lib/jenkins/workspace/elastic+beats+master+multijob-darwin/beat/filebeat/label/macosx/src/github.com/elastic/beats/filebeat/build/system-tests/run/test_registrar.Test.test_clean_removed_with_clean_inactive/log/input2
2018-07-31T03:48:56.542-0700	DEBUG	[input]	log/input.go:494	Update existing file for harvesting: /private/var/lib/jenkins/workspace/elastic+beats+master+multijob-darwin/beat/filebeat/label/macosx/src/github.com/elastic/beats/filebeat/build/system-tests/run/test_registrar.Test.test_clean_removed_with_clean_inactive/log/input2, offset: 32
2018-07-31T03:48:56.542-0700	DEBUG	[input]	log/input.go:546	Harvester for file is still running: /private/var/lib/jenkins/workspace/elastic+beats+master+multijob-darwin/beat/filebeat/label/macosx/src/github.com/elastic/beats/filebeat/build/system-tests/run/test_registrar.Test.test_clean_removed_with_clean_inactive/log/input2
2018-07-31T03:48:56.543-0700	DEBUG	[input]	log/input.go:195	input states cleaned up. Before: 1, After: 1, Pending: 1
2018-07-31T03:48:56.712-0700	DEBUG	[input]	input/input.go:152	Run input
2018-07-31T03:48:56.712-0700	DEBUG	[harvester]	log/log.go:102	End of file reached: /private/var/lib/jenkins/workspace/elastic+beats+master+multijob-darwin/beat/filebeat/label/macosx/src/github.com/elastic/beats/filebeat/build/system-tests/run/test_registrar.Test.test_clean_removed_with_clean_inactive/log/input2; Backoff now.
2018-07-31T03:48:56.712-0700	DEBUG	[input]	log/input.go:174	Start next scan
2018-07-31T03:48:56.713-0700	DEBUG	[input]	log/input.go:404	Check file for harvesting: /private/var/lib/jenkins/workspace/elastic+beats+master+multijob-darwin/beat/filebeat/label/macosx/src/github.com/elastic/beats/filebeat/build/system-tests/run/test_registrar.Test.test_clean_removed_with_clean_inactive/log/input2
2018-07-31T03:48:56.713-0700	DEBUG	[input]	log/input.go:494	Update existing file for harvesting: /private/var/lib/jenkins/workspace/elastic+beats+master+multijob-darwin/beat/filebeat/label/macosx/src/github.com/elastic/beats/filebeat/build/system-tests/run/test_registrar.Test.test_clean_removed_with_clean_inactive/log/input2, offset: 32
2018-07-31T03:48:56.713-0700	DEBUG	[input]	log/input.go:546	Harvester for file is still running: /private/var/lib/jenkins/workspace/elastic+beats+master+multijob-darwin/beat/filebeat/label/macosx/src/github.com/elastic/beats/filebeat/build/system-tests/run/test_registrar.Test.test_clean_removed_with_clean_inactive/log/input2
2018-07-31T03:48:56.713-0700	DEBUG	[input]	log/input.go:195	input states cleaned up. Before: 1, After: 1, Pending: 1
2018-07-31T03:48:56.883-0700	DEBUG	[harvester]	log/log.go:102	End of file reached: /private/var/lib/jenkins/workspace/elastic+beats+master+multijob-darwin/beat/filebeat/label/macosx/src/github.com/elastic/beats/filebeat/build/system-tests/run/test_registrar.Test.test_clean_removed_with_clean_inactive/log/input2; Backoff now.
2018-07-31T03:48:56.888-0700	DEBUG	[input]	input/input.go:152	Run input
2018-07-31T03:48:56.888-0700	DEBUG	[input]	log/input.go:174	Start next scan
2018-07-31T03:48:56.889-0700	DEBUG	[input]	log/input.go:404	Check file for harvesting: /private/var/lib/jenkins/workspace/elastic+beats+master+multijob-darwin/beat/filebeat/label/macosx/src/github.com/elastic/beats/filebeat/build/system-tests/run/test_registrar.Test.test_clean_removed_with_clean_inactive/log/input2
2018-07-31T03:48:56.889-0700	DEBUG	[input]	log/input.go:494	Update existing file for harvesting: /private/var/lib/jenkins/workspace/elastic+beats+master+multijob-darwin/beat/filebeat/label/macosx/src/github.com/elastic/beats/filebeat/build/system-tests/run/test_registrar.Test.test_clean_removed_with_clean_inactive/log/input2, offset: 32
2018-07-31T03:48:56.889-0700	DEBUG	[input]	log/input.go:546	Harvester for file is still running: /private/var/lib/jenkins/workspace/elastic+beats+master+multijob-darwin/beat/filebeat/label/macosx/src/github.com/elastic/beats/filebeat/build/system-tests/run/test_registrar.Test.test_clean_removed_with_clean_inactive/log/input2
2018-07-31T03:48:56.890-0700	DEBUG	[input]	log/input.go:195	input states cleaned up. Before: 1, After: 1, Pending: 1
2018-07-31T03:48:57.049-0700	DEBUG	[input]	input/input.go:152	Run input
2018-07-31T03:48:57.049-0700	DEBUG	[harvester]	log/log.go:102	End of file reached: /private/var/lib/jenkins/workspace/elastic+beats+master+multijob-darwin/beat/filebeat/label/macosx/src/github.com/elastic/beats/filebeat/build/system-tests/run/test_registrar.Test.test_clean_removed_with_clean_inactive/log/input2; Backoff now.
2018-07-31T03:48:57.049-0700	DEBUG	[input]	log/input.go:174	Start next scan
2018-07-31T03:48:57.050-0700	DEBUG	[input]	log/input.go:404	Check file for harvesting: /private/var/lib/jenkins/workspace/elastic+beats+master+multijob-darwin/beat/filebeat/label/macosx/src/github.com/elastic/beats/filebeat/build/system-tests/run/test_registrar.Test.test_clean_removed_with_clean_inactive/log/input2
2018-07-31T03:48:57.050-0700	DEBUG	[input]	log/input.go:494	Update existing file for harvesting: /private/var/lib/jenkins/workspace/elastic+beats+master+multijob-darwin/beat/filebeat/label/macosx/src/github.com/elastic/beats/filebeat/build/system-tests/run/test_registrar.Test.test_clean_removed_with_clean_inactive/log/input2, offset: 32
2018-07-31T03:48:57.050-0700	DEBUG	[input]	log/input.go:546	Harvester for file is still running: /private/var/lib/jenkins/workspace/elastic+beats+master+multijob-darwin/beat/filebeat/label/macosx/src/github.com/elastic/beats/filebeat/build/system-tests/run/test_registrar.Test.test_clean_removed_with_clean_inactive/log/input2
2018-07-31T03:48:57.050-0700	DEBUG	[input]	log/input.go:195	input states cleaned up. Before: 1, After: 1, Pending: 1
2018-07-31T03:48:57.219-0700	DEBUG	[input]	input/input.go:152	Run input
2018-07-31T03:48:57.219-0700	DEBUG	[input]	log/input.go:174	Start next scan
2018-07-31T03:48:57.219-0700	DEBUG	[harvester]	log/log.go:102	End of file reached: /private/var/lib/jenkins/workspace/elastic+beats+master+multijob-darwin/beat/filebeat/label/macosx/src/github.com/elastic/beats/filebeat/build/system-tests/run/test_registrar.Test.test_clean_removed_with_clean_inactive/log/input2; Backoff now.
2018-07-31T03:48:57.219-0700	DEBUG	[input]	log/input.go:404	Check file for harvesting: /private/var/lib/jenkins/workspace/elastic+beats+master+multijob-darwin/beat/filebeat/label/macosx/src/github.com/elastic/beats/filebeat/build/system-tests/run/test_registrar.Test.test_clean_removed_with_clean_inactive/log/input2
2018-07-31T03:48:57.219-0700	DEBUG	[input]	log/input.go:494	Update existing file for harvesting: /private/var/lib/jenkins/workspace/elastic+beats+master+multijob-darwin/beat/filebeat/label/macosx/src/github.com/elastic/beats/filebeat/build/system-tests/run/test_registrar.Test.test_clean_removed_with_clean_inactive/log/input2, offset: 32
2018-07-31T03:48:57.219-0700	DEBUG	[input]	log/input.go:546	Harvester for file is still running: /private/var/lib/jenkins/workspace/elastic+beats+master+multijob-darwin/beat/filebeat/label/macosx/src/github.com/elastic/beats/filebeat/build/system-tests/run/test_registrar.Test.test_clean_removed_with_clean_inactive/log/input2
2018-07-31T03:48:57.220-0700	DEBUG	[input]	log/input.go:195	input states cleaned up. Before: 1, After: 1, Pending: 1
2018-07-31T03:48:57.392-0700	DEBUG	[harvester]	log/log.go:102	End of file reached: /private/var/lib/jenkins/workspace/elastic+beats+master+multijob-darwin/beat/filebeat/label/macosx/src/github.com/elastic/beats/filebeat/build/system-tests/run/test_registrar.Test.test_clean_removed_with_clean_inactive/log/input2; Backoff now.
2018-07-31T03:48:57.394-0700	DEBUG	[input]	input/input.go:152	Run input
2018-07-31T03:48:57.394-0700	DEBUG	[input]	log/input.go:174	Start next scan
2018-07-31T03:48:57.395-0700	DEBUG	[input]	log/input.go:404	Check file for harvesting: /private/var/lib/jenkins/workspace/elastic+beats+master+multijob-darwin/beat/filebeat/label/macosx/src/github.com/elastic/beats/filebeat/build/system-tests/run/test_registrar.Test.test_clean_removed_with_clean_inactive/log/input2
2018-07-31T03:48:57.395-0700	DEBUG	[input]	log/input.go:494	Update existing file for harvesting: /private/var/lib/jenkins/workspace/elastic+beats+master+multijob-darwin/beat/filebeat/label/macosx/src/github.com/elastic/beats/filebeat/build/system-tests/run/test_registrar.Test.test_clean_removed_with_clean_inactive/log/input2, offset: 32
2018-07-31T03:48:57.395-0700	DEBUG	[input]	log/input.go:546	Harvester for file is still running: /private/var/lib/jenkins/workspace/elastic+beats+master+multijob-darwin/beat/filebeat/label/macosx/src/github.com/elastic/beats/filebeat/build/system-tests/run/test_registrar.Test.test_clean_removed_with_clean_inactive/log/input2
2018-07-31T03:48:57.395-0700	DEBUG	[input]	log/input.go:195	input states cleaned up. Before: 1, After: 1, Pending: 1
2018-07-31T03:48:57.562-0700	DEBUG	[harvester]	log/log.go:102	End of file reached: /private/var/lib/jenkins/workspace/elastic+beats+master+multijob-darwin/beat/filebeat/label/macosx/src/github.com/elastic/beats/filebeat/build/system-tests/run/test_registrar.Test.test_clean_removed_with_clean_inactive/log/input2; Backoff now.
2018-07-31T03:48:57.569-0700	DEBUG	[input]	input/input.go:152	Run input
2018-07-31T03:48:57.569-0700	DEBUG	[input]	log/input.go:174	Start next scan
2018-07-31T03:48:57.570-0700	DEBUG	[input]	log/input.go:404	Check file for harvesting: /private/var/lib/jenkins/workspace/elastic+beats+master+multijob-darwin/beat/filebeat/label/macosx/src/github.com/elastic/beats/filebeat/build/system-tests/run/test_registrar.Test.test_clean_removed_with_clean_inactive/log/input2
2018-07-31T03:48:57.570-0700	DEBUG	[input]	log/input.go:494	Update existing file for harvesting: /private/var/lib/jenkins/workspace/elastic+beats+master+multijob-darwin/beat/filebeat/label/macosx/src/github.com/elastic/beats/filebeat/build/system-tests/run/test_registrar.Test.test_clean_removed_with_clean_inactive/log/input2, offset: 32
2018-07-31T03:48:57.570-0700	DEBUG	[input]	log/input.go:546	Harvester for file is still running: /private/var/lib/jenkins/workspace/elastic+beats+master+multijob-darwin/beat/filebeat/label/macosx/src/github.com/elastic/beats/filebeat/build/system-tests/run/test_registrar.Test.test_clean_removed_with_clean_inactive/log/input2
2018-07-31T03:48:57.570-0700	DEBUG	[input]	log/input.go:195	input states cleaned up. Before: 1, After: 1, Pending: 1
2018-07-31T03:48:57.735-0700	DEBUG	[harvester]	log/log.go:102	End of file reached: /private/var/lib/jenkins/workspace/elastic+beats+master+multijob-darwin/beat/filebeat/label/macosx/src/github.com/elastic/beats/filebeat/build/system-tests/run/test_registrar.Test.test_clean_removed_with_clean_inactive/log/input2; Backoff now.
2018-07-31T03:48:57.742-0700	DEBUG	[input]	input/input.go:152	Run input
2018-07-31T03:48:57.742-0700	DEBUG	[input]	log/input.go:174	Start next scan
2018-07-31T03:48:57.743-0700	DEBUG	[input]	log/input.go:404	Check file for harvesting: /private/var/lib/jenkins/workspace/elastic+beats+master+multijob-darwin/beat/filebeat/label/macosx/src/github.com/elastic/beats/filebeat/build/system-tests/run/test_registrar.Test.test_clean_removed_with_clean_inactive/log/input2
2018-07-31T03:48:57.743-0700	DEBUG	[input]	log/input.go:494	Update existing file for harvesting: /private/var/lib/jenkins/workspace/elastic+beats+master+multijob-darwin/beat/filebeat/label/macosx/src/github.com/elastic/beats/filebeat/build/system-tests/run/test_registrar.Test.test_clean_removed_with_clean_inactive/log/input2, offset: 32
2018-07-31T03:48:57.743-0700	DEBUG	[input]	log/input.go:546	Harvester for file is still running: /private/var/lib/jenkins/workspace/elastic+beats+master+multijob-darwin/beat/filebeat/label/macosx/src/github.com/elastic/beats/filebeat/build/system-tests/run/test_registrar.Test.test_clean_removed_with_clean_inactive/log/input2
2018-07-31T03:48:57.743-0700	DEBUG	[input]	log/input.go:195	input states cleaned up. Before: 1, After: 1, Pending: 1
2018-07-31T03:48:57.909-0700	DEBUG	[harvester]	log/log.go:102	End of file reached: /private/var/lib/jenkins/workspace/elastic+beats+master+multijob-darwin/beat/filebeat/label/macosx/src/github.com/elastic/beats/filebeat/build/system-tests/run/test_registrar.Test.test_clean_removed_with_clean_inactive/log/input2; Backoff now.
2018-07-31T03:48:57.912-0700	DEBUG	[input]	input/input.go:152	Run input
2018-07-31T03:48:57.912-0700	DEBUG	[input]	log/input.go:174	Start next scan
2018-07-31T03:48:57.913-0700	DEBUG	[input]	log/input.go:404	Check file for harvesting: /private/var/lib/jenkins/workspace/elastic+beats+master+multijob-darwin/beat/filebeat/label/macosx/src/github.com/elastic/beats/filebeat/build/system-tests/run/test_registrar.Test.test_clean_removed_with_clean_inactive/log/input2
2018-07-31T03:48:57.913-0700	DEBUG	[input]	log/input.go:494	Update existing file for harvesting: /private/var/lib/jenkins/workspace/elastic+beats+master+multijob-darwin/beat/filebeat/label/macosx/src/github.com/elastic/beats/filebeat/build/system-tests/run/test_registrar.Test.test_clean_removed_with_clean_inactive/log/input2, offset: 32
2018-07-31T03:48:57.913-0700	DEBUG	[input]	log/input.go:546	Harvester for file is still running: /private/var/lib/jenkins/workspace/elastic+beats+master+multijob-darwin/beat/filebeat/label/macosx/src/github.com/elastic/beats/filebeat/build/system-tests/run/test_registrar.Test.test_clean_removed_with_clean_inactive/log/input2
2018-07-31T03:48:57.913-0700	DEBUG	[input]	log/input.go:195	input states cleaned up. Before: 1, After: 1, Pending: 1
2018-07-31T03:48:58.082-0700	DEBUG	[input]	input/input.go:152	Run input
2018-07-31T03:48:58.082-0700	DEBUG	[harvester]	log/log.go:102	End of file reached: /private/var/lib/jenkins/workspace/elastic+beats+master+multijob-darwin/beat/filebeat/label/macosx/src/github.com/elastic/beats/filebeat/build/system-tests/run/test_registrar.Test.test_clean_removed_with_clean_inactive/log/input2; Backoff now.
2018-07-31T03:48:58.082-0700	DEBUG	[input]	log/input.go:174	Start next scan
2018-07-31T03:48:58.082-0700	DEBUG	[input]	log/input.go:404	Check file for harvesting: /private/var/lib/jenkins/workspace/elastic+beats+master+multijob-darwin/beat/filebeat/label/macosx/src/github.com/elastic/beats/filebeat/build/system-tests/run/test_registrar.Test.test_clean_removed_with_clean_inactive/log/input2
2018-07-31T03:48:58.082-0700	DEBUG	[input]	log/input.go:494	Update existing file for harvesting: /private/var/lib/jenkins/workspace/elastic+beats+master+multijob-darwin/beat/filebeat/label/macosx/src/github.com/elastic/beats/filebeat/build/system-tests/run/test_registrar.Test.test_clean_removed_with_clean_inactive/log/input2, offset: 32
2018-07-31T03:48:58.082-0700	DEBUG	[input]	log/input.go:546	Harvester for file is still running: /private/var/lib/jenkins/workspace/elastic+beats+master+multijob-darwin/beat/filebeat/label/macosx/src/github.com/elastic/beats/filebeat/build/system-tests/run/test_registrar.Test.test_clean_removed_with_clean_inactive/log/input2
2018-07-31T03:48:58.083-0700	DEBUG	[input]	log/input.go:195	input states cleaned up. Before: 1, After: 1, Pending: 1
2018-07-31T03:48:58.222-0700	DEBUG	[input]	input/input.go:152	Run input
2018-07-31T03:48:58.222-0700	DEBUG	[harvester]	log/log.go:102	End of file reached: /private/var/lib/jenkins/workspace/elastic+beats+master+multijob-darwin/beat/filebeat/label/macosx/src/github.com/elastic/beats/filebeat/build/system-tests/run/test_registrar.Test.test_clean_removed_with_clean_inactive/log/input2; Backoff now.
2018-07-31T03:48:58.222-0700	DEBUG	[input]	log/input.go:174	Start next scan
2018-07-31T03:48:58.222-0700	DEBUG	[input]	log/input.go:404	Check file for harvesting: /private/var/lib/jenkins/workspace/elastic+beats+master+multijob-darwin/beat/filebeat/label/macosx/src/github.com/elastic/beats/filebeat/build/system-tests/run/test_registrar.Test.test_clean_removed_with_clean_inactive/log/input2
2018-07-31T03:48:58.222-0700	DEBUG	[input]	log/input.go:494	Update existing file for harvesting: /private/var/lib/jenkins/workspace/elastic+beats+master+multijob-darwin/beat/filebeat/label/macosx/src/github.com/elastic/beats/filebeat/build/system-tests/run/test_registrar.Test.test_clean_removed_with_clean_inactive/log/input2, offset: 32
2018-07-31T03:48:58.223-0700	DEBUG	[input]	log/input.go:546	Harvester for file is still running: /private/var/lib/jenkins/workspace/elastic+beats+master+multijob-darwin/beat/filebeat/label/macosx/src/github.com/elastic/beats/filebeat/build/system-tests/run/test_registrar.Test.test_clean_removed_with_clean_inactive/log/input2
2018-07-31T03:48:58.223-0700	DEBUG	[input]	log/input.go:195	input states cleaned up. Before: 1, After: 1, Pending: 1
2018-07-31T03:48:58.392-0700	DEBUG	[input]	input/input.go:152	Run input
2018-07-31T03:48:58.392-0700	DEBUG	[harvester]	log/log.go:102	End of file reached: /private/var/lib/jenkins/workspace/elastic+beats+master+multijob-darwin/beat/filebeat/label/macosx/src/github.com/elastic/beats/filebeat/build/system-tests/run/test_registrar.Test.test_clean_removed_with_clean_inactive/log/input2; Backoff now.
2018-07-31T03:48:58.392-0700	DEBUG	[input]	log/input.go:174	Start next scan
2018-07-31T03:48:58.392-0700	DEBUG	[input]	log/input.go:404	Check file for harvesting: /private/var/lib/jenkins/workspace/elastic+beats+master+multijob-darwin/beat/filebeat/label/macosx/src/github.com/elastic/beats/filebeat/build/system-tests/run/test_registrar.Test.test_clean_removed_with_clean_inactive/log/input2
2018-07-31T03:48:58.392-0700	DEBUG	[input]	log/input.go:494	Update existing file for harvesting: /private/var/lib/jenkins/workspace/elastic+beats+master+multijob-darwin/beat/filebeat/label/macosx/src/github.com/elastic/beats/filebeat/build/system-tests/run/test_registrar.Test.test_clean_removed_with_clean_inactive/log/input2, offset: 32
2018-07-31T03:48:58.393-0700	DEBUG	[input]	log/input.go:546	Harvester for file is still running: /private/var/lib/jenkins/workspace/elastic+beats+master+multijob-darwin/beat/filebeat/label/macosx/src/github.com/elastic/beats/filebeat/build/system-tests/run/test_registrar.Test.test_clean_removed_with_clean_inactive/log/input2
2018-07-31T03:48:58.393-0700	DEBUG	[input]	log/input.go:195	input states cleaned up. Before: 1, After: 1, Pending: 1
2018-07-31T03:48:58.562-0700	DEBUG	[input]	input/input.go:152	Run input
2018-07-31T03:48:58.562-0700	DEBUG	[harvester]	log/log.go:102	End of file reached: /private/var/lib/jenkins/workspace/elastic+beats+master+multijob-darwin/beat/filebeat/label/macosx/src/github.com/elastic/beats/filebeat/build/system-tests/run/test_registrar.Test.test_clean_removed_with_clean_inactive/log/input2; Backoff now.
2018-07-31T03:48:58.562-0700	DEBUG	[input]	log/input.go:174	Start next scan
2018-07-31T03:48:58.562-0700	DEBUG	[input]	log/input.go:404	Check file for harvesting: /private/var/lib/jenkins/workspace/elastic+beats+master+multijob-darwin/beat/filebeat/label/macosx/src/github.com/elastic/beats/filebeat/build/system-tests/run/test_registrar.Test.test_clean_removed_with_clean_inactive/log/input2
2018-07-31T03:48:58.562-0700	DEBUG	[input]	log/input.go:494	Update existing file for harvesting: /private/var/lib/jenkins/workspace/elastic+beats+master+multijob-darwin/beat/filebeat/label/macosx/src/github.com/elastic/beats/filebeat/build/system-tests/run/test_registrar.Test.test_clean_removed_with_clean_inactive/log/input2, offset: 32
2018-07-31T03:48:58.562-0700	DEBUG	[input]	log/input.go:546	Harvester for file is still running: /private/var/lib/jenkins/workspace/elastic+beats+master+multijob-darwin/beat/filebeat/label/macosx/src/github.com/elastic/beats/filebeat/build/system-tests/run/test_registrar.Test.test_clean_removed_with_clean_inactive/log/input2
2018-07-31T03:48:58.562-0700	DEBUG	[input]	log/input.go:195	input states cleaned up. Before: 1, After: 1, Pending: 1
2018-07-31T03:48:58.619-0700	DEBUG	[service]	service/service.go:51	Received sigterm/sigint, stopping
2018-07-31T03:48:58.619-0700	INFO	beater/filebeat.go:437	Stopping filebeat
2018-07-31T03:48:58.620-0700	INFO	crawler/crawler.go:139	Stopping Crawler
2018-07-31T03:48:58.620-0700	INFO	crawler/crawler.go:149	Stopping 1 inputs
2018-07-31T03:48:58.620-0700	INFO	input/input.go:149	input ticker stopped
2018-07-31T03:48:58.620-0700	INFO	input/input.go:167	Stopping Input: 8456096243672335408
2018-07-31T03:48:58.620-0700	INFO	log/harvester.go:272	Reader was closed: /private/var/lib/jenkins/workspace/elastic+beats+master+multijob-darwin/beat/filebeat/label/macosx/src/github.com/elastic/beats/filebeat/build/system-tests/run/test_registrar.Test.test_clean_removed_with_clean_inactive/log/input2. Closing.
2018-07-31T03:48:58.620-0700	DEBUG	[harvester]	log/harvester.go:507	Stopping harvester for file: /private/var/lib/jenkins/workspace/elastic+beats+master+multijob-darwin/beat/filebeat/label/macosx/src/github.com/elastic/beats/filebeat/build/system-tests/run/test_registrar.Test.test_clean_removed_with_clean_inactive/log/input2
2018-07-31T03:48:58.621-0700	DEBUG	[harvester]	log/harvester.go:517	Closing file: /private/var/lib/jenkins/workspace/elastic+beats+master+multijob-darwin/beat/filebeat/label/macosx/src/github.com/elastic/beats/filebeat/build/system-tests/run/test_registrar.Test.test_clean_removed_with_clean_inactive/log/input2
2018-07-31T03:48:58.621-0700	DEBUG	[harvester]	log/harvester.go:387	Update state: /private/var/lib/jenkins/workspace/elastic+beats+master+multijob-darwin/beat/filebeat/label/macosx/src/github.com/elastic/beats/filebeat/build/system-tests/run/test_registrar.Test.test_clean_removed_with_clean_inactive/log/input2, offset: 32
2018-07-31T03:48:58.621-0700	DEBUG	[harvester]	log/harvester.go:528	harvester cleanup finished for file: /private/var/lib/jenkins/workspace/elastic+beats+master+multijob-darwin/beat/filebeat/label/macosx/src/github.com/elastic/beats/filebeat/build/system-tests/run/test_registrar.Test.test_clean_removed_with_clean_inactive/log/input2
2018-07-31T03:48:58.621-0700	DEBUG	[publish]	pipeline/client.go:148	client: closing acker
2018-07-31T03:48:58.621-0700	DEBUG	[publish]	pipeline/client.go:150	client: done closing acker
2018-07-31T03:48:58.621-0700	DEBUG	[publish]	pipeline/client.go:154	client: cancelled 0 events
2018-07-31T03:48:58.621-0700	INFO	crawler/crawler.go:165	Crawler stopped
2018-07-31T03:48:58.621-0700	INFO	registrar/registrar.go:356	Stopping Registrar
2018-07-31T03:48:58.621-0700	INFO	registrar/registrar.go:282	Ending Registrar
2018-07-31T03:48:58.621-0700	DEBUG	[registrar]	registrar/registrar.go:400	Write registry file: /private/var/lib/jenkins/workspace/elastic+beats+master+multijob-darwin/beat/filebeat/label/macosx/src/github.com/elastic/beats/filebeat/build/system-tests/run/test_registrar.Test.test_clean_removed_with_clean_inactive/registry
2018-07-31T03:48:58.622-0700	DEBUG	[registrar]	registrar/registrar.go:393	Registry file updated. 2 states written.
2018-07-31T03:48:58.629-0700	INFO	[monitoring]	log/log.go:149	Total non-zero metrics	{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":46,"time":{"ms":46}},"total":{"ticks":192,"time":{"ms":192},"value":192},"user":{"ticks":146,"time":{"ms":146}}},"info":{"ephemeral_id":"69be3864-44f5-4de8-b562-045a12599822","uptime":{"ms":3868}},"memstats":{"gc_next":4194304,"memory_alloc":1740528,"memory_total":4757104,"rss":48955392}},"filebeat":{"events":{"added":7,"done":7},"harvester":{"closed":2,"open_files":0,"running":0,"started":2}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"acked":3,"batches":2,"total":3},"type":"file","write":{"bytes":1779}},"pipeline":{"clients":0,"events":{"active":0,"filtered":4,"published":3,"total":7},"queue":{"acked":3}}},"registrar":{"states":{"cleanup":1,"current":2,"update":7},"writes":{"success":8,"total":8}},"system":{"cpu":{"cores":4},"load":{"1":3.3101,"15":3.8535,"5":3.6777,"norm":{"1":0.8275,"15":0.9634,"5":0.9194}}}}}}
2018-07-31T03:48:58.630-0700	INFO	[monitoring]	log/log.go:150	Uptime: 3.875943551s
2018-07-31T03:48:58.630-0700	INFO	[monitoring]	log/log.go:127	Stopping metrics logging.
2018-07-31T03:48:58.630-0700	INFO	instance/beat.go:373	filebeat stopped.
PASS
coverage: 23.0% of statements in github.com/elastic/beats/filebeat, github.com/elastic/beats/filebeat/autodiscover, github.com/elastic/beats/filebeat/autodiscover/builder/hints, github.com/elastic/beats/filebeat/beater, github.com/elastic/beats/filebeat/channel, github.com/elastic/beats/filebeat/cmd, github.com/elastic/beats/filebeat/config, github.com/elastic/beats/filebeat/crawler, github.com/elastic/beats/filebeat/fileset, github.com/elastic/beats/filebeat/harvester, github.com/elastic/beats/filebeat/include, github.com/elastic/beats/filebeat/input, github.com/elastic/beats/filebeat/input/docker, github.com/elastic/beats/filebeat/input/file, github.com/elastic/beats/filebeat/input/log, github.com/elastic/beats/filebeat/input/redis, github.com/elastic/beats/filebeat/input/stdin, github.com/elastic/beats/filebeat/input/syslog, github.com/elastic/beats/filebeat/input/tcp, github.com/elastic/beats/filebeat/input/udp, github.com/elastic/beats/filebeat/inputsource, github.com/elastic/beats/filebeat/inputsource/tcp, github.com/elastic/beats/filebeat/inputsource/udp, github.com/elastic/beats/filebeat/processor/add_kubernetes_metadata, github.com/elastic/beats/filebeat/prospector, github.com/elastic/beats/filebeat/reader, github.com/elastic/beats/filebeat/reader/docker_json, github.com/elastic/beats/filebeat/reader/encode, github.com/elastic/beats/filebeat/reader/encode/encoding, github.com/elastic/beats/filebeat/reader/json, github.com/elastic/beats/filebeat/reader/limit, github.com/elastic/beats/filebeat/reader/line, github.com/elastic/beats/filebeat/reader/multiline, github.com/elastic/beats/filebeat/reader/strip_newline, github.com/elastic/beats/filebeat/reader/timeout, github.com/elastic/beats/filebeat/registrar, github.com/elastic/beats/filebeat/scripts/generator, github.com/elastic/beats/filebeat/scripts/generator/fields, github.com/elastic/beats/filebeat/scripts/generator/fileset, github.com/elastic/beats/filebeat/scripts/generator/module, github.com/elastic/beats/filebeat/scripts/tester, github.com/elastic/beats/filebeat/util
@ruflin ruflin added Filebeat Filebeat flaky-test Unstable or unreliable test cases. labels Aug 2, 2018
@ruflin
Copy link
Contributor Author

ruflin commented Aug 27, 2018

The issue here seems to be very similar to #7690 There needs to be a better way to wait for the clean_removed to happen.

ruflin added a commit to ruflin/beats that referenced this issue Aug 27, 2018
@ruflin
Fix test_clean_removed_with_clean_inactive test
bac2d19 
This removes the time.sleep part from the test in the hope it will make it more stable. I am not 100% confident that this will fully remove the flakyness of test_clean_removed_with_clean_inactive.

If it fails again the future, the full log line + the registry should be added to Github issue to see what is still inside the registry.

Closes elastic#7842
@ruflin ruflin mentioned this issue Aug 27, 2018
Merged
jsoriano pushed a commit that referenced this issue Aug 27, 2018
@ruflin @jsoriano
Fix test_clean_removed_with_clean_inactive test (#8093)
3c1c6c1 
This removes the time.sleep part from the test in the hope it will make it more stable. I am not 100% confident that this will fully remove the flakyness of test_clean_removed_with_clean_inactive.

If it fails again the future, the full log line + the registry should be added to Github issue to see what is still inside the registry.

Closes #7842
ruflin added a commit to ruflin/beats that referenced this issue Aug 28, 2018
@ruflin
Fix test_clean_removed_with_clean_inactive test (elastic#8093)
d51204b 
This removes the time.sleep part from the test in the hope it will make it more stable. I am not 100% confident that this will fully remove the flakyness of test_clean_removed_with_clean_inactive.

If it fails again the future, the full log line + the registry should be added to Github issue to see what is still inside the registry.

Closes elastic#7842

(cherry picked from commit 3c1c6c1)
@ruflin ruflin mentioned this issue Aug 28, 2018
Merged
ruflin added a commit to ruflin/beats that referenced this issue Aug 28, 2018
@ruflin
Fix test_clean_removed_with_clean_inactive test (elastic#8093)
8f0a8eb 
This removes the time.sleep part from the test in the hope it will make it more stable. I am not 100% confident that this will fully remove the flakyness of test_clean_removed_with_clean_inactive.

If it fails again the future, the full log line + the registry should be added to Github issue to see what is still inside the registry.

Closes elastic#7842

(cherry picked from commit 3c1c6c1)
@ruflin ruflin mentioned this issue Aug 28, 2018
Merged
exekias pushed a commit that referenced this issue Aug 28, 2018
@ruflin @exekias
Fix test_clean_removed_with_clean_inactive test (#8093) (#8120)
2464d52 
This removes the time.sleep part from the test in the hope it will make it more stable. I am not 100% confident that this will fully remove the flakyness of test_clean_removed_with_clean_inactive.

If it fails again the future, the full log line + the registry should be added to Github issue to see what is still inside the registry.

Closes #7842

(cherry picked from commit 3c1c6c1)
ruflin added a commit that referenced this issue Aug 29, 2018
@ruflin
Fix test_clean_removed_with_clean_inactive test (#8093) (#8121)
74b71ee 
This removes the time.sleep part from the test in the hope it will make it more stable. I am not 100% confident that this will fully remove the flakyness of test_clean_removed_with_clean_inactive.

If it fails again the future, the full log line + the registry should be added to Github issue to see what is still inside the registry.

Closes #7842

(cherry picked from commit 3c1c6c1)
@ph ph mentioned this issue Oct 31, 2018
Merged
leweafan pushed a commit to leweafan/beats that referenced this issue Apr 28, 2023
@ruflin @exekias
Fix test_clean_removed_with_clean_inactive test (elastic#8093) (elast…
1594960 
…ic#8120)

This removes the time.sleep part from the test in the hope it will make it more stable. I am not 100% confident that this will fully remove the flakyness of test_clean_removed_with_clean_inactive.

If it fails again the future, the full log line + the registry should be added to Github issue to see what is still inside the registry.

Closes elastic#7842

(cherry picked from commit 602f0c3)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Filebeat Filebeat flaky-test Unstable or unreliable test cases.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@ruflin