Add permissions configuration for file output #4638

clery · 2017-07-10T08:28:58Z

Add the ability to change file permissions on file output for Metricbeat.

karmi · 2017-07-10T08:28:59Z

Hi @plassa-b, we have found your signature in our records, but it seems like you have signed with a different e-mail than the one used in yout Git commit. Can you please add both of these e-mails into your Github profile (they can be hidden), so we can match your e-mails to your Github profile?

elasticmachine · 2017-07-10T08:28:59Z

Can one of the admins verify this patch?

elasticmachine · 2017-07-10T08:32:02Z

Since this is a community submitted pull request, a Jenkins build has not been kicked off automatically on build-eu-00. Can an Elastic organization member please verify the contents of this patch and then kick off a build manually?

clery · 2017-07-10T08:46:25Z

Hi @karmi, I added the email as you asked. Tell me if you need anything else !

clery · 2017-07-13T15:30:37Z

Hey there, any news on this ? Should I make any change ? :)

karmi · 2017-07-17T16:31:05Z

Thanks for adding the e-mail, @plassa-b, the CLA check is now green! I imagine the Beats team will get back to you shortly.

andrewkroh · 2017-07-18T20:43:07Z

libbeat/docs/outputconfig.asciidoc

@@ -958,6 +958,7 @@ output.file:
  filename: {beatname_lc}
  #rotate_every_kb: 10000
  #number_of_files: 7
+  #permissions: 0666


The commented out value should reflect the default value.

andrewkroh · 2017-07-18T20:47:35Z

metricbeat/metricbeat.full.yml

@@ -953,6 +953,9 @@ output.elasticsearch:
  # default is 7 files.
  #number_of_files: 7

+  # Permissions to use for file creation. The default is 0600.
+  #permissions: 0666


This file is generated based on libbeat/_meta/config.reference.yml. After updating the file in libbeat you should run make update from the beats dir to re-generate the config files.

…e to config.go and file_rotator.go

clery · 2017-07-19T09:31:01Z

Hi @andrewkroh, I just made the changes you asked for. I also merged with a more recent version of beats (i realised the config.full.yml was renamed as config.reference.yml, so I figured it would be simpler for you guys)

Also, I realised that I left a segfault when no value was given for permissions in the config file (when logging the value, I was dereferencing the value without checking it wasn't nil), so I fixed that and added the default properly where needed.

Hope it does the trick. Feel free to ask me if you need anything else.

clery · 2017-08-16T16:48:56Z

Hey guys, any news about this PR ?

ejsears · 2017-08-21T15:48:53Z

Just checking into this as we have no other way to import data. Thanks!

andrewkroh · 2017-08-24T15:11:51Z

LGTM. Could you add this configuration option to the documentation for the file output. https://github.com/elastic/beats/blob/master/libbeat/docs/outputconfig.asciidoc#configuration-options-4

clery · 2017-09-04T15:07:00Z

Sorry for the delay.

I'm not sure I understand, if I'm correct, the documentation was already added (https://github.com/plassa-b/beats/blob/master/libbeat/docs/outputconfig.asciidoc#file-output)

Here's the diff https://github.com/elastic/beats/pull/4638/files#diff-f1a89e6c3a1ce6f296f47ad83b989ff7R961

andrewkroh · 2017-09-07T15:09:51Z

jenkins, test it

andrewkroh · 2017-09-07T16:18:54Z

I'm not sure I understand, if I'm correct, the documentation was already added

Sorry, I missed that in the diff.

Can you please check the tests. It looks like this change is causing a panic in a test. https://beats-ci.elastic.co/job/elastic+beats+pull-request+multijob-windows/1451/beat=libbeat,label=windows/testReport/junit/(root)/logp/Test_Rotator/

In some cases pod annotations are neede after the pod is deleted, for instance when filebeat is reading the log behind the container. This change makes sure we keep metadata after a pod is gone. By storing access times we ensure that it's available as long as it's being used

Update to go-libaudit v0.0.6 Closes elastic#4850

`alias` was removed in elastic#4502.

…c#5102)

6.0.0-beta1 snapshot does not exist as its not a snapshot anymore. Also master should be tested againts ES 7.0.0-alpha1.

* Use `beat.name` instead of `beat.hostname` in visualizations. See elastic#5276 for the motivation * Add a "tip" widget that tells user how they can select another host. * Automatically set a search pattern of the form `beat.name: "HOSTNAME"` where HOSTNAME is the Beat name (hostname) that uploads the dashboards. The way the last point works is that I saved the dashbaord using this filter: `beat.name:"CHANGEME_HOSTNAME"`. The kibana loader code does a string replacement and replaces `CHANGEME_HOSTNAME` with the actual hostname. The disadvantage of this approach (vs doing JSON parsing) is that we must remember to always save that dashboard with the `CHANGEME_HOSTNAME` wildcard in place. I have added a unit test to the system module that checks for that, so it should be relatively hard for us to forget. Closes elastic#5276.

andrewkroh · 2017-10-06T21:41:50Z

Sorry for such a late response. I just realized that this never merged and it's a great enhancement.

So I'm wondering wether the test should initialize the Permissions (to avoid calling CheckIfConfigSane) or if they should be defaultly initialized somewhere else.

For simplicity I would just fix the test to initialize Permissions.

Thanks!

This PR adds support for enabling TLS renegotiation. The setting is `ssl.renegotiation` and the options are `never` (default), `once`, and `freely`. This exposes the three options from https://golang.org/pkg/crypto/tls/#RenegotiationSupport. Fixes elastic#4386

The command was initially designed for the apm-server to modify the index pattern. But now a nicer solution was found and this is not needed anymore.

Already the compose timeout is set to 300. Now also setting the retries in the healtcheck to 300.

This were broken on case-sensitive file systems, because the name of the files didn't match the name of the IDs from the dashboards. Fixes elastic#5277.

jenkins_setup in common.bash will also be used for the generator tests.

* Move queue setting to TOC + address some review * Remove flush_interval and spooler from outputconfig * remove 'slot' * add [float] tags * Update queueconfig.asciidoc

codecov · 2017-10-10T08:12:55Z

Codecov Report

Merging #4638 into master will increase coverage by 12.06%.
The diff coverage is n/a.

@@             Coverage Diff             @@
##           master    #4638       +/-   ##
===========================================
+ Coverage    53.9%   65.97%   +12.06%     
===========================================
  Files         555      410      -145     
  Lines       34594    30589     -4005     
===========================================
+ Hits        18647    20180     +1533     
+ Misses      14250     8746     -5504     
+ Partials     1697     1663       -34

Impacted Files	Coverage Δ
metricbeat/helper/server/tcp/tcp.go	`44.26% <0%> (-14.76%)`	⬇️
filebeat/fileset/factory.go	`65.07% <0%> (-7.94%)`	⬇️
auditbeat/module/audit/kernel/audit_linux.go	`43.04% <0%> (-6.1%)`	⬇️
metricbeat/helper/server/http/http.go	`52.11% <0%> (-4.23%)`	⬇️
filebeat/config/config.go	`56.66% <0%> (-2.99%)`	⬇️
filebeat/registrar/registrar.go	`75.59% <0%> (-2.26%)`	⬇️
filebeat/crawler/crawler.go	`78.31% <0%> (-1%)`	⬇️
filebeat/prospector/prospector.go	`89.28% <0%> (-0.25%)`	⬇️
metricbeat/module/elasticsearch/node_stats/data.go	`100% <0%> (ø)`	⬆️
metricbeat/mb/module/runner.go	`100% <0%> (ø)`	⬆️
... and 397 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e1060ff...4f13672. Read the comment docs.

Remove default golang version as not needed anymore. All repos / branches have now a .go-version file.

andrewkroh · 2017-10-10T15:09:53Z

jenkins, test it

* Add Azure VM Cloud Metadata Support * Update CHANGELOG.asciidoc * Update processors-using.asciidoc

andrewkroh · 2017-10-10T15:31:10Z

Could you please rebase the PR on master. We added some new scripts in master that are used by Jenkins for testing.

…e to config.go and file_rotator.go

clery · 2017-10-11T08:13:08Z

So I just actually did a git rebase from master, but giving that the PR is filled with commits that aren't from me, I'm wondering if that was actually what you wanted

ruflin · 2017-10-12T13:24:04Z

@plassa-b You probably have to fetch the most recent version of based and rebase on that.

This PR adds a `output.file.permissions` configuration option to control the file mode used when created the output file.

andrewkroh · 2017-10-12T22:10:43Z

Merged in 5c51448

I used the manual instructions to rebase and squash the PR. Thanks @plassa-b

Cléry Plassat added 4 commits July 7, 2017 18:30

Add permissions configuration for file output

1cd38f7

Add permissions configuration for file output

e699a6f

Merge branch 'master' of github.com:plassa-b/beats

855af00

Small change

aca6d5e

andrewkroh requested changes Jul 18, 2017

View reviewed changes

Cléry Plassat added 3 commits July 19, 2017 10:44

Merge remote-tracking branch 'upstream/master'

aedc2ae

Change config.reference.yml appropriatly & fix wrong value in docs

2f407bd

Fix bug when no value given for permissions & add proper default valu…

e2c5ace

…e to config.go and file_rotator.go

andrewkroh approved these changes Sep 7, 2017

View reviewed changes

Merge branch 'master' into master

2380939

exekias and others added 7 commits September 7, 2017 21:49

Support receiving events from audit multicast group (elastic#5081)

c76f14f

Update to go-libaudit v0.0.6 Closes elastic#4850

Remove alias from perfmon docs (elastic#5130)

ac9561d

`alias` was removed in elastic#4502.

should use strings.Contains(string(cmdline), process) instead (elasti…

01841ce

…c#5102)

Add missing link texts to fields references (elastic#4919)

9f2795e

Update ES test version to 7.0.0-alpha1 (elastic#5142)

68a6f35

6.0.0-beta1 snapshot does not exist as its not a snapshot anymore. Also master should be tested againts ES 7.0.0-alpha1.

Add flush timeout setting to filebeat registrar (elastic#5146)

0d89cc7

andrewkroh added enhancement libbeat labels Oct 6, 2017

andrewkroh and others added 7 commits October 9, 2017 12:13

Remove post-update command (elastic#5306)

a5d5fbe

The command was initially designed for the apm-server to modify the index pattern. But now a nicer solution was found and this is not needed anymore.

Switch to Kibana 6.0.0-rc1 and increase retry to 300 (elastic#5358)

67b6648

Already the compose timeout is set to 300. Now also setting the retries in the healtcheck to 300.

Fix RabbitMQ 5.x dashboard names (elastic#5359)

3ffa7c5

This were broken on case-sensitive file systems, because the name of the files didn't match the name of the IDs from the dashboards. Fixes elastic#5277.

Introduce jenkins_setup command to be reused (elastic#5315)

f9ac23f

jenkins_setup in common.bash will also be used for the generator tests.

Add queue settings to docs (elastic#4884)

54a4481

* Move queue setting to TOC + address some review * Remove flush_interval and spooler from outputconfig * remove 'slot' * add [float] tags * Update queueconfig.asciidoc

Fix Test_Rotator's segmentation fault

4f13672

Only read go version from .go-version file (elastic#5367)

1a98c80

Remove default golang version as not needed anymore. All repos / branches have now a .go-version file.

Add Azure VM support for add_cloud_metadata processor (elastic#5355)

483982b

* Add Azure VM Cloud Metadata Support * Update CHANGELOG.asciidoc * Update processors-using.asciidoc

Cléry Plassat added 7 commits October 11, 2017 10:08

Add permissions configuration for file output

d6c118c

Add permissions configuration for file output

f98de95

Small change

8876ded

Change config.reference.yml appropriatly & fix wrong value in docs

d825830

Fix bug when no value given for permissions & add proper default valu…

8227731

…e to config.go and file_rotator.go

Fix Test_Rotator's segmentation fault

9a136fb

Merge branch 'master' of github.com:plassa-b/beats

d9f265b

andrewkroh pushed a commit that referenced this pull request Oct 12, 2017

Add permissions configuration for file output (#4638)

5c51448

This PR adds a `output.file.permissions` configuration option to control the file mode used when created the output file.

andrewkroh closed this Oct 12, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add permissions configuration for file output #4638

Add permissions configuration for file output #4638

clery commented Jul 10, 2017

karmi commented Jul 10, 2017

elasticmachine commented Jul 10, 2017

elasticmachine commented Jul 10, 2017

clery commented Jul 10, 2017

clery commented Jul 13, 2017

karmi commented Jul 17, 2017 •

edited

Loading

andrewkroh Jul 18, 2017

andrewkroh Jul 18, 2017

clery commented Jul 19, 2017

clery commented Aug 16, 2017

ejsears commented Aug 21, 2017

andrewkroh commented Aug 24, 2017

clery commented Sep 4, 2017

andrewkroh commented Sep 7, 2017

andrewkroh commented Sep 7, 2017

andrewkroh commented Oct 6, 2017

codecov bot commented Oct 10, 2017 •

edited

Loading

andrewkroh commented Oct 10, 2017

andrewkroh commented Oct 10, 2017

clery commented Oct 11, 2017

ruflin commented Oct 12, 2017

andrewkroh commented Oct 12, 2017

Add permissions configuration for file output #4638

Add permissions configuration for file output #4638

Conversation

clery commented Jul 10, 2017

karmi commented Jul 10, 2017

elasticmachine commented Jul 10, 2017

elasticmachine commented Jul 10, 2017

clery commented Jul 10, 2017

clery commented Jul 13, 2017

karmi commented Jul 17, 2017 • edited Loading

andrewkroh Jul 18, 2017

Choose a reason for hiding this comment

andrewkroh Jul 18, 2017

Choose a reason for hiding this comment

clery commented Jul 19, 2017

clery commented Aug 16, 2017

ejsears commented Aug 21, 2017

andrewkroh commented Aug 24, 2017

clery commented Sep 4, 2017

andrewkroh commented Sep 7, 2017

andrewkroh commented Sep 7, 2017

andrewkroh commented Oct 6, 2017

codecov bot commented Oct 10, 2017 • edited Loading

Codecov Report

andrewkroh commented Oct 10, 2017

andrewkroh commented Oct 10, 2017

clery commented Oct 11, 2017

ruflin commented Oct 12, 2017

andrewkroh commented Oct 12, 2017

karmi commented Jul 17, 2017 •

edited

Loading

codecov bot commented Oct 10, 2017 •

edited

Loading