Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Osquerybeat: Set the raw index name to suppress the timestamp suffix #26545

Merged
merged 1 commit into from
Jun 28, 2021
Merged

Osquerybeat: Set the raw index name to suppress the timestamp suffix #26545

merged 1 commit into from
Jun 28, 2021

Conversation

aleksmaus
Copy link
Member

What does this PR do?

Set the raw index name to suppress the timestamp suffix
This fixes the issue where the osquerybeat can't create the index for the result data, due to recent permissions tightening in kibana:

(status=403): {\"type\":\"security_exception\",\"reason\":\"action [indices:admin/auto_create] is unauthorized for API key id [uEnGPnoB_V-56sCMhHs-] of user [elastic] on indices [logs-osquery_manager.result-default-2021.06.24], this action is granted by the index privileges [auto_configure,create_index,manage,all]\"}

Also eliminates the need for special casing for osquerybeat elastic/kibana#103319.

Based on discussion:
https://groups.google.com/a/elastic.co/g/agent-team/c/Syc0bE12aK4/m/uuYWmILkBgAJ

Why is it important?

Fixes the breakage due to permissions tightening in kibana.

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas

@aleksmaus aleksmaus added bug backport-v7.14.0 Automated backport with mergify Team:Asset Mgt labels Jun 28, 2021
@botelastic botelastic bot added needs_team Indicates that the issue/PR needs a Team:* label and removed needs_team Indicates that the issue/PR needs a Team:* label labels Jun 28, 2021
james-elastic
james-elastic approved these changes Jun 28, 2021
View reviewed changes
Copy link

@james-elastic james-elastic left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need to do anything for upgrades?

@aleksmaus
Copy link
Member Author

Do we need to do anything for upgrades?

the new data will end up in the index without date suffix. Everything should still be queryable.

@jen-huang jen-huang mentioned this pull request Jun 28, 2021
Closed
@elasticmachine
Copy link
Collaborator

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview

Expand to view the summary

Build stats

  • Build Cause: Pull request #26545 opened

  • Start Time: 2021-06-28T19:47:58.822+0000

  • Duration: 62 min 30 sec

  • Commit: 28e0abf

Test stats 🧪

Test Results
Failed 0
Passed 128
Skipped 0
Total 128

Trends 🧪

Image of Build Times

Image of Tests

💚 Flaky test report

Tests succeeded.

Expand to view the summary

Test stats 🧪

Test Results
Failed 0
Passed 128
Skipped 0
Total 128

@aleksmaus aleksmaus merged commit 67cf2c6 into elastic:master Jun 28, 2021
mergify bot pushed a commit that referenced this pull request Jun 28, 2021
@aleksmaus
Osquerybeat: set the raw index name to supress the timestamp suffix (#…
0e53a9b 
…26545)

(cherry picked from commit 67cf2c6)
@mergify mergify bot mentioned this pull request Jun 28, 2021
Merged
mdelapenya added a commit to mdelapenya/beats that referenced this pull request Jun 29, 2021
@mdelapenya
Merge branch 'master' into archive-system-tests
d4a87cb 
* master:
  Osquerybeat: set the raw index name to supress the timestamp suffix (elastic#26545)
  [Heartbeat] add screenshots config to synthetics (elastic#26455)
  [Elastic Agent] Use http2 to connect to Fleet Server. (elastic#26474)
  Remove all docs about  Beats central management (elastic#26399)
  update data.json for gcp billing (elastic#26506)
  Skip x-pack metricbeat tests (elastic#26537)
  [Elastic Agent] Fix issue with FLEET_CA not being used with Fleet Server in container (elastic#26529)
  Add changelog entry for  elastic#26224 (elastic#26531)
  Add inttests for the x-pack/metricbeat on a PR/branches basis (elastic#26526)
  Suppress too many errors (elastic#26224)
  Fix master's linting issue (elastic#26517)
  [libbeat] Fix encoding and file offset issues in the disk queue (elastic#26484)
  Add log_group_name_prefix config option for aws-cloudwatch input (elastic#26187)
  Update shared-deduplication.asciidoc (elastic#26492)
  Add Recorded Future support to threatintel module (elastic#26481)
v1v added a commit to v1v/beats that referenced this pull request Jun 29, 2021
@v1v
Merge remote-tracking branch 'upstream/master' into feature/support-d…
2c9f25c 
…arwin-arm64

* upstream/master: (295 commits)
  Update urllib to 1.26.5. (elastic#26380)
  Update golang.org/x/crypto (elastic#26448)
  [Filebeat] Update Fortinet Ingest Pipeline (elastic#24816)
  Move parsers outside of filestream input so others can use them as well (elastic#26541)
  [Filebeat] Fix `threatintel.indicator.url.full` field not populating (elastic#26508)
  [Filebeat] Add network direction processor to Zeek and Suricata modules (elastic#24620)
  Logging code cleanup related to Nomad auto-discovery (elastic#26498)
  [Metricbeat] Add Couchbase's Sync Gateway module (elastic#25599)
  Refactor add_cloud_metadata to handle ECS fields easier (elastic#26438)
  [Elastic Agent] Improper casting of int64 (elastic#26520)
  [Elastic Agent] Enable configuring monitoring namespace (elastic#26439)
  [Heartbeat] configure permissions for synthetics config (elastic#26393)
  Osquerybeat: set the raw index name to supress the timestamp suffix (elastic#26545)
  [Heartbeat] add screenshots config to synthetics (elastic#26455)
  [Elastic Agent] Use http2 to connect to Fleet Server. (elastic#26474)
  Remove all docs about  Beats central management (elastic#26399)
  update data.json for gcp billing (elastic#26506)
  Skip x-pack metricbeat tests (elastic#26537)
  [Elastic Agent] Fix issue with FLEET_CA not being used with Fleet Server in container (elastic#26529)
  Add changelog entry for  elastic#26224 (elastic#26531)
  ...
aleksmaus added a commit that referenced this pull request Jun 29, 2021
@mergify @aleksmaus
Osquerybeat: set the raw index name to supress the timestamp suffix (#…
3de67ff 
…26545) (#26549)

(cherry picked from commit 67cf2c6)

Co-authored-by: Aleksandr Maus <aleksandr.maus@elastic.co>
@ruflin
Copy link
Contributor

ruflin commented Jun 30, 2021

Thanks for the fix @aleksmaus . I consider the previous behaviour a bug. But there is one issue that we need to communicate to customers now. Using 7.13 Elastic Agent with osquery will not be able to ship data to 7.14 because of the permissions.

@aleksmaus
Copy link
Member Author

Thanks for the fix @aleksmaus . I consider the previous behaviour a bug. But there is one issue that we need to communicate to customers now. Using 7.13 Elastic Agent with osquery will not be able to ship data to 7.14 because of the permissions.

Thank you! Will communicate to our team management.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@james-elastic james-elastic james-elastic approved these changes

@urso urso urso approved these changes

@ruflin ruflin Awaiting requested review from ruflin

Assignees
No one assigned
Labels
backport-v7.14.0 Automated backport with mergify bug Team:Asset Mgt
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@aleksmaus @elasticmachine @ruflin @urso @james-elastic