Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Filebeat][GSuite] Initial implementation of SAML and User Accounts filesets #19329

Merged
merged 18 commits into from
Jul 7, 2020

Conversation

marc-gr
Copy link
Contributor

@marc-gr marc-gr commented Jun 23, 2020

What does this PR do?

This PR adds a GSuite module to filebeat that uses httpjson as input, and creates a SAML and User accounts filesets for it, which consumes events from https://developers.google.com/admin-sdk/reports/v1/appendix/activity/[saml|user-accounts]

Why is it important?

It is the first step to allow users to consume gsuite activity reports. It sets the common pieces for the next filesets and since SAML and User accounts are the simplest ones makes it easy to test and validate the module functionality.

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Author's Checklist

  • Both GSuite common fields and fileset specific ones are documented
  • Added test files for filesets

Depends on

#19246

@marc-gr marc-gr added enhancement in progress Pull request is currently in progress. Filebeat Filebeat Team:SIEM v7.9.0 labels Jun 23, 2020
@elasticmachine
Copy link
Collaborator

Pinging @elastic/siem (Team:SIEM)

@botelastic botelastic bot added needs_team Indicates that the issue/PR needs a Team:* label and removed needs_team Indicates that the issue/PR needs a Team:* label labels Jun 23, 2020
@elasticmachine
Copy link
Collaborator

elasticmachine commented Jun 23, 2020

💚 Build Succeeded

Pipeline View Test View Changes Artifacts preview

Expand to view the summary

Build stats

  • Build Cause: [Pull request #19329 updated]

  • Start Time: 2020-07-06T14:51:07.423+0000

  • Duration: 56 min 26 sec

Test stats 🧪

Test Results
Failed 0
Passed 4228
Skipped 677
Total 4905

@marc-gr marc-gr force-pushed the feature_filebeat_gsuite branch from 68fe137 to dd80383 Compare June 25, 2020 08:53
@marc-gr marc-gr changed the title [Filebeat][GSuite] Initial implementation of SAML fileset [Filebeat][GSuite] Initial implementation of SAML and User Accounts filesets Jun 25, 2020
@marc-gr marc-gr force-pushed the feature_filebeat_gsuite branch 2 times, most recently from 4af96b3 to f558ba5 Compare June 25, 2020 15:11
@marc-gr marc-gr force-pushed the feature_filebeat_gsuite branch 3 times, most recently from d43792e to 1e391e5 Compare July 1, 2020 10:39
@marc-gr marc-gr added review and removed in progress Pull request is currently in progress. labels Jul 1, 2020
@marc-gr marc-gr force-pushed the feature_filebeat_gsuite branch from 1e391e5 to 9544599 Compare July 1, 2020 12:17
@marc-gr marc-gr requested a review from andrewkroh July 1, 2020 12:38
@marc-gr marc-gr force-pushed the feature_filebeat_gsuite branch 3 times, most recently from c34f88e to 24fd4e6 Compare July 2, 2020 15:01
Copy link
Contributor

@leehinman leehinman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks really really good. Couple of small questions on ECS types & categories.

@marc-gr marc-gr force-pushed the feature_filebeat_gsuite branch from 24fd4e6 to 5d94306 Compare July 3, 2020 08:13
@marc-gr marc-gr requested review from leehinman and andrewstucki July 3, 2020 08:17
@marc-gr marc-gr force-pushed the feature_filebeat_gsuite branch from 0e5c307 to e66f25e Compare July 6, 2020 14:50
@marc-gr marc-gr requested a review from andrewstucki July 6, 2020 14:50
Copy link

@andrewstucki andrewstucki left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm good with this, but it'd be nice if @leehinman or @andrewkroh could sign off on the updated changes

Copy link
Contributor

@leehinman leehinman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@marc-gr marc-gr merged commit 7abd67d into elastic:master Jul 7, 2020
@marc-gr marc-gr deleted the feature_filebeat_gsuite branch July 7, 2020 14:02
marc-gr added a commit to marc-gr/beats that referenced this pull request Jul 14, 2020
…ilesets (elastic#19329)

* GSuite initial implementation of SAML fileset

* Document fields and generate test file

* Add documentation

* Split fields and improve docs

* Add change to CHANGELOG

* Rename config file and clean docs

* Adds user accounts fileset

* Add delegated user to google oauth

* Add types and make changes to common pipeline

* Do not stop input if array key not found

* Fix docs

* Setup for date cursor

* Add beta tag

* CHANGELOG message

* Improve ECS mappings

* Change cateogrization and types of various fields

* Change event.type to start

* Improve doc references

(cherry picked from commit 7abd67d)
marc-gr added a commit that referenced this pull request Jul 15, 2020
…ilesets (#19329) (#19726)

* GSuite initial implementation of SAML fileset

* Document fields and generate test file

* Add documentation

* Split fields and improve docs

* Add change to CHANGELOG

* Rename config file and clean docs

* Adds user accounts fileset

* Add delegated user to google oauth

* Add types and make changes to common pipeline

* Do not stop input if array key not found

* Fix docs

* Setup for date cursor

* Add beta tag

* CHANGELOG message

* Improve ECS mappings

* Change cateogrization and types of various fields

* Change event.type to start

* Improve doc references

(cherry picked from commit 7abd67d)
melchiormoulin pushed a commit to melchiormoulin/beats that referenced this pull request Oct 14, 2020
…ilesets (elastic#19329)

* GSuite initial implementation of SAML fileset

* Document fields and generate test file

* Add documentation

* Split fields and improve docs

* Add change to CHANGELOG

* Rename config file and clean docs

* Adds user accounts fileset

* Add delegated user to google oauth

* Add types and make changes to common pipeline

* Do not stop input if array key not found

* Fix docs

* Setup for date cursor

* Add beta tag

* CHANGELOG message

* Improve ECS mappings

* Change cateogrization and types of various fields

* Change event.type to start

* Improve doc references
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants