[apps] adding G Suite app for all activity audit reports #426

ryandeivert · 2017-10-27T01:49:29Z

to: @javuto and @austinbyers
cc: @airbnb/streamalert-maintainers
size: large
resolves #348

Background

tl;dr - Consume all GSuite logs:

Changes

Adding initial implementation of G Suite StreamAlertApp
This app utilizes the google-api-python-client module, which has been added to the requirements.txt (along with other dependencies) and to the 3rd party libraries that get packaged with the Apps lambda function.
All GSuite apps must use a Google service account key which must be configured by an admin, along with access to the Admin SDK api.
- The service account key type must be in the Google-recommended JSON format (not p12):
- Attempting to create a GSuite application with a p12 keyfile will fail and inform the user that a JSON keyfile must be provided.

Other Changes

Adding an auth property helper to the AppConfig so the auth subdictionary within the config dict can now be accessed via 'AppConfig.auth' instead of requiring apps to do a dictionary lookup of 'auth' any time authentication info needs to be retrieved.
- Also updating all unit tests to go with the above change.
Making the AppIntegration.required_auth_info method a classmethod to the app does not need to be instantiated in the CLI to access this method.
- Also making corresponding change to the cli's runner.py and config.py so the apps do not get instantiated when prompting the user for required authentication information.
Adding support for complex input validation for items that are provided by the user.
- A function can now be passed to the user_input cli helper that can be used to properly validate input. Previously only a simple x is not in y or regex comparison could be used.

Bug Fixes

Marking the AppIntegration._more_to_poll to False if there was nothing returned from an app's gather function. This avoids potentially bad looping in the subclasses.
Fixing potential issue where casting auth values to a string could potentially cast dict values. This is meant to only case unicode to str, so checking for unicode first.
Fixing bug in validate-schemas command that caused the --test-files flag to not work as expected

Testing

Added unit tests for the GSuiteReportsApps.
Also added the schema for gsuite:reports and added a test event for validation purposes only to test the schema.
All tests passing.

coveralls · 2017-10-27T17:08:37Z

Coverage increased (+0.2%) to 95.769% when pulling e362b65 on ryandeivert-gsuite-app-support into 1372b2b on master.

ghost · 2017-10-27T17:12:29Z

@ryandeivert can you add support for https://developers.google.com/admin-sdk/reports/v1/guides/manage-audit-mobile? It appears to be the last remaining one per my Google foo: inurl:developers.google.com inurl:manage-audit

ryandeivert · 2017-10-27T17:19:06Z

@mime-frame your google foo is excellent :) however, there are a couple of others we aren't doing. See the applicationName parameter here:

https://developers.google.com/admin-sdk/reports/v1/reference/activities/list

Additional apps:

calendar
groups
gplus
mobile
rules
saml (https://developers.google.com/admin-sdk/reports/v1/appendix/activity/saml)

Do we want to support all of the above ^^ ?

ghost · 2017-10-27T17:33:41Z

@ryandeivert - yup, it's worth supporting all of the above - you think its worth doing this in a 2nd PR or this one? There's also a SAML one too it seems: https://developers.google.com/admin-sdk/reports/v1/appendix/activity/saml

ryandeivert · 2017-10-27T17:38:29Z

Supporting them is a very simple change tbh so it doesn't really matter to me either way. Really annoying that googles docs say one thing one place and other thing somewhere else 🤔

ghost · 2017-10-27T17:41:04Z

Supporting them is a very simple change tbh so it doesn't really matter to me either way.

Cool, lets get it into this PR then :)

ryandeivert · 2017-10-27T17:48:17Z

@mime-frame done

coveralls · 2017-10-27T17:50:36Z

Coverage increased (+0.2%) to 95.815% when pulling c89e6ab on ryandeivert-gsuite-app-support into 1372b2b on master.

* The 'auth' subdictionary within the config dict can now be accessed via the 'AppConfig.auth' property instead of requiring apps to do a dictionary lookup of 'auth' any time authentication info needs to be retrieved.

* Marking the `AppIntegration._more_to_poll` to `False` if there was nothing returned from an app's gather function * Fixing potential issue where casting auth values to a string could potentially cast dict values. This is meant to only case `unicode` to str, so checking for unicode first.

* Fixing bug in `validate-schemas` command that caused the `--test-files` flag to not work as expected

…onfig

coveralls · 2017-10-27T18:02:37Z

Coverage increased (+0.2%) to 95.815% when pulling 1448066 on ryandeivert-gsuite-app-support into f7be9e9 on master.

austinbyers

Great work! Can you remind me what happens to API tokens? They are encrypted in SSM, correct? The app documentation explains how the user configures the auth token, but not what happens to it

austinbyers · 2017-10-27T17:56:43Z

app_integrations/apps/gsuite.py

+            userKey='all',
+            applicationName=self._type(),
+            startTime=self._last_timestamp,
+            pageToken=self._next_page_token if self._next_page_token else None


Is there some reason that self._next_page_token could be an empty string? If so, you could do simply:

pageToken = self._next_page_token or None

The case of it potentially being an empty string is exactly why I added this, so good catch :). I don't know for sure if passing '' to pageToken would have a different effect than passing None so I just wanted to guard against it.

I added this because I'm not sure what the default value (if any) is within the response for nextPageToken that gets returned. It's unclear if 1) the key exists at all if there is no value, and 2) what the value would be if it does exist but is empty (is it an empty string or a json null)?

The code where this is set is here, using a .get:

streamalert/app_integrations/apps/gsuite.py

Line 121 in 1448066

self._next_page_token = results.get('nextPageToken')

Maybe a second set of eyes on the docs would help unearth something I'm not seeing ;)

ryandeivert · 2017-10-27T18:48:25Z

@austinbyers excellent question - the answer is yes, the API tokens/secrets are sent to SSM as a SecureString:

streamalert/stream_alert_cli/apps.py

Lines 41 to 49 in 1448066

    
           def save(overwrite=False): 
        
               ssm_client.put_parameter( 
        
                   Name=name, 
        
                   Description=description, 
        
                   Value=param_value, 
        
                   Type='SecureString', 
        
                   Overwrite=overwrite 
        
               )

ryandeivert force-pushed the ryandeivert-gsuite-app-support branch 2 times, most recently from 0f54afb to e362b65 Compare October 27, 2017 17:05

airbnb deleted a comment from coveralls Oct 27, 2017

ghost changed the title ~~[apps] adding G Suite app for admin, drive, login and token reports~~ [apps] adding G Suite app for all activity audit reports Oct 27, 2017

ryandeivert added 8 commits October 27, 2017 10:58

[apps] adding 'auth' property helper to config

c563805

* The 'auth' subdictionary within the config dict can now be accessed via the 'AppConfig.auth' property instead of requiring apps to do a dictionary lookup of 'auth' any time authentication info needs to be retrieved.

[cli] user input restrictions now support a validation function

30c5f4c

[apps] adding gsuite app for admin/login/token/drive reports

363ec4b

[schemas] adding gsuite reports schema and validation event

c752b55

* Fixing bug in `validate-schemas` command that caused the `--test-files` flag to not work as expected

[tests][apps] updating previous tests to use new auth property on c…

a27a5cc

…onfig

[tests][apps] adding unit tests for gsuite app(s)

1bda2b6

[apps][gsuite] calendar, groups, gplus, mobile, rules, and saml apps

1448066

ryandeivert force-pushed the ryandeivert-gsuite-app-support branch from c89e6ab to 1448066 Compare October 27, 2017 17:59

austinbyers approved these changes Oct 27, 2017

View reviewed changes

ryandeivert merged commit b0c016d into master Oct 27, 2017

ryandeivert deleted the ryandeivert-gsuite-app-support branch October 27, 2017 19:39

ryandeivert mentioned this pull request Oct 27, 2017

[docs] updates for gsuite apps #429

Merged

ryandeivert added the apps label Oct 27, 2017

ryandeivert added this to the 1.6.0 milestone Oct 27, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[apps] adding G Suite app for all activity audit reports #426

[apps] adding G Suite app for all activity audit reports #426

ryandeivert commented Oct 27, 2017 •

edited by ghost

Loading

coveralls commented Oct 27, 2017

ghost commented Oct 27, 2017

ryandeivert commented Oct 27, 2017 •

edited

Loading

ghost commented Oct 27, 2017

ryandeivert commented Oct 27, 2017

ghost commented Oct 27, 2017

ryandeivert commented Oct 27, 2017

coveralls commented Oct 27, 2017

coveralls commented Oct 27, 2017

austinbyers left a comment

austinbyers Oct 27, 2017

ryandeivert Oct 27, 2017

ryandeivert Oct 27, 2017

ryandeivert commented Oct 27, 2017

[apps] adding G Suite app for all activity audit reports #426

[apps] adding G Suite app for all activity audit reports #426

Conversation

ryandeivert commented Oct 27, 2017 • edited by ghost Loading

Background

Changes

Other Changes

Bug Fixes

Testing

coveralls commented Oct 27, 2017

ghost commented Oct 27, 2017

ryandeivert commented Oct 27, 2017 • edited Loading

Additional apps:

ghost commented Oct 27, 2017

ryandeivert commented Oct 27, 2017

ghost commented Oct 27, 2017

ryandeivert commented Oct 27, 2017

coveralls commented Oct 27, 2017

coveralls commented Oct 27, 2017

austinbyers left a comment

Choose a reason for hiding this comment

austinbyers Oct 27, 2017

Choose a reason for hiding this comment

ryandeivert Oct 27, 2017

Choose a reason for hiding this comment

ryandeivert Oct 27, 2017

Choose a reason for hiding this comment

ryandeivert commented Oct 27, 2017

ryandeivert commented Oct 27, 2017 •

edited by ghost

Loading

ryandeivert commented Oct 27, 2017 •

edited

Loading