v1.6.0
New Features
Threat Intelligence
This release includes a new (beta) threat intelligence feature to enable analysis and identification of suspicious activity in your infrastructure based on IP address, domain and file hash indicators.
StreamAlert compares these indicators (stored in a DynamoDB table) to incoming data in real-time, and generates an alert if any matches are found.
To compliment this feature, it also includes a Threat Intel Downloader which is a Lambda function to collect and update the latest IP addresses, domains and file hashes mentioned above to the DynamoDB table. Currently, the Threat Intel Downloader supports fetching of data from Anomali’s ThreatStream API.
StreamAlert Apps
StreamAlert Apps enable you to easily retrieve data from any service with a RESTful API to send to StreamAlert for processing and alerting. The infrastructure is serverless, the configuration takes minutes, and the deployment is automated via Terraform.
Read more about this feature in our blog post, or learn how to get started with StreamAlert Apps in the documentation.
This release includes several apps, with more coming in future releases:
- Duo Admin & Auth Logs
- OneLogin Events
- GSuite Admin Reports
- Admin, Calendar, Drive, GPlus, Groups, Login, Mobile, Rules, SAML, Token
- Box Admin Events
Historical Search of Data
As announced in the last release (v1.5.0), StreamAlert can be configured to search generated alerts with AWS Athena.
This feature has been extended to support delivery of all incoming logs into Amazon S3 via AWS Firehose, and to be searched by AWS Athena in the streamalert database. This allows users to query data for long periods of time, and perform statistics, joins, and other analysis.
The StreamAlert CLI also manages the setup, creation, and provisioning of data tables and required AWS infrastructure. To get started, check out our Athena setup instructions.
PagerDuty Events API v2 and Incidents API Output Support
StreamAlert now includes support for two new PagerDuty API outputs:
- PagerDuty Events API (v2) - Be sure to upgrade your outputs, as the Events API V1 has been deprecated.
- PagerDuty Incidents API - This allows for the usage of more advanced PagerDuty features, such as assigning an incident to a specific user or setting a priority, directly from within a rule.
Improvements
Local Rule Testing Enhancements
Rule test events can now be configured to indicate which rules they will trigger, and includes the log schema that this event corresponds to. The CLI also now reports on hard-to-diagnose errors related to rule tests. See the documentation for more information on the new test event structure.
Rule Helpers for Finding Key Items
StreamAlert now includes rule helper functions which help you recursively find key-values in records without worrying about the schema or nesting.
Security Linting via Bandit
Bandit is a Python scanner which checks for common security issues in Python source code. The project has now been updated to run bandit on the StreamAlert source as part the CI pipeline.
User-configurable Kinesis Shard-Level Metrics
Kinesis shard-level metrics, via enhanced monitoring, will now be disabled by default with the optional ability to configure specific metrics to log. This will greatly reduce AWS costs for end-users.
Core Infrastructure Improvements
- Global Alerts Firehose - Enables high throughput delivery of alerts to S3
- Optional Kinesis Modules - Adds modularity to StreamAlert clusters, making Kinesis optional. This fully enables a purely S3 based cluster, where Kinesis is not necessary to deliver data into Lambda.
- Cross-Account CloudTrail - Supports receiving CloudTrail data from multiple AWS accounts into the StreamAlert CloudTrail module.
- S3 Event Filtering - Added support for suffix and prefix filtering of event notifications of objects in S3
Bug Fixes
- #339 - Fix for nested data normalization
- #361 - Classifier TypeError fix when casting to list/dict
- #367, #381 - Fixed various bugs related to data normalization
- #393 , #564 - Faster unit tests
- #449, #431 - Fixing various CLI bugs
- #453 - Fixed json parser bug related to json path
- #456 - Fixed classifier type conversion of nested values
- #548 - Fixed bug with total records metric
- #578 - Athena Partition Refresh KeyError bug fix
Updates to various Carbon Black schemas fixes for Carbon Black and CloudTrail logs.
Shout-outs
Special thanks for the following external contributions from @armtash and @javefang: