Skip to content

Commit

Permalink
Merge pull request #28 from airbnb/docs-conf-authors-contribs-reqs
Browse files Browse the repository at this point in the history
[docs] Add Authors, Contributors, and additional doc changes
  • Loading branch information
jacknagz authored Feb 8, 2017
2 parents 3772933 + 40489d5 commit c60e8cf
Show file tree
Hide file tree
Showing 14 changed files with 294 additions and 112 deletions.
10 changes: 10 additions & 0 deletions AUTHORS.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
Below are a list of both core members and contributors to StreamAlert.

Core Members:
* `Jack Naglieri <https://www.github.com/jacknagz>`_
* `Mime Frame <https://www.github.com/mime-frame>`_
* `Jamie Brim <https://www.github.com/strcrzy>`_
* `Austin Byers <https://github.com/austinbyers>`_
* `Chunyong Lin <https://github.com/chunyong-lin>`_

Contributors:
128 changes: 128 additions & 0 deletions CONTRIBUTING.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,128 @@
Contributing
============

Installing Dependencies
-----------------------

Make sure you are running Python 2.7::

$ python -V
Python 2.7.x

Install:

1. `Pip <https://pip.pypa.io/en/stable/installing/>`_
2. `Terraform <https://www.terraform.io/intro/getting-started/install.html>`_

Fork the StreamAlert repository from the UI, and then clone it locally::

$ git clone https://github.com/<your-username>/streamalert.git

Change into the cloned StreamAlert repo, and install Python dependencies::
$ cd streamalert
$ pip install -r requirements.txt

Making Changes
--------------

Checkout a new branch::
git checkout -b <branch-name>

Make changes, add features, or fix bugs.

When writing commit messages, make sure to prefix with one of the following tags::

[docs] # changes to StreamAlert documentation
[cli] # stream_alert_cli changes
[lambda][alert] # stream_alert lambda changes
[lambda][output] # stream_alert_output lambda changes
[terraform] # terraform changes
[core] # changes with core stream_alert classes used across both functions
[testing] # changes with testing infrastructure or processes
[setup] # StreamAlert development setup changes

The first line of your commit message should be short. Use newlines to explain further::

[tag] short description

* longer explanation of what the change is
* with added context

.. note:: Please squash your similar commits into one. This keeps the repositories commit history easy to read.

Commit Squashing
~~~~~~~~~~~~~~~~

If all of your commits fall within the same tag, you can squash them during the pull request process via the Github UI. If not, follow the steps below.

Scenario: You have unstaged changes that you want to add into your last commit::

$ git add -u # add all tracked files
$ git commit --amend # commit these changes to the last commit

Scenario: You have two commits you want to merge into one::

$ git log --oneline
c3dbbe9 [docs] add contributing guide
f5b038e [docs] add streamalert authors
04e52c1 [setup] add jinja2 to requirements.txt
...
In this case, let's merge the two ``[docs]`` commits::

$ git rebase -i f5b038e~1
pick f5b038e [docs] add streamalert authors
squash c3dbbe9 [docs] add contributing guide
If your editor is `vim`, type `:wq` once you enter the above changes.

This will now open a new window to modify your combined commit message. Make your edits, and exit once again.

After rebasing, you will need to force push your branch if it already exists upstream::

$ git push origin <mybranch> -f

Tests
-----

Unit Testing
~~~~~~~~~~~~

StreamAlert contains unit tests for many parts of the code. When making changes, you need to ensure that you do not break existing functionality. To run unit tests locally::

# run this from the repo root
$ nosetests -v test/unit

Each test should end with ``... ok``, and finally you should see ``OK`` at the end.

If you are making changes which require unit test refactoring, please do so.

If you are adding features to existing classes with tests, you must add test cases to verify expected behavior.

Integration Testing
~~~~~~~~~~~~~~~~~~~

To verify StreamAlert works from end-to-end, locally, follow the testing instructions `here <https://streamalert.io/rules.html>`_.

Pull Request
------------

Once your code is ready for review, push the branch to your forked repository, and make a pull-request to the main ``airbnb/streamalert`` repo.

The title of your pull request should be a short description of your changes.

In your pull request body, use the following template::

to @airbnb/streamalert-maintainers

size: small|medium|large
resolves: #1 # only add this if there's a relevant open issue related to this PR

* summary of changes 1
* summary of changes 2
All pull requests must pass continuous integration tests (nosetests) and receive a code review from one of the maintainers.

After your branch has been accepted, you are free to merge into ``master``!
12 changes: 6 additions & 6 deletions README.rst
Original file line number Diff line number Diff line change
@@ -1,15 +1,13 @@
**********************************************************
StreamAlert - Serverless, Realtime Data Analysis Framework
**********************************************************
==========================================================

.. image:: https://travis-ci.org/airbnb/streamalert.svg?branch=master
:target: https://travis-ci.org/airbnb/streamalert

StreamAlert is a serverless, realtime data analysis framework which empowers you to ingest, analyze, and alert on data from any environment, using datasources and alerting logic you define.

For more details, see our announcement post `here <https://medium.com/@airbnbeng/e8619e3e5043>`_

*High-level*:
High-level
~~~~~~~~~~

* Deployment is automated: simple, safe and repeatable for any AWS account
* Easily scalable from megabytes to terabytes per day
Expand All @@ -20,8 +18,10 @@ For more details, see our announcement post `here <https://medium.com/@airbnbeng
* Supports different types of data (ex: JSON, CSV, Key-Value, or Syslog)
* Supports different use-cases like security, infrastructure, compliance and more

*Links*:
Links
~~~~~

* `Announcement Post <https://medium.com/@airbnbeng/e8619e3e5043>`_
* `User Guide <https://streamalert.readthedocs.io/>`_
* `Twitter <https://twitter.com/streamalert_io>`_ (unofficial)
* `Slack <https://streamalert.herokuapp.com/>`_ (unofficial)
4 changes: 2 additions & 2 deletions conf/sample_sources.json
Original file line number Diff line number Diff line change
Expand Up @@ -7,13 +7,13 @@ $ ./stream_alert_cli.py terraform status
*/
{
"kinesis": {
"cluster-1--kinesis-stream": {
"prefix_cluster1_stream_alert_kinesis": {
"logs": [
"json_log_name",
"csv_log_name"
]
},
"cluster-2-kinesis-stream": {
"prefix_cluster2_stream_alert_kinesis": {
"logs": [
"json_log_name"
]
Expand Down
Binary file added docs/images/streamalert-logo.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
2 changes: 1 addition & 1 deletion docs/source/account.rst
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ Open ``variables.json`` and set ``account_id`` to this value.
prefix
~~~~~~

Open ``variables.json`` and set ``prefix`` to your company name.
Open ``variables.json`` and set ``prefix`` to your company or organization name.

Administrator
~~~~~~~~~~~~~
Expand Down
2 changes: 1 addition & 1 deletion docs/source/clusters.rst
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ StreamAlert will deploy separate infrastructure for each ``cluster`` (environmen

What constitutes a ``cluster`` is up to you.

Example: You could define ``IT``, ``PCI`` and ``Production`` clusters
Example: You could define ``IT``, ``PCI`` and ``Production`` clusters.

**Strategy**

Expand Down
19 changes: 12 additions & 7 deletions docs/source/conf-datasources.rst
Original file line number Diff line number Diff line change
@@ -1,22 +1,27 @@
Datasources
===========
Datasource Configuration
========================

For background on supported datasource types, read `datasources <datasources.html>`_.

Overview
--------

Datasources are defined in ``conf/sources.json``
* Datasources defined in ``conf/sources.json`` control which datasources can send to and be analyzed by StreamAlert.
* Each datasource (``kinesis``, ``s3``) contains a mapping of specific datasource names (kinesis stream names, s3 bucket IDs) along with a list of logs coming from that source.
* Logs are defined in ``conf/logs.json``
* Each log in the list of ``logs`` dictates to StreamAlert how to parse incoming data from that entity. Data will only be analyzed if their type is defined here.

Example::

{
"kinesis": { # define each kinesis stream w/respective logs
"abc_corporate_stream": { # kinesis stream name
"logs": [ # expected log types
"kinesis": { # define each kinesis stream w/respective logs
"abc_corporate_stream_alert_kinesis": { # kinesis stream name
"logs": [ # expected log types
"box",
"pan"
]
},
"abc_production_stream": {
"abc_production_stream_stream_alert_kinesis": {
"logs": [
"inspec",
"osquery"
Expand Down
2 changes: 1 addition & 1 deletion docs/source/conf-schemas.rst
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,7 @@ Here are the basics:
* An empty hash ({}) implies zero or more elements
* Schemas can be as tight or as loose as you want (see Example: osquery)

JSON Example: inspec
JSON Example: Inspec
--------------------
Schema::

Expand Down
46 changes: 28 additions & 18 deletions docs/source/datasources.rst
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
Datasources
=============
===========

StreamAlert supports:

Expand All @@ -8,10 +8,12 @@ StreamAlert supports:
* Log Forwarding Agents\*
* Code/Applications\*

\* *must send to a Kinesis Stream*
.. note:: \* *Must send to a Kinesis Stream*

To configure datasources, read `datasource configuration <conf-datasources.html>`_

Amazon S3
----------
---------

StreamAlert supports data analysis and alerting for logs written to configured Amazon S3 buckets.
This is achieved via Amazon S3 Event Notifications looking for an event type of ``s3:ObjectCreated:*``
Expand All @@ -29,9 +31,30 @@ Example non-AWS use-cases:
* Web Application logs (Apache, nginx, ...)
* SaaS logs (Box, OneLogin, …)

AWS Kinesis Streams
-------------------

StreamAlert also supports data analysis and alerting for logs written to AWS Kinesis Streams.
By default, StreamAlert configures an AWS Kinesis stream per `cluster <clusters.html>`_.

Optionally, StreamAlert can also utilize existing streams as an additional source
by adding the following into your generated Terraform cluster file (found in ``terraform/cluster-name.tf``)::

// Enable a Kinesis Stream to send events to Lambda
module "kinesis_events_<cluster-name>" {
source = "modules/tf_stream_alert_kinesis_events"
lambda_staging_enabled = true
lambda_production_enabled = true
lambda_role_id = "${module.stream_alert_<cluster-name>.lambda_role_id}"
lambda_function_arn = "${module.stream_alert_<cluster-name>.lambda_arn}"
kinesis_stream_arn = "<add-kinesis-stream-ARN-here>"
role_policy_prefix = "<cluster-name>"
}
There are several ways to send data into AWS Kinesis, as listed below.

Log Forwarding Agents
----------------------
~~~~~~~~~~~~~~~~~~~~~

StreamAlert utilizes AWS Kinesis Streams for real-time data ingestion.

Expand All @@ -42,25 +65,12 @@ Log forwarding agents that support AWS Kinesis Streams:
* **aws-kinesis-agent**
* **osquery**


Code/Applications
----------------------
~~~~~~~~~~~~~~~~~

StreamAlert utilizes AWS Kinesis Streams for real-time data ingestion.

Your code can send data to an AWS Kinesis Stream via:

* AWS SDK (Streams API)
* KPL (Amazon Kinesis Producer Library)


AWS Kinesis Streams
-------------------

StreamAlert can utilize existing stream(s) and/or create/deploy new streams to support log forwarding agents or code/applications.






26 changes: 13 additions & 13 deletions docs/source/getting-started.rst
Original file line number Diff line number Diff line change
@@ -1,16 +1,15 @@
Getting Started
===============

Perform the following on your laptop or development machine
Perform the following steps on your laptop or development machine

Dependencies
------------

1. Install Python2.7
2. Install `pip <https://pip.pypa.io/en/stable/installing/>`_
3. Install `Terraform <https://www.terraform.io/intro/getting-started/install.html>`_
4. Download StreamAlert: ``git clone https://github.com/airbnb/streamalert.git``
5. Install dependencies: ``pip install -r requirements.txt``
1. Install Python 2.7 and `pip <https://pip.pypa.io/en/stable/installing/>`_
2. Install `Terraform <https://www.terraform.io/intro/getting-started/install.html>`_
3. Download StreamAlert: ``git clone https://github.com/airbnb/streamalert.git``
4. Install dependencies: ``pip install -r requirements.txt``

.. note:: For Mac OSX/Homebrew users, add the following to ~/.pydistutils.cfg:

Expand All @@ -19,16 +18,17 @@ Dependencies
[install]
prefix=
Configuration
-------------
Quick Start
-----------

1. Define your `account <account.html>`_
1. Define your `AWS account <account.html>`_
2. Define your `clusters <clusters.html>`_
3. Define your `datasources <conf-datasources.html>`_
4. Define your `schemas <conf-schemas.html>`_
5. Define your `kinesis streams <kinesis-streams.html>`_
6. Define your `kinesis firehose <kinesis-firehose.html>`_
7. Define your `lambda settings <lambda.html>`_
8. Define/deploy your `secrets <secrets.html>`_
5. Configure your `kinesis streams <kinesis-streams.html>`_
6. Configure your `kinesis firehose <kinesis-firehose.html>`_
7. Configure your `lambda settings <lambda.html>`_
8. Write your `rules <rules.html>`_
9. Define/deploy your `secrets <secrets.html>`_

Now it's time to `deploy <deployment.html>`_!
Loading

0 comments on commit c60e8cf

Please sign in to comment.