Elastic Architecture

Introduction

Below are several diagrams to represent the current architecture and deployment scenarios for Security Onion on the Elastic Stack.

High-Level Architecture Diagram

Core Components

Logstash - Parse and format logs.
Elasticsearch - Ingest and index logs.
Kibana - Visualize ingested log data.

Auxilliary Components

Curator - Manage indices through scheduled maintenance.
ElastAlert - Query Elasticsearch and alert on user-defined anomalous behavior or other interesting bits of information.
FreqServer -Detect DGAs and find random file names, script names, process names, service names, workstation names, TLS certificate subjects and issuer subjects, etc.
DomainStats - Get additional info about a domain by providing additional context, such as creation time, age, reputation, etc.

Detailed Data Flow Diagram

(click for larger version)

Deployment Types

Distributed

• Recommended deployment type
• Consists of a master server, one or more forward nodes, and one or more storage nodes.

Heavy Distributed

• Recommended only if a standard distributed deployment is not possible.
• Consists of a master server, and one or more heavy nodes.

Standalone

• Not recommended for monitoring high-throughput links
• Consists of a single server running master server components, sensor, and Elastic stack components.

Node Types

Master

The master server runs it's own local copy of Elasticsearch, which manages cross-cluster search configuration for the deployment. This includes configuration for heavy nodes and storage nodes (where applicable), but not forward nodes, as they do not run Elastic Stack components.

The Master Server runs the following components (Production Mode w/ Best Practices):

• Elasticsearch
• Logstash
• Kibana
• Curator
• Elastalert
• Redis (Only if configured to output to a storage node)
• OSSEC
• Sguild

Forward Node

When using a forward node, Elastic Stack components are not installed. Syslog-NG forwards all logs to Logstash on the master server via an autossh tunnel, where they are stored in Elasticsearch on the master server, or forwarded to storage node's Elasticsearch instance (if the master server has been configured to use a storage node). From there, the data can be queried through the use of cross-cluster search.

Forward Nodes run the following components (Production Mode w/ Best Practices):

• Bro
• Snort/Suricata
• Netsniff-NG
• OSSEC
• Syslog-NG

Heavy Node

When using a heavy node, Security Onion implements distributed deployments using Elasticsearch's cross cluster search. When you run Setup and choose Heavy Node, it will create a local Elasticsearch instance and then configure the master server to query that instance (similar to ELSA distributed deployments). This is done by constructing an autossh tunnel from the heavy node to the master server, configuring reverse port forwarding to allow the master server to connect to the local Elasticsearch instance, and updating _cluster/settings on the master server so that it will query the local Elasticsearch instance.

• Elasticsearch
• Logstash
• Curator
• Bro
• Snort/Suricata
• Netsniff-NG
• OSSEC
• Syslog-NG (forwards logs locally to Logstash)

Storage Node

Storage nodes extend the storage and processing capabilities of the master server. Just like heavy nodes, storage nodes are added to the master's cluster search configuration, so the data that resides on the nodes can be queried from the master.

Storage Nodes run the following components (Production Mode w/ Best Practices):

• Elasticsearch
• Logstash
• Curator
• OSSEC