Elastic Stack Release Candidate 2

[dougburks]

• Elasticsearch

  • preserve custom cluster.routing.allocation.disk settings

• Kibana

  • disable replicas on .kibana index
  • avoid multiple logstash-* Index Patterns

• Logstash

  • support Bro logs in JSON format - Security-Onion-Solutions/securityonion-elastic@75f9f62
  • avoid _grokparsefailure on winlogbeats logs (consider replacing brackets)

• Curator

  • add new option to /etc/nsm/securityonion.conf called CURATOR_CLOSE_DAYS and default to 30
  • modify so-elastic-configure-curator to update /etc/curator/action/close.yml with CURATOR_CLOSE_DAYS
  • remove --dry-run from /etc/cron.d/curator-close
  • update /etc/cron.d/curator-close to copy CURATOR_CLOSE_DAYS into config before running curator
  • update /etc/cron.d/curator-delete to copy LOG_SIZE_LIMIT into config before running curator
  • update https://github.com/Security-Onion-Solutions/security-onion/wiki/Curator

• ElastAlert

  • disable replicas on elastalert indices

• Docker

  • when suspending/resuming a VM with VMware Tools installed, Docker interfaces drop and Elastic components no longer able to communicate - workaround is to remove /etc/vmware-tools/scripts/vmware/network
  • include Docker --memory and --memory-swap options so that Docker enforces the limits we've already placed on Elastic - cancelled since this requires changes to grub boot loader and "Memory and swap accounting incur an overhead of about 1% of the total available memory and a 10% overall performance degradation, even if Docker is not running" - recommend disabling swap altogether in host

• so-autossh-*

  • so-autossh-start - log errors to a file
  • add so-autossh-status
  • add so-autossh-stop
  • add so-autossh-restart

• so-elastic-configure-apache

  • re-running gives errors when trying to ln -s, check to see if files exist first

• sosetup-elastic

  • avoid configuring and starting Kibana and ElastAlert when configuring sensor-only
  • avoid configuring and starting Elasticsearch and Logstash when running forward-only sensor
  • avoid configuring cross cluster search when running forward-only
  • remove disclaimers and warnings
  • add support for storage node consuming from redis on master server
  • improve user experience - start a new deployment or join an existing deployment

• so-user-*

  • so-user-add
  • so-user-disable
  • so-user-list
  • so-user-passwd

• ELSA

  • document manual steps necessary to upgrade distributed deployment
  • rename so-migrate-elsa-data-to-elastic to so-elsa-export and just export data without trying to import to Logstash

• update Wiki

  • document data fields (including fields that are renamed and fields whose value we modify)
  • add more Hardware requirements to Hardware page including SSD recommendation

[dougburks]

submitted for testing:
https://groups.google.com/d/topic/security-onion-testing/Fytp1gfeN8A/discussion
https://groups.google.com/d/topic/security-onion-testing/_NuzYTnN38c/discussion

published:
http://blog.securityonion.net/2018/02/security-onion-elastic-stack-release.html