List of changes in Malcolm v3.0.0:

- Change base for Elasticsearch and Kibana Docker images (version 7.6.2) from Elastic.co to Open Distro for Elastic (based on Elastic 7.10.0); see idaholab#15. This is a major change which **breaks backwards compatibility** for several features (listed below). If you are using these features, you will need to back up the data and/or configuration associated with them and migrate them manually to the new tools. No automatic migration or upgrade of these features is performed. It's recommended that you re-run `install.py --configure` (see [System configuration and tuning](https://github.com/cisagov/Malcolm#ConfigAndTuning)) prior to running Malcolm v3.0.0.
  - Kibana [comments](https://github.com/gwintzer/kibana-comments-app-plugin) replaced with [Notebooks](https://opendistro.github.io/for-elasticsearch-docs/docs/notebooks/)
  - Kibana [elastalert](https://github.com/nsano-rururu/elastalert-kibana-plugin) plugin replaced with [Alerting](https://opendistro.github.io/for-elasticsearch-docs/docs/alerting/) plugin
  - Elasticsearch [curator](https://www.elastic.co/guide/en/elasticsearch/client/curator/current/index.html) replaced with [Index Management](https://opendistro.github.io/for-elasticsearch-docs/docs/ism/) plugin
  - The third-party [Sankey visualization plugin](https://github.com/mmguero-dev/kbn_sankey_vis] has been temporarily removed due to compatibility issues, although it is planned to be reintegrated in a Malcolm point release in the near future (see uniberg/kbn_sankey_vis#15)
  - The third-party [Kibana drill-down plugin](https://github.com/mmguero-dev/kibana-plugin-drilldownmenu/) providing Kibana-to-Moloch pivoting has been temporarily removed due to compatibility issues, although it is planned to be reintegrated in a Malcolm point release in the near future (see goodlabs-studio/kibana-plugin-drilldownmenu#5)
  - In addition to those replacements, the Real Time Anomaly Detection feature is now available:
    - [Real Time Anomaly Detection in Open Distro for Elasticsearch](https://opendistro.github.io/for-elasticsearch/blog/odfe-updates/2019/11/real-time-anomaly-detection-in-open-distro-for-elasticsearch/) blog announcement
     - [Anomaly Detection](https://opendistro.github.io/for-elasticsearch-docs/docs/ad/) documentation and source code for [Elasticsearch](https://github.com/opendistro-for-elasticsearch/anomaly-detection) and [Kibana](https://github.com/opendistro-for-elasticsearch/anomaly-detection-kibana-plugin) components
     - [Random Cut Forests writeup](https://opendistro.github.io/for-elasticsearch/blog/odfe-updates/2019/11/random-cut-forests/)

- Malcolm startup time (especially the Logstash container) has been reduced drastically

- Improvements to Malcolm's prebuilt Kibana dashboards

- Improvements to build scripts

- Minor tweaks and bugfixes for ISO-installed environments for Malcolm and Hedgehog Linux

- Minor other bug fixes and performance improvements

- Version bump
  - Yara [v4.0.5](https://github.com/VirusTotal/yara/releases/tag/v4.0.5)

.gitignore

@@ -15,7 +15,6 @@
 /nginx/nginx_ldap.conf
 /htadmin/config.ini
 /htadmin/metadata
-/elastalert/config/smtp-auth.yaml
 
 # development
 .vagrant

Dockerfiles/arkime.Dockerfile

@@ -214,10 +214,8 @@ RUN groupadd --gid $DEFAULT_GID $PGROUP && \
     ln -sfr /data/pcap_moloch_and_zeek_processor.py /data/pcap_moloch_processor.py && \
     cp -f /data/moloch_update_geo.sh $ARKIMEDIR/bin/moloch_update_geo.sh && \
     chmod u+s $ARKIMEDIR/bin/moloch-capture && \
-    chown -R $PUSER:$PGROUP $ARKIMEDIR/logs && \
     mkdir -p /var/run/moloch && \
-    chown -R $PUSER:$PGROUP /var/run/moloch
-
+    chown -R $PUSER:$PGROUP $ARKIMEDIR/etc $ARKIMEDIR/logs /var/run/moloch
 #Update Path
 ENV PATH="/data:$ARKIMEDIR/bin:${PATH}"
 

Dockerfiles/elasticsearch.Dockerfile

@@ -1,4 +1,4 @@
-FROM docker.elastic.co/elasticsearch/elasticsearch-oss:7.6.2
+FROM amazon/opendistro-for-elasticsearch:1.12.0
 
 # Copyright (c) 2021 Battelle Energy Alliance, LLC.  All rights reserved.
 LABEL maintainer="malcolm.netsec@gmail.com"
@@ -7,8 +7,8 @@ LABEL org.opencontainers.image.url='https://github.com/cisagov/Malcolm'
 LABEL org.opencontainers.image.documentation='https://github.com/cisagov/Malcolm/blob/master/README.md'
 LABEL org.opencontainers.image.source='https://github.com/cisagov/Malcolm'
 LABEL org.opencontainers.image.vendor='Cybersecurity and Infrastructure Security Agency'
-LABEL org.opencontainers.image.title='malcolmnetsec/elasticsearch-oss'
-LABEL org.opencontainers.image.description='Malcolm container providing Elasticsearch (the Apache-licensed variant)'
+LABEL org.opencontainers.image.title='malcolmnetsec/elasticsearch-od'
+LABEL org.opencontainers.image.description='Malcolm container providing Elasticsearch (the Apache-licensed Open Distro variant)'
 
 ARG DEFAULT_UID=1000
 ARG DEFAULT_GID=1000
@@ -20,11 +20,23 @@ ENV PUSER_PRIV_DROP true
 
 ENV TERM xterm
 
+ARG GITHUB_OAUTH_TOKEN=""
+ARG DISABLE_INSTALL_DEMO_CONFIG=true
+ENV DISABLE_INSTALL_DEMO_CONFIG $DISABLE_INSTALL_DEMO_CONFIG
+
+# Malcolm manages authentication and encryption via NGINX reverse proxy
+# https://opendistro.github.io/for-elasticsearch-docs/docs/security/configuration/disable/
+# https://opendistro.github.io/for-elasticsearch-docs/docs/install/docker/#customize-the-docker-image
+# https://github.com/opendistro-for-elasticsearch/opendistro-build/issues/613
+RUN /usr/share/elasticsearch/bin/elasticsearch-plugin remove opendistro_security && \
+    /usr/share/elasticsearch/bin/elasticsearch-plugin remove opendistro_performance_analyzer && \
+  echo -e 'cluster.name: "docker-cluster"\nnetwork.host: 0.0.0.0' > /usr/share/elasticsearch/config/elasticsearch.yml && \
+  chown -R $PUSER:$PGROUP /usr/share/elasticsearch/config/elasticsearch.yml && \
+  sed -i "s/\b1000\b/\${PUID:-${DEFAULT_UID}}/g" /usr/local/bin/docker-entrypoint.sh
+
+# just used for initial keystore creation
 ADD shared/bin/docker-uid-gid-setup.sh /usr/local/bin/
 
-ENTRYPOINT ["/usr/local/bin/docker-uid-gid-setup.sh", "/usr/local/bin/docker-entrypoint.sh"]
-
-
 # to be populated at build-time:
 ARG BUILD_DATE
 ARG MALCOLM_VERSION

Dockerfiles/file-monitor.Dockerfile

@@ -75,7 +75,7 @@ ENV EXTRACTED_FILE_ENABLE_CAPA $EXTRACTED_FILE_ENABLE_CAPA
 ENV EXTRACTED_FILE_CAPA_VERBOSE $EXTRACTED_FILE_CAPA_VERBOSE
 ENV SRC_BASE_DIR "/usr/local/src"
 ENV CLAMAV_RULES_DIR "/var/lib/clamav"
-ENV YARA_VERSION "4.0.4"
+ENV YARA_VERSION "4.0.5"
 ENV YARA_URL "https://github.com/VirusTotal/yara/archive/v${YARA_VERSION}.tar.gz"
 ENV YARA_RULES_URL "https://github.com/Neo23x0/signature-base"
 ENV YARA_RULES_DIR "/yara-rules"
@@ -228,6 +228,7 @@ VOLUME ["$YARA_RULES_DIR"]
 VOLUME ["$YARA_RULES_SRC_DIR"]
 
 EXPOSE 3310
+EXPOSE $EXTRACTED_FILE_HTTP_SERVER_PORT
 
 ENTRYPOINT ["/usr/local/bin/docker-uid-gid-setup.sh", "/docker-entrypoint.sh"]
 

Dockerfiles/filebeat.Dockerfile

@@ -1,4 +1,4 @@
-FROM docker.elastic.co/beats/filebeat-oss:7.6.2
+FROM docker.elastic.co/beats/filebeat-oss:7.10.0
 
 # Copyright (c) 2021 Battelle Energy Alliance, LLC.  All rights reserved.
 LABEL maintainer="malcolm.netsec@gmail.com"

Dockerfiles/kibana-helper.Dockerfile

@@ -0,0 +1,92 @@
+FROM alpine:3.12
+
+# Copyright (c) 2020 Battelle Energy Alliance, LLC.  All rights reserved.
+LABEL maintainer="malcolm.netsec@gmail.com"
+LABEL org.opencontainers.image.authors='malcolm.netsec@gmail.com'
+LABEL org.opencontainers.image.url='https://github.com/cisagov/Malcolm'
+LABEL org.opencontainers.image.documentation='https://github.com/cisagov/Malcolm/blob/master/README.md'
+LABEL org.opencontainers.image.source='https://github.com/cisagov/Malcolm'
+LABEL org.opencontainers.image.vendor='Cybersecurity and Infrastructure Security Agency'
+LABEL org.opencontainers.image.title='malcolmnetsec/kibana-helper'
+LABEL org.opencontainers.image.description='Malcolm container providing Kibana support functions'
+
+ARG DEFAULT_UID=1000
+ARG DEFAULT_GID=1000
+ENV DEFAULT_UID $DEFAULT_UID
+ENV DEFAULT_GID $DEFAULT_GID
+ENV PUSER "helper"
+ENV PGROUP "helper"
+ENV PUSER_PRIV_DROP true
+
+ENV TERM xterm
+
+ARG ARKIME_INDEX_PATTERN="sessions2-*"
+ARG ARKIME_INDEX_PATTERN_ID="sessions2-*"
+ARG ARKIME_INDEX_TIME_FIELD="firstPacket"
+ARG CREATE_ES_ARKIME_SESSION_INDEX="true"
+ARG ELASTICSEARCH_URL="http://elasticsearch:9200"
+ARG ISM_SNAPSHOT_COMPRESSED=false
+ARG ISM_SNAPSHOT_REPO=logs
+ARG KIBANA_OFFLINE_REGION_MAPS_PORT="28991"
+ARG KIBANA_URL="http://kibana:5601/kibana"
+
+ENV ARKIME_INDEX_PATTERN $ARKIME_INDEX_PATTERN
+ENV ARKIME_INDEX_PATTERN_ID $ARKIME_INDEX_PATTERN_ID
+ENV ARKIME_INDEX_TIME_FIELD $ARKIME_INDEX_TIME_FIELD
+ENV CREATE_ES_ARKIME_SESSION_INDEX $CREATE_ES_ARKIME_SESSION_INDEX
+ENV ELASTICSEARCH_URL $ELASTICSEARCH_URL
+ENV ISM_SNAPSHOT_COMPRESSED $ISM_SNAPSHOT_COMPRESSED
+ENV ISM_SNAPSHOT_REPO $ISM_SNAPSHOT_REPO
+ENV KIBANA_OFFLINE_REGION_MAPS_PORT $KIBANA_OFFLINE_REGION_MAPS_PORT
+ENV KIBANA_URL $KIBANA_URL
+ENV PATH="/data:${PATH}"
+
+ENV SUPERCRONIC_VERSION "0.1.12"
+ENV SUPERCRONIC_URL "https://github.com/aptible/supercronic/releases/download/v$SUPERCRONIC_VERSION/supercronic-linux-amd64"
+ENV SUPERCRONIC "supercronic-linux-amd64"
+ENV SUPERCRONIC_SHA1SUM "048b95b48b708983effb2e5c935a1ef8483d9e3e"
+ENV SUPERCRONIC_CRONTAB "/etc/crontab"
+
+ADD kibana/dashboards /opt/kibana/dashboards
+ADD kibana/maps /opt/maps
+ADD kibana/scripts /data/
+ADD kibana/supervisord.conf /etc/supervisord.conf
+ADD kibana/zeek_template.json /data/zeek_template.json
+ADD shared/bin/docker-uid-gid-setup.sh /usr/local/bin/
+ADD shared/bin/elastic_search_status.sh /data/
+ADD shared/bin/elastic_index_size_prune.py /data/
+
+RUN apk --no-cache add bash python3 py3-pip curl procps psmisc npm shadow jq && \
+    npm install -g http-server && \
+    pip3 install supervisor humanfriendly && \
+    curl -fsSLO "$SUPERCRONIC_URL" && \
+      echo "${SUPERCRONIC_SHA1SUM}  ${SUPERCRONIC}" | sha1sum -c - && \
+      chmod +x "$SUPERCRONIC" && \
+      mv "$SUPERCRONIC" "/usr/local/bin/${SUPERCRONIC}" && \
+      ln -s "/usr/local/bin/${SUPERCRONIC}" /usr/local/bin/supercronic && \
+    addgroup -g ${DEFAULT_GID} ${PGROUP} ; \
+      adduser -D -H -u ${DEFAULT_UID} -h /nonexistant -s /sbin/nologin -G ${PGROUP} -g ${PUSER} ${PUSER} ; \
+      addgroup ${PUSER} tty ; \
+      addgroup ${PUSER} shadow ; \
+    mkdir -p /data/init && \
+    chown -R ${PUSER}:${PGROUP} /opt/kibana/dashboards /opt/maps /data/init && \
+    chmod 755 /data/*.sh /data/*.py /data/init && \
+    chmod 400 /opt/maps/* && \
+    (echo -e "*/2 * * * * /data/kibana-create-moloch-sessions-index.sh\n0 10 * * * /data/kibana_index_refresh.py --template zeek_template\n*/20 * * * * /data/elastic_index_size_prune.py" > ${SUPERCRONIC_CRONTAB})
+
+EXPOSE $KIBANA_OFFLINE_REGION_MAPS_PORT
+
+ENTRYPOINT ["/usr/local/bin/docker-uid-gid-setup.sh"]
+
+CMD ["/usr/bin/supervisord", "-c", "/etc/supervisord.conf", "-n"]
+
+VOLUME ["/data/init"]
+
+# to be populated at build-time:
+ARG BUILD_DATE
+ARG MALCOLM_VERSION
+ARG VCS_REVISION
+
+LABEL org.opencontainers.image.created=$BUILD_DATE
+LABEL org.opencontainers.image.version=$MALCOLM_VERSION
+LABEL org.opencontainers.image.revision=$VCS_REVISION