Merge pull request #308 from cisagov/v24.04.0_merge_cisagov

Malcolm v24.04.0 * Features and enhancements - Zeek-extracted files scanned and preserved on a [Hedgehog Linux](https://cisagov.github.io/Malcolm/docs/malcolm-hedgehog-e2e-iso-install.html#HedgehogZeekFileExtraction) sensor can now be accessed via [the extracted files download user interface](https://cisagov.github.io/Malcolm/docs/file-scanning.html#ZeekFileExtractionUI) (idaholab#331). - Improvements to creation of index templates, dashboards, and other saved objects on startup (idaholab#208) to ensure that saved objects get created correctly upon upgrade (see [this comment](idaholab#208 (comment)) for more details on this feature). - [Populating the NetBox inventory via passively-gathered network traffic metadata](https://cisagov.github.io/Malcolm/docs/asset-interaction-analysis.html#NetBoxPopPassive) now uses network traffic logs for DNS, NTLM, and DHCP to identify assets' host names when possible for use when populating device and VM names (idaholab#415). Autopopulated devices now have their *status* field set to `Active` rather than `Stage`, and uses *tags* instead to indicated that they were created through autopopulation. - Users can now specify pruning thresholds for [carved files](https://cisagov.github.io/Malcolm/docs/file-scanning.html#ZeekFileExtraction) so that old files are deleted in order to avoid filling available storage (idaholab#453). See a new section of documentation on [Managing disk usage](https://cisagov.github.io/Malcolm/docs/malcolm-config.html#DiskUsage) for more information about this and similar settings. - Users can now specify a prefix that will be prepended to dashboards as they are imported into OpenSearch Dashboards or Kibana, allowing users who have dashboards from other sources to differentiate between those and Malcolm's (idaholab#455). - The default anomaly detectors created for the OpenSearch Anomaly Detection plugin are now created with [category fields for high cardinality](https://opensearch.org/docs/latest/observing-your-data/ad/index/#optional-set-category-fields-for-high-cardinality) to allow for better breakdown of contributing values to anomalies discovered (idaholab#464). - Include [JA4+ plugin in Arkime](https://arkime.com/settings#ja4plus). See idaholab#419 for status on upcoming full JA4+ support in Malcolm. - Hedgehog Linux sensors can now [periodically refresh](https://github.com/idaholab/Malcolm/blob/bceee4616dd5676a010a3dd7b0410856257948e8/hedgehog-iso/interface/sensor_ctl/control_vars.conf#L75) their [Zeek inteligence files](https://idaholab.github.io/Malcolm/docs/hedgehog-config-zeek-intel.html#HedgehogZeekIntel). **NOTE**: due to an oversight, a necessary variable is missing in this release that is required for this to work. Appending the line `export INTEL_DIR=/opt/sensor/sensor_ctl/zeek/intel` to `/opt/sensor/sensor_ctl/control_vars.conf` will correct this. This will be corrected in the next Malcolm release. - Assorted documentation improvements. * Component version updates - Arkime to [v5.1.2](https://github.com/arkime/arkime/blob/bcd9d7e68be8e4a52a17c35211c5d5a7fdcc1a1c/CHANGELOG#L36-L41) - OpenSearch and OpenSearch Dashboards to [v2.13.0](https://github.com/opensearch-project/opensearch-build/blob/main/release-notes/opensearch-release-notes-2.13.0.md) - Beats to [v8.13.2](https://www.elastic.co/guide/en/beats/libbeat/current/release-notes-8.13.2.html) - Logstash to [v8.13.2](https://www.elastic.co/guide/en/logstash/current/logstash-8-13-2.html) - gunicorn to v22.0.0 to address [CVE-2024-1135](GHSA-w3h3-4rj7-4ph4). - elasticsearch-dsl to [v8.13.0](https://github.com/elastic/elasticsearch-dsl-py/releases/tag/v8.13.0) - elasticsearch-py to [v8.13.0](https://github.com/elastic/elasticsearch-py/releases/tag/v8.13.0) - idna to v3.7 to address [CVE-2024-3651](GHSA-jjg7-2v4v-x38h) - Fluent Bit to [v3.0.3](https://fluentbit.io/announcements/v3.0.3/) * Bug fixes - The documentation for [Windows host system configuration](https://cisagov.github.io/Malcolm/docs/host-config-windows.html#HostSystemConfigWindows) was out of date and has been updated for the latest version of Microsoft Windows Subsystem for Linux (idaholab#421). - An issue was fixed in which Malcolm's list of users and their password hashes could become corrupted if the file did not initially end with a newline character (idaholab#426). - The manner in which Zeek intel files are generated has been changed to avoid problems found in Kubernetes deployments when scaling out the number of `zeek-live` containers (idaholab#456). See [this comment](idaholab#456 (comment)) for more details. - Removed the version top-level element from `docker-compose.yml` files as it is [now obsolete](https://docs.docker.com/compose/compose-file/04-version-and-name/) and caused a warning message that sometimes was not handled correctly. - Fix Malcolm ISO not correctly detecting if it's in a live boot ISO environment or installed mode. - Restart live Zeek instances with `zeekctl deploy` instead of `zeekctl restart`. * Configuration changes (in [environment variables](https://cisagov.github.io/Malcolm/docs/malcolm-config.html#MalcolmConfigEnvVars) in [`./config/`](https://github.com/idaholab/Malcolm/blob/v24.04.0/config)) - `ARKIME_QUERY_ALL_INDICES` in [`arkime.env`](https://github.com/idaholab/Malcolm/blob/bceee4616dd5676a010a3dd7b0410856257948e8/config/arkime.env.example#L9-L11) can be set to control the [`queryAllIndices` setting](https://arkime.com/settings#queryAllIndices) in Arkime's `config.ini`. - `DASHBOARDS_PREFIX` in [`dashboards-helper.env`](https://github.com/idaholab/Malcolm/blob/bceee4616dd5676a010a3dd7b0410856257948e8/config/dashboards-helper.env.example#L3C1-L4C19) has been added for idaholab#455 (see above in **Features and Enhancements**). - `LOGSTASH_NETBOX_ENRICHMENT_DATASETS` in [`logstash.env`](https://github.com/idaholab/Malcolm/blob/bceee4616dd5676a010a3dd7b0410856257948e8/config/logstash.env.example#L13) has been changed to include `zeek.dhcp`, `zeek.dns`, and `zeek.ntlm` to support idaholab#415 (see above in **Features and Enhancements**). - `LOGSTASH_ZEEK_IGNORED_LOGS` in [`logstash.env`](https://github.com/idaholab/Malcolm/blob/bceee4616dd5676a010a3dd7b0410856257948e8/config/logstash.env.example#L15) has been changed to remove `capture_loss` and `stats` so that those diagnostic Zeek logs can be parsed without the user having to manually change this variable. - `ZEEK_CRON` has been removed from [`zeek-live.env`](https://github.com/idaholab/Malcolm/blob/bceee4616dd5676a010a3dd7b0410856257948e8/config/zeek-live.env.example) and `ZEEK_INTEL_REFRESH_CRON_EXPRESSION` was removed from [`zeek.env`](https://github.com/idaholab/Malcolm/blob/bceee4616dd5676a010a3dd7b0410856257948e8/config/zeek.env.example) and moved to the "offline" version of the container in [`zeek-offline.env`](https://github.com/idaholab/Malcolm/blob/bceee4616dd5676a010a3dd7b0410856257948e8/config/zeek-offline.env.example#L17-L19) for idaholab#456. - `EXTRACTED_FILE_PRUNE_THRESHOLD_MAX_SIZE`, `EXTRACTED_FILE_PRUNE_THRESHOLD_TOTAL_DISK_USAGE_PERCENT`, and `EXTRACTED_FILE_PRUNE_INTERVAL_SECONDS` were added to [`zeek.env`](https://github.com/idaholab/Malcolm/blob/bceee4616dd5676a010a3dd7b0410856257948e8/config/zeek.env.example#L32-L37) for idaholab#453. See a new section of documentation on [Managing disk usage](https://cisagov.github.io/Malcolm/docs/malcolm-config.html#DiskUsage) for more information about these and similar settings.
cisagov · Apr 30, 2024 · 8467930 · 8467930
2 parents 618fc30 + b606169
commit 8467930
Show file tree

Hide file tree

Showing 230 changed files with 2,858 additions and 1,601 deletions.
diff --git a/.github/workflows/api-build-and-push-ghcr.yml b/.github/workflows/api-build-and-push-ghcr.yml
@@ -13,6 +13,8 @@ on:
       - '!shared/bin/common-init.sh'
       - '!shared/bin/sensor-init.sh'
       - '!shared/bin/os-disk-config.py'
+      - '!shared/bin/extracted_files_http_server.py'
+      - '!shared/bin/web-ui-asset-download.sh'
       - '!shared/bin/preseed_late_user_config.sh'
       - '!shared/bin/configure-interfaces.py'
       - '!shared/bin/configure-capture.py'

diff --git a/.github/workflows/arkime-build-and-push-ghcr.yml b/.github/workflows/arkime-build-and-push-ghcr.yml
@@ -12,6 +12,8 @@ on:
       - '!shared/bin/agg-init.sh'
       - '!shared/bin/common-init.sh'
       - '!shared/bin/sensor-init.sh'
+      - '!shared/bin/extracted_files_http_server.py'
+      - '!shared/bin/web-ui-asset-download.sh'
       - '!shared/bin/os-disk-config.py'
       - '!shared/bin/preseed_late_user_config.sh'
       - '!shared/bin/configure-interfaces.py'

diff --git a/.github/workflows/dashboards-build-and-push-ghcr.yml b/.github/workflows/dashboards-build-and-push-ghcr.yml
@@ -12,6 +12,8 @@ on:
       - '!shared/bin/agg-init.sh'
       - '!shared/bin/common-init.sh'
       - '!shared/bin/sensor-init.sh'
+      - '!shared/bin/extracted_files_http_server.py'
+      - '!shared/bin/web-ui-asset-download.sh'
       - '!shared/bin/os-disk-config.py'
       - '!shared/bin/preseed_late_user_config.sh'
       - '!shared/bin/configure-interfaces.py'

diff --git a/.github/workflows/dashboards-helper-build-and-push-ghcr.yml b/.github/workflows/dashboards-helper-build-and-push-ghcr.yml
@@ -12,6 +12,8 @@ on:
       - '!shared/bin/agg-init.sh'
       - '!shared/bin/common-init.sh'
       - '!shared/bin/sensor-init.sh'
+      - '!shared/bin/extracted_files_http_server.py'
+      - '!shared/bin/web-ui-asset-download.sh'
       - '!shared/bin/os-disk-config.py'
       - '!shared/bin/preseed_late_user_config.sh'
       - '!shared/bin/configure-interfaces.py'

diff --git a/.github/workflows/file-upload-build-and-push-ghcr.yml b/.github/workflows/file-upload-build-and-push-ghcr.yml
@@ -12,6 +12,8 @@ on:
       - '!shared/bin/agg-init.sh'
       - '!shared/bin/common-init.sh'
       - '!shared/bin/sensor-init.sh'
+      - '!shared/bin/extracted_files_http_server.py'
+      - '!shared/bin/web-ui-asset-download.sh'
       - '!shared/bin/os-disk-config.py'
       - '!shared/bin/preseed_late_user_config.sh'
       - '!shared/bin/configure-interfaces.py'

diff --git a/.github/workflows/filebeat-build-and-push-ghcr.yml b/.github/workflows/filebeat-build-and-push-ghcr.yml
@@ -12,6 +12,8 @@ on:
       - '!shared/bin/agg-init.sh'
       - '!shared/bin/common-init.sh'
       - '!shared/bin/sensor-init.sh'
+      - '!shared/bin/extracted_files_http_server.py'
+      - '!shared/bin/web-ui-asset-download.sh'
       - '!shared/bin/os-disk-config.py'
       - '!shared/bin/preseed_late_user_config.sh'
       - '!shared/bin/configure-interfaces.py'

diff --git a/.github/workflows/freq-build-and-push-ghcr.yml b/.github/workflows/freq-build-and-push-ghcr.yml
@@ -12,6 +12,8 @@ on:
       - '!shared/bin/agg-init.sh'
       - '!shared/bin/common-init.sh'
       - '!shared/bin/sensor-init.sh'
+      - '!shared/bin/extracted_files_http_server.py'
+      - '!shared/bin/web-ui-asset-download.sh'
       - '!shared/bin/os-disk-config.py'
       - '!shared/bin/preseed_late_user_config.sh'
       - '!shared/bin/configure-interfaces.py'

diff --git a/.github/workflows/hedgehog-iso-build-docker-wrap-push-ghcr.yml b/.github/workflows/hedgehog-iso-build-docker-wrap-push-ghcr.yml
@@ -99,6 +99,9 @@ jobs:
           cp -r ./arkime/patch ./hedgehog-iso/shared/arkime_patch
           mkdir -p ./hedgehog-iso/suricata
           cp -r ./suricata/rules-default ./hedgehog-iso/suricata/
+          mkdir -p ./hedgehog-iso/nginx
+          cp -r ./nginx/landingpage/css ./hedgehog-iso/nginx/
+          cp -r ./nginx/landingpage/js ./hedgehog-iso/nginx/
           pushd ./hedgehog-iso
           echo "${{ steps.extract_malcolm_version.outputs.mversion }}" > ./shared/version.txt
           echo "${{ secrets.MAXMIND_GEOIP_DB_LICENSE_KEY }}" > ./shared/maxmind_license.txt

diff --git a/.github/workflows/htadmin-build-and-push-ghcr.yml b/.github/workflows/htadmin-build-and-push-ghcr.yml
@@ -12,6 +12,8 @@ on:
       - '!shared/bin/agg-init.sh'
       - '!shared/bin/common-init.sh'
       - '!shared/bin/sensor-init.sh'
+      - '!shared/bin/extracted_files_http_server.py'
+      - '!shared/bin/web-ui-asset-download.sh'
       - '!shared/bin/os-disk-config.py'
       - '!shared/bin/preseed_late_user_config.sh'
       - '!shared/bin/configure-interfaces.py'

diff --git a/.github/workflows/logstash-build-and-push-ghcr.yml b/.github/workflows/logstash-build-and-push-ghcr.yml
@@ -12,6 +12,8 @@ on:
       - '!shared/bin/agg-init.sh'
       - '!shared/bin/common-init.sh'
       - '!shared/bin/sensor-init.sh'
+      - '!shared/bin/extracted_files_http_server.py'
+      - '!shared/bin/web-ui-asset-download.sh'
       - '!shared/bin/os-disk-config.py'
       - '!shared/bin/preseed_late_user_config.sh'
       - '!shared/bin/configure-interfaces.py'

diff --git a/.github/workflows/malcolm-iso-build-docker-wrap-push-ghcr.yml b/.github/workflows/malcolm-iso-build-docker-wrap-push-ghcr.yml
@@ -9,6 +9,8 @@ on:
       - 'malcolm-iso/**'
       - 'shared/bin/*'
       - '!shared/bin/configure-capture.py'
+      - '!shared/bin/extracted_files_http_server.py'
+      - '!shared/bin/web-ui-asset-download.sh'
       - '!shared/bin/zeek*'
       - '!shared/bin/suricata*'
       - '.trigger_iso_workflow_build'

diff --git a/.github/workflows/netbox-build-and-push-ghcr.yml b/.github/workflows/netbox-build-and-push-ghcr.yml
@@ -12,6 +12,8 @@ on:
       - '!shared/bin/agg-init.sh'
       - '!shared/bin/common-init.sh'
       - '!shared/bin/sensor-init.sh'
+      - '!shared/bin/extracted_files_http_server.py'
+      - '!shared/bin/web-ui-asset-download.sh'
       - '!shared/bin/os-disk-config.py'
       - '!shared/bin/preseed_late_user_config.sh'
       - '!shared/bin/configure-interfaces.py'

diff --git a/.github/workflows/nginx-build-and-push-ghcr.yml b/.github/workflows/nginx-build-and-push-ghcr.yml
@@ -12,6 +12,8 @@ on:
       - '!shared/bin/agg-init.sh'
       - '!shared/bin/common-init.sh'
       - '!shared/bin/sensor-init.sh'
+      - '!shared/bin/extracted_files_http_server.py'
+      - '!shared/bin/web-ui-asset-download.sh'
       - '!shared/bin/os-disk-config.py'
       - '!shared/bin/preseed_late_user_config.sh'
       - '!shared/bin/configure-interfaces.py'

diff --git a/.github/workflows/opensearch-build-and-push-ghcr.yml b/.github/workflows/opensearch-build-and-push-ghcr.yml
@@ -12,6 +12,8 @@ on:
       - '!shared/bin/common-init.sh'
       - '!shared/bin/sensor-init.sh'
       - '!shared/bin/os-disk-config.py'
+      - '!shared/bin/extracted_files_http_server.py'
+      - '!shared/bin/web-ui-asset-download.sh'
       - '!shared/bin/preseed_late_user_config.sh'
       - '!shared/bin/configure-interfaces.py'
       - '!shared/bin/configure-capture.py'

diff --git a/.github/workflows/pcap-capture-build-and-push-ghcr.yml b/.github/workflows/pcap-capture-build-and-push-ghcr.yml
@@ -13,6 +13,8 @@ on:
       - '!shared/bin/common-init.sh'
       - '!shared/bin/sensor-init.sh'
       - '!shared/bin/os-disk-config.py'
+      - '!shared/bin/extracted_files_http_server.py'
+      - '!shared/bin/web-ui-asset-download.sh'
       - '!shared/bin/preseed_late_user_config.sh'
       - '!shared/bin/configure-interfaces.py'
       - '!shared/bin/configure-capture.py'

diff --git a/.github/workflows/pcap-monitor-build-and-push-ghcr.yml b/.github/workflows/pcap-monitor-build-and-push-ghcr.yml
@@ -13,6 +13,8 @@ on:
       - '!shared/bin/common-init.sh'
       - '!shared/bin/sensor-init.sh'
       - '!shared/bin/os-disk-config.py'
+      - '!shared/bin/extracted_files_http_server.py'
+      - '!shared/bin/web-ui-asset-download.sh'
       - '!shared/bin/preseed_late_user_config.sh'
       - '!shared/bin/configure-interfaces.py'
       - '!shared/bin/configure-capture.py'

diff --git a/.github/workflows/postgresql-build-and-push-ghcr.yml b/.github/workflows/postgresql-build-and-push-ghcr.yml
@@ -12,6 +12,8 @@ on:
       - '!shared/bin/common-init.sh'
       - '!shared/bin/sensor-init.sh'
       - '!shared/bin/os-disk-config.py'
+      - '!shared/bin/extracted_files_http_server.py'
+      - '!shared/bin/web-ui-asset-download.sh'
       - '!shared/bin/preseed_late_user_config.sh'
       - '!shared/bin/configure-interfaces.py'
       - '!shared/bin/configure-capture.py'

diff --git a/.github/workflows/redis-build-and-push-ghcr.yml b/.github/workflows/redis-build-and-push-ghcr.yml
@@ -12,6 +12,8 @@ on:
       - '!shared/bin/common-init.sh'
       - '!shared/bin/sensor-init.sh'
       - '!shared/bin/os-disk-config.py'
+      - '!shared/bin/extracted_files_http_server.py'
+      - '!shared/bin/web-ui-asset-download.sh'
       - '!shared/bin/preseed_late_user_config.sh'
       - '!shared/bin/configure-interfaces.py'
       - '!shared/bin/configure-capture.py'

diff --git a/.github/workflows/suricata-build-and-push-ghcr.yml b/.github/workflows/suricata-build-and-push-ghcr.yml
@@ -13,6 +13,8 @@ on:
       - '!shared/bin/common-init.sh'
       - '!shared/bin/sensor-init.sh'
       - '!shared/bin/os-disk-config.py'
+      - '!shared/bin/extracted_files_http_server.py'
+      - '!shared/bin/web-ui-asset-download.sh'
       - '!shared/bin/preseed_late_user_config.sh'
       - '!shared/bin/configure-interfaces.py'
       - '!shared/bin/configure-capture.py'

diff --git a/.github/workflows/zeek-build-and-push-ghcr.yml b/.github/workflows/zeek-build-and-push-ghcr.yml
@@ -13,6 +13,8 @@ on:
       - '!shared/bin/common-init.sh'
       - '!shared/bin/sensor-init.sh'
       - '!shared/bin/os-disk-config.py'
+      - '!shared/bin/extracted_files_http_server.py'
+      - '!shared/bin/web-ui-asset-download.sh'
       - '!shared/bin/preseed_late_user_config.sh'
       - '!shared/bin/configure-interfaces.py'
       - '!shared/bin/configure-capture.py'

diff --git a/Dockerfiles/arkime.Dockerfile b/Dockerfiles/arkime.Dockerfile
@@ -7,7 +7,7 @@ ENV TERM xterm
 ENV PYTHONDONTWRITEBYTECODE 1
 ENV PYTHONUNBUFFERED 1
 
-ENV ARKIME_VERSION "v5.0.1"
+ENV ARKIME_VERSION "v5.1.2"
 ENV ARKIME_DIR "/opt/arkime"
 ENV ARKIME_URL "https://github.com/arkime/arkime.git"
 ENV ARKIME_LOCALELASTICSEARCH no
@@ -16,7 +16,8 @@ ENV ARKIME_INET yes
 ADD arkime/scripts/bs4_remove_div.py /opt/
 ADD arkime/patch/* /opt/patches/
 
-RUN apt-get -q update && \
+RUN export DEBARCH=$(dpkg --print-architecture) && \
+    apt-get -q update && \
     apt-get -y -q --no-install-recommends upgrade && \
     apt-get install -q -y --no-install-recommends \
         binutils \
@@ -73,7 +74,10 @@ RUN apt-get -q update && \
     make install && \
     npm cache clean --force && \
     rm -f ${ARKIME_DIR}/wiseService/source.* ${ARKIME_DIR}/etc/*.systemd.service && \
-    bash -c "file ${ARKIME_DIR}/bin/* ${ARKIME_DIR}/node-v*/bin/* | grep 'ELF 64-bit' | sed 's/:.*//' | xargs -l -r strip -v --strip-unneeded"
+    bash -c "file ${ARKIME_DIR}/bin/* ${ARKIME_DIR}/node-v*/bin/* | grep 'ELF 64-bit' | sed 's/:.*//' | xargs -l -r strip -v --strip-unneeded" && \
+    mkdir -p "${ARKIME_DIR}"/plugins && \
+    curl -fsSL -o "${ARKIME_DIR}/plugins/ja4plus.${DEBARCH}.so" "https://github.com/arkime/arkime/releases/download/${ARKIME_VERSION}/ja4plus.${DEBARCH}.so" && \
+    chmod 755 "${ARKIME_DIR}/plugins/ja4plus.${DEBARCH}.so"
 
 FROM debian:12-slim
 

diff --git a/Dockerfiles/dashboards-helper.Dockerfile b/Dockerfiles/dashboards-helper.Dockerfile
@@ -61,7 +61,7 @@ ADD scripts/malcolm_utils.py /data/
 
 RUN apk update --no-cache && \
     apk upgrade --no-cache && \
-    apk --no-cache add bash python3 py3-pip curl openssl procps psmisc npm rsync shadow jq tini && \
+    apk --no-cache add bash python3 py3-pip curl openssl procps psmisc moreutils npm rsync shadow jq tini && \
     npm install -g http-server && \
     pip3 install --break-system-packages supervisor humanfriendly requests && \
     curl -fsSLO "$SUPERCRONIC_URL" && \
@@ -95,7 +95,7 @@ RUN apk update --no-cache && \
                                 /opt/templates && \
     chmod 755 /data/*.sh /data/*.py /data/init && \
     chmod 400 /opt/maps/* && \
-    (echo -e "*/2 * * * * /data/create-arkime-sessions-index.sh\n0 10 * * * /data/index-refresh.py --index MALCOLM_NETWORK_INDEX_PATTERN --template malcolm_template --unassigned\n30 */2 * * * /data/index-refresh.py --index MALCOLM_OTHER_INDEX_PATTERN --template malcolm_beats_template --unassigned\n*/20 * * * * /data/opensearch_index_size_prune.py" > ${SUPERCRONIC_CRONTAB})
+    (echo -e "*/2 * * * * /data/shared-object-creation.sh\n0 10 * * * /data/index-refresh.py --index MALCOLM_NETWORK_INDEX_PATTERN --template malcolm_template --unassigned\n30 */2 * * * /data/index-refresh.py --index MALCOLM_OTHER_INDEX_PATTERN --template malcolm_beats_template --unassigned\n*/20 * * * * /data/opensearch_index_size_prune.py" > ${SUPERCRONIC_CRONTAB})
 
 EXPOSE $OFFLINE_REGION_MAPS_PORT
 

diff --git a/Dockerfiles/dashboards.Dockerfile b/Dockerfiles/dashboards.Dockerfile
@@ -1,4 +1,4 @@
-FROM opensearchproject/opensearch-dashboards:2.12.0
+FROM opensearchproject/opensearch-dashboards:2.13.0
 
 LABEL maintainer="malcolm@inl.gov"
 LABEL org.opencontainers.image.authors='malcolm@inl.gov'
@@ -20,7 +20,7 @@ ENV PUSER_PRIV_DROP true
 ENV TERM xterm
 
 ENV TINI_VERSION v0.19.0
-ENV OSD_TRANSFORM_VIS_VERSION 2.12.0
+ENV OSD_TRANSFORM_VIS_VERSION 2.13.0
 
 ARG NODE_OPTIONS="--max_old_space_size=4096"
 ENV NODE_OPTIONS $NODE_OPTIONS
@@ -40,8 +40,8 @@ RUN yum upgrade -y && \
     /usr/share/opensearch-dashboards/bin/opensearch-dashboards-plugin remove securityDashboards --allow-root && \
     cd /tmp && \
         # unzip transformVis.zip opensearch-dashboards/transformVis/opensearch_dashboards.json opensearch-dashboards/transformVis/package.json && \
-        # sed -i "s/2\.12\.0/2\.12\.0/g" opensearch-dashboards/transformVis/opensearch_dashboards.json && \
-        # sed -i "s/2\.12\.0/2\.12\.0/g" opensearch-dashboards/transformVis/package.json && \
+        # sed -i "s/2\.12\.0/2\.13\.0/g" opensearch-dashboards/transformVis/opensearch_dashboards.json && \
+        # sed -i "s/2\.12\.0/2\.13\.0/g" opensearch-dashboards/transformVis/package.json && \
         # zip transformVis.zip opensearch-dashboards/transformVis/opensearch_dashboards.json opensearch-dashboards/transformVis/package.json && \
         cd /usr/share/opensearch-dashboards/plugins && \
         /usr/share/opensearch-dashboards/bin/opensearch-dashboards-plugin install file:///tmp/transformVis.zip --allow-root && \

diff --git a/Dockerfiles/file-monitor.Dockerfile b/Dockerfiles/file-monitor.Dockerfile
@@ -34,6 +34,9 @@ ARG EXTRACTED_FILE_SCANNER_START_SLEEP=10
 ARG EXTRACTED_FILE_LOGGER_START_SLEEP=5
 ARG EXTRACTED_FILE_MIN_BYTES=64
 ARG EXTRACTED_FILE_MAX_BYTES=134217728
+ARG EXTRACTED_FILE_PRUNE_THRESHOLD_MAX_SIZE=1TB
+ARG EXTRACTED_FILE_PRUNE_THRESHOLD_TOTAL_DISK_USAGE_PERCENT=0
+ARG EXTRACTED_FILE_PRUNE_INTERVAL_SECONDS=300
 ARG VTOT_API2_KEY=0
 ARG VTOT_REQUESTS_PER_MINUTE=4
 ARG EXTRACTED_FILE_ENABLE_CLAMAV=false
@@ -65,6 +68,9 @@ ENV EXTRACTED_FILE_SCANNER_START_SLEEP $EXTRACTED_FILE_SCANNER_START_SLEEP
 ENV EXTRACTED_FILE_LOGGER_START_SLEEP $EXTRACTED_FILE_LOGGER_START_SLEEP
 ENV EXTRACTED_FILE_MIN_BYTES $EXTRACTED_FILE_MIN_BYTES
 ENV EXTRACTED_FILE_MAX_BYTES $EXTRACTED_FILE_MAX_BYTES
+ENV EXTRACTED_FILE_PRUNE_THRESHOLD_MAX_SIZE $EXTRACTED_FILE_PRUNE_THRESHOLD_MAX_SIZE
+ENV EXTRACTED_FILE_PRUNE_THRESHOLD_TOTAL_DISK_USAGE_PERCENT $EXTRACTED_FILE_PRUNE_THRESHOLD_TOTAL_DISK_USAGE_PERCENT
+ENV EXTRACTED_FILE_PRUNE_INTERVAL_SECONDS $EXTRACTED_FILE_PRUNE_INTERVAL_SECONDS
 ENV VTOT_API2_KEY $VTOT_API2_KEY
 ENV VTOT_REQUESTS_PER_MINUTE $VTOT_REQUESTS_PER_MINUTE
 ENV EXTRACTED_FILE_ENABLE_CLAMAV $EXTRACTED_FILE_ENABLE_CLAMAV
@@ -103,6 +109,11 @@ ENV SUPERCRONIC_SHA1SUM "cd48d45c4b10f3f0bfdd3a57d054cd05ac96812b"
 ENV SUPERCRONIC_CRONTAB "/etc/crontab"
 
 COPY --chmod=755 shared/bin/yara_rules_setup.sh /usr/local/bin/
+ADD nginx/landingpage/css "${EXTRACTED_FILE_HTTP_SERVER_ASSETS_DIR}/css"
+ADD nginx/landingpage/js "${EXTRACTED_FILE_HTTP_SERVER_ASSETS_DIR}/js"
+ADD --chmod=644 docs/images/logo/Malcolm_background.png "${EXTRACTED_FILE_HTTP_SERVER_ASSETS_DIR}/assets/img/bg-masthead.png"
+COPY --chmod=644 docs/images/icon/favicon.ico "${EXTRACTED_FILE_HTTP_SERVER_ASSETS_DIR}/favicon.ico"
+COPY --chmod=755 shared/bin/web-ui-asset-download.sh /usr/local/bin/
 
 RUN sed -i "s/main$/main contrib non-free/g" /etc/apt/sources.list.d/debian.sources && \
     apt-get -q update && \
@@ -129,7 +140,7 @@ RUN sed -i "s/main$/main contrib non-free/g" /etc/apt/sources.list.d/debian.sour
       pkg-config \
       tini \
       unzip && \
-    apt-get  -y -q install \
+    apt-get -y -q install \
       inotify-tools \
       libzmq5 \
       psmisc \
@@ -143,6 +154,7 @@ RUN sed -i "s/main$/main contrib non-free/g" /etc/apt/sources.list.d/debian.sour
     python3 -m pip install --break-system-packages --no-compile --no-cache-dir \
       clamd \
       dominate \
+      humanfriendly \
       psutil \
       pycryptodome \
       python-magic \
@@ -170,6 +182,8 @@ RUN sed -i "s/main$/main contrib non-free/g" /etc/apt/sources.list.d/debian.sour
       rm -rf "${SRC_BASE_DIR}"/yara* && \
     cd "${YARA_RULES_SRC_DIR}" && \
       /usr/local/bin/yara_rules_setup.sh -r "${YARA_RULES_SRC_DIR}" -y "${YARA_RULES_DIR}" && \
+    cd /tmp && \
+      /usr/local/bin/web-ui-asset-download.sh -o "${EXTRACTED_FILE_HTTP_SERVER_ASSETS_DIR}/css" && \
     cd /tmp && \
       curl -fsSL -o ./capa.zip "${CAPA_URL}" && \
       unzip ./capa.zip && \
@@ -190,9 +204,6 @@ RUN sed -i "s/main$/main contrib non-free/g" /etc/apt/sources.list.d/debian.sour
         libtool \
         make \
         python3-dev && \
-      apt-get -y -q --allow-downgrades --allow-remove-essential --allow-change-held-packages autoremove && \
-      apt-get clean && \
-      rm -rf /var/lib/apt/lists/* /tmp/* && \
     mkdir -p /var/log/clamav "${CLAMAV_RULES_DIR}" && \
     groupadd --gid ${DEFAULT_GID} ${PGROUP} && \
       useradd -m --uid ${DEFAULT_UID} --gid ${DEFAULT_GID} ${PUSER} && \
@@ -214,31 +225,22 @@ RUN sed -i "s/main$/main contrib non-free/g" /etc/apt/sources.list.d/debian.sour
       ln -r -s /usr/local/bin/zeek_carve_scanner.py /usr/local/bin/clam_scan.py && \
       ln -r -s /usr/local/bin/zeek_carve_scanner.py /usr/local/bin/yara_scan.py && \
       ln -r -s /usr/local/bin/zeek_carve_scanner.py /usr/local/bin/capa_scan.py && \
-      echo "0 */6 * * * /bin/bash /usr/local/bin/capa-update.sh\n0 */6 * * * /usr/local/bin/yara_rules_setup.sh -r \"${YARA_RULES_SRC_DIR}\" -y \"${YARA_RULES_DIR}\"" > ${SUPERCRONIC_CRONTAB}
+      echo "0 */6 * * * /bin/bash /usr/local/bin/capa-update.sh\n0 */6 * * * /usr/local/bin/yara_rules_setup.sh -r \"${YARA_RULES_SRC_DIR}\" -y \"${YARA_RULES_DIR}\"" > ${SUPERCRONIC_CRONTAB} && \
+  apt-get -y -q --allow-downgrades --allow-remove-essential --allow-change-held-packages autoremove && \
+  apt-get clean && \
+  rm -rf /var/lib/apt/lists/* /tmp/*
 
 USER ${PUSER}
 
 RUN /usr/bin/freshclam freshclam --config-file=/etc/clamav/freshclam.conf
 
 USER root
 
-ADD nginx/landingpage/css "${EXTRACTED_FILE_HTTP_SERVER_ASSETS_DIR}/css"
-ADD nginx/landingpage/js "${EXTRACTED_FILE_HTTP_SERVER_ASSETS_DIR}/js"
-ADD --chmod=644 docs/images/logo/Malcolm_background.png "${EXTRACTED_FILE_HTTP_SERVER_ASSETS_DIR}/assets/img/bg-masthead.png"
-ADD --chmod=644 https://fonts.gstatic.com/s/lato/v24/S6u_w4BMUTPHjxsI9w2_Gwfo.ttf "${EXTRACTED_FILE_HTTP_SERVER_ASSETS_DIR}/css/"
-ADD --chmod=644 https://fonts.gstatic.com/s/lato/v24/S6u8w4BMUTPHjxsAXC-v.ttf "${EXTRACTED_FILE_HTTP_SERVER_ASSETS_DIR}/css/"
-ADD --chmod=644 https://fonts.gstatic.com/s/lato/v24/S6u_w4BMUTPHjxsI5wq_Gwfo.ttf "${EXTRACTED_FILE_HTTP_SERVER_ASSETS_DIR}/css/"
-ADD --chmod=644 https://fonts.gstatic.com/s/lato/v24/S6u9w4BMUTPHh7USSwiPHA.ttf "${EXTRACTED_FILE_HTTP_SERVER_ASSETS_DIR}/css/"
-ADD --chmod=644 https://fonts.gstatic.com/s/lato/v24/S6uyw4BMUTPHjx4wWw.ttf "${EXTRACTED_FILE_HTTP_SERVER_ASSETS_DIR}/css/"
-ADD --chmod=644 https://fonts.gstatic.com/s/lato/v24/S6u9w4BMUTPHh6UVSwiPHA.ttf "${EXTRACTED_FILE_HTTP_SERVER_ASSETS_DIR}/css/"
-ADD --chmod=644 'https://cdn.jsdelivr.net/npm/bootstrap-icons@1.5.0/font/fonts/bootstrap-icons.woff2?856008caa5eb66df68595e734e59580d' "${EXTRACTED_FILE_HTTP_SERVER_ASSETS_DIR}/css/bootstrap-icons.woff2"
-ADD --chmod=644 'https://cdn.jsdelivr.net/npm/bootstrap-icons@1.5.0/font/fonts/bootstrap-icons.woff?856008caa5eb66df68595e734e59580d' "${EXTRACTED_FILE_HTTP_SERVER_ASSETS_DIR}/css/bootstrap-icons.woff"
-
-COPY --chmod=644 docs/images/icon/favicon.ico "${EXTRACTED_FILE_HTTP_SERVER_ASSETS_DIR}/favicon.ico"
 COPY --chmod=755 shared/bin/docker-uid-gid-setup.sh /usr/local/bin/
+COPY --chmod=755 shared/bin/prune_files.sh /usr/local/bin/
 COPY --chmod=755 shared/bin/service_check_passthrough.sh /usr/local/bin/
 COPY --chmod=755 shared/bin/zeek_carve*.py /usr/local/bin/
-COPY --chmod=755 file-monitor/scripts/*.py /usr/local/bin/
+COPY --chmod=755 shared/bin/extracted_files_http_server.py /usr/local/bin/
 COPY --chmod=644 shared/bin/watch_common.py /usr/local/bin/
 COPY --chmod=644 scripts/malcolm_utils.py /usr/local/bin/
 COPY --chmod=644 file-monitor/supervisord.conf /etc/supervisord.conf

diff --git a/Dockerfiles/filebeat.Dockerfile b/Dockerfiles/filebeat.Dockerfile
@@ -1,4 +1,4 @@
-FROM docker.elastic.co/beats/filebeat-oss:8.12.1
+FROM docker.elastic.co/beats/filebeat-oss:8.13.2
 
 # Copyright (c) 2024 Battelle Energy Alliance, LLC.  All rights reserved.
 LABEL maintainer="malcolm@inl.gov"

diff --git a/Dockerfiles/logstash.Dockerfile b/Dockerfiles/logstash.Dockerfile
@@ -1,4 +1,4 @@
-FROM docker.elastic.co/logstash/logstash-oss:8.12.1
+FROM docker.elastic.co/logstash/logstash-oss:8.13.2
 
 LABEL maintainer="malcolm@inl.gov"
 LABEL org.opencontainers.image.authors='malcolm@inl.gov'

diff --git a/Dockerfiles/opensearch.Dockerfile b/Dockerfiles/opensearch.Dockerfile
@@ -1,4 +1,4 @@
-FROM opensearchproject/opensearch:2.12.0
+FROM opensearchproject/opensearch:2.13.0
 
 # Copyright (c) 2024 Battelle Energy Alliance, LLC.  All rights reserved.
 LABEL maintainer="malcolm@inl.gov"