Malcolm v24.04.0

[mmguero]

Malcolm v24.04.0 contains new features, improvements, bug fixes and component version updates.

v24.03.1...v24.04.0

Because some of the environment variables used for configuring Malcolm have been reorganized in the .env files found in the ./config directory, it is recommended you re-run ./scripts/configure for this release.

• Features and enhancements
  • Zeek-extracted files scanned and preserved on a Hedgehog Linux sensor can now be accessed via the extracted files download user interface (provide way to access extracted_files on Hedgehog from Malcolm idaholab/Malcolm#331).
  • Improvements to creation of index templates, dashboards, and other saved objects on startup (improvements to creation of templates, component templates, and dashboards on startup idaholab/Malcolm#208) to ensure that saved objects get created correctly upon upgrade (see this comment for more details on this feature).
  • Populating the NetBox inventory via passively-gathered network traffic metadata now uses network traffic logs for DNS, NTLM, and DHCP to identify assets' host names when possible for use when populating device and VM names (utilize DNS/NTLM/DHCP/etc. when populating NetBox inventory via passively-gathered network traffic metadata idaholab/Malcolm#415). Autopopulated devices now have their status field set to Active rather than Stage, and uses tags instead to indicated that they were created through autopopulation.
  • Users can now specify pruning thresholds for carved files so that old files are deleted in order to avoid filling available storage (provide threshold for pruning extracted files idaholab/Malcolm#453). See a new section of documentation on Managing disk usage for more information about this and similar settings.
  • Users can now specify a prefix that will be prepended to dashboards as they are imported into OpenSearch Dashboards or Kibana, allowing users who have dashboards from other sources to differentiate between those and Malcolm's (allow user to specify prefix for dashboards idaholab/Malcolm#455).
  • The default anomaly detectors created for the OpenSearch Anomaly Detection plugin are now created with category fields for high cardinality to allow for better breakdown of contributing values to anomalies discovered (set category fields in default anomaly detectors to give a better breakdown of contributors idaholab/Malcolm#464).
  • Include JA4+ plugin in Arkime. See integrate JA4 zeek plugin idaholab/Malcolm#419 for status on upcoming full JA4+ support in Malcolm.
  • Hedgehog Linux sensors can now periodically refresh their Zeek inteligence files. NOTE: due to an oversight, a necessary variable is missing in this release that is required for this to work. Appending the line export INTEL_DIR=/opt/sensor/sensor_ctl/zeek/intel to /opt/sensor/sensor_ctl/control_vars.conf will correct this. This will be corrected in the next Malcolm release.
  • Assorted documentation improvements.
• Component version updates
  • Arkime to v5.1.2
  • OpenSearch and OpenSearch Dashboards to v2.13.0
  • Beats to v8.13.2
  • Logstash to v8.13.2
  • gunicorn to v22.0.0 to address CVE-2024-1135.
  • elasticsearch-dsl to v8.13.0
  • elasticsearch-py to v8.13.0
  • idna to v3.7 to address CVE-2024-3651
  • Fluent Bit to v3.0.3
• Bug fixes
  • The documentation for Windows host system configuration was out of date and has been updated for the latest version of Microsoft Windows Subsystem for Linux (review and update Windows prep and installation documentation idaholab/Malcolm#421).
  • An issue was fixed in which Malcolm's list of users and their password hashes could become corrupted if the file did not initially end with a newline character (htadmin creating entries without a newline between them in the htpasswd file idaholab/Malcolm#426).
  • The manner in which Zeek intel files are generated has been changed to avoid problems found in Kubernetes deployments when scaling out the number of zeek-live containers (adjustments to how Zeek intel files get generated among Malcolm's containers idaholab/Malcolm#456). See this comment for more details.
  • Removed the version top-level element from docker-compose.yml files as it is now obsolete and caused a warning message that sometimes was not handled correctly.
  • Fix Malcolm ISO not correctly detecting if it's in a live boot ISO environment or installed mode.
  • Restart live Zeek instances with zeekctl deploy instead of zeekctl restart.
• Configuration changes (in environment variables in ./config/)
  • ARKIME_QUERY_ALL_INDICES in arkime.env can be set to control the queryAllIndices setting in Arkime's config.ini.
  • DASHBOARDS_PREFIX in dashboards-helper.env has been added for allow user to specify prefix for dashboards idaholab/Malcolm#455 (see above in Features and Enhancements).
  • LOGSTASH_NETBOX_ENRICHMENT_DATASETS in logstash.env has been changed to include zeek.dhcp, zeek.dns, and zeek.ntlm to support utilize DNS/NTLM/DHCP/etc. when populating NetBox inventory via passively-gathered network traffic metadata idaholab/Malcolm#415 (see above in Features and Enhancements).
  • LOGSTASH_ZEEK_IGNORED_LOGS in logstash.env has been changed to remove capture_loss and stats so that those diagnostic Zeek logs can be parsed without the user having to manually change this variable.
  • ZEEK_CRON has been removed from zeek-live.env and ZEEK_INTEL_REFRESH_CRON_EXPRESSION was removed from zeek.env and moved to the "offline" version of the container in zeek-offline.env for adjustments to how Zeek intel files get generated among Malcolm's containers idaholab/Malcolm#456.
  • EXTRACTED_FILE_PRUNE_THRESHOLD_MAX_SIZE, EXTRACTED_FILE_PRUNE_THRESHOLD_TOTAL_DISK_USAGE_PERCENT, and EXTRACTED_FILE_PRUNE_INTERVAL_SECONDS were added to zeek.env for provide threshold for pruning extracted files idaholab/Malcolm#453. See a new section of documentation on Managing disk usage for more information about these and similar settings.