Malcolm v24.03.1

[mmguero]

Malcolm v24.03.1 contains new features, improvements, bug fixes and component version updates.

v24.03.0...v24.03.1

Because some of the environment variables used for configuring Malcolm have been reorganized in the .env files found in the ./config directory, it is strongly recommended you re-run ./scripts/configure for this release.

• Features and enhancements
  • Malcolm instances created using the installer ISO will now detect and format any large (>100GB) storage devices and automatically set them up for use for storing the OpenSearch data store, PCAP files, and/or log storage, similar to what Hedgehog Linux does. (Malcolm ISO should format bigger drives for index and artifact storage #266)
  • Since v24.01.0, Malcolm has allowed users to specify custom index patterns for Zeek and Suricata logs (see issue 313). This release now also provides the capability for Arkime to know about those indexes so that those documents also appear in Arkime search results. (allow customizing indexes for logs written to OpenSearch/Elasticsearch #313, Add support for queryExtraIndices option in config.ini to include other index pattern(s) in DB queries arkime/arkime#2705) As this is not released in Arkime yet, Malcolm is using a local patch with these changes, to be released upstream in Arkime v5.0.2.
  • A new setting for Logstash has been added to allow autocreation and assignment of NetBox subnets during enrichment. If "Should Malcolm automatically create missing NetBox subnet prefixes based on observed network traffic?" is answered to the affirmative during configuration, observed traffic that does not fall into any existing NetBox prefix will cause one to automatically be created, creating them one level down (e.g., 8 additional masked bits) from the RFC1918 address space definitions. This relplaces an earlier feature (controlled by the NETBOX_PRELOAD_PREFIXES variable, which no longer has any effect) in which three giant buckets (one for each block in the RFC1918 private address definitions) could be created once at startup. (autocreation and assignment of NetBox subnets in Logstash #436). So, for example:
    • 10./16 (255.255.0.0)
      • the IP address 10.9.0.215 would cause us to create and assign it to a 10.9.0.0/16 subnet
    • 192.168./24 (255.255.255.0)
      • the IP address 192.168.100.123 would cause us to create and assign it to a 192.168.100.0/24 subnet
    • 172.16./20 (255.255.240.0)
      • the IP address 172.16.29.10 would cause us to create and assign it to a 172.16.29.10/20 subnet
  • New configuration settings have been added to specify creation and rotation of Suricata's EVE JSON log files, including controls for threaded file output and file rotation. See this comment for a full description of the changes (add ability to generate suricata.yaml such that suricata's eve.json is split/rotated so it doesn't grow as large #445). Most noteworthy are:
    • SURICATA_EVE_THREADED - controls threaded file output (default false)
    • SURICATA_EVE_ROTATE_INTERVAL - controls eve.json file rotation (default 1h)
  • Table visualisations in Malcolm's prebuilt OpenSearch Dashboards were not consistent in the number of rows returned. This has been standardized to 100 and otherBucket: true has been set for all of these table visualizations to ensure that the end user knows that Other rows may also exist outside of the rows shown. (unformly increase number of results for table visualizations in Dashboards #447)
  • Some some field mappings were moved from malcolm_template.json to the composable template malcolm_common.json
  • Documenation improvements
  • Minor update to slides
  • Some directories named like bro_logs were renamed to zeek_logs on Hedgehog Linux
  • The Community ID field is now being added to Zeek's notice.log
  • Attempt to install necessary Python 3 packages at the beginning of install.py instead of just failing
• Component version updates
  • Zeek to v6.2.0
  • opensearch-py to v2.5.0
  • Fluent Bit to v3.0.0
  • Moved from the no-longer-maintained Salesforce repo for HASSH to Corelight's
• Bug fixes
  • AF_PACKET was not being utilized for capturing traffic on Malcolm in the zeek-live container. This did not affect Suricata, Arkime capture, or any of the Hedgehog Linux processes. (AF_PACKET isn't being enabled for zeek-live container capture #437)
  • The Packet Capture Statistics dashboard was not correctly computing seen and dropped packets for Suricata. (suricata stats on "Packet Capture Statistics" not reflecting search time frame #442)
  • A STDERR warning from the new Docker Compose v2.25 was messing up the creation of the OpenSearch keystore file. (stderr from new docker compose v2.25 will mess up creation of OpenSearch keystore #452)
  • Fixed an issue in which the Dashboards for non-network data (e.g., temperatures, resource usage, etc.) would not see the correct data if the MALCOLM_OTHER_INDEX_PATTERN variable had been set to something other than the default.
  • Ensure that index names created for use by Logstash sending to OpenSearch/Elasticsearch are lowercase
  • Major cleanup and refactoring of the NetBox enrichment code used by Logstash
• Configuration changes (in environment variables in ./config/)
  • ARKIME_DEBUG_LEVEL=0 has been added to arkime.env to control the debug level for Arkime's config.ini.
  • Additions/deletions in netbox-common.env (also, see below for some existing variables that were moved from logstash.env):
    • NETBOX_PRELOAD_PREFIXES has been removed and replaced with NETBOX_AUTO_CREATE_PREFIX for autocreation and assignment of NetBox subnets in Logstash #436
    • NETBOX_ENRICHMENT_LOOKUP_SERVICE=true has been added to indicate whether or not services (i.e., destination IP/port) should be looked up during NetBox enrichment
  • Comments were added to opensearch.env to give examples of how to set the time bucketing for OpenSearch/Elasticsearch indexes
  • In addition to the new variables mentioned above, some cleanup and organization was done in the environment variable files used for configuring Malcolm:
    • LOG_CLEANUP_MINUTES and ZIP_CLEANUP_MINUTES are now in filebeat.env, moved from upload-common.env
    • Some NetBox related variables have been moved from logstash.env to netbox-common.env and renamed:
      • LOGSTASH_NETBOX_ENRICHMENT is now NETBOX_ENRICHMENT
      • LOGSTASH_NETBOX_AUTO_POPULATE is now NETBOX_AUTO_POPULATE
      • LOGSTASH_NETBOX_CACHE_SIZE is now NETBOX_CACHE_SIZE
      • LOGSTASH_NETBOX_CACHE_TTL is now LOGSTASH_NETBOX_CACHE_TTL