diff --git a/README.md b/README.md index fb4a574d1..f8dde1e6a 100644 --- a/README.md +++ b/README.md @@ -4,7 +4,7 @@ Malcolm is a powerful network traffic analysis tool suite designed with the following goals in mind: -* **Easy to use** – Malcolm accepts network traffic data in the form of full packet capture (PCAP) files and Zeek (formerly Bro) logs. These artifacts can be uploaded via a simple browser-based interface or captured live and forwarded to Malcolm using lightweight forwarders. In either case, the data is automatically normalized, enriched, and correlated for analysis. +* **Easy to use** – Malcolm accepts network traffic data in the form of full packet capture (PCAP) files and Zeek logs. These artifacts can be uploaded via a simple browser-based interface or captured live and forwarded to Malcolm using lightweight forwarders. In either case, the data is automatically normalized, enriched, and correlated for analysis. * **Powerful traffic analysis** – Visibility into network communications is provided through two intuitive interfaces: OpenSearch Dashboard, a flexible data visualization plugin with dozens of prebuilt dashboards providing an at-a-glance overview of network protocols; and Arkime (formerly Moloch), a powerful tool for finding and identifying the network sessions comprising suspected security incidents. * **Streamlined deployment** – Malcolm operates as a cluster of Docker containers – isolated sandboxes that each serve a dedicated function of the system. This Docker-based deployment model, combined with a few simple scripts for setup and run-time management, makes Malcolm suitable to be deployed quickly across a variety of platforms and use cases; whether it be for long-term deployment on a Linux server in a security operations center (SOC) or for incident response on a Macbook for an individual engagement. * **Secure communications** – All communications with Malcolm, both from the user interface and from remote log forwarders, are secured with industry standard encryption protocols. diff --git a/docs/arkime.md b/docs/arkime.md index 41d9fdc35..557a33193 100644 --- a/docs/arkime.md +++ b/docs/arkime.md @@ -17,7 +17,7 @@ The Arkime interface will be accessible over HTTPS on port 443 at the docker hos ## Zeek log integration -A stock installation of Arkime extracts all its network connection ("session") metadata ("SPI" or "Session Profile Information") from full packet capture artifacts (PCAP files). Zeek (formerly Bro) generates similar session metadata, linking network events to sessions via a connection UID. Malcolm aims to facilitate analysis of Zeek logs by mapping values from Zeek logs to the Arkime session database schema for equivalent fields, and by creating new "native" Arkime database fields for all other Zeek log values for which there is not currently an equivalent in Arkime: +A stock installation of Arkime extracts all its network connection ("session") metadata ("SPI" or "Session Profile Information") from full packet capture artifacts (PCAP files). Zeek generates similar session metadata, linking network events to sessions via a connection UID. Malcolm aims to facilitate analysis of Zeek logs by mapping values from Zeek logs to the Arkime session database schema for equivalent fields, and by creating new "native" Arkime database fields for all other Zeek log values for which there is not currently an equivalent in Arkime: ![Zeek log session record](./images/screenshots/arkime_session_zeek.png) diff --git a/docs/malcolm-hedgehog-e2e-iso-install.md b/docs/malcolm-hedgehog-e2e-iso-install.md index 1c0b2a7a9..7e675bee0 100644 --- a/docs/malcolm-hedgehog-e2e-iso-install.md +++ b/docs/malcolm-hedgehog-e2e-iso-install.md @@ -463,7 +463,7 @@ To specify which files should be extracted, specify the Zeek file carving mode: If unsure what mode to choose, both **mapped (except common plain text files)** (to carve and scan almost all files) and **interesting** (to only carve and scan files with [mime types of common attack vectors]({{ site.github.repository_url }}/blob/{{ site.github.build_revision }}/hedgehog-iso/interface/sensor_ctl/zeek/extractor_override.interesting.zeek)) are probably good choices. -Next, specify which carved files to preserve (saved on the sensor under `/capture/bro/capture/extract_files/quarantine` by default). In order to not consume all the sensor's available storage space, the oldest preserved files will be pruned along with the oldest Zeek logs as described below with **AUTOSTART_PRUNE_ZEEK** in the [autostart services](#HedgehogConfigAutostart) section. +Next, specify which carved files to preserve (saved on the sensor under `/capture/zeek/capture/extract_files/quarantine` by default). In order to not consume all the sensor's available storage space, the oldest preserved files will be pruned along with the oldest Zeek logs as described below with **AUTOSTART_PRUNE_ZEEK** in the [autostart services](#HedgehogConfigAutostart) section. Users will prompted to specify which engine(s) to use to analyze extracted files. Extracted files can be examined through any of three methods: diff --git a/hedgehog-iso/interface/sensor_ctl/control_vars.conf b/hedgehog-iso/interface/sensor_ctl/control_vars.conf index a3b2c76ac..66a9ec266 100644 --- a/hedgehog-iso/interface/sensor_ctl/control_vars.conf +++ b/hedgehog-iso/interface/sensor_ctl/control_vars.conf @@ -32,7 +32,7 @@ export FLUENTBIT_METRICS_INTERVAL=30 export FLUENTBIT_THERMAL_INTERVAL=10 export FLUENTBIT_AIDE_INTERVAL=86400 -export ZEEK_LOG_PATH=/home/sensor/bro_logs +export ZEEK_LOG_PATH=/home/sensor/zeek_logs export ZEEK_MAX_DISK_FILL=90 export ZEEK_PRUNE_CHECK_SECONDS=90 diff --git a/hedgehog-iso/interface/sensor_ctl/filebeat/filebeat.yml b/hedgehog-iso/interface/sensor_ctl/filebeat/filebeat.yml index 7d122e151..adb9ae734 100644 --- a/hedgehog-iso/interface/sensor_ctl/filebeat/filebeat.yml +++ b/hedgehog-iso/interface/sensor_ctl/filebeat/filebeat.yml @@ -5,8 +5,8 @@ logging.metrics.enabled: false filebeat.inputs: - type: log paths: - - ${BEAT_LOG_PATTERN:/home/sensor/bro_logs/*.log} - - ${BEAT_STATIC_LOG_PATTERN:/home/sensor/bro_logs/static/*.log} + - ${BEAT_LOG_PATTERN:/home/sensor/zeek_logs/*.log} + - ${BEAT_STATIC_LOG_PATTERN:/home/sensor/zeek_logs/static/*.log} symlinks: true fields_under_root: true tags: ["_filebeat_zeek_hedgehog_live"] @@ -24,7 +24,7 @@ filebeat.inputs: - type: log paths: - - ${BEAT_SURICATA_LOG_PATTERN:/home/sensor/bro_logs/suricata/eve*.json} + - ${BEAT_SURICATA_LOG_PATTERN:/home/sensor/zeek_logs/suricata/eve*.json} symlinks: true fields_under_root: true tags: ["_filebeat_suricata_hedgehog_live"] diff --git a/hedgehog-iso/interface/sensor_ctl/filebeat/sensor_filebeat_local.sh b/hedgehog-iso/interface/sensor_ctl/filebeat/sensor_filebeat_local.sh index 212a1d1ad..7a6da4307 100755 --- a/hedgehog-iso/interface/sensor_ctl/filebeat/sensor_filebeat_local.sh +++ b/hedgehog-iso/interface/sensor_ctl/filebeat/sensor_filebeat_local.sh @@ -3,10 +3,10 @@ # Copyright (c) 2024 Battelle Energy Alliance, LLC. All rights reserved. if [[ -z "$ZEEK_CAPTURE_PATH" ]]; then - ZEEK_CAPTURE_PATH="$HOME/bro_logs" + ZEEK_CAPTURE_PATH="$HOME/zeek_logs" fi if [[ -z "$SURICATA_CAPTURE_PATH" ]]; then - SURICATA_CAPTURE_PATH="$HOME/bro_logs/suricata" + SURICATA_CAPTURE_PATH="$HOME/zeek_logs/suricata" fi export ZEEK_CAPTURE_PATH export SURICATA_CAPTURE_PATH diff --git a/hedgehog-iso/interface/sensor_interface/static/js/custom.js b/hedgehog-iso/interface/sensor_interface/static/js/custom.js index 9e7373347..347e16927 100644 --- a/hedgehog-iso/interface/sensor_interface/static/js/custom.js +++ b/hedgehog-iso/interface/sensor_interface/static/js/custom.js @@ -28,7 +28,7 @@ function stop_all() { } -function start_bro() { +function start_zeek() { var xhttp = new XMLHttpRequest(); loadingBar('on'); xhttp.onreadystatechange = function () { @@ -41,7 +41,7 @@ function start_bro() { xhttp.send(); } -function stop_bro() { +function stop_zeek() { var xhttp = new XMLHttpRequest(); loadingBar('on'); xhttp.onreadystatechange = function () { diff --git a/hedgehog-iso/interface/sensor_interface/templates/buttons.html b/hedgehog-iso/interface/sensor_interface/templates/buttons.html index 84eae9cdb..652e67dfe 100644 --- a/hedgehog-iso/interface/sensor_interface/templates/buttons.html +++ b/hedgehog-iso/interface/sensor_interface/templates/buttons.html @@ -55,7 +55,7 @@

+ onclick="start_zeek()"> Start
@@ -116,7 +116,7 @@

+ onclick="stop_zeek()"> Stop
diff --git a/pcap-capture/scripts/supervisor.sh b/pcap-capture/scripts/supervisor.sh index 618e775ff..73c4a869a 100755 --- a/pcap-capture/scripts/supervisor.sh +++ b/pcap-capture/scripts/supervisor.sh @@ -19,7 +19,7 @@ function SetCaptureCapabilities() { # Create config files for each capture interface for the various capture programs (tcpdump, netsniff) # so that supervisord can manage instances of each of these programs for each interface. -# bro is now managed by broctl (via brodeploy.sh) rather than individually by supervisord so that +# zeek is now managed by zeekctl (via zeekdeploy.sh) rather than individually by supervisord so that # we can use pf_ring function CreateCaptureConfigs() { diff --git a/shared/bin/os-disk-config.py b/shared/bin/os-disk-config.py index 4e7243583..16ce559dc 100755 --- a/shared/bin/os-disk-config.py +++ b/shared/bin/os-disk-config.py @@ -27,7 +27,7 @@ OS_MODE_MALCOLM = 'malcolm' HEDGEHOG_PCAP_DIR = "pcap" -HEDGEHOG_ZEEK_DIR = "bro" +HEDGEHOG_ZEEK_DIR = "zeek" MALCOLM_DB_DIR = "datastore" MALCOLM_PCAP_DIR = "pcap" MALCOLM_LOGS_DIR = "logs" @@ -204,7 +204,7 @@ def main(): nargs='?', const=True, default=False, - help="Unmount capture directories before determining candidate drives", + help="Unmount storage directories before determining candidate drives", ) parser.add_argument( '-v', '--verbose', dest='debug', type=str2bool, nargs='?', const=True, default=False, help="Verbose output" @@ -250,9 +250,9 @@ def main(): # unmount existing mounts if requested if args.umount and (not args.dryrun): - if (not args.interactive) or YesOrNo('Unmount any mounted capture path(s)?'): + if (not args.interactive) or YesOrNo('Unmount any mounted storage path(s)?'): if debug: - eprint("Attempting unmount of capture path(s)...") + eprint("Attempting unmount of storage path(s)...") for subdir in OS_PARAMS[osMode][MOUNT_DIRS]: run_subprocess(f"umount {os.path.join(OS_PARAMS[osMode][MOUNT_ROOT_PATH], subdir)}") run_subprocess(f"umount {OS_PARAMS[osMode][MOUNT_ROOT_PATH]}") @@ -271,7 +271,7 @@ def main(): eprint(f"\t{line}") _, reloadOut = run_subprocess("systemctl daemon-reload") - # check existing mounts, if the capture path(s) are already mounted, then abort + # check existing mounts, if the path(s) are already mounted, then abort with open('/proc/mounts', 'r') as f: for line in f.readlines(): mountDetails = line.split() @@ -366,7 +366,7 @@ def main(): if len(candidateDevs) > 0: if args.encrypt: - # create keyfile (will be on the encrypted system drive, and used to automatically unlock the encrypted capture drives) + # create keyfile (will be on the encrypted system drive, and used to automatically unlock the encrypted drives) with open(OS_PARAMS[osMode][CRYPT_KEYFILE], 'wb') as f: f.write(os.urandom(4096)) os.chown(OS_PARAMS[osMode][CRYPT_KEYFILE], 0, 0) @@ -574,17 +574,17 @@ def main(): # reload tab files with systemctl _, reloadOut = run_subprocess("systemctl daemon-reload") - # get the GID of the group of the user(s) that will be doing the capture + # get the GID of the group of the user(s) under which the processes will be run try: ecode, guidGetOut = run_subprocess( f"getent group {OS_PARAMS[osMode][GROUP_OWNER]}", stdout=True, stderr=True ) if (ecode == 0) and (len(guidGetOut) > 0): - netdevGuid = int(guidGetOut[0].split(':')[2]) + ownerGuid = int(guidGetOut[0].split(':')[2]) else: - netdevGuid = -1 + ownerGuid = -1 except Exception: - netdevGuid = -1 + ownerGuid = -1 # rmdir any mount directories that might be interfering from previous configurations if os.path.isdir(OS_PARAMS[osMode][MOUNT_ROOT_PATH]): @@ -599,7 +599,7 @@ def main(): if debug: eprint(f"Creating {OS_PARAMS[osMode][MOUNT_ROOT_PATH]}") os.makedirs(OS_PARAMS[osMode][MOUNT_ROOT_PATH], exist_ok=True) - os.chown(OS_PARAMS[osMode][MOUNT_ROOT_PATH], -1, netdevGuid) + os.chown(OS_PARAMS[osMode][MOUNT_ROOT_PATH], -1, ownerGuid) os.chmod(OS_PARAMS[osMode][MOUNT_ROOT_PATH], OS_PARAMS[osMode][DIR_PERMS]) # add crypttab entries @@ -656,23 +656,23 @@ def main(): for subdir in OS_PARAMS[osMode][MOUNT_DIRS]: userDirs.append(os.path.join(par.mount, subdir)) else: - # we're mounted somewhere *underneath* /capture, so create a user-writeable subdirectory where we are + # we're mounted somewhere *underneath* /{MOUNT_ROOT_PATH}, so create a user-writeable subdirectory where we are userDirs.append(os.path.join(par.mount, OS_PARAMS[osMode][MOUNT_ROOT_PATH].strip(os.path.sep))) # set permissions on user dirs createdUserDirs = defaultdict(lambda: None) for userDir in userDirs: os.makedirs(userDir, exist_ok=True) - os.chown(userDir, OS_PARAMS[osMode][USER_UID], netdevGuid) + os.chown(userDir, OS_PARAMS[osMode][USER_UID], ownerGuid) os.chmod(userDir, OS_PARAMS[osMode][SUBDIR_PERMS]) if debug: - eprint(f'Created "{userDir}" for writing by capture user') + eprint(f'Created "{userDir}" for writing by unprivileged user') for subdir in OS_PARAMS[osMode][MOUNT_DIRS]: if f"{os.path.sep}{subdir}{os.path.sep}" in userDir: createdUserDirs[subDir] = userDir break - # replace capture paths in-place in control_vars.conf + # replace paths in-place in control_vars.conf if (osMode == OS_MODE_HEDGEHOG) and os.path.isfile(OS_PARAMS[osMode][SYSTEM_CONFIG_FILE]): capture_re = re.compile(r"\b(?PPCAP_PATH|ZEEK_LOG_PATH)\s*=\s*.*?$") with fileinput.FileInput(OS_PARAMS[osMode][SYSTEM_CONFIG_FILE], inplace=True, backup='.bak') as f: