- Enriching network traffic metadata via NetBox lookups
- Compare and highlight discrepancies between NetBox inventory and observed network traffic
- Populating the NetBox inventory
- Compare NetBox inventory with database of known vulnerabilities
- Preloading NetBox inventory
- Backup and restore
Malcolm provides an instance of NetBox, an open-source "solution for modeling and documenting modern networks." The NetBox web interface is available at at https://localhost/netbox/ if connecting locally.
The design of a potentially deeper integration between Malcolm and Netbox is a work in progress.
Please see the NetBox page on GitHub, its documentation and its public demo for more information.
As Zeek logs and Suricata alerts are parsed and enriched (if the LOGSTASH_NETBOX_ENRICHMENT
environment variable in ./config/logstash.env
is set to true
), the NetBox API will be queried for the associated hosts' information. If found, the information retrieved by NetBox will be used to enrich these logs through the creation of the following new fields. See the NetBox API documentation and the NetBox documentation for more information.
destination.…
destination.device.cluster
(/virtualization/clusters/
) (for Virtual Machine device types)destination.device.device_type
(/dcim/device-types/
)destination.device.id
(/dcim/devices/{id}
)destination.device.manufacturer
(/dcim/manufacturers/
)destination.device.name
(/dcim/devices/
)destination.device.role
(/dcim/device-roles/
)destination.device.service
(/ipam/services/
)destination.device.site
(/dcim/sites/
)destination.device.url
(/dcim/devices/
)destination.device.details
(full JSON object, only withLOGSTASH_NETBOX_ENRICHMENT_VERBOSE: 'true'
)destination.segment.id
(/ipam/vrfs/{id}
)destination.segment.name
(/ipam/vrfs/
)destination.segment.site
(/dcim/sites/
)destination.segment.tenant
(/tenancy/tenants/
)destination.segment.url
(/ipam/vrfs/
)destination.segment.details
(full JSON object, only withLOGSTASH_NETBOX_ENRICHMENT_VERBOSE: 'true'
)
source.…
same asdestination.…
- collected as
related
fields (the same approach used in ECS)related.device_type
related.device_name
related.manufacturer
related.role
related.segment
related.service
related.site
For Malcolm's purposes, both physical devices and virtualized hosts will be stored as described above: the device_type
field can be used to distinguish between them.
NetBox has the concept of sites. Sites can have overlapping IP address ranges. The value of the NETBOX_DEFAULT_SITE
variable in environment variable in netbox-common.env
will be used as a query parameter for these enrichment lookups.
This feature was implemented as described in idaholab/Malcolm#132.
As Malcolm cross-checks network traffic with NetBox's model (as described above), the resulting enrichment data (or lack thereof) can highlight devices and services observed in network traffic for which there is no corresponding entry in the list of inventoried assets.
These uninventoried devices and services are highlighted in two dashboards:
- Zeek Known Summary - this dashboard draws from the periodically-generated
known_
logs andsoftware
logs to provide a summary of the known devices and services in the network. The Uninventoried Observed Services and Uninventoried Observed Hosts tables show services and hosts (by IP address) that weren't found when searched via the NetBox API.
- Asset Interaction Analysis - this dashboard contains much of the same information from the Zeek Known Summary dashboard, but it is from a traffic standpoint rather than just an "observed" standpoint. The Uninventoried Internal Source IPs, Uninventoried Internal Destination IPs and Uninventoried Internal Assets - Logs tables highlight communications involving devices not found when searched via the NetBox API.
This feature was implemented as described in idaholab/Malcolm#133.
While the initial effort of populating NetBox's network segment and device inventory manually is high, it is the preferred method to ensure creation of an accurate model of the intended network design.
The Populating Data section of the NetBox documentation outlines mechanisms available to populate data in NetBox, including manual object creation, bulk import, scripting and the NetBox REST API.
The following elements of the NetBox data model are used by Malcolm for Asset Interaction Analysis.
- Network segments
- Network Hosts
- Devices
- Virtual Machines
- IP Addresses
- Can be assigned to devices and virtual machines
- Other
If the LOGSTASH_NETBOX_AUTO_POPULATE
environment variable in ./config/logstash.env
is set to true
, uninventoried devices with private IP addresses (as defined in RFC 1918 and RFC 4193) observed in known network segments will be automatically created in the NetBox inventory based on the information available. This value is set to true
by answering Y to "Should Malcolm automatically populate NetBox inventory based on observed network traffic?" during configuration.
However, careful consideration should be made before enabling this feature: the purpose of an asset management system is to document the intended state of a network: with Malcolm configured to populate NetBox with the live network state, a network misconfiguration fault could result in an incorrect documented configuration.
Devices created using this autopopulate method will have their status
field set to staged
. It is recommended that users periodically review automatically-created devices for correctness and to fill in known details that couldn't be determined from network traffic. For example, the manufacturer
field for automatically-created devices will be set based on the organizational unique identifier (OUI) determined from the first three bytes of the observed MAC address, which may not be accurate if the device's traffic was observed across a router. If possible, observed hostnames will be used in the naming of the automatically-created devices, falling back to the device manufacturer otherwise (e.g., MYHOSTNAME @ 10.10.0.123
vs. Schweitzer Engineering @ 10.10.0.123
).
Since device autocreation is based on IP address, information about network segments (including virtual routing and forwarding (VRF) and prefixes) must be first manually specified in NetBox in order for devices to be automatically populated.
Although network devices can be automatically created using this method, services should inventoried manually. The Uninventoried Observed Services visualization in the Zeek Known Summary dashboard can help users review network services to be created in NetBox.
See idaholab/Malcolm#135 for more information on this feature.
See idaholab/Malcolm#136.
See idaholab/Malcolm#134.
YML files in [./netbox/preload
]({{ site.github.repository_url }}/tree/{{ site.github.build_revision }}/netbox/preload/) under the Malcolm installation directory will be preloaded upon startup using the third-party netbox-initializers plugin. Examples illustrating the format of these YML files can be found at its GitHub repository.
[workflow files]
The NetBox database may be backed up and restored using ./scripts/netbox-backup
and ./scripts/netbox-restore
, respectively. While Malcolm is running, run the following command from within the Malcolm installation directory to backup the entire NetBox database:
$ ./scripts/netbox-backup
NetBox configuration database saved to ('malcolm_netbox_backup_20230110-133855.gz', 'malcolm_netbox_backup_20230110-133855.media.tar.gz')
To clear the existing NetBox database and restore a previous backup, run the following command (substituting the filename of the netbox_….gz
you wish to restore) from within the Malcolm installation directory while Malcolm is running:
./scripts/netbox-restore --netbox-restore ./malcolm_netbox_backup_20230110-125756.gz
Note that some of the data in the NetBox database is cryptographically signed with the value of the SECRET_KEY
environment variable in the ./netbox/env/netbox-secret.env
environment file. A restored NetBox backup will not work if this value is different from when it was created.