allow splitting out indexes by other field values

[mmguero]

As of release v24.01.0, the MALCOLM_NETWORK_INDEX_PATTERN and MALCOLM_NETWORK_INDEX_SUFFIX environment variables allow splitting out Suricata and Zeek to a different index pattern from the one Arkime creates.

It may be useful to add another replacer to the pattern definable in either one or the other of these variables (probably the suffix?) to further allow it to be split out based on the event.provider variable (suricata vs. zeek, etc.).

[mmguero]

Values from event fields will now be expanded in the logstash filter that figures out the index to use when writing documents. From opensearch.env, MALCOLM_NETWORK_INDEX_SUFFIX and MALCOLM_OTHER_INDEX_SUFFIX now support expanding dot-separated field names to their values.

So, for example:

MALCOLM_NETWORK_INDEX_SUFFIX={{event.provider}}-{{event.dataset}}-%{%y%m%d}

could produce something like this:

$ docker compose exec api curl -sSL 'localhost:5000/mapi/indices' | jq -r '.indices[].index' | sort | grep sessions
arkime_sessions3-200428
arkime_sessions3-suricata-alert-200428
arkime_sessions3-zeek-analyzer-200428
arkime_sessions3-zeek-bestguess-200428
arkime_sessions3-zeek-cip-200428
arkime_sessions3-zeek-conn-200428
arkime_sessions3-zeek-cotp-200428
arkime_sessions3-zeek-dce_rpc-200428
arkime_sessions3-zeek-dhcp-200428
arkime_sessions3-zeek-dns-200428
arkime_sessions3-zeek-dpd-200428
arkime_sessions3-zeek-enip-200428
arkime_sessions3-zeek-files-200428
arkime_sessions3-zeek-http-200428
arkime_sessions3-zeek-ja4ssh-200428
arkime_sessions3-zeek-kerberos-200428
arkime_sessions3-zeek-known_hosts-200428
arkime_sessions3-zeek-known_modbus-200428
arkime_sessions3-zeek-known_services-200428
arkime_sessions3-zeek-login-200428
arkime_sessions3-zeek-modbus-200428
arkime_sessions3-zeek-modbus_detailed-200428
arkime_sessions3-zeek-notice-200428
arkime_sessions3-zeek-ntlm-200428
arkime_sessions3-zeek-pe-200428
arkime_sessions3-zeek-rdp-200428
arkime_sessions3-zeek-rfb-200428
arkime_sessions3-zeek-s7comm-200428
arkime_sessions3-zeek-s7comm_read_szl-200428
arkime_sessions3-zeek-s7comm_upload_download-200428
arkime_sessions3-zeek-signatures-200428
arkime_sessions3-zeek-signatures-240828
arkime_sessions3-zeek-smb_files-200428
arkime_sessions3-zeek-smb_filessmb_cmd-200428
arkime_sessions3-zeek-smb_mapping-200428
arkime_sessions3-zeek-software-200428
arkime_sessions3-zeek-ssh-200428
arkime_sessions3-zeek-ssl-200428
arkime_sessions3-zeek-tds-200428
arkime_sessions3-zeek-weird-200428
arkime_sessions3-zeek-x509-200428

Of course, these ONLY apply to events that get indexed through the logstash pipeline (not ones indexed by Arkime capture) itself and for which the user isn't somehow overriding the index value in another way.

Also, updated the documentation on environment variables:

• The following variables control the OpenSearch indices to which network traffic metadata are written. Changing them from their defaults may cause logs from non-Arkime data sources (i.e., Zeek, Suricata) to not show up correctly in Arkime.
  • MALCOLM_NETWORK_INDEX_PATTERN - Index pattern for network traffic logs written via Logstash (default is arkime_sessions3-*)
  • MALCOLM_NETWORK_INDEX_TIME_FIELD - Default time field to use for network traffic logs in Logstash and Dashboards (default is firstPacket)
  • MALCOLM_NETWORK_INDEX_SUFFIX - Suffix used to create index to which network traffic logs are written
    • supports Ruby strftime strings in ％{}) (e.g., hourly: ％{％y％m％dh％H}, twice daily: ％{％P％y％m％d}, daily (default): ％{％y％m％d}, weekly: ％{％yw％U}, monthly: ％{％ym％m}
    • supports expanding dot-delimited field names in {{ }} (e.g., {{event.provider}}％{％y％m％d})
• The following variables control the OpenSearch indices to which other logs (third-party logs, resource utilization reports from network sensors, etc.) are written.
  • MALCOLM_OTHER_INDEX_PATTERN - Index pattern for other logs written via Logstash (default is malcolm_beats_*)
  • MALCOLM_OTHER_INDEX_TIME_FIELD - Default time field to use for other logs in Logstash and Dashboards (default is @timestamp)
  • MALCOLM_OTHER_INDEX_SUFFIX - Suffix used to create index to which other logs are written (with the same rules as MALCOLM_NETWORK_INDEX_SUFFIX above) (default is ％{％y％m％d})