Skip to content

Commit

Permalink
Update parser to support all available reports.
Browse files Browse the repository at this point in the history 
Update to existing test cases to match current report types.

New tests for added report types.

pycodestyle fixes

add testdata licenses

pycodestyle fix

Added reports parameter

Suggested changes to the parser

Proposed details for the release

Test script updates for suggested changes

Test input updates

Realign columns

Update compromised_website.csv

Update scan_adb.csv

Update scan_adb.csv

Update scan_ftp.csv

Update scan_ipp.csv

Update scan_snmp.csv

Realign columns

Remove duplicates

Changed malware.name to extra.infection

Updated SPDX-FileCopyrightText

shadowserver api: document and warn on old parameter

document the old parameter `country` and its status
warn if used
adapt the test

DOC: fix NEWS entry of PR#2143

Added the sector field to scan_amqp, scan_cwmp, and scan_vnc.

Copyright and raw field updates

Added the sector field to scan_amqp, scan_cwmp, and scan_vnc.

Copyright updates

Added phish_url and scan_modbus reports.

Update source.url and source.fqdn for phish_url and malware_url reports.  Update classification.taxonomy and classification.type for scan_modbus report.

* additional field type validation changes
* added count, bytes, duration, avg_pps, and max_pps fields to event_honeypot_ddos_amp
* added 'protocol.application': 'https' to scan_ssl, scan_ssl_freak, and scan_ssl_poodle
* added 'extra.tag' to scan_* and device_id

Replaced scan_modbus with scan_ics

Addeed event4_honeypot_ddos, event4_honeypot_ddos_target, scan_dvr_dhcpdiscover, and scan_socks.

Tests for event4_honeypot_ddos.

Tests for event4_honeypot_ddos_target.

Tests for scan_dvr_dhcpdiscover.

Tests for scan_socks.

Rename file

Rename file

update:scan_mdns, scan_smb, and special; add:scan_ddos_middle_box

cleanup renamed license files

updated scan_mdns test files

updated scan_smb test files

updated special test files

add scan_ddos_middlebox test files

add scan_ddos_middlebox test

updated schema

Updated scan_smb tests

Updated scan_ntp tests

Updated scan_snmp tests

New scan_docker test

New scan_kubernetes test

New scan_mysql test

Updated report schema for June 2022

Added scan_epmd test

Revert "Added scan_epmd test"

This reverts commit 01edea1.

Revert: Fix for recover_line method as commited in #2192

Added scan_couchdb

Test case for scan_couchdb

Added scan6_rpd

Added/updated README with maintainer details

Restored feed names and classification.identifiers to minimize upgrade impact.

Merge repair

pycodestyle repairs

codespell fixes

license compliance fixes

pycodestyle fixes

Feed configuration updates for compatibility with the original.

Added scan_postgres test

Added additional IPv6 aliases

Fix for recover_line method as commited in #2192
  • Loading branch information
@elsif2
elsif2 committed Aug 1, 2022
1 parent e0086c6 commit 6194014
Show file tree
Hide file tree
Showing 183 changed files with 7,797 additions and 4,937 deletions.
126 changes: 126 additions & 0 deletions NEWS.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -9,11 +9,53 @@ NEWS
This file lists all changes which have an affect on the administration of IntelMQ and contains steps that you need to be aware off for the upgrade.
Please refer to the changelog for a full list of changes.


3.1.0 Feature release (unreleased)
----------------------------------

### Requirements

### Bots
#### ShadowServer Reports API collector
The misleading `country` parameter has been depreciated and a `reports` parameter has been added.
The backwards-compatibility will be removed in IntelMQ version 4.0.0.
See the [Shadowserver Reports API bot's documentation](https://intelmq.readthedocs.io/en/latest/user/bots.html#shadowserver-reports-api).

#### ShadowServer parser
Previously, mappings used a mix of `extra.naics` and `extra.source.naics`. The parser has been updated to use the more specific term (`extra.source.naics`).

A number of the _classification.identifier_ values have been updated to follow a common naming convention based on their canonical report name:

| before IntelMQ 3.1.0 | in IntelMQ 3.1.0 and higher |
| --- | --- |
| accessible-adb | open-adb |
| accessible-afp | open-afp |
| accessible-amqp | open-amqp |
| accessible-ard | open-ard |
| accessible-cisco-smart-install | open-cisco-smart-install |
| accessible-coap | open-coap |
| accessible-ftp | open-ftp |
| accessible-hadoop | open-hadoop |
| accessible-http | open-http |
| accessible-msrdpeudp | open-rdpeudp |
| accessible-radmin | open-radmin |
| accessible-rsync | open-rsync |
| accessible-ubiquiti-discovery-service | open-ubiquiti |
| amplification-ddos-victim | honeypot-ddos-amp |
| blacklisted-ip | blocklist |
| dns-open-resolver | open-dns |
| honeypot-http-scan | honeypot-http-scan |
| ics | honeypot-ics-scan |
| ntp-monitor | open-ntpmonitor |
| ntp-version | open-ntp |
| open-db2-discovery-service | open-db2 |
| open-ike | open-isakmp |
| open-ldap | open-ldap-tcp |
| open-natpmp | open-nat-pmp |
| open-netbios-nameservice | open-netbios |
| open-netis | open-netis-router |
| sinkholedns | sinkhole-dns |

### Tools

### Data Format
Expand All @@ -37,6 +79,90 @@ The parameter `timeout` has been merged into `redis_cache_ttl`.
### Libraries

### Postgres databases
The following statements optionally update existing data for the harmonization classification changes:
```sql
UPDATE events
SET "classification.identifier" = 'open-adb'
WHERE "classification.identifier" = 'accessible-adb';
UPDATE events
SET "classification.identifier" = 'open-afp'
WHERE "classification.identifier" = 'accessible-afp';
UPDATE events
SET "classification.identifier" = 'open-amqp'
WHERE "classification.identifier" = 'accessible-amqp';
UPDATE events
SET "classification.identifier" = 'open-ard'
WHERE "classification.identifier" = 'accessible-ard';
UPDATE events
SET "classification.identifier" = 'open-cisco-smart-install'
WHERE "classification.identifier" = 'accessible-cisco-smart-install';
UPDATE events
SET "classification.identifier" = 'open-coap'
WHERE "classification.identifier" = 'accessible-coap';
UPDATE events
SET "classification.identifier" = 'open-ftp'
WHERE "classification.identifier" = 'accessible-ftp';
UPDATE events
SET "classification.identifier" = 'open-hadoop'
WHERE "classification.identifier" = 'accessible-hadoop';
UPDATE events
SET "classification.identifier" = 'open-http'
WHERE "classification.identifier" = 'accessible-http';
UPDATE events
SET "classification.identifier" = 'open-rdpeudp'
WHERE "classification.identifier" = 'accessible-msrdpeudp';
UPDATE events
SET "classification.identifier" = 'open-radmin'
WHERE "classification.identifier" = 'accessible-radmin';
UPDATE events
SET "classification.identifier" = 'open-rsync'
WHERE "classification.identifier" = 'accessible-rsync';
UPDATE events
SET "classification.identifier" = 'open-ubiquiti'
WHERE "classification.identifier" = 'accessible-ubiquiti-discovery-service';
UPDATE events
SET "classification.identifier" = 'honeypot-ddos-amp'
WHERE "classification.identifier" = 'amplification-ddos-victim';
UPDATE events
SET "classification.identifier" = 'blocklist'
WHERE "classification.identifier" = 'blacklisted-ip';
UPDATE events
SET "classification.identifier" = 'open-dns'
WHERE "classification.identifier" = 'dns-open-resolver';
UPDATE events
SET "classification.identifier" = 'honeypot-http-scan'
WHERE "classification.identifier" = 'honeypot-http-scan';
UPDATE events
SET "classification.identifier" = 'honeypot-ics-scan'
WHERE "classification.identifier" = 'ics';
UPDATE events
SET "classification.identifier" = 'open-ntpmonitor'
WHERE "classification.identifier" = 'ntp-monitor';
UPDATE events
SET "classification.identifier" = 'open-ntp'
WHERE "classification.identifier" = 'ntp-version';
UPDATE events
SET "classification.identifier" = 'open-db2'
WHERE "classification.identifier" = 'open-db2-discovery-service';
UPDATE events
SET "classification.identifier" = 'open-isakmp'
WHERE "classification.identifier" = 'open-ike';
UPDATE events
SET "classification.identifier" = 'open-ldap-tcp'
WHERE "classification.identifier" = 'open-ldap';
UPDATE events
SET "classification.identifier" = 'open-nat-pmp'
WHERE "classification.identifier" = 'open-natpmp';
UPDATE events
SET "classification.identifier" = 'open-netbios'
WHERE "classification.identifier" = 'open-netbios-nameservice';
UPDATE events
SET "classification.identifier" = 'open-netis-router'
WHERE "classification.identifier" = 'open-netis';
UPDATE events
SET "classification.identifier" = 'sinkhole-dns'
WHERE "classification.identifier" = 'sinkholedns';
```


### Bots
Expand Down
3 changes: 2 additions & 1 deletion docs/user/bots.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -654,9 +654,10 @@ The Cache is required to memorize which files have already been processed (TTL n

**Configuration Parameters**

* `country`: The country you want to download the reports for
* `country`: **Deprecated:** The country you want to download the reports for. Will be removed in IntelMQ version 4.0.0, use *reports* instead.
* `apikey`: Your Shadowserver API key
* `secret`: Your Shadowserver API secret
* `reports`: A list of strings or a comma-separated list of the mailing lists you want to process.
* `types`: A list of strings or a string of comma-separated values with the names of report types you want to process. If you leave this empty, all the available reports will be downloaded and processed (i.e. 'scan', 'drones', 'intel', 'sandbox_connection', 'sinkhole_combined'). The possible report types are equivalent to the file names given in the section :ref:`Supported Reports <shadowserver-supported-reports>` of the Shadowserver parser.
* **Cache parameters** (see in section :ref:`common-parameters`, the default TTL is set to 10 days)

Expand Down
9 changes: 9 additions & 0 deletions intelmq/bots/collectors/shadowserver/README.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
<!--
SPDX-FileCopyrightText: 2022 The Shadowserver Foundation
SPDX-License-Identifier: AGPL-3.0-or-later
-->

This module is maintained by [The Shadowserver Foundation](https://www.shadowserver.org/).

Please contact intelmq@shadowserver.org with any issues or concerns.

9 changes: 9 additions & 0 deletions intelmq/bots/parsers/shadowserver/README.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
<!--
SPDX-FileCopyrightText: 2022 The Shadowserver Foundation
SPDX-License-Identifier: AGPL-3.0-or-later
-->

This module is maintained by [The Shadowserver Foundation](https://www.shadowserver.org/).

Please contact intelmq@shadowserver.org with any issues or concerns.

Loading

0 comments on commit 6194014

Please sign in to comment.