Skip to content

cert-se/megatron-java

Repository files navigation

Megatron - A System for Abuse- and Incident Handling

Megatron is a tool implemented by CERT-SE which collects and analyses log files with bad machines, e.g. from Shadowserver. Apart from abuse mail handling, Megatron can be used to collect statistics, convert log files, and do log file analysis during incident handling.

Major features:

  • Flexible parsing: many input sources are supported without any coding. Regular expressions are used to extract tokens which are bound to variables. A variable corresponds to a field in the database and can be used in for example filters and templates.
  • Organization matching: Megatron tries to match every input record to an organization using its IP-blocks, ASNs, and domain names. Each organization have a priority which makes it possible to filter out important records.
  • Database: input records are stored in a relational database (MySQL is default), making data mining possible.
  • Filtering: a wide variety of filters can be used to filter input records.
  • Data decoration: data may be added to the record before it is saved to the database. Types of lookups: IP to ASN, IP to country code, IP to hostname, hostname to IP, URL to hostname, hostname to country code, and IP to geolocation (longitude, latitude, and city). Three of MaxMind's databases are supported (ASN, City, and Country).
  • Performance: large data volumes can be handled. All lookups except DNS queries are local, for instance ASN lookups uses database queries by importing a BGP routing table to the database.

For all features, see readme-general.txt

Example of Usage

Convert a file with IP addresses to a pipe-separated file with IP, AS, country code, and hostname:

$ megatron.sh --job-type ip-flowing --export --no-db test-data/multiple-ips-per-line3.log

As above but output file is tab-separated and contains geolocation data:

$ megatron.sh --job-type ip-flowing-verbose --export --no-db test-data/multiple-ips-per-line3.log

Prints information about two IP addresses (hostnames or URLs works as well):

$ megatron.sh --whois 130.238.7.133 217.21.237.35
IP              | AS     | CC | Hostname                                      | AS Name                                       | Organization
130.238.7.133   | 1653   | SE | live.webb.uu.se                               | SUNET Swedish University Network              | Uppsala universitet
217.21.237.35   | 29672  | SE | stockholm.se                                  | S:t Erik Kommunikation AB                     | Stockholms stad

Prints information about URLs in specified file:

$ megatron.sh --whois infection-urls.txt

Process file and save result in the database:

$ megatron.sh --job-type shadowserver-drone test-data/2009-06-08-drone-report-se.log

Preview of mail to be sent:

$ megatron.sh --job shadowserver-drone_2013-06-22_160142 --id 4242 --mail-dry-run

Send emails for the job:

$ megatron.sh --job shadowserver-drone_2009-06-22_160142 --id 4242 --mail

Megatron is designed to automate day-to-day abuse handling. Running Megatron from the command line is only necessary in special cases.

Download

Binaries, config files, and source code are included in this tarball:

MD5: 816a6b9644cc929822e861221b07ea37
SHA1: 77c8c1bedd6318574b34d85daaaae8a6d94e6b1e

Installation

Megatron is cross-platform as it is implemented in Java and Python. It has been tested on Windows 7 and Ubuntu 12.04. Megatron requires a MySQL database. Do not worry, the installation is easy and described in detail in this document: readme-install.txt

Documentation

To get a grasp of Megatron, we recommend skimming the following:

Other resources are this presentation and the projects Wiki.

License

Megatron is distributed under the terms of the Apache Software Foundation license version 2.0, which is included in the file LICENSE in the root of the project.

Support and Warranty

CERT-SE does not offer any support, and Megatron is provided "as is" without warranty of any kind.

However, we are a bunch of nerds at CERT-SE and we would love to hear from you and discuss technical stuff. Send feedback to cert@cert.se. Our PGP keys can be found on CERT-SE's contact page.

Mailing List

Megatron Hacking is the mailing list for Megatron. Join by sending an empty email to:

megatron-hacking+subscribe@googlegroups.com

Or you can go to the following URL:

https://groups.google.com/group/megatron-hacking/subscribe

The mailing list is for discussions about Megatron (installation problems, bug fixes, feature requests, etc.). Keep in mind that the list is public.