DNS Log Analyzer

DNS Log Analyzer is a powerful tool designed to provide enhanced understanding and in-depth analysis of log files generated by a DNS server. This project efficiently monitors, collects, enriches, and indexes DNS log data for comprehensive analysis using Elasticsearch queries.

Table of Contents

• Prerequisites
• Installation
• Configuration
• Usage
• Features
• How It Works
• First Test Run
• License

Prerequisites

Before running DNS Log Analyzer, ensure that you have the following prerequisites installed:

• Java: The program is written in Java, so you need Java Runtime Environment (JRE) or Java Development Kit (JDK) installed on your system. You can download Java from Oracle's website.

• RabbitMQ: DNS Log Analyzer uses RabbitMQ as a message broker to efficiently transfer log data between Producers and Consumers. You can download and install RabbitMQ from the official RabbitMQ website.

• Elasticsearch: Elasticsearch is used for indexing and storing log data. You can download and install Elasticsearch from the official Elasticsearch website.

• Apache Maven - You can download and install Maven from the official Maven website

Installation

1. Clone this repository to your local machine using the following command:

   git clone https://github.com/iso53/dns-log-listener-analyzer.git
   

2. Navigate to the project directory:

   cd dns-log-listener-analyzer
   

3. Build the project using your preferred Java build tool (e.g., Maven or Gradle). For Maven, you can use the following command:

   mvn clean install
   

This will compile the source code and generate the necessary executable JAR files.

Configuration

To configure DNS Log Analyzer, follow these steps:

1. Open the config.txt file located in the project's root directory.

2. In the <start_log_files> section, add the directories containing DNS log files that you want to monitor. You can specify these directories manually or use the program's CLI to add them.

3. Save the config.txt file.

Usage

To use DNS Log Analyzer, follow these steps:

1. Ensure that RabbitMQ and Elasticsearch are running on your system on default ports.

   On default RabbitMQ should be runing on port 15672 and Elasticsearch should be running on port 9200

2. Run the program by running the following commands on terminal in your project folder:

   1. cd LogListener

   2. mvn clean package

   3. java -jar target/LogListener-1.0-SNAPSHOT.jar
      

The DirectoryWatcher threads will start monitoring the specified directories for log file changes.

Log entries from the monitored files will be continuously collected, enriched with additional information, and indexed into Elasticsearch.

Analyze the log data using Elasticsearch queries to gain insights into DNS server activity.

Features

DNS Log Analyzer offers a range of features to facilitate effective log analysis:

• Multithreaded Architecture: The project efficiently utilizes multithreading to monitor multiple directories and log files simultaneously, improving performance and responsiveness.

• File Locking: File locking mechanisms are implemented to ensure exclusive access to log files while reading, preventing race conditions and data corruption.

• Continuous Data Collection: Log entries are continuously collected and sent to RabbitMQ, ensuring that no log data is missed, even during high traffic periods.

• Data Enrichment: Log entries are enriched with additional information such as local IP, host address, and MAC address. This enrichment enhances the quality of data available for analysis.

• Data Persistence: The program saves the status of each Watcher in the config.txt file to prevent data loss in case of unexpected shutdowns or interruptions.

• Elasticsearch Integration: Log entries are indexed into Elasticsearch, enabling powerful search and analysis using Elasticsearch queries.

How It Works

The DNS Log Analyzer project follows a straightforward workflow:

• Configuration: Directories containing DNS log files are specified in the config.txt file.

• Monitoring: DirectoryWatcher threads monitor the specified directories for changes. When a change is detected, a Watcher is created for the log file.

• Data Collection: Each Watcher continuously collects log entries from the log file, ensuring that all changes are captured and then sends them to RabbitMQ Queue, (acting as a message broker for efficient and reliable data transfer).

• Data Enrichment: Consumers read log entries from RabbitMQ queue, enrich them with additional information, such as local IP, host address, and MAC address.

• Data Indexing: Enriched log entries are then indexed into Elasticsearch for efficient storage and retrieval.

• Analysis: Users can analyze the log data using Elasticsearch queries, gaining valuable insights into DNS server activity.

This comprehensive workflow ensures that log data is seamlessly processed, enriched, and made available for in-depth analysis while maintaining data integrity and reliability through RabbitMQ integration.

First Test Run

1. Find the foo.log file in the root directory of the project and copy it's path.

2. Paste the path in between <start_log_files> and <end_log_files> inside the config.txt folder.

3. Run the program.

4. Run the create_mock_dns_log_data.py to create mock DNS logs with the following command.

   python create_mock_dns_log_data.py

5. This will continously generate DNS logs until the program is stopped. You can stop the program by pressing CTRL + C.

6. Check the Elasticsearch indexes to see the data. First time? Use the Multi Elasticsearch Head chrome extension to see the Elasticsearch data. No installation required!

License

This project is licensed under the GNU General Public License v3.0.