MongoDB Reporting

[kchason]

Proposed changes

Allows exporting of scan results to a MongoDB database for storage and reporting. As there are several options for JSON formatting and other datastore exports exist (e.g., Elasticsearch, Splunk), this seemed like a natural capability to support.

Closes #5700

It utilizes the same issue-tracker-config.yml format with the following options:

mongodb:
  # the connection string to the MongoDB database
  # (e.g., mongodb://root:example@localhost:27017/nuclei?ssl=false&authSource=admin)
  connection-string: ""
  # the name of the collection to store the issues
  collection-name: ""
  # excludes the Request and Response from the results (helps with filesize)
  omit-raw: false
  # determines the number of results to be kept in memory before writing it to the database or 0 to
  # persist all in memory and write all results at the end (default)
  batch-size: 0

It also optionally allows the use of the environment variable MONGO_CONNECTION_STRING as the connection string may contain passwords or other sensitive information that users may not want to store in a config file.

To run this locally:

Run a Docker Compose stack for MongoDB (source):

docker-compose.yml

services:
  mongo:
    image: mongo
    restart: always
    environment:
      MONGO_INITDB_ROOT_USERNAME: root
      MONGO_INITDB_ROOT_PASSWORD: example

  mongo-express:
    image: mongo-express
    restart: always
    ports:
      - 8081:8081
    environment:
      ME_CONFIG_MONGODB_ADMINUSERNAME: root
      ME_CONFIG_MONGODB_ADMINPASSWORD: example
      ME_CONFIG_MONGODB_URL: mongodb://root:example@mongo:27017/
      ME_CONFIG_BASICAUTH: false

Run the stack with:

docker compose up -d

Navigate to http://localhost:8081 and create a nuclei database.

Create the config file reporters.yml (or update one in place) with the contents:

mongodb:
  connection-string: "mongodb://root:example@localhost:27017/nuclei?ssl=false&authSource=admin"
  collection-name: "results"
  omit-raw: false
  batch-size: 0

Run Nuclei with the config, adding any additional parameters to tune the scan as desired.

nuclei -report-config reporters.yml -template-id ssl-issuer,http-missing-security-headers -target hackthissite.org

Checklist

• Pull request is created against the dev branch
• All checks passed (lint, unit/integration/regression tests etc.) with my changes
• I have added tests that prove my fix is effective or that my feature works
• I have added necessary documentation (if appropriate)

[dwisiswant0]

Why do we need "database-name" field when we can just fill it in with a DSN/conn string?

[kchason]

Why do we need "database-name" field when we can just fill it in with a DSN/conn string?

Good call, it now parses it from the connection string as of 5d934aa

[ehsandeep]

@kchason thank you for this addition, whenever you get a chance, please update the docs for the same.

[kchason]

@kchason thank you for this addition, whenever you get a chance, please update the docs for the same.

projectdiscovery/docs#93