DOR - Domain Ranker

Fast HTTP service which shows a specified domain rank from following providers:

• Alexa
• Majestic
• Umbrella OpenDNS
• Open PageRank
• Tranco List
• Quantcast
• YandexRadar

Can be used as a base for a domain categorization / network filters / suspicious domain detection.

Data is updated once a day automatically.

Supported types of storages:

• Clickhouse (recommended)
• MongoDB
• In-Memory

You can easily add the storage you like by implementing Storage interface.

Installation

Check out the releases page.

HTTP service usage

Use Clickhouse storage located at clickhouse and bind to port 8080:

go run service/dor-http/dor.go \
    -storage=clickhouse \
    -storage-url=tcp://clickhouse:9000 \
    -listen-addr=:8080

Fill database with the data

go run cmd/dor-insert/dor-insert \
    -storage=clickhouse \
    -storage-url=tcp://clickhouse:9000

Docker usage

Project has docker-compose that uses Clickhouse as a storage. Make changes here accordingly if any (folder for data persistence, ports etc).

docker-compose up -d

Client usage

$: curl 127.0.0.1:8080/rank/github.com

{
  "data": "github.com",
  "ranks": [
    {
      "domain": "github.com",
      "rank": 2698,
      "date": "2019-09-07T00:00:00Z",
      "source": "umbrella",
      "raw": ""
    },
    {
      "domain": "github.com",
      "rank": 29,
      "date": "2019-09-07T00:00:00Z",
      "source": "majestic",
      "raw": "29,24,github.com,com,176946,489686,github.com,com,29,24,176096,487221"
    },
    {
      "domain": "github.com",
      "rank": 26,
      "date": "2019-09-07T00:00:00Z",
      "source": "pagerank",
      "raw": ""
    },
    {
      "domain": "github.com",
      "rank": 32,
      "date": "2019-09-07T00:00:00Z",
      "source": "alexa",
      "raw": ""
    },
    {
      "domain": "github.com",
      "rank": 467,
      "date": "2019-09-07T00:00:00Z",
      "source": "yandex-radar",
      "raw": "The world’s leading software development platform · GitHub,github.com,,Сервисы,,,1520000,2340000,,,"
    },
    {
      "domain": "github.com",
      "rank": 43,
      "date": "2019-09-07T00:00:00Z",
      "source": "tranco",
      "raw": ""
    },
    {
      "domain": "github.com",
      "rank": 168,
      "date": "2019-09-07T00:00:00Z",
      "source": "quantcast",
      "raw": ""
    }
  ],
  "timestamp": "2019-09-07T14:32:32.9725943Z"
}