This repository has been archived by the owner on Jul 10, 2019. It is now read-only.

Behemoth Modules

Jump to bottom

Julien Nioche edited this page Apr 8, 2015 · 10 revisions

Core module - Load documents into Behemoth, inspect Behemoth output
GATE module - Process documents using GATE
IO module - Load documents from Nutch or WARC
Language ID - Identifies the language of Behemoth documents
Mahout module - Generates vectors for clustering with Mahout
SOLR module - Load Behemoth documents into a SOLR server
Tika module - Parse Behemoth documents using Tika
UIMA module - Process documents using UIMA

External modules

CommonCrawl - Imports documents from the CommonCrawl dataset on Amazon S3 and converts them to the BehemothDocument format for further processing. Supports the 2012 format as well as the old one.
Text Classification - Classifies Behemoth documents with a model generated by our Text Classification API.
ElasticSearch - Sends documents to ElasticSearch for indexing.