Skip to content

Latest commit

 

History

History
40 lines (29 loc) · 1.18 KB

README.md

File metadata and controls

40 lines (29 loc) · 1.18 KB
Raw

CsvToEs

This is a POC to parse a CSV in a streaming fashion and store a JSON-object derived from a CSV-line in Elasticsearch, using the Bulk-API of Elasticsearch.

Build

mix escript.build

Load a CSV

./csv_to_es <FILENAME>

Limitations

  • Currently only a ;-separated file is supported which must have a header-line for naming the ES-doc-fields. This project has only been tested with a bagadres-full.csv file downloaded from NLExtract.nl download
  • Elasticsearch is expected to run at localhost
  • As we don't create an _id field explicitly, multiple runs of the tool will create duplicates
  • The batch-size is fixed at 1_000 this figure has been made up with no test or knowledge whatsoever
  • The time-out of 60s has been chose as "large enough" to avoid timeouts
  • No error-handling is implemented
  • The target index is hardcoded to elixir-csv

Tip

Before running this tool you'd best set the number_of_replicas to 0 and the refresh_interval to -1 for the target-elasticsearch-index elixir-csv

Can I use this in Production

Probably not as is