Error during import

[roja]

After a reasonably long period of successful importing the following error occurs and halts import entirely:

2017-06-07T12:03:16.494Z - ^[[31merror^[[39m: [whosonfirst] Inability to parse JSON usually means that Who's on First has been cloned without using git-lfs, please see instructions here: https://github.c$
events.js:160
throw er; // Unhandled 'error' event
^

TypeError: Path must be a string. Received undefined
at assertPath (path.js:7:11)
at Object.join (path.js:1211:7)
at /data/src/whosonfirst/src/components/loadJSON.js:11:33
at DestroyableTransform._transform (/data/src/whosonfirst/node_modules/pelias-parallel-stream/index.js:16:5)
at DestroyableTransform.Transform._read (/data/src/whosonfirst/node_modules/through2/node_modules/readable-stream/lib/_stream_transform.js:159:10)
at DestroyableTransform.Transform._write (/data/src/whosonfirst/node_modules/through2/node_modules/readable-stream/lib/_stream_transform.js:147:83)
at doWrite (/data/src/whosonfirst/node_modules/through2/node_modules/readable-stream/lib/_stream_writable.js:337:64)
at clearBuffer (/data/src/whosonfirst/node_modules/through2/node_modules/readable-stream/lib/_stream_writable.js:437:7)
at onwrite (/data/src/whosonfirst/node_modules/through2/node_modules/readable-stream/lib/_stream_writable.js:368:7)
at WritableState.onwrite (/data/src/whosonfirst/node_modules/through2/node_modules/readable-stream/lib/_stream_writable.js:118:5)

npm ERR! Linux 4.4.38-std-1
npm ERR! argv "/usr/bin/nodejs" "/usr/bin/npm" "start"
npm ERR! node v6.10.3

[missinglink]

I take it you have git-lfs installed as per the error message?

Inability to parse JSON usually means that Who's on First has been cloned without using git-lfs, please see instructions here: https://github.com/whosonfirst/whosonfirst-data#git-and-large-files

I think New Zealand (85633345) is the only file in LFS at the moment, could you check that file is valid JSON?

[missinglink]

$ head /data/whosonfirst-data/data/856/333/45/85633345.geojson 
{
  "id": 85633345,
  "type": "Feature",
  "properties": {
    "edtf:cessation":"uuuu",
    "edtf:inception":"uuuu",
    "geom:area":29.187792,
    "geom:bbox":"166.426148132,-47.2899924965,178.577243781,-33.9584981387",
    "geom:latitude":-41.735409,
    "geom:longitude":172.590761,

[trescube]

The initial error is happening when path.join receives an undefined for record.path.

[roja]

root@scw-d180db:~# head /data/whosonfirst/data/856/333/45/85633345.geojson
{
"id": 85633345,
"type": "Feature",
"properties": {
"edtf:cessation":"uuuu",
"edtf:inception":"uuuu",
"geom:area":29.187792,
"geom:bbox":"166.426148132,-47.2899924965,178.577243781,-33.9584981387",
"geom:latitude":-41.735409,
"geom:longitude":172.590761,

[missinglink]

looks like maybe a data issue. @trescube we should really warn here instead of fatal

[trescube]

Sure, it can be warn but first I have to figure out how it's happening and I'm unable to replicate the error. record.path is undefined for some unknown reason. The flow is:

1. read metadata via csv-stream
2. reject null island records
3. load JSON (error happens here)
4. other stuff

I have no idea how undefined is coming thru unless a line in the metafile just doesn't have enough fields as csv-stream returns an empty string when record.path is empty.

There are 2 errors happening here. First, path.join is throwing an error since record.path is undefined. Second, because the path.join error was neither expected nor handled, fs.readFile is throwing an error because full_file_path is undefined which is handled but gives a misleading error message.

@roja if you downloaded the Who's on First data via pelias/whosonfirst npm run download then you won't have git-lfs issues.

[trescube]

@roja Is it your intention to import venues?

[trescube]

@roja We don't currently import who's on first venues into our production environment so unless you really want venues, please set imports.whosonfirst.importVenues to false in our pelias.json config.

I tried importing just the file that's causing an error (wof-venue-cz-latest) but didn't encounter an error which makes me think something else is up with the sheer amount of data in venues.

[roja]

Right. Will give it a retry and see where we get.

[orangejulius]

As this was a venue related issue, and official support for venue importing is a low priority task (see #94), I'm going to mark this closed. If similar issues happen when importing admin records or postal codes, don't hesitate to reach out!