Memory and recursion issues in tilelive-copy with --parts

[jflemer-ndp]

Using tilelive-copy with --parts on a large copy seems to have a few issues. In both cases, I was using:
tilelive-copy --concurrency=24 --withoutprogress --timeout=900000 --minzoom=0 --maxzoom=12 --retry=1000 --parts=$PARTS --part=$n mapnik://./project.xml .../foo.mbtiles

Initially, I tried PARTS=1000, and got a Maximum call stack size exceeded. The stack (now gone, sorry) looked to involve a massive recursion on skip()/get(), right about here:

tilelive/lib/stream-scanline.js

Line 120 in efdfe7d

get(push);

I figured even with the recursion, that I'd avoid blowing the stack with PARTS=20, and I did. But each part ended up getting killed or dying due to out of memory. The same command without --parts at all runs just fine (no out of memory errors).

I did not try the pyramid or list scheme, so this may or may not be limited to scanline.

Here are some of the versions in use:

| +-- tilelive@5.12.2
| | +-- minimist@0.2.0
| | `-- progress-stream@0.5.0
| |   +-- single-line-log@0.3.1
| |   +-- speedometer@0.1.4
| |   `-- through2@0.2.3
| |     +-- readable-stream@1.1.14
| |     | `-- isarray@0.0.1
| |     `-- xtend@2.1.2
| |       `-- object-keys@0.4.0
| +-- tilelive-cache@0.6.5
| | +-- locking-cache@0.2.1
| | `-- tilelive-streaming@0.6.4
| |   `-- async@0.9.2
| `-- tilelive-modules@0.2.1

[rclark]

@jflemer-ndp thanks for the report - @jakepruitt also ran into this recently and we'll see if we can pick it apart this week.