{ buffer: false } seems to impose a limit on the number/size of files

[agross]

I've just started with gulp trying to automate mostly product image processing. As you can imagine we have a large number of product images. For my tests I started with 5000 images, exactly 1 MB each (they were generated by a script during my tests).

Setup

My first objective is to just copy the files from source to destination without further processing. As gulp does not need to read the file contents (yet), I pass { buffer: false } to gulp.src:

gulp.task('test', function() {
  return gulp
    .src('source/data/images/*.jpg'), { buffer: false })
    .pipe(gulp.dest('dist/products'));
});

When I run gulp the process starts and then just stops without further information (no "finished" line):

$ gulp test
[21:22:06] Using gulpfile C:\somewhere\gulp-static-site\gulpfile.js
[21:22:06] Starting 'test'...

C:\somewhere\gulp-static-site
$

Results

After that I have exactly 1985 files in my dist/products directory. 1984 of these are 1 MB in size, 1 has 0 bytes. The sum of these files is 2 GB, and that limit seems somewhat special. This is reproducible.

If I pass { buffer: false } (or omit it), all files get copied, but the memory footprint is much higher.

$ gulp test
[21:26:51] Using gulpfile C:\somewhere\gulp-static-site\gulpfile.js
[21:26:51] Starting 'test'...
[21:27:50] Finished 'test' after 58 s

C:\somewhere\gulp-static-site
$

Technical information

I tested both on Windows:

$ node --version
v0.12.0

$ gulp --version
[21:23:39] CLI version 3.8.11
[21:23:39] Local version 3.8.11

And on Fedora 21, where only 985 files are copied:

$ node --version
v0.10.36

$ gulp --version
[22:05:13] CLI version 3.8.11
[22:05:13] Local version 3.8.11

[yocontra]

Probably JS running out of memory which I think was the whole thing streaming mode was supposed to solve. Hmm. Can you try messing with the highWaterMark in gulp.src (its an option passed to through2) and see if that fixes anything?

[agross]

The node executable on Windows has a memory footprint around 70 MB. So I guess memory is not the issue here.

But as you suggested I modified through2's index.js and set highWaterMark: 1. Unfortunately, that doesn't change anything.

What's really strange is that the copy process succeeds if { buffer: false } is unset.

[agross]

FYI, I'm running node 64-bit.

[agross]

I traced this down to an EMFILE error happening while opening the source file:

$ NODE_DEBUG=GFS gulp test

...
GFS 12840: open done { [Error: EMFILE, open 'C:\somewhere\gulp-static-site\dist\products\2838.jpg']
  errno: -4066,
  code: 'EMFILE',
  path: 'C:\\somewhere\\gulp-static-site\\dist\\products\\2838.jpg' } undefined
GFS 12840: enqueue 1 OpenReq { path: 'C:\\somewhere\\gulp-static-site\\dist\\products\\2838.jpg',
  flags: 'w',
  mode: 33206,
  cb: [Function],
  done: [Function],
  failures: 1 }
GFS 12840: fds { fd3: 'C:\\somewhere\\gulp-static-site\\dist\\products\\1.jpg',
  fd4: 'C:\\somewhere\\gulp-static-site\\dist\\products\\10.jpg',
  fd5: 'C:\\somewhere\\gulp-static-site\\dist\\products\\100.jpg',
  fd6: 'C:\\somewhere\\gulp-static-site\\dist\\products\\1000.jpg',
  fd7: 'C:\\somewhere\\gulp-static-site\\dist\\products\\1001.jpg',
... (lots more)

GFS 12840: [ { path: 'C:\\somewhere\\gulp-static-site\\dist\\products\\2838.jpg',
    flags: 'w',
    mode: 33206,
    cb: [Function],
    done: [Function],
    failures: 1 } ]

assert.js:86
  throw new assert.AssertionError({
        ^
AssertionError: 1 == 0
    at process.<anonymous> (C:\somewhere\gulp-static-site\node_modules\gulp\node_modules\vinyl-fs\node_modules\graceful-fs\graceful-fs.js:28:12)
    at process.emit (events.js:129:20)

I wonder, why does node not close the destination fd once the file has been copied?

[agross]

Possibly related: nodejs/node-v0.x-archive#6041, isaacs/node-graceful-fs#20

[yocontra]

Moving to gulpjs/vinyl-fs#56