Send multiple files

[efsantana]

I'm trying to send multiple files to the ftp server and I'm usually losing connection after sending the first file. What's the best way to send more than one file to an ftp server.

I'm getting the files as base 64 strings through web sockets using socket.io and trying to send them using the put method.

[xiaomeng]

I tried to use Ftp.raw.mput(), seems it is not supported by jsftp?

[evanplaice]

I'm having the same issue, except I'm using `ftp.put()'

Note: The following code is from the node-ftpsync project.

Here's my upload function:

  // upload a file to the remote server
  upload: function(file, callback) {
    var local = settings.local + file;
    var remote = settings.remote + file;
    fs.readFile(local, function(err, buffer) {
      if(err) {
        console.error(err);
        callback(err);
      }
      else {
        ftp.put(buffer, remote, function(err) {
          if (err) {
            console.error(err);
            callback(err);
          }
          else {
            console.log(file + " - uploaded successfuly");
            callback();
          }
        });
      }
    });
  },

• ftp holds the jsftp instance.

The number of parallel connections are limited via async.mapLimit()

        async.mapLimit(sync.update, settings.connections, utils.upload, function (err) {
          if (err) {
            callback(err, 'updates failed');
          }
          else {
            callback(null, 'updates complete');
          }
        });

• sync.update - contains the list of files queued for upload
• settings.connections - is the max number of connections
• utils.upload - is the upload function I posted above.

For each item in sync.update it calls the utils.upload() function but the maximum number of uploads running in parallel is limited to whatever settings.connections is set to. Currently it only works if settings.connections is set to 1.

It would make sense that a jsftp instance only has the ability to buffer one connection at a time.

If that's the case then maybe it's possible to overcome the limitation by creating a new jsftp instance for each connection by cloning the template jsftp instance each time an action (ex upload) is performed. In theory, if the instance is cloned into a local variable the memory used by the clone should be recovered when the action is complete and the local scope is closed.

Ex:

  upload: function(file, callback) {
    var newFTP = new ftp();
    var local = settings.local + file;
    var remote = settings.remote + file;
    fs.readFile(local, function(err, buffer) {
      if(err) {
        console.error(err);
        callback(err);
      }
      else {
        newFtp.put(buffer, remote, function(err) {
          if (err) {
            console.error(err);
            callback(err);
          }
          else {
            console.log(file + " - uploaded successfuly");
            callback();
          }
        });
      }
    });
  },

[evanplaice]

My assumptions seem to have been correct.

Simply creating a copy of the main ftp instance didn't work and I don't want to delve into the dark art of deep cloning objects so I chose the simpler approach.

Instead I just created a fresh jsftp instance for each upload.

  upload: function(file, callback) {
    var ftp = new jsftp({
      host: settings.host,
      port: settings.port,
      user: settings.user,
      pass: settings.pass
    });
    var local = settings.local + file;
    var remote = settings.remote + file;
    fs.readFile(local, function(err, buffer) {
      if(err) {
        console.error(err);
        callback(err);
      }
      else {
        newFtp.put(buffer, remote, function(err) {
          if (err) {
            console.error(err);
            callback(err);
          }
          else {
            console.log(file + " - uploaded successfuly");
            callback();
          }
        });
      }
    });
  },

For quicker tasks such as rmdir, mkdir, and delete I just limit to one connection and use the main ftp instance. For uploads, downloads I fire off a new jsftp instance for each connection.

I tested this uploading ten copies of a 8MB file with 2, 3, 4, 5, and 10 concurrent connections and it hasn't failed yet. The more connections I added, the more memory my application used but I didn't see any other noticeable side-effects.

IMHO, this issue is solved.

[mrameezraja]

does anyone able to solve the issue without creating a new instance for every file?

[JulienLeal]

This just happens yet. How I fix it ??

[juampi92]

IDK if some of you figured it out, but in I had the same problem, and I tried using a mapSeries instead of parallel. Seems that one FTP session can only do one task at once. If you'd like parallelism, use multiple sessions (or sockets)