Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Resumable uploads aren't tested #825

Closed
stephenplusplus opened this issue Aug 26, 2015 · 0 comments
Closed

Resumable uploads aren't tested #825

stephenplusplus opened this issue Aug 26, 2015 · 0 comments
Assignees
Labels
api: storage Issues related to the Cloud Storage API.

Comments

@stephenplusplus
Copy link
Contributor

We kind of are, but not correctly. We need to actually destroy the event mid-stream. Our test should look more like this:

it('should resume an upload after an interruption', function(done) {
  fs.stat(files.big.path, function(err, metadata) {
    assert.ifError(err);

    // Use a random name to force an empty ConfigStore cache.
    var file = bucket.file('LargeFile' + Date.now());
    var fileSize = metadata.size;

    upload({ interrupt: true }, function(err) {
      assert.strictEqual(err.message, 'Interrupted.');

      upload({ interrupt: false }, function(err) {
        assert.ifError(err);

        assert.equal(file.metadata.size, fileSize);
        file.delete(done);
      });
    });

    function upload(opts, callback) {
      var ws = file.createWriteStream();
      var sizeStreamed = 0;

      fs.createReadStream(files.big.path)
        .pipe(through(function(chunk, enc, next) {
          sizeStreamed += chunk.length;

          if (opts.interrupt && sizeStreamed >= fileSize / 2) {
            // stop sending data half way through.
            this.push(chunk);
            this.destroy();
            ws.destroy(new Error('Interrupted.'));
          } else {
            this.push(chunk);
            next();
          }
        }))
        .pipe(ws)
        .on('error', callback)
        .on('finish', callback);
    }
  });
});

But more importantly, the file we're using is apparently too small. Either the chunks aren't actually being emitted by request (I think they are, though, cause they're supposed to be drained in 16kb chunks), or GCS doesn't recognize storing any data for a resumable upload until it meets some threshold.

Our test file is 3MB and cuts off half way though. I tried with 10MB, 20MB, 31MB, and 42MB. 42 and higher were the only ones that got a response from the API call to see where the request left off:

PUT {resumable_uri}
Content-Length: 0
Content-Range: bytes */*

(Note that the amount stored by the API would have been halved (so 21 when 42 worked))

// @jgeewax

@stephenplusplus stephenplusplus added api: storage Issues related to the Cloud Storage API. testing labels Aug 26, 2015
@callmehiphop callmehiphop self-assigned this Aug 28, 2015
sofisl pushed a commit that referenced this issue Nov 11, 2022
sofisl pushed a commit that referenced this issue Nov 11, 2022
🤖 I have created a release \*beep\* \*boop\*
---
## [4.1.0](https://www.github.com/googleapis/nodejs-dialogflow/compare/v4.0.0...v4.1.0) (2021-06-15)


### Features

* added Automated agent reply type and allow cancellation flag for partial response feature ([#825](https://www.github.com/googleapis/nodejs-dialogflow/issues/825)) ([5b128d9](https://www.github.com/googleapis/nodejs-dialogflow/commit/5b128d97eeb63e7d7666e800e5a62789f6f56a65))
---


This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).
sofisl pushed a commit that referenced this issue Jan 17, 2023
Co-authored-by: release-please[bot] <55107282+release-please[bot]@users.noreply.github.com>
sofisl pushed a commit that referenced this issue Jan 24, 2023
PiperOrigin-RevId: 413728495
Source-Link: googleapis/googleapis@10f96cb
Source-Link: googleapis/googleapis-gen@f464864
Copy-Tag: eyJwIjoiLmdpdGh1Yi8uT3dsQm90LnlhbWwiLCJoIjoiZjQ2NDg2NDI5MmViNmIwNTVmMWUxZDg0MDhhZmY4NzgxN2NjYTY0YyJ9
See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md
See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md
Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com>
Co-authored-by: Jeff Ching <chingor@google.com>
Co-authored-by: nicain <nicain.seattle@gmail.com>
Co-authored-by: Benjamin E. Coe <bencoe@google.com>
sofisl pushed a commit that referenced this issue Jan 25, 2023
PiperOrigin-RevId: 413728495
Source-Link: googleapis/googleapis@10f96cb
Source-Link: googleapis/googleapis-gen@f464864
Copy-Tag: eyJwIjoiLmdpdGh1Yi8uT3dsQm90LnlhbWwiLCJoIjoiZjQ2NDg2NDI5MmViNmIwNTVmMWUxZDg0MDhhZmY4NzgxN2NjYTY0YyJ9
See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md
See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md
Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com>
Co-authored-by: Jeff Ching <chingor@google.com>
Co-authored-by: nicain <nicain.seattle@gmail.com>
Co-authored-by: Benjamin E. Coe <bencoe@google.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
api: storage Issues related to the Cloud Storage API.
Projects
None yet
Development

No branches or pull requests

2 participants