Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature request: support for aws-sdk v3 #241

Open
neverendingqs opened this issue Jun 14, 2021 · 4 comments · May be fixed by #277
Open

Feature request: support for aws-sdk v3 #241

neverendingqs opened this issue Jun 14, 2021 · 4 comments · May be fixed by #277

Comments

@neverendingqs
Copy link

neverendingqs commented Jun 14, 2021

The S3 client for v3 of the SDK signature has changed when trying to create a read stream.

The signature has changed so that instead of

stream: function(offset,length) {
var d = {};
for (var key in params)
d[key] = params[key];
d.Range = 'bytes='+offset+'-' + (length ? length : '');
return client.getObject(d).createReadStream();
}
, it would have to look something like

stream: function(offset,length) { 
  var d = {}; 
  for (var key in params) 
    d[key] = params[key]; 
  d.Range = 'bytes='+offset+'-' + (length ? length : ''); 
  return new Promise(function(resolve,reject) {
    client.getObject(d, function(err, data) {
      if (err)
        reject(err);
      else
        resolve(data.Body);
    });
 });
}

, although it's unclear how to distinguish between a v2 client and a v3 client and if the caller would be happy with a Promise being returned instead of the stream directly.

If a v3 client is passed in, the following error is thrown in the current version of unzipper:

ERROR TypeError: client.getObject(...).createReadStream is not a function
@openwebsolns
Copy link

Any updates on this issue?

@alvin-nt
Copy link

alvin-nt commented May 10, 2023

AFAIK the getObject method of SDK v3 does not support the 'synchronous' way of creating stream; the response.Body is available asynchronously (via callback/promise). Should the support will be implemented, some refactoring of existing code will be required. This may involve some breaking changes, unfortunately, with node v8.

CMIIW, the SDK v3 also bumps the minimum supported node version to v10

@alvin-nt alvin-nt linked a pull request May 10, 2023 that will close this issue
@alvin-nt
Copy link

I have created PR #277. Please check if any changes are required.

Suggestions for test scripts are welcome, since I have yet to figure out how to implement one.

@Sljux
Copy link

Sljux commented Mar 18, 2024

Workaround I did that worked:

const directory = await unzipper.Open.custom({
    async size() {
        const info = await s3.send(
            new HeadObjectCommand({
                Bucket: bucket,
                Key: key,
            }),
        );

        return info.ContentLength;
    },

    stream(offset, length) {
        const stream = new PassThrough();

        s3.send(
            new GetObjectCommand({
                Bucket: bucket,
                Key: key,
                Range: `bytes=${offset}-${length ?? ""}`,
            }),
        )
            .then((response) => {
                response.Body.pipe(stream);
             })
            .catch((error) => {
                stream.emit("error", error);
            });

        return stream;
     },
});

alice-was-here added a commit to alice-was-here/node-unzipper that referenced this issue Jun 25, 2024
This should resolve this issue: ZJONSSON#241
I have used a variation of @Sljux's solution to make the interface compatible with the current `stream` interface.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants