[storage-blob] processing downloadBlockBlobResponse.readableStreamBody stucks/hangs - and doesn't throw #2

[PedroFonsecaDEV]

• Package Name:
• Package Version:
• Operating system:
• [x ] nodejs
  • version: 16
• browser
  • name/version:
• [x ] typescript
  • version: 4.4.4
• Is the bug related to documentation in
  • README.md
  • source code documentation
  • SDK API docs on https://docs.microsoft.com

Describe the bug
#16987 should be re-opened.
Error is still happening.

No events or errors occurs, the for await of just pre mature exit the process.
code:

get the readableStream from the download method and then:
 const parser = readableStream.pipe(csvParser());
        for await (const row of parser) {
                console.log(row)
        }

. The best way of reproducing is disconnecting your computer from the network when reading rows from the storage account test with a huge file of at least 500mb.

To Reproduce
Steps to reproduce the behavior:

1. The best way to reproduce is disconnecting your computer from the network during the process.(500mb file)

Expected behavior
throw an error upon disconnecting.
Screenshots
If applicable, add screenshots to help explain your problem.

Additional context
Add any other context about the problem here.

[EmmaZhu]

Hi @PedroFonsecaDEV ,

Current retry only happens when it meets error before or during getting the response headers. If the network failure happens when downloading the response body, neither of timeout or retry logic would work.

Besides that, it's not recommend downloading a large file with only one download operation. It'd be more robust to split a large blob to small pieces, and download each piece with one download call.

Following is a sample shows how to split a large to small piece. Please be noted, it's only a sample without fully test, it's just used to show you the possible workaround logic. Please do make fully validation before using it in your production.

const fd = await fs.open(path, 'w');

  const propertes = await blockBlobClient.getProperties();
  const buffer = Buffer.alloc(4 * 1024 * 1024);
  const count = 4194304;

  for (let offset = 0; offset < propertes.contentLength!;){
    const downloadCount = Math.min(propertes.contentLength! - offset, count);
    offset += downloadCount;
    let retryMax = 3;
    let shouldRetry = true;

    while (retryMax >= 0 && shouldRetry) {
      try {
        await blockBlobClient.downloadToBuffer(buffer, offset, downloadCount, {
          abortSignal: AbortController.timeout(10 *1000), // Abort uploading with timeout in 10 secs
          blockSize: 4 * 1024 * 1024, // 4MB block size
          onProgress: (ev) => console.log(ev),
        });
        shouldRetry = false;
        console.log("downloadToBuffer succeeds");
      } catch (err: any) {
        shouldRetry = true;
      }
      --retryMax;
    }

    // shouldRetry is false, means that download succeeded.
    if (!shouldRetry) {
      await fd.write(buffer, 0, downloadCount, offset);
    }
  }

  await fd.close();

Thanks
Emma

[github-actions]

Hi @PedroFonsecaDEV. Thank you for opening this issue and giving us the opportunity to assist. To help our team better understand your issue and the details of your scenario please provide a response to the question asked above or the information requested above. This will help us more accurately address your issue.

[github-actions]

Hi @PedroFonsecaDEV, we're sending this friendly reminder because we haven't heard back from you in 7 days. We need more information about this issue to help address it. Please be sure to give us your input. If we don't hear back from you within 14 days of this comment the issue will be automatically closed. Thank you!