Investigate storage transient test failures

[mikeharder]

Storage tests have transient failures. To mitigate, #4162 added global retries, but a more narrowly-scoped retry mechanism would be preferable to avoid hiding other transient issues and more closely represent expected customer usage which also needs to handle retries.

Examples:

• https://dev.azure.com/azure-sdk/internal/_build/results?buildId=61115&view=logs&j=073207db-8f6c-5961-98d3-a3ec4eda538e
• https://dev.azure.com/azure-sdk/internal/_build/results?buildId=61127&view=logs&j=3af57de4-db87-5a73-5c41-f28014b64054

  1) Highlevel
       downloadToBuffer should success:
     Error: connect ETIMEDOUT 52.165.136.32:443
      at new RestError (D:\a\1\s\sdk\core\core-http\lib\restError.ts:18:5)
      at AxiosHttpClient.<anonymous> (D:\a\1\s\sdk\core\core-http\lib\axiosHttpClient.ts:162:15)
      at step (D:\a\1\s\common\temp\node_modules\.registry.npmjs.org\tslib\1.10.0\node_modules\tslib\tslib.js:136:27)
      at Object.throw (D:\a\1\s\common\temp\node_modules\.registry.npmjs.org\tslib\1.10.0\node_modules\tslib\tslib.js:117:57)
      at rejected (D:\a\1\s\common\temp\node_modules\.registry.npmjs.org\tslib\1.10.0\node_modules\tslib\tslib.js:108:69)
      at processTicksAndRejections (internal/process/task_queues.js:82:5)

[HarshaNalluru]

Created a new branch AvoidRetries out of feature/storage and removed the mocha retry functionality.

storage-queue tests succeeded.

Occasional failures in storage-blob tests -
https://dev.azure.com/azure-sdk/internal/_build/results?buildId=63958&view=results
https://dev.azure.com/azure-sdk/internal/_build/results?buildId=63952&view=results

[kurtzeborn]

Assigning this to @HarshaNalluru and attaching it to the preview 3 release for storage.

[HarshaNalluru]

@jiacfan @XiaoningLiu

Please remove the retry option in a branch for the tests and verify them by doing multiple runs to find the causes behind these transient issues.
Probably, need be handled on a case-to-case basis in a better way.
It is in our best interests to remove the retries and run the tests so that we may uncover any hidden bugs with the SDK or even the service.

[mikeharder]

I am currently working to get all live tests passing at 90% (with no test-level retries). As part of this, we will need to disable retries in the JS tests and investigate the root causes.

[mikeharder]

PR to revert retries: #6910

[jiacfan]

• @XiaoningLiu per DRI assigned.

[ramya-rao-a]

It does not look like there is any pending work here.
The retries that were added have been removed.

Please post a comment if there is anything else to do here