VM: create LRO polling mechanism improvement

[yugangw-msft]

For now, we pull every 5 seconds.
We need to have this aggressive polling only for the first 400 seconds. After that we can have an incremental back-off. Having a flat polling every 5 seconds will have high risk of running into the ARM throttling limits.

[tjprescott]

The whole point of the 5 second polling interval was so that truly long running LROs (like VM create) didn't incur a penalty of roughly 1/2 the polling delay. If we end up using a short delay for short LROs and a long delay for long LROs, it seems like we have the same perf hit for the long LROs while just adding more requests for the short ones for no discernible benefit (don't know of anyone complaining about the relatively quick LROs).

[yugangw-msft]

Agreed with your point.
Since it is for VM, and most time, I saw the VM gets created within 3 minutes(180 seconds). So for now, I think this is for edge case when the service was slowed down for reasons.

[yugangw-msft]

Also, this bug was opened per folks doing perf evaluation from other teams

[tjprescott]

If we are going to end up with long polling intervals for long LROs, then why not just go back to the default of 30 sec?

[yugangw-msft]

agree

[yugangw-msft]

I suggest we live with it, for 2 reasons

1. 400 seconds is 6 minutes and 20 seconds. The VM creation currently takes about 2~4 minutes, so the new logic will not make an impact mostly.
   2 . Python AutoRestAzure LRO polling doesn't appear to have the extensibility to dynamically update the polling interval, unless we hack it. //cc: @annatisch @lmazuel

If #1 becomes a problem say it starts to take more than 400 seconds, we can consider to updating the AutoRestAzure

[lmazuel]

Are you saying we should ignore the RetryAfter Header of the ARM spec? Or provide a callback or something like this to open to customisation easily?

[yugangw-msft]

Are you saying we should ignore the RetryAfter Header of the ARM spec?

@lmazuel, no, we should always respect 'RetryAfter'. For 'vm create' the service doesn't respond with RetryAfter, so autorest_azure ends up with a sleep

or provide a callback or something like this to open to customisation easily?

If needed, but like I mentioned early on, I suggest not to do it right now till we have user data

[yugangw-msft]

Closing per reasons I provided. We can reactivate it when we have new data