[Performance] Refactor Azure SDK usage

[yika-luo]

The previous deployment SDK doesn't support community image usage, so upgrading to use the latest 2024 SDK. This can improve VM creation part by ~7s

Tested (run the relevant ones):

• Code formatting: bash format.sh
• Manual tests:
  sky launch --cloud=azure -y -t Standard_DS1_v2 --region westus2 --num-nodes=2
sky stop, sky start, sky autostop, sky autostop --down
• Smoke test: pytest tests/test_smoke.py::test_minimal --azure
• Backward compatibility tests: conda deactivate; bash -i tests/backward_compatibility_tests.sh

[Michaelvll]

Thanks for refactoring the implementation of Azure @yika-luo! Could we do a quick profiling for how fast this new SDK is for single and multi-node on Azure, as IIRC, the old deployment is API is very slow.

Also, it worth checking if we should get rid of the deployments in the following place as well.

skypilot/sky/provision/azure/config.py

Line 45 in f5d4f64

def bootstrap_instances(

[yika-luo]

Thanks for refactoring the implementation of Azure @yika-luo! Could we do a quick profiling for how fast this new SDK is for single and multi-node on Azure, as IIRC, the old deployment is API is very slow.

Also, it worth checking if we should get rid of the deployments in the following place as well.

skypilot/sky/provision/azure/config.py

Line 45 in f5d4f64

def bootstrap_instances(

It speeds up about 7s. Yes we should refactor config too, I added a TODO to address later

[Michaelvll]

Thanks for the refactoring @yika-luo! It looks mostly good to me. We probably need to check a few basic things, especially for multi-node:

1. sky launch
2. sky stop
3. sky start
4. sky autostop (both with and without --down)
5. Also, the backward compatibility on Azure (use the backward compatibility script in the PR description, with the cloud modified in the script)

[Michaelvll]

Thanks for the refactoring and tests @yika-luo! LGTM.