Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Core: Fix AWS/GCP autostop with new provisioner. #2719

Merged
merged 4 commits into from
Oct 18, 2023

Conversation

concretevitamin
Copy link
Member

@concretevitamin concretevitamin commented Oct 18, 2023

Previously, the regex used is incorrect, causing failures to autostop an AWS cluster that just got updated to use the new provisioner.

Before

git checkout b1d8c9d8
sky launch -c dbg --cloud aws --cpus 4+

git checkout bf336028  # master
sky launch -c dbg --cloud aws --cpus 4+ -i0

Observe failure to autostop in the VM's skylet.log:

..
KeyError: 'head_setup_commands'
E 10-18 05:07:42 events.py:51] AutostopEvent error: Command '['ray', 'down', '-y', '/home/ubuntu/.sky/sky_ray.yml']' returned non-zero exit status 1.
E 10-18 05:07:42 events.py:53] Traceback (most recent call last):
E 10-18 05:07:42 events.py:53]   File "/opt/conda/lib/python3.10/site-packages/sky/skylet/events.py", line 48, in run
E 10-18 05:07:42 events.py:53]     self._run()
E 10-18 05:07:42 events.py:53]   File "/opt/conda/lib/python3.10/site-packages/sky/skylet/events.py", line 122, in _run
E 10-18 05:07:42 events.py:53]     self._stop_cluster(autostop_config)
E 10-18 05:07:42 events.py:53]   File "/opt/conda/lib/python3.10/site-packages/sky/skylet/events.py", line 194, in _stop_cluster
E 10-18 05:07:42 events.py:53]     subprocess.run(
E 10-18 05:07:42 events.py:53]   File "/opt/conda/lib/python3.10/subprocess.py", line 524, in run
E 10-18 05:07:42 events.py:53]     raise CalledProcessError(retcode, process.args,
E 10-18 05:07:42 events.py:53] subprocess.CalledProcessError: Command '['ray', 'down', '-y', '/home/ubuntu/.sky/sky_ray.yml']' returned non-zero exit status 1.
E 10-18 05:07:42 events.py:53]

With this PR

  • after running the above, switch to this branch and run
sky launch -c dbg --cloud aws --cpus 4+ -i0
  • observe that autostop kicks in correctly

Tested (run the relevant ones):

  • Code formatting: bash format.sh
  • Any manual or new tests for this PR (please specify below): above
  • All smoke tests: pytest tests/test_smoke.py
    • pytest tests/test_smoke.py::test_autostop --aws
  • Relevant individual smoke tests: pytest tests/test_smoke.py::test_fill_in_the_name
  • Backward compatibility tests: bash tests/backward_comaptibility_tests.sh

Copy link
Collaborator

@Michaelvll Michaelvll left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for identifying and fixing this issue @concretevitamin! Left two comments. We should get this in once the test is passed

sky/skylet/events.py Outdated Show resolved Hide resolved
sky/skylet/events.py Outdated Show resolved Hide resolved
@concretevitamin
Copy link
Member Author

Smoke test (autostop) and manual back-compat tests passed. Merging.

@concretevitamin concretevitamin merged commit 7a54940 into master Oct 18, 2023
18 checks passed
@concretevitamin concretevitamin deleted the fix-autostop-newprov branch October 18, 2023 17:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants