[k8s] SkyServe on Kubernetes #3377

romilbhardwaj · 2024-03-28T02:15:22Z

Adds SkyServe support on Kubernetes. Replicas and controller can now be run on Kubernetes. Supersedes #3109.

Changes in this PR:

Support for service accounts [k8s] Use service acccount based auth instead of copying over kubeconfig #3277
Docs on minimal permissions [k8s] Add docs on minimal permissions required on Kubernetes cluster #2806
Refactor get_endpoints to backend_utils
Use get_endpoints API in serve controller and LB
Optimizations to reduce dependencies installed on the controller based on enabled clouds

Tested:

Code formatting: bash format.sh
HTTP Serve Example on GKE with ingress
HTTP Serve Example on GKE with LoadBalancer
LLM Example on GKE with ingress
HTTP Serve Example on EKS with ingress
HTTP Serve Example on EKS with LoadBalancer
Onprem testing through design partner
All smoke tests, including serve tests now enabled for k8s (in progress)

…o serve_k8s_playground # Conflicts: # sky/clouds/kubernetes.py # sky/serve/core.py

…o serve_k8s_playground # Conflicts: # sky/cli.py

…o serve_k8s_playground

…t-org/skypilot into serve_k8s_playground # Conflicts: # sky/serve/core.py # sky/serve/replica_managers.py

romilbhardwaj · 2024-04-29T20:17:06Z

Thanks for the reviews @Michaelvll! Ready for another look. If it looks ok at a high level, I'll start running smoke tests.

Michaelvll

Thank you for the update @romilbhardwaj and sorry for the delay! LGTM. Could you help resolve the conflict? If the tests work fine, I think it is good to go.

sky/backends/backend_utils.py

sky/utils/common_utils.py

sky/utils/controller_utils.py

Michaelvll · 2024-05-05T18:07:04Z

docs/source/serving/sky-serve.rst

@@ -308,11 +308,12 @@ Let's bring up a real LLM chat service with FastChat + Vicuna. We'll use the `Vi
      conda activate chatbot

      echo 'Starting controller...'
-      python -u -m fastchat.serve.controller > ~/controller.log 2>&1 &


nit: should we also mention serving on kubernetes a bit in the serving doc and refer to the cloud-permissions for the setup?
We may also want to explicitly list out why people should use skyserve on kubernetes than just using deployments, such as:

easier customization for load balancer and autoscaling policy

simple interface and easy to setup

spillover to clouds.

Good idea - filed #3518, will submit a separate PR for docs.

…o serve_k8s_final # Conflicts: # sky/backends/backend_utils.py # sky/cli.py # sky/utils/controller_utils.py # sky/utils/schemas.py

romilbhardwaj · 2024-05-07T22:53:54Z

Thanks @Michaelvll! Addressed comments. Running smoke and backcompat tests now:

Cloud smoke tests, including serve tests - pytest -v tests/test_smoke.py
Kubernetes smoke tests, including serve tests - pytest -v tests/test_smoke.py --kubernetes
Backward compatibility - run a service from master, switch branch and launch a new svc. Try to access both services.

romilbhardwaj · 2024-05-08T05:28:08Z

Update - had to add skip_status_check arg to backend_utils.get_endpoints() to disable checks for cluster status (i.e., do not fail early if cluster is not UP). This is similar to the previous behavior, where head_ip:port was used without checking cluster status. This is required when multiple sky serve up are run concurrently (e.g., in smoke tests), since each call would put the cluster in INIT state and get_endpoints call at the end of serve.core.up would fail:

  File "/home/gcpuser/.sky/file_mounts/sky_repo/sky/serve/core.py", line 257, in up
    endpoint = backend_utils.get_endpoints(
  File "/home/gcpuser/.sky/file_mounts/sky_repo/sky/backends/backend_utils.py", line 2756, in get_endpoints
    raise exceptions.ClusterNotUpError(
sky.exceptions.ClusterNotUpError: Cluster 'sky-serve-controller-c3f7de47' is not in UP status.

Smoke tests pass now, this is ready to be merged.

Michaelvll · 2024-05-08T05:57:38Z

tests/test_yamls/minimal.yaml

@@ -2,6 +2,7 @@ name: min

 setup: |
  echo "running setup"
+  sudo apt-get install -y jq


Would be nice to move this to test_minimal in test_smoke.py only. : )

good point, since this minimal.yaml is also used in a bunch of other tests. moved jq installation to test_minimal.yaml.

romilbhardwaj added 30 commits February 1, 2024 08:05

playing around

34598a2

wip with hacks

95b1eb0

wip refactor get_endpoints

f93e6dd

working get_endpoints

c8276bb

wip

c713293

fixed circular import

aa286e8

Working for ingress and loadbalancer svc

9f768a2

lint

1848b3a

add purging from #3094

ba0ff60

Use local catalog on the controller too

13f7241

use externalip if available

996742a

Merge branch 'master' of https://github.com/skypilot-org/skypilot int…

7c8b5cd

…o serve_k8s_playground # Conflicts: # sky/clouds/kubernetes.py # sky/serve/core.py

Merge branch 'master' of https://github.com/skypilot-org/skypilot int…

b222186

…o serve_k8s_playground # Conflicts: # sky/cli.py

add dshm_size_limit

d852c61

optimize dependency installation

b9014df

Add todo

282bcef

Merge branch 'master' of https://github.com/skypilot-org/skypilot int…

d10fb24

…o serve_k8s_playground

optimize ingress

3cb948f

fix

4a7e10a

fix

9e0d910

Merge branch 'k8s_ingress_optimization' of https://github.com/skypilo…

9e51793

…t-org/skypilot into serve_k8s_playground # Conflicts: # sky/serve/core.py # sky/serve/replica_managers.py

remove autostop timing

68bb6e8

Fix URLs for raw IP:ports

b72a814

fixes

a22b6d3

wip

4637f91

SA wip

d27ae5f

Allow use of service accounts through remote_identity field

cdc2e3f

Make purge work for no clusters in kubeconfig

643fb77

Handle ingress namespace not present

7649fcb

setup optimizations and critical SA key fix

8c76eb6

Michaelvll mentioned this pull request May 3, 2024

[Core/UX] Add Job API and support managed job (both on-demand and spot) #3419

Merged

12 tasks

Michaelvll approved these changes May 5, 2024

View reviewed changes

sky/backends/backend_utils.py Outdated Show resolved Hide resolved

sky/utils/common_utils.py Outdated Show resolved Hide resolved

sky/utils/controller_utils.py Outdated Show resolved Hide resolved

Michaelvll reviewed May 5, 2024

View reviewed changes

romilbhardwaj added 8 commits May 7, 2024 12:38

Merge branch 'master' of https://github.com/skypilot-org/skypilot int…

017dcb5

…o serve_k8s_final # Conflicts: # sky/backends/backend_utils.py # sky/cli.py # sky/utils/controller_utils.py # sky/utils/schemas.py

merge fixes

cd08d0f

lint

05f1996

fix smoke tests

97b1d56

fix smoke tests

9a8ccd1

comment

3b7d33f

add enum for remote identity

e029ebc

lint

7b201f2

romilbhardwaj mentioned this pull request May 7, 2024

[docs] Add references to Kubernetes in SkyServe docs #3518

Closed

romilbhardwaj mentioned this pull request May 8, 2024

[k8s] Disable autostop for controller on kubernetes #3521

Merged

3 tasks

romilbhardwaj added 3 commits May 7, 2024 18:31

add skip_status_check

8a54faf

remove zone requirement

5fb4232

fix timings for test

2e8afe9

Michaelvll approved these changes May 8, 2024

View reviewed changes

romilbhardwaj added 3 commits May 7, 2024 23:02

silence curl download

736e087

move jq from yaml to test_minimal

e3768b4

move jq from yaml to test_minimal

26cc380

romilbhardwaj merged commit 0a03995 into master May 8, 2024
20 checks passed

romilbhardwaj deleted the serve_k8s_final branch May 8, 2024 06:49

cblmemo mentioned this pull request May 10, 2024

[Serve] Change back from autodown to autostop #3535

Merged

6 tasks

This was referenced May 22, 2024

[k8s] Use service acccount based auth instead of copying over kubeconfig #3277

Closed

[k8s] Add docs on minimal permissions required on Kubernetes cluster #2806

Closed

romilbhardwaj mentioned this pull request Jun 5, 2024

[Managed Jobs] Support Rsync for managed jobs #3620

Closed

Michaelvll mentioned this pull request Jun 10, 2024

[k8s][spot] Support spot controller on Kubernetes or local clusters #2249

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[k8s] SkyServe on Kubernetes #3377

[k8s] SkyServe on Kubernetes #3377

romilbhardwaj commented Mar 28, 2024 •

edited

Loading

romilbhardwaj commented Apr 29, 2024

Michaelvll left a comment

Michaelvll May 5, 2024

romilbhardwaj May 7, 2024

romilbhardwaj commented May 7, 2024 •

edited

Loading

romilbhardwaj commented May 8, 2024

Michaelvll May 8, 2024

romilbhardwaj May 8, 2024

[k8s] SkyServe on Kubernetes #3377

[k8s] SkyServe on Kubernetes #3377

Conversation

romilbhardwaj commented Mar 28, 2024 • edited Loading

romilbhardwaj commented Apr 29, 2024

Michaelvll left a comment

Choose a reason for hiding this comment

Michaelvll May 5, 2024

Choose a reason for hiding this comment

romilbhardwaj May 7, 2024

Choose a reason for hiding this comment

romilbhardwaj commented May 7, 2024 • edited Loading

romilbhardwaj commented May 8, 2024

Michaelvll May 8, 2024

Choose a reason for hiding this comment

romilbhardwaj May 8, 2024

Choose a reason for hiding this comment

romilbhardwaj commented Mar 28, 2024 •

edited

Loading

romilbhardwaj commented May 7, 2024 •

edited

Loading