Add Kubernetes Support #180

TaekyungHeo · 2024-08-21T21:00:37Z

Summary

Added Kubernetes support with KubernetesSystem, KubernetesJob, and KubernetesRunner classes.
Implemented JsonGenStrategy for generating Kubernetes job specs, including SleepKubernetesJsonGenStrategy.
Introduced KubernetesInstaller for managing Kubernetes installations.
Updated Registry and Tests to include Kubernetes-specific components.

This PR builds upon @wpeng102's work in PR #165.

Test Plan

Install

$ cloudai --mode install --system-config conf/system/kubernetes_cluster.toml --test-templates-dir conf/test_template --tests-dir conf/test/  --test-scenario conf/test_scenario/nccl_test.toml

[INFO] System configuration file: conf/system/kubernetes_cluster.toml
[INFO] Test templates directory: conf/test_template
[INFO] Tests directory: conf/test
[INFO] Test scenario file: conf/test_scenario/nccl_test.toml
[INFO] Output directory: None
[WARNING] No InstallStrategy found for TestTemplateParser and KubernetesSystem
[WARNING] No CommandGenStrategy found for TestTemplateParser and KubernetesSystem
[WARNING] No JsonGenStrategy found for TestTemplateParser and KubernetesSystem
[WARNING] No JobIdRetrievalStrategy found for TestTemplateParser and KubernetesSystem
[WARNING] No JobStatusRetrievalStrategy found for TestTemplateParser and KubernetesSystem
[WARNING] No ReportGenerationStrategy found for TestTemplateParser and KubernetesSystem
[WARNING] No GradingStrategy found for TestTemplateParser and KubernetesSystem
[WARNING] No InstallStrategy found for TestTemplateParser and KubernetesSystem
[WARNING] No CommandGenStrategy found for TestTemplateParser and KubernetesSystem
[WARNING] No JsonGenStrategy found for TestTemplateParser and KubernetesSystem
[WARNING] No JobIdRetrievalStrategy found for TestTemplateParser and KubernetesSystem
[WARNING] No JobStatusRetrievalStrategy found for TestTemplateParser and KubernetesSystem
[WARNING] No ReportGenerationStrategy found for TestTemplateParser and KubernetesSystem
[WARNING] No GradingStrategy found for TestTemplateParser and KubernetesSystem
[WARNING] No InstallStrategy found for TestTemplateParser and KubernetesSystem
[WARNING] No CommandGenStrategy found for TestTemplateParser and KubernetesSystem
[WARNING] No JsonGenStrategy found for TestTemplateParser and KubernetesSystem
[WARNING] No JobIdRetrievalStrategy found for TestTemplateParser and KubernetesSystem
[WARNING] No JobStatusRetrievalStrategy found for TestTemplateParser and KubernetesSystem
[WARNING] No ReportGenerationStrategy found for TestTemplateParser and KubernetesSystem
[WARNING] No GradingStrategy found for TestTemplateParser and KubernetesSystem
[WARNING] No InstallStrategy found for TestTemplateParser and KubernetesSystem
[WARNING] No CommandGenStrategy found for TestTemplateParser and KubernetesSystem
[WARNING] No JobIdRetrievalStrategy found for TestTemplateParser and KubernetesSystem
[WARNING] No ReportGenerationStrategy found for TestTemplateParser and KubernetesSystem
[WARNING] No GradingStrategy found for TestTemplateParser and KubernetesSystem
[WARNING] No InstallStrategy found for TestTemplateParser and KubernetesSystem
[WARNING] No CommandGenStrategy found for TestTemplateParser and KubernetesSystem
[WARNING] No JobIdRetrievalStrategy found for TestTemplateParser and KubernetesSystem
[WARNING] No ReportGenerationStrategy found for TestTemplateParser and KubernetesSystem
[WARNING] No GradingStrategy found for TestTemplateParser and KubernetesSystem
[INFO] System Name: kubernetes-cluster
[INFO] Scheduler: kubernetes
[INFO] CloudAI is already installed.

Sleep test scenario

# cloudai --mode run --system-config conf/system/kubernetes_cluster.toml --test-templates-dir conf/test_template --tests-dir conf/test/  --test-scenario conf/test_scenario/sleep.toml                            


[INFO] System configuration file: conf/system/kubernetes_cluster.toml
[INFO] Test templates directory: conf/test_template
[INFO] Tests directory: conf/test
[INFO] Test scenario file: conf/test_scenario/sleep.toml
[INFO] Output directory: None
[WARNING] No InstallStrategy found for TestTemplateParser and KubernetesSystem
[WARNING] No CommandGenStrategy found for TestTemplateParser and KubernetesSystem
[WARNING] No JsonGenStrategy found for TestTemplateParser and KubernetesSystem
[WARNING] No JobIdRetrievalStrategy found for TestTemplateParser and KubernetesSystem
[WARNING] No JobStatusRetrievalStrategy found for TestTemplateParser and KubernetesSystem
[WARNING] No ReportGenerationStrategy found for TestTemplateParser and KubernetesSystem
[WARNING] No GradingStrategy found for TestTemplateParser and KubernetesSystem
[WARNING] No InstallStrategy found for TestTemplateParser and KubernetesSystem
[WARNING] No CommandGenStrategy found for TestTemplateParser and KubernetesSystem
[WARNING] No JsonGenStrategy found for TestTemplateParser and KubernetesSystem
[WARNING] No JobIdRetrievalStrategy found for TestTemplateParser and KubernetesSystem
[WARNING] No JobStatusRetrievalStrategy found for TestTemplateParser and KubernetesSystem
[WARNING] No ReportGenerationStrategy found for TestTemplateParser and KubernetesSystem
[WARNING] No GradingStrategy found for TestTemplateParser and KubernetesSystem
[WARNING] No InstallStrategy found for TestTemplateParser and KubernetesSystem
[WARNING] No CommandGenStrategy found for TestTemplateParser and KubernetesSystem
[WARNING] No JsonGenStrategy found for TestTemplateParser and KubernetesSystem
[WARNING] No JobIdRetrievalStrategy found for TestTemplateParser and KubernetesSystem
[WARNING] No JobStatusRetrievalStrategy found for TestTemplateParser and KubernetesSystem
[WARNING] No ReportGenerationStrategy found for TestTemplateParser and KubernetesSystem
[WARNING] No GradingStrategy found for TestTemplateParser and KubernetesSystem
[WARNING] No InstallStrategy found for TestTemplateParser and KubernetesSystem
[WARNING] No CommandGenStrategy found for TestTemplateParser and KubernetesSystem
[WARNING] No JobIdRetrievalStrategy found for TestTemplateParser and KubernetesSystem
[WARNING] No ReportGenerationStrategy found for TestTemplateParser and KubernetesSystem
[WARNING] No GradingStrategy found for TestTemplateParser and KubernetesSystem
[WARNING] No CommandGenStrategy found for TestTemplateParser and KubernetesSystem
[WARNING] No JobIdRetrievalStrategy found for TestTemplateParser and KubernetesSystem
[WARNING] No ReportGenerationStrategy found for TestTemplateParser and KubernetesSystem
[WARNING] No GradingStrategy found for TestTemplateParser and KubernetesSystem
[INFO] System Name: kubernetes-cluster
[INFO] Scheduler: kubernetes
[INFO] Test Scenario Name: test_scenario_example
[INFO] Checking if test templates are installed.
[INFO] Test Scenario: test_scenario_example

Section Name: Tests.1
  Test Name: sleep_10
  Description: sleep_10
  No dependencies
Section Name: Tests.2
  Test Name: sleep_5
  Description: sleep_5
  Start Post Init: Tests.1, Time: 5 seconds
Section Name: Tests.3
  Test Name: sleep_5
  Description: sleep_5
  Start Post Comp: Tests.1, Time: 0 seconds
Section Name: Tests.4
  Test Name: sleep_20
  Description: sleep_20
  End Post Comp: Tests.1, Time: 5 seconds
[INFO] Initializing Runner
[INFO] Creating KubernetesRunner
[INFO] Starting test scenario execution.
[INFO] Starting test: Tests.1
[INFO] Running test: Tests.1
[INFO] Generated JSON string for test Tests.1: {'apiVersion': 'batch/v1', 'kind': 'Job', 'metadata': {'name': 'tests-1', 'namespace': 'default'}, 'spec': {'ttlSecondsAfterFinished': 0, 'template': {'spec': {'containers': [{'args': ['sleep 10'], 'command': ['/bin/bash', '-c'], 'image': 'ubuntu:22.04', 'name': 'task'}], 'restartPolicy': 'Never'}}}}
[INFO] Starting test: Tests.4
[INFO] Running test: Tests.4
[INFO] Generated JSON string for test Tests.4: {'apiVersion': 'batch/v1', 'kind': 'Job', 'metadata': {'name': 'tests-4', 'namespace': 'default'}, 'spec': {'ttlSecondsAfterFinished': 0, 'template': {'spec': {'containers': [{'args': ['sleep 20'], 'command': ['/bin/bash', '-c'], 'image': 'ubuntu:22.04', 'name': 'task'}], 'restartPolicy': 'Never'}}}}
[INFO] Delayed start for test Tests.2 by 5 seconds.
[INFO] Starting test: Tests.2
[INFO] Running test: Tests.2
[INFO] Generated JSON string for test Tests.2: {'apiVersion': 'batch/v1', 'kind': 'Job', 'metadata': {'name': 'tests-2', 'namespace': 'default'}, 'spec': {'ttlSecondsAfterFinished': 0, 'template': {'spec': {'containers': [{'args': ['sleep 5'], 'command': ['/bin/bash', '-c'], 'image': 'ubuntu:22.04', 'name': 'task'}], 'restartPolicy': 'Never'}}}}
[INFO] Job completed: Tests.1
[INFO] Delayed start for test Tests.3 by 0 seconds.
[INFO] Starting test: Tests.3
[INFO] Running test: Tests.3
[INFO] Generated JSON string for test Tests.3: {'apiVersion': 'batch/v1', 'kind': 'Job', 'metadata': {'name': 'tests-3', 'namespace': 'default'}, 'spec': {'ttlSecondsAfterFinished': 0, 'template': {'spec': {'containers': [{'args': ['sleep 5'], 'command': ['/bin/bash', '-c'], 'image': 'ubuntu:22.04', 'name': 'task'}], 'restartPolicy': 'Never'}}}}
[INFO] Scheduling termination of job tests-4 after 5 seconds.
[INFO] Job completed: Tests.2
[INFO] Job completed: Tests.4
[INFO] Job completed: Tests.3
[INFO] All test scenario results stored at: /root/cloudai/results/test_scenario_example_2024-08-09_14-17-22
[WARNING] Skipping directory '/root/cloudai/results/test_scenario_example_2024-08-09_14-17-22/Tests.1/0' for test 'sleep_10'
[WARNING] Skipping directory '/root/cloudai/results/test_scenario_example_2024-08-09_14-17-22/Tests.2/0' for test 'sleep_5'
[WARNING] Skipping directory '/root/cloudai/results/test_scenario_example_2024-08-09_14-17-22/Tests.3/0' for test 'sleep_5'
[WARNING] Skipping directory '/root/cloudai/results/test_scenario_example_2024-08-09_14-17-22/Tests.4/0' for test 'sleep_20'
[INFO] All test scenario execution attempts are complete. Please review the 'debug.log' file to confirm successful completion or to identify any issues.

Co-authored-by: Peng Wang <pengwang@nvidia.com>

src/cloudai/_core/json_gen_strategy.py

src/cloudai/installer/kubernetes_installer.py

src/cloudai/systems/kubernetes/kubernetes_system.py

src/cloudai/runner/kubernetes/kubernetes_job.py

conf/common/system/kubernetes_cluster.toml

requirements.txt

src/cloudai/runner/kubernetes/kubernetes_job.py

Co-authored-by: Andrei Maslennikov <andreyma@nvidia.com>

TaekyungHeo · 2024-09-03T12:01:50Z

@TaekyungHeo & @amaslenn 's comment: Let's unblock first. Let's merge without updating this PR.

TaekyungHeo · 2024-09-03T14:36:55Z

Resolving all comments to make forward progress based on the discussion with @amaslenn .

The following comments should be resolved in follow-up PRs

srivatsankrishnan

Left some minor comments. I see them as non-blockers for approval.

conf/common/system/kubernetes_cluster.toml

src/cloudai/_core/test.py

TaekyungHeo added the Oct24 Oct'24 release feature label Aug 21, 2024

TaekyungHeo force-pushed the k8s-base branch 4 times, most recently from 000729c to 76e3c2a Compare August 21, 2024 21:17

TaekyungHeo mentioned this pull request Aug 21, 2024

Update NCCL Test Templates and Introduce Kubernetes-Specific Strategies (No Shared Dir / Mixin / Sep 3rd) #181

Closed

TaekyungHeo added the feature label Aug 26, 2024

TaekyungHeo force-pushed the k8s-base branch from 76e3c2a to 691c6e9 Compare August 29, 2024 18:37

TaekyungHeo and others added 7 commits August 29, 2024 16:03

Add Kubernetes job

92207bc

Co-authored-by: Peng Wang <pengwang@nvidia.com>

Add Kubernetes system

7b630a0

Co-authored-by: Peng Wang <pengwang@nvidia.com>

Add Kubernetes system parser

37ca92e

Co-authored-by: Peng Wang <pengwang@nvidia.com>

Add example Kubernetes system schema

73565af

Co-authored-by: Peng Wang <pengwang@nvidia.com>

Add Kubernetes installer

a9effd6

Co-authored-by: Peng Wang <pengwang@nvidia.com>

Add Kubernetes runner

e8c1139

Co-authored-by: Peng Wang <pengwang@nvidia.com>

Add Kubernetes support for Sleep

c4ec665

Co-authored-by: Peng Wang <pengwang@nvidia.com>

TaekyungHeo force-pushed the k8s-base branch from 691c6e9 to c4ec665 Compare August 29, 2024 20:04

TaekyungHeo requested review from amaslenn, artemry-nv, srivatsankrishnan and srinivas212 August 29, 2024 22:58

amaslenn reviewed Aug 30, 2024

View reviewed changes

src/cloudai/_core/json_gen_strategy.py Outdated Show resolved Hide resolved

src/cloudai/installer/kubernetes_installer.py Show resolved Hide resolved

src/cloudai/systems/kubernetes/kubernetes_system.py Show resolved Hide resolved

TaekyungHeo commented Aug 30, 2024

View reviewed changes

src/cloudai/runner/kubernetes/kubernetes_job.py Outdated Show resolved Hide resolved

TaekyungHeo commented Aug 30, 2024

View reviewed changes

conf/common/system/kubernetes_cluster.toml Show resolved Hide resolved

amaslenn reviewed Aug 30, 2024

View reviewed changes

requirements.txt Show resolved Hide resolved

TaekyungHeo commented Aug 30, 2024

View reviewed changes

src/cloudai/runner/kubernetes/kubernetes_job.py Show resolved Hide resolved

TaekyungHeo commented Aug 30, 2024

View reviewed changes

src/cloudai/runner/kubernetes/kubernetes_job.py Outdated Show resolved Hide resolved

TaekyungHeo and others added 3 commits August 30, 2024 14:41

Merge remote-tracking branch 'origin/main' into k8s-base

7d9fb32

Reflect Andrei's comments

537fd8b

Co-authored-by: Andrei Maslennikov <andreyma@nvidia.com>

Fix bug in job status retrieval

a4d0fef

TaekyungHeo force-pushed the k8s-base branch from 108d4f8 to a4d0fef Compare August 30, 2024 19:51

TaekyungHeo marked this pull request as ready for review September 3, 2024 12:02

TaekyungHeo mentioned this pull request Sep 3, 2024

Update NCCL Test Templates and Introduce Kubernetes-Specific Strategies #186

Merged

srivatsankrishnan approved these changes Sep 3, 2024

View reviewed changes

conf/common/system/kubernetes_cluster.toml Show resolved Hide resolved

src/cloudai/_core/test.py Show resolved Hide resolved

srinivas212 approved these changes Sep 3, 2024

View reviewed changes

srinivas212 merged commit 1e6d618 into NVIDIA:main Sep 3, 2024
2 checks passed

This was referenced Sep 16, 2024

Resolve K8s Comments (Part 1) #195

Draft

Resolve K8s Comments (Part 2) #197

Draft

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Kubernetes Support #180

Add Kubernetes Support #180

TaekyungHeo commented Aug 21, 2024 •

edited

Loading

TaekyungHeo commented Sep 3, 2024

TaekyungHeo commented Sep 3, 2024 •

edited

Loading

srivatsankrishnan left a comment

Add Kubernetes Support #180

Add Kubernetes Support #180

Conversation

TaekyungHeo commented Aug 21, 2024 • edited Loading

Summary

Test Plan

TaekyungHeo commented Sep 3, 2024

TaekyungHeo commented Sep 3, 2024 • edited Loading

srivatsankrishnan left a comment

Choose a reason for hiding this comment

TaekyungHeo commented Aug 21, 2024 •

edited

Loading

TaekyungHeo commented Sep 3, 2024 •

edited

Loading