Redefine Job, Runner, and System Responsibilities; Refactor Classes

[TaekyungHeo]

Summary

• Moved job lifecycle methods (is_running, is_completed, kill) from runner to job classes.
• Enhanced BaseJob with new methods for mode, system, and ID retrieval.
• Fixed a bug in parsing.

Test Plan

1. CI
CI passes. New unit tests were added, but they do not cover all of the changes I made. I did not spend more time adding additional unit tests for the following reasons:

• The changes made in the Job classes are straightforward, and adding unit tests would add complexity without significantly improving coverage.
• The changes made in the System classes cannot be fully tested with unit tests.

2. Ran on a server
Sleep

$ python cloudaix.py --mode run --system-config conf/v0.6/general/system/system.toml --test-templates-dir conf/v0.6/general/test_template --tests-dir conf/v0.6/general/test --test-scenario conf/v0.6/general/test_scenario/sleep.toml 
[INFO] System configuration file: conf/v0.6/general/system/system.toml
[INFO] Test templates directory: conf/v0.6/general/test_template
[INFO] Tests directory: conf/v0.6/general/test
[INFO] Test scenario file: conf/v0.6/general/test_scenario/sleep.toml
[INFO] Output directory: None
[INFO] System Name: System
[INFO] Scheduler: slurm
[INFO] Test Scenario Name: test_scenario_example
[INFO] Checking if test templates are installed.
[INFO] Test Scenario: test_scenario_example

Section Name: Tests.1
  Test Name: sleep_10
  Description: sleep_10
  No dependencies
Section Name: Tests.2
  Test Name: sleep_5
  Description: sleep_5
  Start Post Init: Tests.1, Time: 5 seconds
Section Name: Tests.3
  Test Name: sleep_5
  Description: sleep_5
  Start Post Comp: Tests.1, Time: 0 seconds
Section Name: Tests.4
  Test Name: sleep_20
  Description: sleep_20
  End Post Comp: Tests.1, Time: 5 seconds
[INFO] Initializing Runner
[INFO] Creating SlurmRunner
[INFO] Starting test scenario execution.
[INFO] Starting test: Tests.1
[INFO] Running test: Tests.1
[INFO] Executing command for test Tests.1: sbatch /obfuscated/path/results/test_scenario_example_2024-08-23_22-46-25/Tests.1/0/cloudai_sbatch_script.sh
[INFO] Starting test: Tests.4
[INFO] Running test: Tests.4
[INFO] Executing command for test Tests.4: sbatch /obfuscated/path/results/test_scenario_example_2024-08-23_22-46-25/Tests.4/0/cloudai_sbatch_script.sh
[INFO] Delayed start for test Tests.2 by 5 seconds.
[INFO] Starting test: Tests.2
[INFO] Running test: Tests.2
[INFO] Executing command for test Tests.2: sbatch /obfuscated/path/results/test_scenario_example_2024-08-23_22-46-25/Tests.2/0/cloudai_sbatch_script.sh
[INFO] Job completed: Tests.1
[INFO] Delayed start for test Tests.3 by 0 seconds.
[INFO] Starting test: Tests.3
[INFO] Running test: Tests.3
[INFO] Executing command for test Tests.3: sbatch /obfuscated/path/results/test_scenario_example_2024-08-23_22-46-25/Tests.3/0/cloudai_sbatch_script.sh
[INFO] Scheduling termination of job 286816 after 5 seconds.
[INFO] Job completed: Tests.2
[INFO] Job completed: Tests.4
[INFO] Job completed: Tests.3
[INFO] All test scenario results stored at: /obfuscated/path/results/test_scenario_example_2024-08-23_22-46-25
[WARNING] Skipping directory '/obfuscated/path/results/test_scenario_example_2024-08-23_22-46-25/Tests.1/0' for test 'sleep_10'
[WARNING] Skipping directory '/obfuscated/path/results/test_scenario_example_2024-08-23_22-46-25/Tests.2/0' for test 'sleep_5'
[WARNING] Skipping directory '/obfuscated/path/results/test_scenario_example_2024-08-23_22-46-25/Tests.3/0' for test 'sleep_5'
[WARNING] Skipping directory '/obfuscated/path/results/test_scenario_example_2024-08-23_22-46-25/Tests.4/0' for test 'sleep_20'

NCCL test

$ python cloudaix.py --mode run --system-config conf/v0.6/general/system/system.toml --test-templates-dir conf/v0.6/general/test_template --tests-dir conf/v0.6/general/test --test-scenario conf/v0.6/general/test_scenario/nccl_test.toml
[INFO] System configuration file: conf/v0.6/general/system/system.toml
[INFO] Test templates directory: conf/v0.6/general/test_template
[INFO] Tests directory: conf/v0.6/general/test
[INFO] Test scenario file: conf/v0.6/general/test_scenario/nccl_test.toml
[INFO] Output directory: None
[INFO] System Name: System
[INFO] Scheduler: slurm
[INFO] Test Scenario Name: nccl-test
[INFO] Checking if test templates are installed.
[INFO] Test Scenario: nccl-test

Section Name: Tests.1
  Test Name: nccl_test_all_reduce
  Description: all_reduce
  No dependencies
[INFO] Initializing Runner
[INFO] Creating SlurmRunner
[INFO] Starting test scenario execution.
[INFO] Starting test: Tests.1
[INFO] Running test: Tests.1
[INFO] Executing command for test Tests.1: sbatch /obfuscated/path/results/nccl-test_2024-08-23_22-50-48/Tests.1/0/cloudai_sbatch_script.sh
[INFO] Job completed: Tests.1
[INFO] All test scenario results stored at: /obfuscated/path/results/nccl-test_2024-08-23_22-50-48

[TaekyungHeo]

Rebased to the main branch.

[srivatsankrishnan]

No major issues with merging this. As discussed in the call, I understand this is blocking the K8s PR as well which needs rebasing. LGTM.