Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Refactor test scenario directory structure to remove unnecessary indirection #44

Merged
merged 1 commit into from
May 29, 2024
Merged

Refactor test scenario directory structure to remove unnecessary indirection #44

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,15 +46,15 @@ To simulate running experiments without execution, use the dry-run mode:
cloudai\
--mode dry-run\
--system_config_path conf/v0.6/general/system/example_slurm_cluster.toml\
--test_scenario_path conf/v0.6/general/test_scenario/sleep/test_scenario.toml
--test_scenario_path conf/v0.6/general/test_scenario/sleep.toml
```

To run experiments, execute Cloud AI CLI in run mode:
```bash
cloudai\
--mode run\
--system_config_path conf/v0.6/general/system/example_slurm_cluster.toml\
--test_scenario_path conf/v0.6/general/test_scenario/sleep/test_scenario.toml
--test_scenario_path conf/v0.6/general/test_scenario/sleep.toml
```

To generate reports, execute Cloud AI CLI in generate-report mode:
Expand Down
6 changes: 3 additions & 3 deletions USER_GUIDE.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -80,7 +80,7 @@ python main.py\
```

### Step 5: Run Experiments and Generate Reports
You can find predefined NCCL test schemas under `conf/v0.6/general/test` and a test scenario under `conf/v0.6/general/test_scenario/nccl_test`. In a test schema file, you can adjust arguments as shown below. In the cmd_args section, you can provide different values other than the default values for each argument. In extra_cmd_args, you can provide additional arguments that will be appended after the NCCL test command. You can specify additional environment variables in the extra_env_vars section.
You can find predefined NCCL test schemas under `conf/v0.6/general/test` and a test scenario at `conf/v0.6/general/nccl_test.toml`. In a test schema file, you can adjust arguments as shown below. In the cmd_args section, you can provide different values other than the default values for each argument. In extra_cmd_args, you can provide additional arguments that will be appended after the NCCL test command. You can specify additional environment variables in the extra_env_vars section.
```
name = "nccl_test_bisection"
description = "Bisection"
Expand Down Expand Up @@ -118,13 +118,13 @@ To generate NCCL test commands without actual execution, use the dry-run mode. Y
python main.py\
--mode dry-run\
--system_config_path conf/v0.6/general/system/example_slurm_cluster.toml\
--test_scenario_path conf/v0.6/general/test_scenario/nccl_test/test_scenario.toml
--test_scenario_path conf/v0.6/general/test_scenario/nccl_test.toml
```

You can run NCCL test experiments with the following command. Whenever you run CloudAI in the run mode, a new directory will be created under the results directory with the timestamp. In the directory, you can find the results from the test scenario including stdout and stderr. Once completed successfully, you can find generated reports under the directories as well.
```bash
python main.py\
--mode run\
--system_config_path conf/v0.6/general/system/example_slurm_cluster.toml\
--test_scenario_path conf/v0.6/general/test_scenario/nccl_test/test_scenario.toml
--test_scenario_path conf/v0.6/general/test_scenario/nccl_test.toml
```
0 ...scenario/chakra_replay/test_scenario.toml → .../general/test_scenario/chakra_replay.toml
View file
Open in desktop
File renamed without changes.
0 ...est_scenario/nccl_test/test_scenario.toml → ...v0.6/general/test_scenario/nccl_test.toml
View file
Open in desktop
File renamed without changes.
0 ...scenario/nemo_launcher/test_scenario.toml → .../general/test_scenario/nemo_launcher.toml
View file
Open in desktop
File renamed without changes.
0 ...mo_launcher_with_noise/test_scenario.toml → ...st_scenario/nemo_launcher_with_noise.toml
View file
Open in desktop
File renamed without changes.
0 ...al/test_scenario/sleep/test_scenario.toml → conf/v0.6/general/test_scenario/sleep.toml
View file
Open in desktop
File renamed without changes.
0 ...test_scenario/ucc_test/test_scenario.toml → .../v0.6/general/test_scenario/ucc_test.toml
View file
Open in desktop
File renamed without changes.
4 changes: 2 additions & 2 deletions tests/test_acceptance.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,8 +5,8 @@
from cloudai.__main__ import handle_dry_run_and_run

SLURM_TEST_SCENARIOS = [
Path("conf/v0.6/general/test_scenario/sleep/test_scenario.toml"),
Path("conf/v0.6/general/test_scenario/ucc_test/test_scenario.toml"),
Path("conf/v0.6/general/test_scenario/sleep.toml"),
Path("conf/v0.6/general/test_scenario/ucc_test.toml"),
]


Expand Down