Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Slurm6. Add support for dynamic nodeset. #1986

Merged
merged 1 commit into from
Mar 15, 2024

Conversation

mr0re1
Copy link
Collaborator

@mr0re1 mr0re1 commented Nov 22, 2023

Unlike normal or TPU nodesets, don't add dedicated modules, but instead expect user to specify them. E.g.:

  - id: savanna
    source: community/modules/compute/schedmd-slurm-gcp-v6-partition
    settings:
      partition_name: savanna
      nodeset_dyn:
      - nodeset_name: giraffe
        nodeset_feature: spots
      - nodeset_name: zebra
        nodeset_feature: stripes

@mr0re1 mr0re1 added the release-module-improvements Added to release notes under the "Module Improvements" heading. label Nov 22, 2023
Copy link
Contributor

@danielahlin danielahlin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Testing this I get the following error:

Error: Resource precondition failed

on .terraform/modules/slurm_controller.slurm_partition/terraform/slurm_cluster/modules/slurm_partition/main.tf line 53, in resource "null_resource" "partition":
53: condition = local.has_node || local.has_dyn || local.has_tpu
├────────────────
│ local.has_dyn is false
│ local.has_node is false
│ local.has_tpu is false

Partition must contain at least one type of nodeset.

Error: Resource precondition failed

on .terraform/modules/slurm_controller.slurm_partition/terraform/slurm_cluster/modules/slurm_partition/main.tf line 57, in resource "null_resource" "partition":
57: condition = ((!local.has_node || !local.has_dyn) && local.has_tpu) || ((local.has_node || local.has_dyn) && !local.has_tpu)
├────────────────
│ local.has_dyn is false
│ local.has_node is false
│ local.has_tpu is false

Partition cannot contain TPU and non-TPU nodesets.

@mr0re1 mr0re1 assigned mr0re1 and unassigned danielahlin Nov 30, 2023
@mr0re1 mr0re1 removed their assignment Dec 13, 2023
@mr0re1 mr0re1 closed this Mar 7, 2024
auto-merge was automatically disabled March 7, 2024 04:39

Pull request was closed

@mr0re1 mr0re1 deleted the s6_dyn branch March 7, 2024 04:39
@mr0re1 mr0re1 restored the s6_dyn branch March 8, 2024 01:58
@mr0re1 mr0re1 assigned mr0re1 and unassigned danielahlin Mar 8, 2024
@mr0re1 mr0re1 removed the request for review from danielahlin March 8, 2024 01:59
@mr0re1 mr0re1 reopened this Mar 8, 2024
@mr0re1 mr0re1 assigned danielahlin and unassigned mr0re1 Mar 8, 2024
Unlike normal or TPU nodesets, don't add dedicated modules, but instead
expect user to specify them  DRAFT.
Copy link
Contributor

@danielahlin danielahlin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, this results in the correct configuration of the dynamic partition.

@mr0re1 mr0re1 merged commit 768fb18 into GoogleCloudPlatform:develop Mar 15, 2024
8 of 40 checks passed
@mr0re1 mr0re1 deleted the s6_dyn branch March 15, 2024 23:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
release-module-improvements Added to release notes under the "Module Improvements" heading.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants