v1.16.0: New Lustre Example, Slurm v5.6.2, & HTCondor Improvements

[nick-stroud]

Improvements

• New simple Lustre example. (blueprint, documentation)
• htcondor-execute-point: Added option to HTCondor autoscaler for minimum number of idle VMs to decrease job startup time.
• htcondor-execute-point: Added option to set boot disk size.
• New validator reports unused deployment variables. (documentation)
• Expanded options for skipping individual validators. (documentation)
• terraform.tfvars file generated in the deployment folder is written in stable order, making it easier to track in version control.
• Test and documentation updates.

Version updates

• Slurm V5 modules minor update v5.6.0 > v5.6.2 (full changelog)
  • Now compatible with Terraform 1.4.0.
  • Resume failures now notify srun of the error.
  • setup.log now discoverable in GCP Cloud Logger.
  • Fix slurm and slurm-gcp logs not showing up in Cloud Logging.

What's Changed

• Bump googleapis-common-protos from 1.54.0 to 1.58.0 in /community/front-end/ofe by @dependabot in Bump googleapis-common-protos from 1.54.0 to 1.58.0 in /community/front-end/ofe #1014
• Bump httplib2 from 0.20.4 to 0.21.0 in /community/front-end/ofe by @dependabot in Bump httplib2 from 0.20.4 to 0.21.0 in /community/front-end/ofe #1015
• Bump wrapt from 1.13.3 to 1.15.0 in /community/front-end/ofe by @dependabot in Bump wrapt from 1.13.3 to 1.15.0 in /community/front-end/ofe #1016
• Bump django-extensions from 3.1.5 to 3.2.1 in /community/front-end/ofe by @dependabot in Bump django-extensions from 3.1.5 to 3.2.1 in /community/front-end/ofe #1018
• Remove ModuleToGroup from DeploymentConfig by @mr0re1 in Remove ModuleToGroup from DeploymentConfig #1022
• Add enforcement of minimum # of idle VMs to HTCondor autoscaler by @tpdownes in Add enforcement of minimum # of idle VMs to HTCondor autoscaler #983
• Bump grpcio-status from 1.43.0 to 1.51.3 in /community/front-end/ofe by @dependabot in Bump grpcio-status from 1.43.0 to 1.51.3 in /community/front-end/ofe #1017
• Fix broken markdown link by @mr0re1 in Fix broken markdown link #1024
• Update year of Terraform deployment license by @tpdownes in Update year of Terraform deployment license #1026
• Fix startup-options test by @mr0re1 in Fix startup-options test #1029
• Fix broken test config use-resources.yaml by @mr0re1 in Fix broken test config use-resources.yaml #1030
• Add an integration test for chrome-remote-desktop module by @nick-stroud in Add an integration test for chrome-remote-desktop module #1027
• Fix: dependabot proposed incompatible requirements by @nick-stroud in Fix: dependabot proposed incompatible requirements #1028
• Merge v1.14.1 back to develop by @nick-stroud in Merge v1.14.1 back to develop #1033
• Add retries to apt tasks in chrome-remote-desktop to account for lock contention with unattended-upgrades by @nick-stroud in Add retries to apt tasks in chrome-remote-desktop to account for lock contention with unattended-upgrades #1025
• Bring all provider_meta versions up to current version by @nick-stroud in Bring all provider_meta versions up to current version #1034
• Update pre-commit hook repos by @tpdownes in Update pre-commit hook repos #1035
• Fix dtype of slurm node-group.preemtible to bool by @mr0re1 in Fix dtype of slurm node-group.preemtible to bool #1036
• Set label dtype = map(string) for community/modules by @mr0re1 in Set label dtype = map(string) for community/modules #1038
• Fix labels dtype in DDN-EXAScaler by @mr0re1 in Fix labels dtype in DDN-EXAScaler #1039
• Make readers usable outside of package modulereader by @mr0re1 in Make readers usable outside of package modulereader #1040
• Fix slurm gcp v5 validation message in partition by @cboneti in Fix slurm gcp v5 validation message in partition #1043
• Add a test to enforce contracts on modules interfaces by @mr0re1 in Add a test to enforce contracts on modules interfaces #1041
• Bump google.golang.org/api from 0.112.0 to 0.114.0 by @dependabot in Bump google.golang.org/api from 0.112.0 to 0.114.0 #1044
• Bump github.com/googleapis/gax-go/v2 from 2.7.1 to 2.8.0 by @dependabot in Bump github.com/googleapis/gax-go/v2 from 2.7.1 to 2.8.0 #1045
• Bump cloud.google.com/go/serviceusage from 1.5.0 to 1.6.0 by @dependabot in Bump cloud.google.com/go/serviceusage from 1.5.0 to 1.6.0 #1046
• Bump github.com/zclconf/go-cty from 1.13.0 to 1.13.1 by @dependabot in Bump github.com/zclconf/go-cty from 1.13.0 to 1.13.1 #1047
• Bump github.com/go-git/go-git/v5 from 5.6.0 to 5.6.1 by @dependabot in Bump github.com/go-git/go-git/v5 from 5.6.0 to 5.6.1 #1048
• Add integration test coverage for add_deployment_name_before_prefix by @nick-stroud in Add integration test coverage for add_deployment_name_before_prefix #1037
• Minor refactoring of code around expressions by @mr0re1 in Minor refactoring of code around expressions #1042
• Upgrade pip in builder image by @tpdownes in Upgrade pip in builder image #1056
• Add simple sanity test installing the OFE virtual environment by @tpdownes in Add simple sanity test installing the OFE virtual environment #1057
• Order settings alphabettically and format main.tf by @mr0re1 in Order settings alphabettically and format main.tf #1059
• Add links to related material on YouTube by @nick-stroud in Add links to related material on YouTube #1061
• Resolve anticipated merge conflict between develop and main by @tpdownes in Resolve anticipated merge conflict between develop and main #1066
• Merge v1.15.0 release into develop by @tpdownes in Merge v1.15.0 release into develop #1070
• Bump github.com/hashicorp/go-getter from 1.7.0 to 1.7.1 by @dependabot in Bump github.com/hashicorp/go-getter from 1.7.0 to 1.7.1 #1067
• Use stable order while writing variables and backend configs by @mr0re1 in Use stable order while writing variables and backend configs #1071
• Increase version of Terraform google providers by @tpdownes in Increase version of Terraform google providers #1072
• Implement exponential backoff in startup-script by @tpdownes in Implement exponential backoff in startup-script #1073
• Update to slurm-gcp 5.6.2 (from 5.6.0) by @SkylerMalinowski in Update to slurm-gcp 5.6.2 (from 5.6.0) #1074
• Update slurm images by @tpdownes in Update slurm images #1075
• Add deployment_name to HTCondor example VMs by @tpdownes in Add deployment_name to HTCondor example VMs #1076
• Remove labels-specific logic from tfwriter. by @mr0re1 in Remove labels-specific logic from tfwriter. #1064
• Add additional documentation on enable_reconfigure troubleshooting by @nick-stroud in Add additional documentation on enable_reconfigure troubleshooting #1077
• Use stable order while writing terraform.tfvars by @mr0re1 in Use stable order while writing terraform.tfvars #1080
• Refactoring to support edge-tracking in graph by @tpdownes in Refactoring to support edge-tracking in graph #1078
• Refactor moduleConnections as a map tracking source module of connection by @tpdownes in Refactor moduleConnections as a map tracking source module of connection #1079
• Remove field Module.ModuleName add DeploymentSource instead by @mr0re1 in Remove field Module.ModuleName add DeploymentSource instead #1081
• Improve version constraint on batch-job-template module by @tpdownes in Improve version constraint on batch-job-template module #1084
• Lustre example by @cboneti in Lustre example #1082
• Resolve accidental destruction of startup-scripts by @tpdownes in Resolve accidental destruction of startup-scripts #1085
• Updating image in the slurm-gcp-v5 Ubuntu example by @cboneti in Updating image in the slurm-gcp-v5 Ubuntu example #1087
• Show core examples first by @cboneti in Show core examples first #1088
• Use latest update to startup-script module by @tpdownes in Use latest update to startup-script module #1090
• Bump cloud.google.com/go/compute from 1.18.0 to 1.19.0 by @dependabot in Bump cloud.google.com/go/compute from 1.18.0 to 1.19.0 #1089
• Check validations during make tests; Add flag to skip validators by @mr0re1 in Check validations during make tests; Add flag to skip validators  #1032
• Changing CRD test to us-central1-f by @cboneti in Changing CRD test to us-central1-f #1093
• Changing the GPU type of quantum-circuit-simulator to T4. by @cboneti in Changing the GPU type of quantum-circuit-simulator to T4. #1091
• Unify network_storage variables type by @mr0re1 in Unify network_storage variables type #1092
• Several minor fixes by @tpdownes in Several minor fixes #1095
• Bump google-api-python-client from 2.37.0 to 2.82.0 in /community/front-end/ofe by @dependabot in Bump google-api-python-client from 2.37.0 to 2.82.0 in /community/front-end/ofe #1068
• Bump platformdirs from 2.5.0 to 3.1.1 in /community/front-end/ofe by @dependabot in Bump platformdirs from 2.5.0 to 3.1.1 in /community/front-end/ofe #1053
• Bump pyparsing from 3.0.7 to 3.0.9 in /community/front-end/ofe by @dependabot in Bump pyparsing from 3.0.7 to 3.0.9 in /community/front-end/ofe #1052
• Bump pyjwt from 2.4.0 to 2.6.0 in /community/front-end/ofe by @dependabot in Bump pyjwt from 2.4.0 to 2.6.0 in /community/front-end/ofe #1050
• Bump lazy-object-proxy from 1.7.1 to 1.9.0 in /community/front-end/ofe by @dependabot in Bump lazy-object-proxy from 1.7.1 to 1.9.0 in /community/front-end/ofe #1049
• Updating requirements for ofe by @cboneti in Updating requirements for ofe #1098
• Fixing issue Creating Filestore via Front End is broken due to missing required setting (network_id) #1096, and dependency conflicts for OFE. by @ek-nag in Fixing issue #1096, and dependency conflicts for OFE. #1099
• Track graph edges from modules to deployment variables by @tpdownes in Track graph edges from modules to deployment variables #1094
• Unused deployment variables validator by @tpdownes in Unused deployment variables validator #1101
• Simplify test_module_not_used implementation by @tpdownes in Simplify test_module_not_used implementation #1108
• Bump django-allauth from 0.53.1 to 0.54.0 in /community/front-end/ofe by @dependabot in Bump django-allauth from 0.53.1 to 0.54.0 in /community/front-end/ofe #1104
• Bump google-auth from 2.17.0 to 2.17.1 in /community/front-end/ofe by @dependabot in Bump google-auth from 2.17.0 to 2.17.1 in /community/front-end/ofe #1102
• Bump django-extensions from 3.1.5 to 3.2.1 in /community/front-end/ofe by @dependabot in Bump django-extensions from 3.1.5 to 3.2.1 in /community/front-end/ofe #1103
• Track explicit connections between modules by @tpdownes in Track explicit connections between modules #1109
• Add ability to set disk size for htcondor execute point by @thiagosgobe in Add ability to set disk size for htcondor execute point #1107
• Test infra automation. Init commit by @mr0re1 in Test infra automation. Init commit #1105
• version update by @mr0re1 in version update #1137
• Release v1.16.0 by @mr0re1 in Release v1.16.0 #1147

Full Changelog: v1.15.0...v1.16.0

---

This discussion was created from the release v1.16.0: New Lustre Example, Slurm v5.6.2, & HTCondor Improvements.