DeepOps Release 22.04

[ajdecon]

DeepOps 22.04 Release Notes

Known Issues

• Kubeflow deployment is currently broken due to incompatibility between current Kubeflow and Kubernetes 1.22. Kubeflow deployment will be updated to add support when Kubeflow releases 1.6.

General

• Extensive improvements to automated testing with Jenkins, Ansible Molecule, and ansible-lint
• Update MIG playbook to use the new nvidia-mig-manager systemd service
• Updates to roles for nvidia-docker and GPU driver
• Various bug fixes

Slurm

• Enhanced NCCL tests for Slurm cluster validation
• Make use of pam_slurm_adopt optional
• Break out multiple sections in Slurm inventory file

Kubernetes

• Update to Kubernetes 1.22.6
• Update default container runtime from dockershim to containerd
• Add support for NVIDIA Network Operator
• Add support to deploy NVIDIA Deep Learning Examples on Kubernetes clusters
• Update to GPU Operator 1.9

Changes

Bugs/Enhancements

• Fixes for rsyslog server role (Fixing rsyslog-server role. #1096, Update roles/rsyslog-server/tasks/main.yml #1098)
• Update NetApp Trident default version number and branding (Update NetApp Trident default version number and branding #1105)
• Introduce a common script library (Introduce a common script library, config for env vars, and inject these into all scripts #953)
• Update versions of monitoring stack components (Monitoring update #1107)
• Updates to Jenkins testing (Update Jenkinsfile-nightly to reflect new Jenkins setup #1112, Increase timeout for multinode monitoring test in k8s #1127, Fix multi-node K8s deployment script #1133, Disable k8s container registry by default #1137, Disable container registry test #1138, Add more tests to cover containerd/driver container k8s deployments #1139, Update jenkins matrix test to support driver container vs host driver #1150, Add docker vs containerd to manual jenkins options #1151)
• Fixes for setup script (Fixing scripts/setup.sh. #1114)
• Automated testing of DeepOps roles using Molecule (Add molecule tests for "easy" roles #1094, Temporarily disable CentOS 8 tests in Molecule #1116, Fix Ansible Galaxy dependencies to address Molecule test failures #1158)
• Update nvidia.nvidia_docker role to v1.2.4 (Update nvidia.nvidia_docker dependency to v1.2.4 #1121)
• Automated deployment of Deep Learning Examples (Deep learning examples #1083, Update Deep Learning Examples automation to handle containerd #1145)
• Make it optional to use pam_slurm_adopt (Make it optional to use pam_slurm_adopt #1111)
• Convert MIG playbook to use nvidia-mig-manager service (Convert nvidia-mig.yml to leverage the nvidia-mig-manager systemd service #1106)
• Update to GPU Operator 1.9 (GPU Operator 1.9 support with DGX inclusion #1074)
• Automatically run ansible-lint on each role (Automatically run ansible-lint on each role #1129)
• Update Kubeflow deployment script to Kubeflow 1.4 (Rework deploy_kubeflow.sh for Kubeflow 1.4 #1104)
• Remove old build dirs during Slurm upgrade (Slurm build: remove old build dirs during install #1101)
• Fixes to ood-wrapper role (ood-wrapper fixes for #963 #1125)
• Documentation of network ports (Add doc on network ports for common services #1126)
• Set missing defaults in playbooks (Set missing defaults in playbooks #1134)
• Update to Kubespray v2.18.1 and containerd (Kubespray v2.18.0 and containerd runtime #1043, Misc fixes for Jenkins CI #1141)
• Fix GPU Operator config (Fix GPU Operator runtime configuration #1136)
• Break out functional host groups in Slurm inventory (Break out functional host groups for slurm cluster #1087)
• Fix ordering in k8s cluster deployment (Fix some ordering in k8s-cluster.yml to install Helm properly and run all commands from kube-master[0]- fixing CentOS install #1128)
• Update nvidia.nvidia_driver role to v2.2.0 (Update role nvidia.nvidia_driver to v2.1.0 #1143, Update to nvidia.nvidia_driver role v2.2.0 #1160)
• Add support for NVIDIA Network Operator (Adding Nvidia network operator for DGX #1113, Fix linting errors in network operator role #1156)
• Enhanced NCCL tests for Slurm validation (Enhanced nccl tests slurm validation playbook. #1042)
• Fix git.io shortlinks (git.io is being deprecated, fix shortlinks #1163)
• Check for SELinux disabled in SELinux tasks (Check for SELinux disabled in Ansible tasks #1162)

Upgrade Steps

If you are upgrading to this version of DeepOps from a previous release you will need to follow the upgrade section of the Slurm or Kubernetes Deployment Guides. In addition to this, the ./scripts/setup.sh script must be re-run and any new variables in the config.example files should be added to the existing config. For a full diff from release 22.01 run git diff 22.01 22.04 -- config.example/. If you encounter problem please open a GitHub issue. See the update guide for additional guidance.

Notes

[ajdecon]

@dholt @supertetelman : Please review the release notes and let me know if this looks good! Then we can merge and tag a release.

[dholt]

Looks great to me. I like this better than the way we have been doing releases