Slurm-Mail

Author: Neil Munday (neil at mundayweb.com)

Repository: https://github.com/neilmunday/slurm-mail

Contents

1. Introduction
2. Requirements
3. Installation
4. Configuration
5. SMTP Settings
6. Customising E-mails
7. Validating E-mails
8. Including Job Output in E-mails
9. Job Arrays
10. Development
11. Linting
12. Testing
13. Upgrading from Slurm-Mail version 3 to 4
14. Upgrading from Slurm-Mail version 4.0-4.9 to 4.10
15. Troubleshooting
16. Contributors

Introduction

E-mail notifications from Slurm are rather brief and all the information is contained in the subject of the e-mail - the body is empty.

Slurm-Mail aims to address this by providing a drop in replacement for Slurm's e-mails to give users much more information about their jobs via HTML e-mails which contain the following information:

• Start/End
• Job name
• Partition
• Work dir
• Elapsed time
• Exit code
• Std out file path
• Std err file path
• No. of nodes used
• Node list
• Requested memory per node
• Maximum memory usage per node
• CPU efficiency
• Wallclock
• Wallclock accuracy

E-mails can be easily customised to your needs using the provided templates (see below).

You can also opt to include a number of lines from the end of the job's output files in the job completion e-mails (see below).

Requirements

• cron
• logrotate
• Python 3.6 or newer
• Slurm 22 or 23
• A working e-mail server

Note: earlier versions of Slurm may work but are not tested with this release of Slurm-Mail.

Installation

Amazon Linux, RedHat and SUSE Based Operating Systems

For each release of Slurm-Mail, RPMs for Amazon Linux 2, RedHat 7/8/9 and SUSE 15 based operating systems are provided at releases.

Once downloaded, install using your appropriate package manager, e.g. for Rocky Linux 8:

dnf localinstall ./slurm-mail-*.noarch.rpm

Or for RedHat 7 and Amazon Linux 2:

yum localinstall ./slurm-mail-*.noarch.rpm

For other operating systems that use RPM packages you can create a package for your OS like so:

dnf -y install python36 rpm-build tar
git clone https://github.com/neilmunday/slurm-mail
slurm-mail/build-tools/build-rpm.sh

The RPM will be written to ~/rpmbuild/RPMS/noarch.

Ubuntu 20 and 22

Pre-built Ubuntu 20 and 22 packages are provided at releases.

For other Debian variants you can create a package for your OS like so:

sudo apt-get install -y fakeroot dh-python lintian lsb-release python3 python3-stdeb
git clone https://github.com/neilmunday/slurm-mail
slurm-mail/build-tools/build-deb.sh

At the end of the execution the location of the built package will be written to stdout.

To install the generated package for example:

apt-get install -y cron logrotate python3 slurm-client
dpkg --force-all -i /tmp/slurm-mail_4.3-ubuntu1_all.deb

From source (as root)

git clone https://github.com/neilmunday/slurm-mail
cd slurm-mail
python setup.py install
cp etc/logrotate.d/slurm-mail /etc/logrotate.d/
cp etc/cron.d/slurm-mail /etc/cron.d/
install -d -m 700 -o slurm -g slurm /var/log/slurm-mail

Note: Depending on your operating system's Python set-up, it is possible that setuptools might install Slurm-Mail to /usr/local rather than /usr.

Configuration

Edit /etc/slurm-mail/slurm-mail.conf to suit your needs. For example, check that the location of sacct is correct. If you are installing from source check that the log and spool directories are set to your desired values.

Change the value of MailProg in your slurm.conf file to /usr/bin/slurm-spool-mail. By default the Slurm config file will be located at /etc/slurm/slurm.conf.

Restart slurmctld:

systemctl restart slurmctld

Slurm-Mail will now log e-mail requests from Slurm users to the Slurm-Mail spool directory /var/spool/slurm-mail.

The cron job created during installation at /etc/cron.d/slurm-mail will execute once per minute to process the spool files, thus making sure that slurmctld is not blocked by processing e-mails.

SMTP Settings

By default Slurm-Mail will send e-mails to a mail server running on the same host as Slurm-Mail is installed on, i.e. localhost.

You can edit the smtp configuration options in /etc/slurm-mail/slurm-mail.conf. For example, to send e-mails via Gmail's SMTP server set the following settings:

smtpServer = smtp.gmail.com
smtpPort = 587
smtpUseTls = yes
smtpUserName = your_email@gmail.com
smtpPassword = your_gmail_password

NOTE: As this file will contain your Gmail password make sure that it has the correct owner, group and file access permissions.

If your SMTP server does not require a login, leave smtpUserName and smtpPassword set to null, i.e.

smtpUserName =
smtpPassword =

For SMTP servers that use SSL rather than starttls please set smtpUseSsl = yes.

E-mail retries

By default Slurm-Mail will attempt to resend e-mails when a previous attempt failed. This can result in repeated failed e-mail attempts if for example a user has specified an invalid e-mail address.

If you would prefer to disable this feature, set the following option in /etc/slurm-mail/slurm-mail.conf:

retryOnFailure = no

In either case, errors for failed e-mail delivery will always be logged in /var/log/slurm-mail/slurm-send-mail.log

Customising E-mails

Templates

Slurm-Mail uses Python's string.Template class to create the e-mails it sends. Under Slurm-Mail's /etc/slurm-mail/templates directory you will find the following files that you can edit to customise e-mails to your needs under the html and text directories. Templates under the html directory are used for HTML e-mails and the templates under text are used for plain text e-mails.

Filename	Template Purpose
ended-array.tpl	Used for jobs in an array that have finished.
ended-array_summary	Used when all jobs in an array have finished.
ended.tpl	Used for jobs that have finished.
invalid-dependency.tpl	Used when a job has an invalid dependency.
job-table.tpl	Used to create the job info table in e-mails.
never-ran.tpl	Used for jobs that never ran.
signature.tpl	Used to create the e-mail signature.
staged-out.tpl	Used when a job's burst buffer stage has completed.
started.tpl	Used for jobs that have started.
started-array-summary.tpl	Used when the first job in an array has started.
started-array.tpl	Used for the first job in an array that has started.
time.tpl	Used when a job reaches a percentage of it's time limit.

Each template has a number of variables which can be used in the generation of e-mails. Please see TEMPLATES for futher details.

Styling

You can adjust the font style, size, colours etc. by editing the Cascading Style Sheet (CSS) file /etc/slurm-mail/style.css used for generating the e-mails.

Date/time format

To change the date/time format used for job start and end times in the e-mails, change the datetimeFormat configuration option in /etc/slurm-mail/slurm-mail.conf. The format string used is the same as Python's datetime.strftime function.

E-mail subject

To change the subject of the e-mails, change the emailSubject configuration option in /etc/slurm-mail. You use the following place holders in the string:

Place holder	Value
$CLUSTER	The name of the cluster
$JOB_NAME	The name of the job
$JOB_ID	The Slurm ID of the job
$STATE	The state of the job

Validating E-mails

By default Slurm-Mail will not perform any checks on the destination e-mail address (i.e the value supplied to sbatch via --mail-user). If you would like Slurm-Mail to only send e-mails for jobs that correspond to a valid e-mail address (e.g. user@some.domain) then you can set the validateEmail option in /etc/slurm-mail/slurm-mail.conf to true. E-mail addresses that failed this check will be logged in /var/log/slurm-mail/slurm-send-mail.log as an error.

The regular expression used to validate e-mail addresses can be configured by adjusting the emailRegEx value in /etc/slurm-mail/slurm-mail.conf.

Including Job Output in E-mails

In /etc/slurm-mail/slurm-mail.conf you can set the includeOutputLines to the number of lines to include from the end of each job's standard out and standard error files.

Notes:

• if the user has decided to use the same file for both standard output and standard error then there will be only one section of job output in the job completion e-mails.
• Job output can only be included if the process that is running slurm-send-mail.py is able to read the user's output files.
• Due to the way scontrol reports filenames that use Slurm's filename patterns only these patterns are supported when including job output in e-mails: %A, %a, %j, %u, and %x.

Job Arrays

Slurm-Mail will honour the behaviour of --mail-type option of sbatch for job arrays. If a user specifies --mail-type=ARRAY_TASKS then Slurm-Mail will send notification e-mails for all jobs in the array. If you want to limit the number of e-mails that will be sent in this scenario then change the arrayMaxNotifications parameter in slurm-mail.conf to a value greater than zero.

Development

Clone this repository to your desktop and make sure you have a supported version of Python installed (see Requirements above).

Pull requests are welcome!

A VSCode settings file is provided in this repository and is configured to allow you to run unit tests from the GUI.

Linting

pylint is used to check that the Slurm-Mail Python source code is formatted correctly.

pip3 install --user pylint

Once install you can run pylint against the Slurm-Mail source code:

pylint setup.py src/slurmmail/*.py tests/unit/*.py tests/integration/docker-slurm/*.py tests/integration/*.py

Testing

In order to run the unit tests you will need to install pytest, e.g.

pip3 install --user pytest

The unit tests can be found at tests/unit and can be invoked from either VSCode or from the command line, e.g.

pytest

Integration tests can be found at tests/integration which also contains a Docker compose file which allows you to experiment with a demo of Slurm-Mail complete with MailHog as a working mail server and webmail client.

Upgrading from Slurm-Mail version 3 to 4

Version 4.0 onwards no longer installs to the /opt/slurm-mail. Instead from version 4.0 onwards install using Python's setuptools module. If your current Slurm-Mail 3.x installation was installed by your operating system's package manager, then you can just upgrade your installation using your package manager (e.g. yum, dnf).

If not, then proceed as follows:

1. install Slurm-Mail 4 using one of the methods describe above - take note of where the scripts and config files are installed (e.g. /usr/bin and /etc).

2. If you have not modified any template files you can skip this step.

cp /opt/slurm-mail/conf.d/templates/* /etc/slurm-mail/templates/html/

Now check the contents of the plain text templates located at /etc/slurm-mail/templates/text and adjust as necessary to meet your requirements.

3. If you have not modified Slurm-Mail's style.css file you can skip this step.

cp /opt/slurm-mail/conf.d/style.css /etc/slurm-mail/

4. If you have not modified slurm-mail.conf you can skip this step.

cp /opt/slurm-mail/conf.d/slurm-mail.conf /etc/slurm-mail/

5. Update your Slurm installation's slurm.conf file:

MailProg=/usr/bin/slurm-spool-mail

Now restart slurmctld:

systemctl restart slurmctld

6. Edit /etc/cron.d/slurm-mail and set the path of the executable to: /usr/bin/slurm-send-mail

7. If you are sure you will no longer need the old version:

rm -rf /opt/slurm-mail

Upgrading from Slurm-Mail version 4.0-4.9 to 4.10

Starting from version 4.10, HTML and plain text e-mail templates are provided. If you have adjusted any templates please copy them from /etc/slurm-mail/templates to /etc/slurm-mail/templates/html and adjust any of the plain text templates as necessary under /etc/slurm-mail/templates/text.

Troubleshooting

1. Check that spool files are being created under: /var/spool/slurm-mail. If they are not:

• check cron is working
• check for invocation of /usr/bin/slurm-spool-mail in the slurmctld logs

2. If spool files are being created but not purged please comment out logFile in the [slurm-send-mail] section in /etc/slurm-mail/slurm-mail.conf and run (as root): /usr/bin/slurm-send-mail -v at the console.

3. If /usr/bin/slurm-send-mail is executing ok but you are not receiving e-mails, then check the mail logs on your server for any mail delivery errors.

Contributors

Thank you to the following people who have contributed code improvements, features and aided the development of Slurm-Mail:

• Dan Barker (@danbarke)
• David Murray (@dajamu)
• drhey (@drhey)
• hakasapl (@hakasapl)
• Hugo Larcher @Hugoch
• Jan-Christoph Klie (@jcklie)
• langefa (@langefa)
• Mehran Khodabandeh (@mkhodabandeh)
• Neil Prockter (@mrgum)
• rzegson (@rzegson)
• sdx23 (@sdx23)
• Simon Feltman (@sfeltman)
• Thanasis Georgiou (@thgeorgiou)
• Woody Chang (@jitkang)