Add support for array jobs and include errors and (portion of) log file in the email #7
Conversation
|
If you accept PRs and also think that this PR is worth merging, I can spend some time cleaning it up :) |
|
Hi, Thanks for the PR.
Yes, I do accept pull requests. I'll try to take a look at your modifications later this week. Cheers, Neil. |
| # if this server is an SMTP server: | ||
| s = smtplib.SMTP('localhost') | ||
| # Otherwise use a secondary server for example gmail: | ||
| # s = smtplib.SMTP('smtp.gmail.com', 587, timeout=60) | ||
| # s.ehlo() | ||
| # s.starttls() | ||
| # s.ehlo() | ||
| # s.login(USERNAME, APP_PASSWORD) |
There was a problem hiding this comment.
I think that it might be useful to add additional configuration options in conf.d/slurm-mail.conf to allow the user to define which mail server to use.
I'll add this as a separate issue to be addressed.
There was a problem hiding this comment.
Now implemented in 1.7 branch.
| msg['subject'] = 'Job %s.%d: %s' % (cluster, jobId, state) | ||
| msg['subject'] = 'Job %s.%s: %s' % (cluster, job_name, state) |
There was a problem hiding this comment.
I prefer the job ID to be in the subject line of the e-mail. However, I could add an option to conf.d/slurm-mail.conf that allows the format of the subject to be customisable, a bit like the date format can. I'll add an issue for this.
There was a problem hiding this comment.
Customisable e-mail subject now implemented in 1.7 branch.
| try: | ||
| if '\n' in stdout: | ||
| stdout = stdout.split('\n')[job_count-1] | ||
| except: | ||
| stdout_last_100 += '\n\n Error in parsing this file \n \n' | ||
| rtnCode = 1 | ||
| if rtnCode == 0: | ||
| jobDic = {} | ||
| if IS_PYTHON_3: | ||
| stdout = stdout.decode() | ||
| for i in stdout.split(' '): | ||
| x = i.split('=', 1) | ||
| if len(x) == 2: | ||
| jobDic[x[0]] = x[1] | ||
| stdoutFile = jobDic['StdOut'] | ||
| stdoutFiles += stdoutFile | ||
| logging.info('stdoutFile: {}'.format(stdoutFile)) | ||
| stderrFile = jobDic['StdErr'] | ||
| stderrFile = '' if stderrFile == stdoutFile else stderrFile | ||
| stderrFiles += stderrFile | ||
| try: | ||
| with open(stdoutFile, 'r') as fread: | ||
| stdout_last_100 += stdoutFile+':\n' | ||
| for i, line in enumerate(fread): | ||
| stdout_last_100 += line | ||
| if i == 100: | ||
| break | ||
| stdout_last_100 += '\n\n' | ||
| except: | ||
| logging.exception("couldn't read the stdout file") | ||
| if stderrFile != '': | ||
| try: | ||
| with open(stderrFile, 'r') as fread: | ||
| stderr_content += stderrFile+':\n' | ||
| stderr_content += fread.read()+'\n\n' | ||
| except: | ||
| logging.exception("couldn't read the stderr file") | ||
| else: | ||
| logging.error('failed to run: %s' % cmd) | ||
| logging.error(stdout) | ||
| logging.error(stderr) |
There was a problem hiding this comment.
Nice idea adding some of the standard out to the e-mail. One slight issue though is that it is not always the case that the server running slurmctld also mounts the file system where the user's job output has been written to. For example, on supercomputers that use Slurm, often the parallel file system (e.g. Lustre) where the user's job output is written to is not mounted on the Slurm servers. I think it would better to remove this code for now in this particular pull request and just focus on job arrays.
|
Job arrays now implemented in 1.7 branch. |
|
All features now implemented in version 1.7 plus additional enhancements |
No description provided.