Add a pause between slurm srun tests

[heyealex]

Submission Checklist

• Have you installed and run this change against pre-commit? pre-commit install
• Are all tests passing? make tests
• If applicable, have you written additional unit tests to cover this
  change?
• Is unit test coverage still above 80%?
• Have you updated any application documentation such as READMEs and user
  guides?
• Have you followed the guidelines in our Contributing document?

[heyealex]

This worked overnight, I'm having issues kicking off another run this morning (Internal Error?). I'll try again shortly.

[tpdownes]

I would consider wait_for ansible resource. You could also do something like wait_for the existence of the job output file.

e.g.

- name: Sleep for 300 seconds and continue with play
  wait_for:
    timeout: 300
  delegate_to: localhost

- name: Wait until the file /tmp/foo is present before continuing
  wait_for:
    path: /tmp/foo

(I don't know what delegate_to is doing in this example)

[tpdownes]

You might also consider some until and retry if the point is resiliency:

https://docs.ansible.com/ansible/latest/user_guide/playbooks_loops.html#retrying-a-task-until-a-condition-is-met