Refactor submission templates

[b-butler]

Description

This PR moves much of the task and number of nodes logic to the environment classes where they can specify any needed parameters using the resources template context value now provided to the Jinja templates. This allows more complicated logic in a simpler to understand format compared to Jinja. This also promotes similar computation of resources, and explicit overrides through inheritance. For some more discussion see #702.

Motivation and Context

This is attempting to solve some bugs in the existing template logic. For one, we sometimes fail on multi-nodes GPU submissions (see #702) and related issues. We also currently modify user resource requests (e.g. rounding CPU tasks to the same number per node) rather than submitting as is or erroring.

Checklist:

• I am familiar with the Contributing Guidelines.
• I agree with the terms of the Contributor Agreement.
• My name is on the list of contributors.
• The changes introduced by this pull request are covered by existing or newly introduced tests.
• The package documentation and framework documentation in signac-docs are up to date with these changes.
• I have updated the changelog and added any related issue and pull request numbers for future reference.

[b-butler]

TODO:

• Document _get_scheduler_resources more fully
• Get feedback on new choices on default settings (e.g. --ntasks over --ntasks-per-node)
• Port over all templates (that we have access to)
• Test all templates
• Regenerate template test reference and validate

Note need to make sure to only set -N when required to not over request resources.

Anyone working on this please feel free to modify this.

[bdice]

@b-butler Can we push this through or close it?

[codecov]

Codecov Report

Merging #722 (f8db823) into main (fd1f3d0) will decrease coverage by 0.09%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##             main     #722      +/-   ##
==========================================
- Coverage   69.25%   69.17%   -0.09%     
==========================================
  Files          44       44              
  Lines        4297     4331      +34     
  Branches      950     1052     +102     
==========================================
+ Hits         2976     2996      +20     
- Misses       1109     1129      +20     
+ Partials      212      206       -6     

Files	Coverage Δ
flow/environment.py	82.18% <100.00%> (+3.86%)	⬆️
flow/environments/incite.py	72.58% <100.00%> (-13.79%)	⬇️
flow/environments/umich.py	85.71% <100.00%> (+3.89%)	⬆️
flow/environments/xsede.py	85.07% <100.00%> (+1.74%)	⬆️
flow/project.py	83.15% <100.00%> (+0.13%)	⬆️

[tommy-waltmann]

I have two small suggestions and a question:

This PR touches a lot of cluster templates. Does flow normally need to be validated on each of the clusters in addition to unit tests? If so, has the validation been done after the changes in this PR?

[b-butler]

This PR touches a lot of cluster templates. Does flow normally need to be validated on each of the clusters in addition to unit tests? If so, has the validation been done after the changes in this PR?

@tommy-waltmann that is part of the reason the PR took so long. All of the templates changed have been validated, I believe (need to double check Andes). That is also the reason some are not changed since I could not access them.