WorkflowTemplate resolution should happen once

[jessesuen]

Spawned from #1312 (comment)

When resolving workflow templates, we should do this once when we start processing the workflow, and store the resolved templates in status. This will handle the problem of templates changing their meaning, or disappearing while workflows are running.

There's also the question about how to handle retry of workflows which are using templates. I feel behavior should be as follows:

1. argo retry should not re-resolve workflow templates
2. argo resubmit should re-resolve any workflow templates

We could consider adding flags to re-resolve templates or not during those two commands.

[jessesuen]

One corner case:

3. argo resubmit --memoized should probably re-use resolved workflow templates. I think we need to re-use the resolved templates because it is re-using prior node status information, and allowing templates to change meaning during resubmit may have inconsistent results.

[simster7]

Closing as resolved