Add exponential back-off to retryStrategy

[simster7]

Closes: #700. Supports duration, factor, maxDuration.

Example:

# This example demonstrates the use of retry back offs
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: retry-backoff-
spec:
  entrypoint: retry-backoff
  templates:
  - name: retry-backoff
    retryStrategy:
      limit: 10
      backoff:
        duration: 1       # Default unit is seconds. Could also be a Duration, e.g.: "2m", "6h", "1d"
        factor: 2
        maxDuration: "1m" # Default unit is seconds. Could also be a Duration, e.g.: "2m", "6h", "1d"
    container:
      image: python:alpine3.6
      command: ["python", -c]
      # fail with a 66% probability
      args: ["import random; import sys; exit_code = random.choice([1, 1]); sys.exit(exit_code)"]

When waiting to retry (see parent node message):

Name:                retry-container-69ggh
Namespace:           argo
ServiceAccount:      default
Status:              Running
Created:             Tue Nov 19 12:44:05 -0800 (30 seconds ago)
Started:             Tue Nov 19 12:44:05 -0800 (30 seconds ago)
Duration:            30 seconds

STEP                                             PODNAME                           DURATION  MESSAGE
 ● retry-container-69ggh (retry-container)                                                   Retrying in 12 seconds
 ├-✖ retry-container-69ggh(0) (retry-container)  retry-container-69ggh-4109076362  3s        failed with exit code 1
 ├-✖ retry-container-69ggh(1) (retry-container)  retry-container-69ggh-283735303   4s        failed with exit code 1
 ├-✖ retry-container-69ggh(2) (retry-container)  retry-container-69ggh-1894828012  3s        failed with exit code 1
 └-✖ retry-container-69ggh(3) (retry-container)  retry-container-69ggh-216919017   3s        failed with exit code 1

When max duration limit exceeded:

Name:                retry-container-69ggh
Namespace:           argo
ServiceAccount:      default
Status:              Failed
Message:             Max duration limit exceeded
Created:             Tue Nov 19 12:44:05 -0800 (1 minute ago)
Started:             Tue Nov 19 12:44:05 -0800 (1 minute ago)
Finished:            Tue Nov 19 12:45:08 -0800 (now)
Duration:            1 minute 3 seconds

STEP                                             PODNAME                           DURATION  MESSAGE
 ✖ retry-container-69ggh (retry-container)                                                   Max duration limit exceeded
 ├-✖ retry-container-69ggh(0) (retry-container)  retry-container-69ggh-4109076362  3s        failed with exit code 1
 ├-✖ retry-container-69ggh(1) (retry-container)  retry-container-69ggh-283735303   4s        failed with exit code 1
 ├-✖ retry-container-69ggh(2) (retry-container)  retry-container-69ggh-1894828012  3s        failed with exit code 1
 ├-✖ retry-container-69ggh(3) (retry-container)  retry-container-69ggh-216919017   3s        failed with exit code 1
 └-✖ retry-container-69ggh(4) (retry-container)  retry-container-69ggh-2363615606  3s        failed with exit code 1

[simster7]

A bit of an interesting trade-off here is that since we are changing the parentNode message, the workflow will be immediately re-queued. This makes re-queuing the Workflow with a duration (using woc.requeue(time.Until(waitingDeadline))) moot, and introduces some "polling" behavior in which the workflow is operated on once a second—mostly to update the "Retrying in ..." message.

If we decide that this polling behavior is not acceptable, then could make the "Retrying in ..." message static (e.g. "Retrying at [TIME]") and we could then re-queue with a duration as explained above.

[epa095]

Sorry for the bikeshedding, but should this not be called "base" instead, given that it is, in fact, the base of the exponential function?
Also, good work :-)

[simster7]

Duration is actually what K8s uses :) I think because technically factor is optional; if left at 0 then the backoff would simply apply the same duration wait

[sarabala1979]

Can you resolve the conflicts?