Migrate katib to new test-infra

[PatrickXYS]

What this PR does / why we need it:
Migrate katib to new test-infra

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #1422

Special notes for your reviewer:

1. Please confirm that if this PR changes any image versions, then that's the sole change this PR makes.

Release note:

NONE

/cc @andreyvelich
/hold

[PatrickXYS]

 Failed: cannot unmarshall spec: cannot convert int64 to string

Checking

[PatrickXYS]

Same reason, roll back all the changes and see what's going on

[PatrickXYS]

INFO|2021-01-26T21:17:32|/src/kubeflow/testing/py/kubeflow/testing/argo_client.py|48| Parsing ApiException body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"workflows.argoproj.io \"kubeflow-katib-presubmit-e2e-v1beta1-1423-3f174af-3552-0e2b\" not found","reason":"NotFound","details":{"name":"kubeflow-katib-presubmit-e2e-v1beta1-1423-3f174af-3552-0e2b","group":"argoproj.io","kind":"workflows"},"code":404}

After applying workflow, it can't find corresponding workflow.

[PatrickXYS]

/test kubeflow-katib-presubmit

Kick off again to see if the issue happen

[PatrickXYS]

Same issue... Dig into it to see why it can't apply generated-workflow into remote argo cluster

[PatrickXYS]

It works this time, looks like those TTL configuration causes the issue

[PatrickXYS]

Removed ttl parameters since we have already configured default workflow spec:

configmap -o yaml
apiVersion: v1
data:
  workflowDefaults: |
    metadata:
      annotations:
        argo: workflows
    spec:
      ttlStrategy:
        secondsAfterSuccess: 604800
        secondsAfterCompletion: 604800
        secondsAfterFailure: 604800
      activeDeadlineSeconds: 86400
...

https://argoproj.github.io/argo/fields/#ttlstrategy

That's being said, WG does not need to configure ad-hoc activeDeadlineSeconds and ttlStrategy parameters.

/cc @andreyvelich

[PatrickXYS]

https://argo.kubeflow-testing.com/workflows/kubeflow-test-infra/kubeflow-katib-presubmit-e2e-v1beta1-1423-e99e074-4128-402f?tab=workflow&nodeId=kubeflow-katib-presubmit-e2e-v1beta1-1423-e99e074-4128-402f-1929247164

Works as expected

[andreyvelich]

https://argoproj.github.io/argo/fields/#ttlstrategy

@PatrickXYS That makes sense.
Should we merge this ?

[andreyvelich]

Can you remove this comment, please?

[PatrickXYS]

https://argoproj.github.io/argo/fields/#ttlstrategy

@PatrickXYS That makes sense.
Should we merge this ?

It's already in, our new Argo cluster installed with v2.12.3 argo workflow

[PatrickXYS]

/unhold

@andreyvelich Can you help approve this PR and get it merged?

[andreyvelich]

/approve

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: andreyvelich, PatrickXYS

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [andreyvelich]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[PatrickXYS]

Looks like you also need to lgtm @andreyvelich

[andreyvelich]

/lgtm