Add Azure support for kfctl.sh

[ritazh]

kfctl.sh will be going away eventually, so this is just a stop gap to support Azure until the go client kfctl is ready for other cloud providers.

@jlewi @kkasravi PTAL

cc @sozercan @aronchick

---

This change is 

[k8s-ci-robot]

Hi @ritazh. Thanks for your PR.

I'm waiting for a kubeflow member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[jlewi]

/ok-to-test
/assign @jlewi

[jlewi]

/ok-to-test
/lgtm
/hold to address @sozercan 's comments

[ritazh]

comment has been addressed in new commits

[jlewi]

The test failure looks like #2574

[jlewi]

I think #2555 might provide a temporary fix. You might just need to rebase on the latest master to pull that in.

[ritazh]

@jlewi thanks for the recommendation! Looks like it's still failing after the rebase.

[kkasravi]

We can port this to kfctl (golang) based on kfctl.sh or @ritazh you could do the same.
Another possibility is I could create the Azure plugin with hooks and specific azure API calls could be done by you. It looks like we would use the azure-sdk-for-go?

[ritazh]

@kkasravi I think adding azure under https://github.com/kubeflow/kubeflow/blob/master/bootstrap/pkg/client makes sense to me. we should use azure-sdk-for-go.

[ritazh]

bump
is there anything else I need to do?

[jlewi]

Most recent test failure:
http://testing-argo.kubeflow.org/workflows/kubeflow-test-infra/kubeflow-presubmit-kfctl-2557-72ba595-5954-ecd6?tab=workflow

I had to fetch the logs from stackdriver.

POD=kubeflow-presubmit-kfctl-2557-72ba595-5954-ecd6-625146308
gcloud --project=${PROJECT} logging read --format="table(timestamp, resource.labels.container_name, textPayload)" --freshness=24h      --order asc         "resource.type=\"k8s_container\" resource.labels.pod_name=\"${POD}\"  " > ~/tmp/${POD}.log.txt

The apply step failure is

ERROR: (gcloud.container.clusters.get-credentials) cluster kfctl-ecd6 is missing endpoint. Is it still PROVISIONING?

This is most likely because the cluster couldn't be created because us-east1-d was low on capacity.

So its a test flake.
/test all

[ritazh]

/retest

[ritazh]

@jlewi can you PTAL at the failed job? This PR shouldnt impact any of the tests that run on GCP.

[jlewi]

There was some problems with the test-infra see #324

/test all

[jlewi]

/lgtm
/approve

[jlewi]

latest failure looks like a flake with katib.
/test all

[jlewi]

Most recent failure is

level=error msg="no prototype names matched 'spark-operator'"

Not sure where that would be coming from.

[jlewi]

/test all

[ritazh]

/test all

[jlewi]

Latest failures

Traceback (most recent call last):
 File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
   "__main__", fname, loader, pkg_name)
 File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
   exec code in run_globals
 File "/mnt/test-data-volume/kubeflow-presubmit-kfctl-2557-72ba595-6107-bdc7/src/kubeflow/kubeflow/testing/tf_job_simple_test.py", line 30, in <module>
   from py import tf_job_client
ImportError: cannot import name tf_job_client

This looks like #2574.

@ritazh sorry this is turning into such a marathon. We're hoping to make a big push on our test-infra in Q2/0..

Can you try rebasing on the latest master to pull in the fixes?

[jlewi]

/lgtm
/approve

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: jlewi

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [jlewi]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[jlewi]

/test all

[ritazh]

/retest

[k8s-ci-robot]

@ritazh: The following test failed, say /retest to rerun them all:

Test name	Commit	Details	Rerun command
kubeflow-presubmit	99b1aba	link	/test kubeflow-presubmit

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

[jlewi]

Merging manually.

Sorry @ritazh

[ritazh]

Thanks @jlewi!