feature(backend): Allow KFP apiserver to create recursion pipelineLoop CRs for the users

[Tomcli]

Which issue is resolved by this Pull Request:
Resolves #496

Description of your changes:
Since we don't have support for putting custom task in the task spec. The below diagram shows how we could offload some of the extra api calls from the users to KFP apiserver. Ideally, having taskSpec for custom task is a better solution but it will take a while before it can be used in a Tekton stable release.

This PR handles the first step below where KFP apiserver creates the custom task CR x N for the users. Where N is the number of CRs and each CR needs a new API call.

Although this feature allows users to deploy all the necessary custom resources with one KFP API call, KFP doesn't clean up these custom resources because the same custom resource could have multiple owners. This issue should be resolved in the upcoming taskSpec support for the custom task, so we don't want to create new dependencies on this temporary solution.

Environment tested:

• Python Version (use python --version): 3.7
• Tekton Version (use tkn version): 0.21.0
• Kubernetes Version (use kubectl version): 1.18
• OS (e.g. from /etc/os-release):

Checklist:

• The title for your pull request (PR) should follow our title convention. Learn more about the pull request title convention used in this repository.

• Do you want this pull request (PR) cherry-picked into the current release branch?

  Learn more about cherry-picking updates into the release branch.

[Tomcli]

A new use case for this feature is with recursion. Since recursion loop has to reference a loop resource, we cannot use taskSpec to replace this feature because taskSpec is not a referable resource that can call itself in the same pipeline.

[Tomcli]

/hold
for reviews

[ckadner]

Thanks @Tomcli, I like it :-)

[google-oss-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: ckadner, Tomcli

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [Tomcli,ckadner]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[ckadner]

Good idea to put this PR on hold for others to review, so I can safely add my +1 while still giving others a chance for review 👍🏻

/lgtm

[Tomcli]

@Udiknedormin We are planning to make this feature as default for recursion. Can you give a quick review to make sure this won't break your team's use case? Thanks.

[Udiknedormin]

@Tomcli It looks like it works similar to what we already implemented, i.e. auto-creates resources (although it seems to be at a different time). It also doesn't remove the resources while I think our version does. If I understood correctly, it also requires moving resources to some annotation, which our version doesn't do nor need.

We would rather still use our version, but if it's possible to simply turn this new feature off, I think we should be fine with it.

[Tomcli]

@Udiknedormin I added more details on how to disable this feature both from the API and SDK. The reason I didn't let kfp to remove the resources because there could be multiple pipelines that are referring to the same resource. If we think it is safe enough, I can add owner reference to those resources to bound to the latest pipelineRun that is using the recursive loop.

[Tomcli]

/unhold

[Tomcli]

@pugangxa can you give a final review and lgtm this PR? Thanks.

[Tomcli]

This feature should be enabled by default and can be opt-out with a flog

[pugangxa]

/lgtm