PipelineResources left behind after PipelineRuns are wasteful

[jessm12]

Expected Behavior

When a PipelineRun has completed it is desirable that as few resources as possible are left behind so as to preserve memory and keep systems clean.

Actual Behavior

When creating PipelineRuns, associated PipelineResources are created to reference key inputs or outputs to Tasks within a Pipeline. This leaves behind many resources that are only used during the PipelineRun(s) for which they were created and are not cleaned up.

Steps to Reproduce the Problem

As an example case:

1. Trigger a number of concurrent PipelineRuns for a repository with uniquely named PipelineResources for example of type git, each of which will reference a different commit ID
2. These PipelineResources that must be created are left behind and despite containing data which may only vary by commit ID they require a separate resource for each PipelineRun

Additional Info

Some ideas that we have had so far to tackle this issue:

1.PipelineResources could be automatically cleaned up after their use within a PipelineRun, is there any use in them being kept around indefinitely?

Could this be done by using ownerRefs in that a PipelineResource could have one or many PipelineRuns that have ownership of it and the PipelineResource will only be removed either on completion or deletion of each of its associated PipelineRuns

2.The PipelineResources could be replaced with something like PipelineParams that would be passed directly into a PipelineRun and handled accordingly (such as mounting the git source into the workspace directory)

[bobcatfish]

Awesome write up @jessm12 😻, thanks so much, that captures the problem perfectly!!

[bobcatfish]

Note there was some related work in #200 but definitely looks like we can make this even better :D

[jessm12]

Thanks @bobcatfish! 😄 There are some really interesting ideas and work under that issue! Some of which would definitely come under discussion here as well 👍

[bobcatfish]

I've created a design doc to propose tackling this by embedding PipelineResources in PipelineRuns, and managing their lifecycle in the Pipelines controller. (The resource extensibility proposal from #238 is very relevant also!)

[bobcatfish]

Seems like folks are in agreement about the proposal! And @pritidesai is working on this via #872 so I'm going to close this issue.