Change the Storage Version to V1 Types #6444

JeromeJu · 2023-03-28T14:04:15Z

Changes

Prior to this PR, CRDs are stored in etcd as V1beta1 apiVersion. This commit swaps the storage version to V1 Types for Task, TaskRun, Pipeline and PipelineRun CRDs. From now on we could make changes only to the V1 APIs.

This PR makes the following changes:

switch storage to v1beta1 for Task, TaskRun, Pipeline and PipelineRun CRDs(config/)
use V1 types in controllers
convert v1 CRDs to v1beta1 for reconciling features that are not moved to V1 ie. CustomTask
- it exports the ConvertFrom functions for v1beta1 Params, Workspaces and TaskRefs for v1beta1
  required fields to be reconciled when we swap to v1 storage version.
migrate the v1beta1 Yamls and tests following the migration guide
swap the RefSource and Params in remote resolution API from V1beta1 to V1
for tests and examples that are enabled on default in v1beta1, add beta to required-gates for features that are not defaulted in v1 but in v1beta1 ie. resolver_tests and move the examples under a beta folder

Closes #5541

Submitter Checklist

As the author of this PR, please check off the items in this checklist:

Has Docs included if any changes are user facing
Has Tests included if any functionality added or changed
Follows the commit message standard
Meets the Tekton contributor standards (including
functionality, content, code)
Has a kind label. You can add one by adding a comment on this PR that contains /kind <type>. Valid types are bug, cleanup, design, documentation, feature, flake, misc, question, tep
Release notes block below has been updated with any user facing changes (API changes, bug fixes, changes requiring upgrade notices or deprecation warnings)
Release notes contains the string "action required" if the change requires additional action from users switching to the new release

Release Notes

action required:  for custom resolver users, please update to use v1.Param and v1.RefSource

tekton-robot · 2023-03-28T14:04:17Z

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

pkg/reconciler/taskrun/resources/taskref.go

tekton-robot · 2023-04-11T16:38:19Z

The following is the coverage report on the affected files.
Say /test pull-tekton-pipeline-go-coverage-df to re-run this coverage report

File	Old Coverage	New Coverage	Delta
pkg/reconciler/pipelinerun/resources/pipelineref.go	93.1%	82.8%	-10.3
pkg/reconciler/pipelinerun/resources/resultrefresolution.go	89.1%	79.6%	-9.5
pkg/reconciler/taskrun/resources/taskref.go	89.7%	85.3%	-4.4
pkg/resolution/resolver/cluster/resolver.go	84.0%	63.8%	-20.2
pkg/result/result.go	Do not exist	85.7%
pkg/taskrunmetrics/metrics.go	82.8%	83.1%	0.3

tekton-robot · 2023-04-11T16:42:31Z

The following is the coverage report on the affected files.
Say /test pull-tekton-pipeline-go-coverage to re-run this coverage report

File	Old Coverage	New Coverage	Delta
pkg/reconciler/pipelinerun/resources/pipelineref.go	93.1%	82.8%	-10.3
pkg/reconciler/pipelinerun/resources/resultrefresolution.go	89.1%	79.6%	-9.5
pkg/reconciler/taskrun/resources/taskref.go	89.7%	85.3%	-4.4
pkg/resolution/resolver/cluster/resolver.go	84.0%	63.8%	-20.2
pkg/result/result.go	Do not exist	85.7%
pkg/taskrunmetrics/metrics.go	82.8%	83.1%	0.3

JeromeJu · 2023-06-16T15:36:47Z

Release Notes
for users with TektonCloudEventData, please update to use v1 Structs within the Event Data
The note here is not for users of this go API. This PR changes the content of CloudEvents produced by Tekton from v1beta1 to v1. So there is an action required for all users of CloudEvents produced by Tekton to adapt the logic in the even consumers to expect v1 instead of v1beta1.

The API version is actually not specified, see https://tekton.dev/docs/pipelines/events/, but there are changes between v1 and v1beta1, so we should warn users about that.

I think we should probably add to the events doc that the format of the resources is now v1. We do have a version number on the event type (e.g. dev.tekton.event.taskrun.started.v1), so perhaps we should switch those to v2. I think it would be nice to have a config flag that allows selecting which version of events to send, to start sending v2 but give an option to continue using v1 and deprecate v1.

All this said, perhaps the easiest option for this PR is to convert resources back to v1beta1 for sending CloudEvents, and then implement events v2 in a separate PR.

Thanks @afrittoli for the catch. That said, now we are converting back to v1beta1 TektonCloudEventDataType, we wouldn't need to put the release note for this for 'action required'.

I do appreciate this and opened #6839 to track this. Thanks for the suggestion for the workaround also 🙏

PTAL at the new commit that I pushed for the purpose of easier review, if that looks good, I am happy to squash it.

tekton-robot · 2023-06-16T15:55:25Z

The following is the coverage report on the affected files.
Say /test pull-tekton-pipeline-go-coverage to re-run this coverage report

File	Old Coverage	New Coverage	Delta
pkg/reconciler/events/cloudevent/cloudevent.go	63.8%	64.2%	0.4
pkg/reconciler/pipelinerun/pipelinerun.go	91.2%	91.3%	0.2
pkg/reconciler/pipelinerun/resources/pipelineref.go	93.8%	92.9%	-0.9
pkg/reconciler/taskrun/resources/taskref.go	90.0%	88.4%	-1.6
pkg/reconciler/taskrun/taskrun.go	84.9%	84.8%	-0.1
test/trustedresources.go	10.6%	8.9%	-1.7

JeromeJu · 2023-06-16T16:02:50Z

/test pull-tekton-pipeline-beta-integration-tests
flake: #6624

pkg/reconciler/events/cloudevent/cloudevent.go

tekton-robot · 2023-06-16T16:28:48Z

The following is the coverage report on the affected files.
Say /test pull-tekton-pipeline-go-coverage-df to re-run this coverage report

File	Old Coverage	New Coverage	Delta
pkg/reconciler/events/cloudevent/cloudevent.go	63.8%	62.3%	-1.5
pkg/reconciler/pipelinerun/pipelinerun.go	91.2%	91.3%	0.2
pkg/reconciler/pipelinerun/resources/pipelineref.go	93.8%	92.9%	-0.9
pkg/reconciler/taskrun/resources/taskref.go	90.0%	88.4%	-1.6
pkg/reconciler/taskrun/taskrun.go	84.9%	84.8%	-0.1
test/trustedresources.go	10.6%	8.9%	-1.7

tekton-robot · 2023-06-16T16:31:40Z

The following is the coverage report on the affected files.
Say /test pull-tekton-pipeline-go-coverage to re-run this coverage report

File	Old Coverage	New Coverage	Delta
pkg/reconciler/events/cloudevent/cloudevent.go	63.8%	53.1%	-10.7
pkg/reconciler/pipelinerun/pipelinerun.go	91.2%	91.3%	0.2
pkg/reconciler/pipelinerun/resources/pipelineref.go	93.8%	92.9%	-0.9
pkg/reconciler/taskrun/resources/taskref.go	90.0%	88.4%	-1.6
pkg/reconciler/taskrun/taskrun.go	84.9%	84.8%	-0.1
test/trustedresources.go	10.6%	8.9%	-1.7

tekton-robot · 2023-06-16T16:33:56Z

The following is the coverage report on the affected files.
Say /test pull-tekton-pipeline-go-coverage-df to re-run this coverage report

File	Old Coverage	New Coverage	Delta
pkg/reconciler/events/cloudevent/cloudevent.go	63.8%	53.1%	-10.7
pkg/reconciler/pipelinerun/pipelinerun.go	91.2%	91.3%	0.2
pkg/reconciler/pipelinerun/resources/pipelineref.go	93.8%	92.9%	-0.9
pkg/reconciler/taskrun/resources/taskref.go	90.0%	88.4%	-1.6
pkg/reconciler/taskrun/taskrun.go	84.9%	84.8%	-0.1
test/trustedresources.go	10.6%	8.9%	-1.7

pkg/reconciler/pipelinerun/resources/pipelineref.go

afrittoli · 2023-06-16T17:03:37Z

pkg/reconciler/pipelinerun/pipelinerun.go

+	var customRef *v1beta1.TaskRef
+	if rpt.PipelineTask.TaskRef != nil {
+		customRef = &v1beta1.TaskRef{}
+		customRef.ConvertFrom(ctx, *rpt.PipelineTask.TaskRef)


Could you please add comments in the code about this?

pkg/reconciler/pipelinerun/pipelinerun_test.go

tekton-robot · 2023-06-16T17:37:58Z

The following is the coverage report on the affected files.
Say /test pull-tekton-pipeline-go-coverage to re-run this coverage report

File	Old Coverage	New Coverage	Delta
pkg/reconciler/events/cloudevent/cloudevent.go	63.8%	53.1%	-10.7
pkg/reconciler/pipelinerun/pipelinerun.go	91.2%	91.3%	0.2
pkg/reconciler/pipelinerun/resources/pipelineref.go	93.8%	92.9%	-0.9
pkg/reconciler/taskrun/resources/taskref.go	90.0%	88.4%	-1.6
pkg/reconciler/taskrun/taskrun.go	84.9%	84.8%	-0.1
test/trustedresources.go	10.6%	8.9%	-1.7

tekton-robot · 2023-06-16T17:40:20Z

The following is the coverage report on the affected files.
Say /test pull-tekton-pipeline-go-coverage-df to re-run this coverage report

File	Old Coverage	New Coverage	Delta
pkg/reconciler/events/cloudevent/cloudevent.go	63.8%	53.1%	-10.7
pkg/reconciler/pipelinerun/pipelinerun.go	91.2%	91.3%	0.2
pkg/reconciler/pipelinerun/resources/pipelineref.go	93.8%	92.9%	-0.9
pkg/reconciler/taskrun/resources/taskref.go	90.0%	88.4%	-1.6
pkg/reconciler/taskrun/taskrun.go	84.9%	84.8%	-0.1
test/trustedresources.go	10.6%	8.9%	-1.7

afrittoli · 2023-06-16T18:00:40Z

pkg/reconciler/taskrun/resources/taskref_test.go

 		expectedErr error
 	}{
-		{


NIT: I guess this does not need to be a loop anymore, it's just one test

I think it could be but for the format of the expected error to fit in, it might work better with the tc box testing structs? :)

afrittoli

Thanks for this huge amount of work!

afrittoli

/lgtm

Prior to this PR, CRDs are stored in etcd as V1beta1 apiVersion. This commit swaps the storage version to V1 Types for Task, TaskRun, Pipeline and PipelineRun CRDs. From now on we could make changes only to the V1 APIs. This PR makes the following changes: - switch storage to v1beta1 for Task, TaskRun, Pipeline and PipelineRun CRDs(config/) - use V1 types in controllers - convert v1 CRDs to v1beta1 for reconciling features that are not moved to V1 ie. CustomTask - it exports the ConvertFrom functions for v1beta1 Params, Workspaces and TaskRefs for v1beta1 required fields to be reconciled when we swap to v1 storage version. - migrate the v1beta1 Yamls and tests following the [migration guide](https://github.com/tektoncd/pipeline/blob/main/docs/migrating-v1beta1-to-v1.md) - swap the RefSource and Params in remote resolution API from V1beta1 to V1 - for tests and examples that are enabled on default in v1beta1, add beta to for features that are not defaulted in v1 but in v1beta1 ie. resolver_tests and move the examples which requires enable-api-fields to beta under a beta folder

tekton-robot · 2023-06-16T18:42:08Z

The following is the coverage report on the affected files.
Say /test pull-tekton-pipeline-go-coverage to re-run this coverage report

File	Old Coverage	New Coverage	Delta
pkg/reconciler/events/cloudevent/cloudevent.go	63.8%	53.1%	-10.7
pkg/reconciler/pipelinerun/pipelinerun.go	91.2%	91.3%	0.2
pkg/reconciler/pipelinerun/resources/pipelineref.go	93.8%	92.9%	-0.9
pkg/reconciler/taskrun/resources/taskref.go	90.0%	88.4%	-1.6
pkg/reconciler/taskrun/taskrun.go	84.9%	84.8%	-0.1
test/trustedresources.go	10.6%	8.9%	-1.7

tekton-robot · 2023-06-16T18:43:57Z

The following is the coverage report on the affected files.
Say /test pull-tekton-pipeline-go-coverage-df to re-run this coverage report

File	Old Coverage	New Coverage	Delta
pkg/reconciler/events/cloudevent/cloudevent.go	63.8%	53.1%	-10.7
pkg/reconciler/pipelinerun/pipelinerun.go	91.2%	91.3%	0.2
pkg/reconciler/pipelinerun/resources/pipelineref.go	93.8%	92.9%	-0.9
pkg/reconciler/taskrun/resources/taskref.go	90.0%	88.4%	-1.6
pkg/reconciler/taskrun/taskrun.go	84.9%	84.8%	-0.1
test/trustedresources.go	10.6%	8.9%	-1.7

lbernick · 2023-06-16T18:44:02Z

/lgtm

dibyom · 2023-06-16T19:22:05Z

Wooot!!! Thank you @JeromeJu !!!

lbernick · 2023-06-16T19:29:44Z

/meow

tekton-robot · 2023-06-16T19:29:47Z

@lbernick:

In response to this:

/meow

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

tekton-robot requested review from abayer and lbernick March 28, 2023 14:04

JeromeJu force-pushed the v1-storage branch from e08bd8d to d28850d Compare March 28, 2023 15:35

tekton-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Mar 28, 2023

JeromeJu commented Mar 28, 2023

View reviewed changes

pkg/reconciler/taskrun/resources/taskref.go Outdated Show resolved Hide resolved

tekton-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Mar 30, 2023

JeromeJu force-pushed the v1-storage branch from d28850d to 4646f13 Compare March 31, 2023 09:25

tekton-robot added kind/misc Categorizes issue or PR as a miscellaneuous one. and removed needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. labels Mar 31, 2023

JeromeJu force-pushed the v1-storage branch from 4646f13 to 5582bb6 Compare April 3, 2023 07:43

tekton-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Apr 3, 2023

lbernick mentioned this pull request Apr 7, 2023

Export required Conversions for Reconciling the Deprecated CustomRun #6496

Closed

4 tasks

JeromeJu force-pushed the v1-storage branch 2 times, most recently from 15332e7 to 70f157c Compare April 11, 2023 16:30

tekton-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Apr 11, 2023

tekton-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Apr 13, 2023

JeromeJu closed this Apr 13, 2023

JeromeJu force-pushed the v1-storage branch from 70f157c to abd1849 Compare April 13, 2023 15:13

tekton-robot added size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. and removed size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Apr 13, 2023

JeromeJu reopened this Apr 13, 2023

tekton-robot removed needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. labels Apr 13, 2023

afrittoli reviewed Jun 16, 2023

View reviewed changes

pkg/reconciler/events/cloudevent/cloudevent.go Outdated Show resolved Hide resolved

afrittoli reviewed Jun 16, 2023

View reviewed changes

pkg/reconciler/events/cloudevent/cloudevent.go Outdated Show resolved Hide resolved

JeromeJu force-pushed the v1-storage branch 2 times, most recently from 43f7952 to 37765c9 Compare June 16, 2023 16:25

afrittoli reviewed Jun 16, 2023

View reviewed changes

pkg/reconciler/pipelinerun/resources/pipelineref.go Show resolved Hide resolved

afrittoli reviewed Jun 16, 2023

View reviewed changes

tekton-robot added the lgtm Indicates that a PR is ready to be merged. label Jun 16, 2023

JeromeJu force-pushed the v1-storage branch from 3754774 to cf68fef Compare June 16, 2023 18:35

tekton-robot removed the lgtm Indicates that a PR is ready to be merged. label Jun 16, 2023

tekton-robot added the lgtm Indicates that a PR is ready to be merged. label Jun 16, 2023

tekton-robot merged commit 7384a67 into tektoncd:main Jun 16, 2023

JeromeJu mentioned this pull request Jul 20, 2023

Default enable-api-fields value for opt-in features once feature and API versioning are decoupled #6948

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Change the Storage Version to V1 Types #6444

Change the Storage Version to V1 Types #6444

JeromeJu commented Mar 28, 2023 •

edited

Loading

tekton-robot commented Mar 28, 2023

tekton-robot commented Apr 11, 2023

tekton-robot commented Apr 11, 2023

JeromeJu commented Jun 16, 2023 •

edited

Loading

Release Notes

tekton-robot commented Jun 16, 2023

JeromeJu commented Jun 16, 2023

tekton-robot commented Jun 16, 2023

tekton-robot commented Jun 16, 2023

tekton-robot commented Jun 16, 2023

afrittoli Jun 16, 2023

tekton-robot commented Jun 16, 2023

tekton-robot commented Jun 16, 2023

afrittoli Jun 16, 2023

JeromeJu Jun 16, 2023

afrittoli left a comment

afrittoli left a comment

tekton-robot commented Jun 16, 2023

tekton-robot commented Jun 16, 2023

lbernick commented Jun 16, 2023

dibyom commented Jun 16, 2023

lbernick commented Jun 16, 2023

tekton-robot commented Jun 16, 2023

Change the Storage Version to V1 Types #6444

Change the Storage Version to V1 Types #6444

Conversation

JeromeJu commented Mar 28, 2023 • edited Loading

Changes

Submitter Checklist

Release Notes

tekton-robot commented Mar 28, 2023

tekton-robot commented Apr 11, 2023

tekton-robot commented Apr 11, 2023

JeromeJu commented Jun 16, 2023 • edited Loading

Release Notes

tekton-robot commented Jun 16, 2023

JeromeJu commented Jun 16, 2023

tekton-robot commented Jun 16, 2023

tekton-robot commented Jun 16, 2023

tekton-robot commented Jun 16, 2023

afrittoli Jun 16, 2023

Choose a reason for hiding this comment

tekton-robot commented Jun 16, 2023

tekton-robot commented Jun 16, 2023

afrittoli Jun 16, 2023

Choose a reason for hiding this comment

JeromeJu Jun 16, 2023

Choose a reason for hiding this comment

afrittoli left a comment

Choose a reason for hiding this comment

afrittoli left a comment

Choose a reason for hiding this comment

tekton-robot commented Jun 16, 2023

tekton-robot commented Jun 16, 2023

lbernick commented Jun 16, 2023

dibyom commented Jun 16, 2023

lbernick commented Jun 16, 2023

tekton-robot commented Jun 16, 2023

JeromeJu commented Mar 28, 2023 •

edited

Loading

JeromeJu commented Jun 16, 2023 •

edited

Loading