Split reconcile into prepare and reconcile #2421

afrittoli · 2020-04-16T17:50:59Z

Changes

The current "reconcile" function does a lot of things, which means
it's hard to test properly, it's hard to follow the logic and
it's hard to handle all possible errors and events properly.

This splits reconcile into two parts, the first one that deals
preparing and validating the taskrun and all the resources it
depends on, the second one that actually reconciles the TaskRun
by creating the pod if required and updating the TaskRun from
the pod.

This adds events that were missing for error situation that
happens during preparation and validation.

Submitter Checklist

These are the criteria that every PR should meet, please check them off as you
review them:

[WIP] Includes tests (if functionality changed/added)
[n/a] Includes docs (if user facing)
Commit messages follow commit message best practices

See the contribution guide for more details.

Double check this list of stuff that's easy to miss:

If you are adding a new binary/image to the cmd dir, please update
the release Task to build and release this image.

Reviewer Notes

If API changes are included, additive changes must be approved by at least two OWNERS and backwards incompatible changes must be approved by more than 50% of the OWNERS, and they must first be added in a backwards compatible way.

Release Notes

The TaskRun controller now emits events also when error occurs during the validation of the TaskRun or the resources it depends on.

pkg/reconciler/taskrun/taskrun.go

ghost · 2020-04-17T17:56:10Z

/approve

afrittoli · 2020-04-20T15:26:35Z

/test pull-tekton-pipeline-integration-tests

afrittoli · 2020-04-21T18:26:43Z

I cannot reproduce any of the integration test failures on my test cluster, even after multiple runs. I think that something in the logic that waits for events is not going well, not waiting long enough or perhaps starting to wait too soon.

afrittoli · 2020-04-22T16:58:03Z

/test pull-tekton-pipeline-integration-tests

afrittoli · 2020-04-22T16:59:27Z

/test pull-tekton-pipeline-build-tests

afrittoli · 2020-04-22T20:04:51Z

/test pull-tekton-pipeline-build-tests

afrittoli · 2020-04-22T20:04:58Z

/test pull-tekton-pipeline-integration-tests

pkg/reconciler/taskrun/taskrun.go

test/init_test.go

pkg/reconciler/taskrun/taskrun.go

afrittoli · 2020-04-23T16:10:31Z

/test pull-tekton-pipeline-build-tests

afrittoli · 2020-04-23T16:10:53Z

/test pull-tekton-pipeline-integration-tests

afrittoli · 2020-04-24T10:53:54Z

/test pull-tekton-pipeline-integration-tests

afrittoli · 2020-04-24T15:24:42Z

/test pull-tekton-pipeline-build-tests
/test pull-tekton-pipeline-integration-tests

The current "reconcile" function does a lot of things, which means it's hard to test properly, it's hard to follow the logic and it's hard to handle all possible errors and events properly. This splits reconcile into two parts, the first one that deals preparing and validating the taskrun and all the resources it depends on, the second one that actually reconciles the TaskRun by creating the pod if required and updating the TaskRun from the pod. This adds events that were missing for error situations that happen during preparation and validation. The new events go past the burst limit of the event broadcaster, which delays some events a lot for a pipeline with a few tasks. Tweaking the default burst rate and QPS is required to pass CI. We might want to add those configs to a dedicated config map in future.

dibyom · 2020-04-24T21:21:41Z

pkg/reconciler/reconciler.go

+
+		correlatorOptions := record.CorrelatorOptions{
+			// The default burst size is 25
+			BurstSize: 50,


Probably fine for now but should be configurable right?

Yes, I'll follow up on this

dibyom · 2020-04-24T21:21:59Z

pkg/reconciler/reconciler.go

+		correlatorOptions := record.CorrelatorOptions{
+			// The default burst size is 25
+			BurstSize: 50,
+			QPS:       1,


Is this the number of queries per second that we can emit? I couldn't figure out after reading the godoc for this

Yes, the way I understood this is the QPS we allow after a burst

tekton-robot · 2020-04-24T21:25:31Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: dibyom, sbwsg

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [dibyom,sbwsg]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

dibyom · 2020-04-24T21:26:50Z

/lgtm

tekton-robot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Apr 16, 2020

tekton-robot requested review from dlorenc and a user April 16, 2020 17:51

googlebot added the cla: yes Trying to make the CLA bot happy with ppl from different companies work on one commit label Apr 16, 2020

tekton-robot added the size/M Denotes a PR that changes 30-99 lines, ignoring generated files. label Apr 16, 2020

afrittoli force-pushed the split_prepare_reconcile branch from fca8bbd to 5d67a4d Compare April 16, 2020 18:03

afrittoli commented Apr 16, 2020

View reviewed changes

pkg/reconciler/taskrun/taskrun.go Show resolved Hide resolved

afrittoli force-pushed the split_prepare_reconcile branch from 5d67a4d to af1cbe3 Compare April 17, 2020 12:22

tekton-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Apr 17, 2020

afrittoli changed the title ~~WIP Split reconcile into prepare and reconcile~~ Split reconcile into prepare and reconcile Apr 17, 2020

tekton-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Apr 17, 2020

ghost added the kind/cleanup Categorizes issue or PR as related to cleaning up code, process, or technical debt. label Apr 17, 2020

afrittoli force-pushed the split_prepare_reconcile branch 2 times, most recently from 7ce886b to 9467425 Compare April 17, 2020 17:08

ghost reviewed Apr 17, 2020

View reviewed changes

pkg/reconciler/taskrun/taskrun.go Show resolved Hide resolved

pkg/reconciler/taskrun/taskrun.go Show resolved Hide resolved

tekton-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Apr 17, 2020

afrittoli force-pushed the split_prepare_reconcile branch 4 times, most recently from 9fa0ad0 to bb2ffb4 Compare April 20, 2020 12:04

afrittoli mentioned this pull request Apr 20, 2020

Dereference the TaskSpec into TaskRun.Status. #2444

Merged

3 tasks

afrittoli force-pushed the split_prepare_reconcile branch from bb2ffb4 to b8ff62c Compare April 20, 2020 15:54

afrittoli force-pushed the split_prepare_reconcile branch from b8ff62c to e57c6f5 Compare April 21, 2020 18:32

afrittoli mentioned this pull request Apr 21, 2020

Emit events for all TaskRun lifecycle events #2329

Merged

3 tasks

afrittoli force-pushed the split_prepare_reconcile branch 2 times, most recently from 26768b7 to 7897553 Compare April 21, 2020 20:10

afrittoli force-pushed the split_prepare_reconcile branch from aa74c56 to cdec415 Compare April 22, 2020 16:32

dibyom reviewed Apr 22, 2020

View reviewed changes

pkg/reconciler/taskrun/taskrun.go Outdated Show resolved Hide resolved

test/init_test.go Outdated Show resolved Hide resolved

pkg/reconciler/taskrun/taskrun.go Outdated Show resolved Hide resolved

afrittoli force-pushed the split_prepare_reconcile branch 7 times, most recently from 685a28d to 2e396a1 Compare April 23, 2020 15:18

afrittoli force-pushed the split_prepare_reconcile branch from 2e396a1 to cf305d6 Compare April 24, 2020 09:36

afrittoli force-pushed the split_prepare_reconcile branch from cf305d6 to 5e5bb4e Compare April 24, 2020 15:46

afrittoli force-pushed the split_prepare_reconcile branch from 5e5bb4e to a963e63 Compare April 24, 2020 16:26

dibyom approved these changes Apr 24, 2020

View reviewed changes

tekton-robot assigned dibyom Apr 24, 2020

tekton-robot added the lgtm Indicates that a PR is ready to be merged. label Apr 24, 2020

tekton-robot merged commit c11c6af into tektoncd:master Apr 24, 2020

afrittoli mentioned this pull request Apr 29, 2020

TaskRun emits no event when it fails on timeout #2328

Closed

afrittoli added the kind/feature Categorizes issue or PR as related to a new feature. label Apr 30, 2020

qu1queee mentioned this pull request Jul 24, 2020

Upgrade the Tekton version to the latest v0.12 version shipwright-io/build#221

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Split reconcile into prepare and reconcile #2421

Split reconcile into prepare and reconcile #2421

afrittoli commented Apr 16, 2020

ghost commented Apr 17, 2020

afrittoli commented Apr 20, 2020

afrittoli commented Apr 21, 2020

afrittoli commented Apr 22, 2020

afrittoli commented Apr 22, 2020

afrittoli commented Apr 22, 2020

afrittoli commented Apr 22, 2020

afrittoli commented Apr 23, 2020

afrittoli commented Apr 23, 2020

afrittoli commented Apr 24, 2020

afrittoli commented Apr 24, 2020

dibyom Apr 24, 2020

afrittoli Apr 24, 2020

dibyom Apr 24, 2020

afrittoli Apr 24, 2020

tekton-robot commented Apr 24, 2020

dibyom commented Apr 24, 2020

Split reconcile into prepare and reconcile #2421

Split reconcile into prepare and reconcile #2421

Conversation

afrittoli commented Apr 16, 2020

Changes

Submitter Checklist

Reviewer Notes

Release Notes

ghost commented Apr 17, 2020

afrittoli commented Apr 20, 2020

afrittoli commented Apr 21, 2020

afrittoli commented Apr 22, 2020

afrittoli commented Apr 22, 2020

afrittoli commented Apr 22, 2020

afrittoli commented Apr 22, 2020

afrittoli commented Apr 23, 2020

afrittoli commented Apr 23, 2020

afrittoli commented Apr 24, 2020

afrittoli commented Apr 24, 2020

dibyom Apr 24, 2020

Choose a reason for hiding this comment

afrittoli Apr 24, 2020

Choose a reason for hiding this comment

dibyom Apr 24, 2020

Choose a reason for hiding this comment

afrittoli Apr 24, 2020

Choose a reason for hiding this comment

tekton-robot commented Apr 24, 2020

dibyom commented Apr 24, 2020