Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

E2E: Add additional checks to verify if the components are ready #2202

Merged
merged 1 commit into from
Aug 15, 2023

Conversation

tenzen-y
Copy link
Member

What this PR does / why we need it:
To fix flaky E2E, I added additional checks to verify if the components are ready.

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #

Checklist:

  • Docs included if any changes are user facing

@google-oss-prow
Copy link

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: tenzen-y

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@tenzen-y tenzen-y force-pushed the fix-flaky-e2e branch 3 times, most recently from 5f9095f to e95f4f6 Compare August 13, 2023 13:16
@tenzen-y tenzen-y marked this pull request as draft August 13, 2023 13:18
@google-oss-prow google-oss-prow bot added size/M and removed size/S labels Aug 13, 2023
@tenzen-y tenzen-y force-pushed the fix-flaky-e2e branch 6 times, most recently from d2e0a1d to 0f57b3c Compare August 13, 2023 17:11
@google-oss-prow google-oss-prow bot added size/S and removed size/M labels Aug 13, 2023
@tenzen-y tenzen-y force-pushed the fix-flaky-e2e branch 2 times, most recently from e64253e to 8691375 Compare August 13, 2023 18:47
TIMEOUT=120s
kubectl wait --for=condition=ready --timeout=${TIMEOUT} -l "katib.kubeflow.org/component in ($WITH_DATABASE_TYPE,controller,db-manager,ui)" -n kubeflow pod ||

kubectl wait --for=condition=ContainersReady=True --timeout=${TIMEOUT} -l "katib.kubeflow.org/component in ($WITH_DATABASE_TYPE,controller,db-manager,ui)" -n kubeflow pod ||
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The --for=condition=ready option doesn't confirm if all containers are ready:

/home/runner/work/katib/katib/test/e2e/v1beta1/scripts/gh-actions
pod/katib-db-manager-756ff4d844-wrs27 condition met
All Katib components are running.
Katib deployments
NAME               READY   UP-TO-DATE   AVAILABLE   AGE
katib-controller   0/1     0            0           2s
katib-db-manager   1/1     1            1           2s
katib-mysql        0/1     0            0           2s

https://github.com/kubeflow/katib/actions/runs/5842762640/job/15844213190?pr=2201#step:4:5605

@tenzen-y tenzen-y force-pushed the fix-flaky-e2e branch 2 times, most recently from cef41c0 to 7d28b7f Compare August 13, 2023 21:49
@google-oss-prow google-oss-prow bot removed the size/S label Aug 13, 2023
@andreyvelich
Copy link
Member

/lgtm
Thank you @tenzen-y!

@google-oss-prow google-oss-prow bot added the lgtm label Aug 15, 2023
@tenzen-y tenzen-y marked this pull request as ready for review August 15, 2023 15:29
@google-oss-prow google-oss-prow bot requested a review from andreyvelich August 15, 2023 15:29
Signed-off-by: Yuki Iwai <yuki.iwai.tz@gmail.com>
@google-oss-prow google-oss-prow bot merged commit 2ae3eb5 into kubeflow:master Aug 15, 2023
@tenzen-y tenzen-y deleted the fix-flaky-e2e branch August 15, 2023 18:30
@google-oss-prow
Copy link

New changes are detected. LGTM label has been removed.

@google-oss-prow google-oss-prow bot removed the lgtm label Aug 15, 2023
tenzen-y added a commit to tenzen-y/katib that referenced this pull request Aug 16, 2023
google-oss-prow bot pushed a commit that referenced this pull request Aug 16, 2023
…if the components are ready (#2212)

Signed-off-by: Yuki Iwai <yuki.iwai.tz@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants