Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Errored workspaces aren't removed after PVC cleanup failure #873

Closed
Tracked by #21503
AObuchow opened this issue Jun 17, 2022 · 1 comment · Fixed by #879
Closed
Tracked by #21503

Errored workspaces aren't removed after PVC cleanup failure #873

AObuchow opened this issue Jun 17, 2022 · 1 comment · Fixed by #879
Assignees
Labels
bug Something isn't working
Milestone

Comments

@AObuchow
Copy link
Collaborator

Description

This issue was initially found by @amisevsk here.

If the common PVC cleanup job fails and all workspaces are deleted, the workspace which had the failed PVC cleanup job will be stuck in an errored state and won't be removed.

How To Reproduce

First, ensure the common PVC cleanup job will fail by modifying the PVC cleanup job spec in cleanup.go:

	Args: []string{
		"-c",
- 		fmt.Sprintf(cleanupCommandFmt, path.Join(pvcClaimMountPath, workspaceId)),
+		 "exit 1",
	},

Then do the following:

  1. oc apply -f samples/theia-next.yaml
  2. yq '.metadata.name="theia-next-2"' samples/theia-next.yaml | kubectl apply -f -
  3. Wait for workspaces to start/get finalizers at least
  4. oc delete dw theia-next
  5. Wait for deletion to hit error
  6. oc delete dw --all
  7. The workspace which had the PVC cleanup job that failed should be stuck in the error state, visible when doing kubectl get devworkspace -n $NAMESPACE

(Thank you to @amisevsk for the instructions on how to reproduce)

Expected behavior

The workspaces which are in the error state (from the PVC cleanup job failing) should be removed automatically.

Additional context

This bug arose from #846 and will only be apparent when that PR is merged.

@amisevsk
Copy link
Collaborator

amisevsk commented Jun 28, 2022

Might be incidentally/partially resolved by #879 (sorry @AObuchow :))

@amisevsk amisevsk mentioned this issue Jul 4, 2022
3 tasks
@amisevsk amisevsk added this to the v0.15.x milestone Jul 25, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants