WIP: Add yaml manifest to cleanup all Antrea resources

[antoninbas]

This is still very much a work-in-progress, I'm opening this PR to
gather feedback on the approach.

For someone deleting Antrea, the steps would be as follows:

• kubectl delete -f <path to antrea.yml>
• kubectl apply -f <path to antrea-cleanup.yml>
• check that job has completed with kubectl -n kube-system get jobs
• kubectl delete -f <path to antrea-cleanup.yml>

The cleanup manifest creates a DaemonSet that will perform the necessary
deletion tasks on each Node. After the tasks have been completed, the
"status" is reported to the cleanup controller through a custom
resource. Once the controller has received enough statuses (or after a
timeout of 1 minutes) the controller job completes and the user can
delete the cleanup manifest.

Known remaining items:

• place cleanup binaries (antrea-cleanup-agent and
  antrea-cleanup-controller) in a separate docker image to avoid
  increasing the size of the main Antrea docker image
• generate manifest with kustomize?
• find a way to test this as part of CI?
• update documentation
• additional cleanup tasks: as of now we only take care of deleting the
  OVS bridge
• place cleanup CRD in non-default namespace
• use kubebuilder instead of the code generator directly (related to
  Use kubebuilder 1.16 instead of the code generator directly for CRDs #16); we probably want to punt this task to a future PR.

See #181

[antrea-bot]

Thanks for your PR.
Unit tests and code linters are run automatically every time the PR is updated.
E2e, conformance and network policy tests can only be triggered by a member of the vmware-tanzu organization. Regular contributors to the project should join the org.

The following commands are available:

• /test-e2e: to trigger e2e tests.
• /skip-e2e: to skip e2e tests.
• /test-conformance: to trigger conformance tests.
• /skip-conformance: to skip conformance tests.
• /test-networkpolicy: to trigger networkpolicy tests.
• /skip-networkpolicy: to skip networkpolicy tests.
• /test-all: to trigger all tests.
• /skip-all: to skip all tests.

These commands can only be run by members of the vmware-tanzu organization.

[jianjuns]

@antoninbas: do you think we can reuse the AgentInfo and ControllerInfo CRD for the cleanup task status, instead of adding another CRD?

Also two other thoughts:

1. We need to provide a way to clean up node when K8s is down. Should we consider antctl for that?
2. We might not be able to do cleanup with a DS/Pod for Windows. In that case, do you assume some powershell scripts to start the cleanup service?
   Thinking if it is easier to let the Agent perform the cleanup in that case.

[jianjuns]

2. We might not be able to do cleanup with a DS/Pod for Windows. In that case, do you assume some powershell scripts to start the cleanup service?
   Thinking if it is easier to let the Agent perform the cleanup in that case.

Thought over. For Windows, we should use powershell to install Controllers and Agents, and so it makes sense to use powershell to uninstall and start cleanup. But in that case, we need not coordination between cleanup-agent and cleanup-controller, but just need one-off execution for cleaning up both Controller and Agent resources.
@wenyingd might have a better idea.

[antoninbas]

@jianjuns FYI, I'm planning on starting work again on this PR this week and address these 2 comments from you:

• try to reuse existing CRD API groups to report status back to deletion controller
• provide "local" cleanup mechanism on a single Node when K8s is down (though antctl)

[jianjuns]

@jianjuns FYI, I'm planning on starting work again on this PR this week and address these 2 comments from you:

Sounds great!

* try to reuse existing CRD API groups to report status back to deletion controller

* provide "local" cleanup mechanism on a single Node when K8s is down (though antctl)

Will your cleanup DS also executes antctl?

[antoninbas]

Will your cleanup DS also executes antctl?

My idea was that they would share their implementation in pkg/agent/cleanup, but that the DS would not invoke antctl directly.

[antrea-bot]

Can one of the admins verify this patch?

[github-actions]

This PR is stale because it has been open 180 days with no activity. Remove stale label or comment, or this will be closed in 180 days