Catch signals when running terraform CLI commands

[bobh66]

Description of your changes

This change executes the terraform CLI command inside a wrapper that will terminate the CLI command if the context terminates via timeout or signal before the command finishes.

This will help with handling provider shutdown cleanly but it may also leave the Workspace in an unknown state if the running command was apply or destroy.

I have:

• Run make reviewable to ensure this PR is ready for review.

How has this code been tested

This code needs to soak in a test environment for a while but I wanted to get the changes reviewed in the mean time. I'll take the PR out of WIP when the changes have been tested locally.

Test procedure:

• Create a Workspace that has a terraform time_sleep resource with create_timeout set to 5 minutes and uses the terraform kubernetes backend
• While tailing the pod log apply the workspace and observe that the reconcile of the workspace starts
• Retrieve the list of kubernetes lease resources and observe that there is a lease for the specified workspace
• Terminate the provider pod and observe in the logs that the process gets terminated. Observe that the Lease for the workspace has been deleted.

[ulucinar]

Hi @bobh66,
Thanks for the PR! Left some minor nits and I believe we are good to go when you complete your tests.

I also did some tests with a simple configuration on linux AMD64 & darwin ARM64 systems with the terraform CLI and the time_resource. My observation is that when the Terraform CLI receives the INT signal, it immediately terminates and cleans up the state lock:

terraform {
  required_providers {
    time = {
      source = "hashicorp/time"
      version = "0.10.0"
    }
  }
}

resource "time_sleep" "wait_30_seconds" {
  create_duration = "300s"
}

After we merge this PR, let's also consider putting the async mode of reconciliation for the provider into our roadmap.

[bobh66]

I ran the same tests and it does clean up the lock, however the crossplane.io/external-create-pending annotation is left on the Workspace so reconciliation will still be blocked until someone manually removes the annotation.

[bobh66]

After we merge this PR, let's also consider putting the async mode of reconciliation for the provider into our roadmap.

How does async apply to running the CLI command? It seems like we would still be CPU-bound even if we are running the command async.