Enable deletion of SageMaker job style CR from API server if job is complete

[surajkota]

Description of Changes

• Enable deletion of SageMaker job style CR from API server if the job is complete(i.e. JobStatus = Completed, Failed, Stopped)
• Reduce/clean up custom code by using callback hooks See: Callback hooks PR, Templates path PR
  • Now we no longer need to match the method signature of the parent operation for adding custom code unless we intend to use all the parameters.
• Address comment: Add Requeue logic for Job Style resources till completion #10 (comment)

Testing

Tested manually - let the job(training, process, transform & hpo) run till completion and ensured running kubectl delete on the CR removes the resource from management. No error on controller side.

Without this PR, the controller gets stuck in backoff because of an error from SageMaker

2021-03-25T07:41:11.356Z	DEBUG	ackrt	starting reconciliation	{"kind": "TrainingJob", "namespace": "default", "name": "xgboost-trainingjob-z61ptgyri5m1", "generation": 2, "account": "XXXX", "role": "", "region": "us-west-2"}
2021-03-25T07:41:11.707Z	ERROR	controller-runtime.controller	Reconciler error	{"controller": "trainingjob", "request": "default/xgboost-trainingjob-z61ptgyri5m1", "error": "ValidationException: The request was rejected because the training job is in status Completed.\n\tstatus code: 400, request id: 8e6a1690-9d35-4529-a2b9-3e9e011fcef2"}
github.com/go-logr/zapr.(*zapLogger).Error
	/home/ubuntu/go/pkg/mod/github.com/go-logr/zapr@v0.1.0/zapr.go:128
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
	/home/ubuntu/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.0/pkg/internal/controller/controller.go:258
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/home/ubuntu/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.0/pkg/internal/controller/controller.go:232
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker
	/home/ubuntu/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.0/pkg/internal/controller/controller.go:211
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1
	/home/ubuntu/go/pkg/mod/k8s.io/apimachinery@v0.18.6/pkg/util/wait/wait.go:155
k8s.io/apimachinery/pkg/util/wait.BackoffUntil
	/home/ubuntu/go/pkg/mod/k8s.io/apimachinery@v0.18.6/pkg/util/wait/wait.go:156
k8s.io/apimachinery/pkg/util/wait.JitterUntil
	/home/ubuntu/go/pkg/mod/k8s.io/apimachinery@v0.18.6/pkg/util/wait/wait.go:133
k8s.io/apimachinery/pkg/util/wait.Until
	/home/ubuntu/go/pkg/mod/k8s.io/apimachinery@v0.18.6/pkg/util/wait/wait.go:90

automated tests will be part of a separate follow up of @mbaijal's PR

Related PRs

Both merged
aws-controllers-k8s/code-generator#45
aws-controllers-k8s/code-generator#26

[jkuruba]

Can you update the testing to include the specific tests run and the test outputs?

Otherwise, changes look good.