Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: add aws batch compute and job queue #1489

Closed
wants to merge 49 commits into from
Closed

feat: add aws batch compute and job queue #1489

wants to merge 49 commits into from

Conversation

sachinmalanki
Copy link
Contributor

Description of your changes

Adding AWS Batch Compute Environment(aws_batch_compute_environment) and Job Queue(aws_batch_job_queue)

Fixes #

I have:

  • Read and followed Crossplane's contribution process.
  • Run make reviewable to ensure this PR is ready for review.
  • Added backport release-x.y labels to auto-backport this PR if necessary.

How has this code been tested

Manually tested as per the guide
Batch compute environment
image

Job Queue
image

@sachinmalanki
Copy link
Contributor Author

/test-examples="examples/batch/v1beta1/computeenvironment.yaml"

1 similar comment
@jeanduplessis
Copy link
Collaborator

/test-examples="examples/batch/v1beta1/computeenvironment.yaml"

@jeanduplessis
Copy link
Collaborator

/test-examples="examples/batch/v1beta1/jobqueue.yaml"

@jeanduplessis
Copy link
Collaborator

/test-examples="examples/batch/v1beta1/jobqueue.yaml"

1 similar comment
@jeanduplessis
Copy link
Collaborator

/test-examples="examples/batch/v1beta1/jobqueue.yaml"

@jeanduplessis
Copy link
Collaborator

/test-examples="examples/batch/v1beta1/computeenvironment.yaml"

1 similar comment
@sachinmalanki
Copy link
Contributor Author

/test-examples="examples/batch/v1beta1/computeenvironment.yaml"

@sachinmalanki
Copy link
Contributor Author

/test-examples="examples/batch/v1beta1/jobqueue.yaml"

@jeanduplessis
Copy link
Collaborator

/test-examples="examples/batch/v1beta1/computeenvironment.yaml"

@jeanduplessis
Copy link
Collaborator

/test-examples="examples/batch/v1beta1/jobqueue.yaml"

1 similar comment
@turkenf
Copy link
Collaborator

turkenf commented Sep 23, 2024

/test-examples="examples/batch/v1beta1/jobqueue.yaml"

@turkenf
Copy link
Collaborator

turkenf commented Sep 23, 2024

/test-examples="examples/batch/v1beta1/computeenvironment.yaml"

@turkenf
Copy link
Collaborator

turkenf commented Sep 24, 2024

/test-examples="examples/batch/v1beta1/jobqueue.yaml"

@sachinmalanki
feat: add aws batch compute and job queue
6a5434d 
Signed-off-by: sachinmalanki <sachin.chini@gmail.com>
@sachinmalanki
fix: update exmaple for compute environment and job queue
b7c0ba2 
Signed-off-by: sachinmalanki <sachin.chini@gmail.com>
@sachinmalanki
fix: update example to missing policy
5862c06 
Signed-off-by: sachinmalanki <sachin.chini@gmail.com>
@sachinmalanki
fix: update example to add subnets and security id
06686aa 
Signed-off-by: sachinmalanki <sachin.chini@gmail.com>
@sachinmalanki
fix: updated example role and policy
4a45ef6 
Signed-off-by: sachinmalanki <sachin.chini@gmail.com>
@jeanduplessis
Copy link
Collaborator

/test-examples="examples/batch/v1beta1/computeenvironment.yaml"

@jeanduplessis
Copy link
Collaborator

/test-examples="examples/batch/v1beta1/jobqueue.yaml"

@sachinmalanki
Copy link
Contributor Author

sachinmalanki commented Sep 25, 2024 •
edited
Loading

The test fails because of hardcoding securityGroupIds: and subnets: values. Can I know how to fix this? The computeenvironment crd does not allow subnet and securitygroupid reference.

@mergenci
Copy link
Collaborator

@sachinmalanki, here's an example configuration: https://github.com/crossplane-contrib/provider-upjet-aws/blob/v1.14.0/config/apprunner/config.go#L14-L23

sergenyalcin and others added 2 commits September 27, 2024 18:27
@sergenyalcin @sachinmalanki
Use different parameters name for auth and config group overrides
10b10b1 
Signed-off-by: Sergen Yalçın <yalcinsergen97@gmail.com>
@sachinmalanki
fix: add subnet and security ref and selector
0707cd4 
Signed-off-by: sachinmalanki <sachin.chini@gmail.com>
turkenf and others added 20 commits October 23, 2024 16:43
@turkenf @sachinmalanki
Convert some fields to list
0e8f715 
Signed-off-by: Fatih Türken <turkenf@gmail.com>
@blakeromano @sachinmalanki
Add Bedrock Agent Support
b8e4d1b 
Signed-off-by: Blake R <85771645+blakeromano@users.noreply.github.com>
@blakeromano @sachinmalanki
make reviewable
153fe7b 
Signed-off-by: Blake R <85771645+blakeromano@users.noreply.github.com>
@turkenf @sachinmalanki
Run make generate after rebase
015c274 
Signed-off-by: Fatih Türken <turkenf@gmail.com>
@erhancagirici @sachinmalanki
add custom diff for LBListener forward options
af6e285 
Signed-off-by: Erhan Cagirici <erhan@upbound.io>
@mergenci @sachinmalanki
nolint:gocyclo resource configuration.
85a5172 
Signed-off-by: Cem Mergenci <cmergenci@gmail.com>
@mergenci @sachinmalanki
Bump Terraform provider fork version.
4d2648b 
Signed-off-by: Cem Mergenci <cmergenci@gmail.com>
@erhancagirici @sachinmalanki
ignore non-effective diff for LBTargetGroup unhealthy_draining_interv…
9a475df 
…al defaulting

Signed-off-by: Erhan Cagirici <erhan@upbound.io>
@erhancagirici @sachinmalanki
add LBListener examples with forward default action
200d789 
Signed-off-by: Erhan Cagirici <erhan@upbound.io>
@turkenf @sachinmalanki
Add PathARNExtractor for kms_key_id to rds cluster
3d4079b 
Signed-off-by: Fatih Türken <turkenf@gmail.com>
@turkenf @sachinmalanki
Backport the changes to v1beta1 and add a new example
cabed02 
Signed-off-by: Fatih Türken <turkenf@gmail.com>
@erhancagirici @sachinmalanki
fix custom diff for Domain.opensearch ignoring valid diff
3a2399e 
Signed-off-by: Erhan Cagirici <erhan@upbound.io>
@renovate @sachinmalanki
Update actions/cache digest to 6849a64
3729a4b
@sachinmalanki
feat: add aws batch compute and job queue
9886641 
Signed-off-by: sachinmalanki <sachin.chini@gmail.com>
@sachinmalanki
fix: rebase and update external config
f3d3a19 
Signed-off-by: sachinmalanki <sachin.chini@gmail.com>
@sachinmalanki
fix: rebase and update external config
fa7f65d 
Signed-off-by: sachinmalanki <sachin.chini@gmail.com>
@sachinmalanki
fix: rebase and update external config
8acc89a 
Signed-off-by: sachinmalanki <sachin.chini@gmail.com>
@sachinmalanki
Merge branch 'main' into aws-batch-resources
4fbfd64
@sachinmalanki
fix: rebase and update example-generated
f8fb54d 
Signed-off-by: sachinmalanki <sachin.chini@gmail.com>
@sachinmalanki
fix: rebase and update example-generated
8811242 
Signed-off-by: sachinmalanki <sachin.chini@gmail.com>
@sachinmalanki
Copy link
Contributor Author

Hi @turkenf. Thanks for the suggestions, I have rebased and made the changes. Could you please review ?
When I was testing this locally, the service_role gets deleted first and leaves the computeenvironment in a INVALID hanging state. Any idea how this could be fixed? This could cause the uptest to fail.

@turkenf
Copy link
Collaborator

turkenf commented Oct 24, 2024 •
edited
Loading

When I was testing this locally, the service_role gets deleted first and leaves the computeenvironment in a INVALID hanging state. Any idea how this could be fixed? This could cause the uptest to fail.

@sachinmalanki, we have hooks for this kind of cases. We can use pre-delete-hook. Here is an example: yaml and sh files.

I noticed that there are currently 47 commits in the PR’s commit history. When you have a chance, could you please fork your PR from the latest main branch and squash your commits? It would help keep the history clean. Thanks so much!

turkenf
turkenf reviewed Oct 24, 2024
View reviewed changes
config/batch/config.go Outdated
Comment on lines 10 to 21
r.References = config.References{
"compute_resources.subnets": config.Reference{
TerraformName: "aws_subnet",
RefFieldName: "SubnetRefs",
SelectorFieldName: "SubnetSelector",
},
"compute_resources.security_group_ids": config.Reference{
TerraformName: "aws_security_group",
RefFieldName: "SecurityGroupRefs",
SelectorFieldName: "SecurityGroupSelector",
},
}
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
r.References = config.References{
"compute_resources.subnets": config.Reference{
TerraformName: "aws_subnet",
RefFieldName: "SubnetRefs",
SelectorFieldName: "SubnetSelector",
},
"compute_resources.security_group_ids": config.Reference{
TerraformName: "aws_security_group",
RefFieldName: "SecurityGroupRefs",
SelectorFieldName: "SecurityGroupSelector",
},
}
r.References["compute_resources.subnets"] = config.Reference{
TerraformName: "aws_subnet",
RefFieldName: "SubnetRefs",
SelectorFieldName: "SubnetSelector",
}
r.References["compute_resources.security_group_ids"] = config.Reference{
TerraformName: "aws_security_group",
RefFieldName: "SecurityGroupRefs",
SelectorFieldName: "SecurityGroupSelector",
}

@sachinmalanki sachinmalanki mentioned this pull request Oct 25, 2024
Merged
3 tasks
@sachinmalanki
Copy link
Contributor Author

I have raised another PR - #1542 with all the changes suggested.

@sachinmalanki
feat: aws-batch resources - compute environment and job queue
cd184a6 
Signed-off-by: sachinmalanki <sachin.chini@gmail.com>
@turkenf
Copy link
Collaborator

turkenf commented Oct 25, 2024

Hi @sachinmalanki, which PR will we continue with?

@sachinmalanki
Copy link
Contributor Author

Lets continue with #1542. let me close this PR

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@turkenf turkenf turkenf left review comments

@ulucinar ulucinar Awaiting requested review from ulucinar ulucinar is a code owner

@sergenyalcin sergenyalcin Awaiting requested review from sergenyalcin sergenyalcin is a code owner

@mbbush mbbush Awaiting requested review from mbbush mbbush is a code owner

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
8 participants
@sachinmalanki @jeanduplessis @turkenf @mergenci @sergenyalcin @blakeromano @ravilr @erhancagirici