Add aws_codebuild_webhook resource for creating GitHub webhooks for CodeBuild projects #4473

joestump · 2018-05-07T21:46:40Z

Changes proposed in this pull request:

Introduces a resource called aws_codebuild_webhook that creates GitHub webhook for a CodeBuild project that has its source stored on GitHub.

This works builds on #2814 and adds branch filtering.

Output from acceptance testing:

$ make testacc TESTARGS='-run=TestAccAwsCodeBuildWebhook_basic'
==> Checking that code complies with gofmt requirements...
TF_ACC=1 go test ./... -v -run=TestAccAwsCodeBuildWebhook_basic -timeout 120m
?   	github.com/terraform-providers/terraform-provider-aws	[no test files]
=== RUN   TestAccAwsCodeBuildWebhook_basic
--- FAIL: TestAccAwsCodeBuildWebhook_basic (41.83s)
	testing.go:518: Step 0 error: Error applying: 1 error(s) occurred:
		
		* aws_codebuild_webhook.test: 1 error(s) occurred:
		
		* aws_codebuild_webhook.test: ResourceNotFoundException: Could not find access token for server type github
			status code: 400, request id: fe4a925b-523e-11e8-8f47-a9109ac40b1a
FAIL
FAIL	github.com/terraform-providers/terraform-provider-aws/aws	41.863s
make: *** [testacc] Error 1

The reason because this fails is because:

The AWS user Terraform runs as must have already done an OAuth login with GitHub on an existing CodeBuild project.
The GitHub user that the AWS user logs in as must have access to the GitHub repository listed in the source.

I've tried to test this, but passing AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY to tests appears to be ignored. Creating a webhook with the credentials I created works via the CLI, but not via the API call this resource makes.

How does one use their own AWS credentials when running make test?

NOTE: Once this passes, the upstream Terraform test environment will need to address this in order to run these tests. Alternatively, I could implement something like the Heroku provider has for skipping tests.

…aform-provider-aws into jstump-add-codepipeline-webhook

gthole · 2018-05-18T17:09:35Z

Not sure this will help much without tracking the secret and payload_url returned from AWS and making them available as attributes.

For example, this is the syntax I'd expect:

resource "aws_codebuild_webhook" "my_webhook" {
    name = "${aws_codebuild_project.some_project.name}"
    branch_filter = "master"
}

resource "github_repository_webhook" "my_webhook" {
    repository = "${github_repository.repo.name}"
    name = "some_project_deploy"

    configuration {
        url = "${aws_codebuild_webhook.my_webhook.payload_url}"
        secret = "${aws_codebuld_webhook.my_webhook.secret}"
        content_type = "json"
        insecure_ssl = false
    }

    active = true
    events = ["push"]
}

Unfortunately, it doesn't look like the secret is ever returned from batchGetProjects, despite what the documentation says. (Otherwise I'd just create a provisioner and external data source to work around this.)

So if the secret can be stored in the state, that'd be ideal.

Awesome to see progress on this - it'd be a huge win for our devops stack.

bflad · 2018-05-24T21:28:21Z

I'll be taking a look at this tomorrow. 👍

bflad

Pull request review below for those curious -- I will be handling the feedback for this one and getting this merged shortly. 😄

bflad · 2018-05-25T19:31:47Z

aws/resource_aws_codebuild_project_test.go

+	// Additionally, the GitHub user that the Terraform AWS user logs in as must have
+	// access to the GitHub repository. This allows others to run tests for the webhook
+	// without having to have access to the Packer GitHub repository.
+	sourceURL, ok := os.LookupEnv("CODEBUILD_GITHUB_SOURCE_URL")


This is a great idea -- I'll make sure this gets ported over in some fashion when I rebase this PR.

bflad · 2018-05-25T19:33:38Z

aws/resource_aws_codebuild_webhook.go

+		Update: resourceAwsCodeBuildWebhookUpdate,
+
+		Schema: map[string]*schema.Schema{
+			"name": {


This might seem pedantic for now, but on merge will rename this attribute to project_name to better align with the API and help operators understand the association with projects.

bflad · 2018-05-25T19:34:37Z

aws/resource_aws_codebuild_webhook.go

+	}
+
+	d.SetId(d.Get("name").(string))
+	d.Set("branch_filter", d.Get("branch_filter").(string))


d.Set() should only be called in the read function 😄 -- will fix this on merge.

bflad · 2018-05-25T19:38:39Z

aws/resource_aws_codebuild_webhook.go

+	})
+
+	if err != nil {
+		return err


We should check for "resource not found" type errors and d.SetId("") similar to how the project resource does this. Will fix on merge. 👍

bflad · 2018-05-25T19:40:05Z

aws/resource_aws_codebuild_webhook.go

+	}
+
+	if len(resp.Projects) == 0 {
+		d.SetId("")


When we're removing a resource from the state, we should log an appropriate warning message in the logs, e.g.

log.Printf("[WARN] CodeBuild Project %q not found, removing from state", d.Id())

bflad · 2018-05-25T19:43:43Z

aws/resource_aws_codebuild_webhook_test.go

+				Config: testAccCodeBuildWebhookConfig_basic(acctest.RandString(5)),
+				Check: resource.ComposeTestCheckFunc(
+					testAccCheckAwsCodeBuildWebhookExists("aws_codebuild_webhook.test"),
+					resource.TestCheckResourceAttrSet("aws_codebuild_webhook.test", "url"),


We should probably be a little more cautious and at least partially validate the value as resource.TestCheckResourceAttrSet() will pass with empty strings and I believe even d.Set("XXX", nil)

e.g.

resource.TestMatchResourceAttr("aws_codebuild_webhook.test", "url", regexp.MustCompile(`^https://`))

bflad · 2018-05-25T19:45:14Z

aws/resource_aws_codebuild_webhook_test.go

+		})
+
+		if err != nil {
+			return err


A resource not found exception here would be a good thing and should return nil -- will fix on merge.

bflad · 2018-05-25T19:46:23Z

aws/resource_aws_codebuild_webhook_test.go

+		if !ok {
+			return fmt.Errorf("Not found: %s", name)
+		}
+


Similar to how the destroy acceptance test function above is checking against the API, we should do the same thing here to ensure the "physical" webhook actually exists.

bflad · 2018-05-25T19:47:21Z

website/docs/r/codebuild_webhook.html.markdown

+
+Provides a CodeBuild Webhook resource.
+
+~> **Note:** The AWS account that Terraform uses to create this resource *must* have authorized CodeBuild to access GitHub's OAuth API. This is a manual step that must be done *before* creating webhooks with this resource. If OAuth is not configured, AWS will return an error similar to `ResourceNotFoundException: Could not find access token for server type github`.


This manual step must also be done in every applicable region where the resource will be used. We should probably also point to the AWS documentation surrounding project webhooks here. Will update on merge.

bflad · 2018-05-25T19:50:46Z

aws/resource_aws_codebuild_webhook.go

+	"github.com/hashicorp/terraform/helper/schema"
+)
+
+func resourceAwsCodeBuildWebhook() *schema.Resource {


We can easily add import support for this resource as the ID is already handled properly in the read function. Will add the passthrough importer to this resource and document how to import on merge.

bflad · 2018-05-25T20:00:16Z

Not sure this will help much without tracking the secret and payload_url returned from AWS and making them available as attributes.

AWS is using "your" GitHub OAuth credentials to manage the actual GitHub webhook (physically creating, updating, and deleting on the GitHub side with this resource). ~~I'm not sure they expect you to be managing the webhook outside of their workflow.~~

Edit: Ah, but upon further reading of the documentation -- this looks like its required for GitHub Enterprise. I'll make sure the additional attributes are exported. 👍 Sorry for jumping the gun.

bflad · 2018-05-25T23:05:06Z

The new aws_codebuild_webhook resource will be available in version 1.21.0 of the AWS provider, likely releasing middle of next week. 🎉

gthole · 2018-05-25T23:25:30Z

Hooray! This is a big improvement for our stacks. Thanks @bflad, @atsushi-ishibashi @joestump!

joestump · 2018-05-29T17:29:01Z

@bflad thanks for the notes and merge! 🎉

bflad · 2018-05-31T04:01:29Z

This has been released in version 1.21.0 of the AWS provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

salvianreynaldi · 2018-05-31T07:01:27Z

I think the documentation needs to be updated. It still list only name and branch_filter as the supported arguments, while the code has more than that (and name should be project_name too).

ghost · 2020-04-05T17:42:09Z

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!

atsushi-ishibashi and others added 7 commits January 1, 2018 01:07

wip

19e6763

Merge branch 'codebuild_webhook' of github.com:atsushi-ishibashi/terr…

f6aa617

…aform-provider-aws into jstump-add-codepipeline-webhook

Fix up docs.

b7f1069

Add branch filtering support.

b40dc8e

Fix tests and add an update function.

8522c88

Fix docs and add a note about the OAuth dance.

22e8140

Allow people to override the sourceURL in the basic config.

9abbc82

ghost added the size/L Managed by automation to categorize the size of a PR. label May 7, 2018

joestump mentioned this pull request May 7, 2018

[WIP, NeedHelp]New Resource: codebuild_webhook #2814

Merged

bflad added new-resource Introduces a new resource. service/codebuild Issues and PRs that pertain to the codebuild service. labels May 8, 2018

paultyng assigned bflad May 21, 2018

paultyng requested a review from bflad May 21, 2018 18:26

bflad reviewed May 25, 2018

View reviewed changes

bflad merged commit 9abbc82 into hashicorp:master May 25, 2018

bflad added this to the v1.21.0 milestone May 25, 2018

bflad added a commit that referenced this pull request May 25, 2018

Update CHANGELOG for #4473

50b338c

joestump deleted the jstump-codebuild-webhook branch May 29, 2018 17:29

gthole mentioned this pull request May 31, 2018

aws_codebuild_webhook: Secret not propagated to GHE webhook as in documentation example #4720

Closed

ghost locked and limited conversation to collaborators Apr 5, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add aws_codebuild_webhook resource for creating GitHub webhooks for CodeBuild projects #4473

Add aws_codebuild_webhook resource for creating GitHub webhooks for CodeBuild projects #4473

joestump commented May 7, 2018

gthole commented May 18, 2018

bflad commented May 24, 2018

bflad left a comment

bflad May 25, 2018

bflad May 25, 2018

bflad May 25, 2018

bflad May 25, 2018

bflad May 25, 2018

bflad May 25, 2018

bflad May 25, 2018

bflad May 25, 2018

bflad May 25, 2018

bflad May 25, 2018

bflad commented May 25, 2018 •

edited

Loading

bflad commented May 25, 2018

gthole commented May 25, 2018

joestump commented May 29, 2018

bflad commented May 31, 2018

salvianreynaldi commented May 31, 2018

ghost commented Apr 5, 2020


		Provides a CodeBuild Webhook resource.

		~> Note: The AWS account that Terraform uses to create this resource must have authorized CodeBuild to access GitHub's OAuth API. This is a manual step that must be done before creating webhooks with this resource. If OAuth is not configured, AWS will return an error similar to `ResourceNotFoundException: Could not find access token for server type github`.

Add aws_codebuild_webhook resource for creating GitHub webhooks for CodeBuild projects #4473

Add aws_codebuild_webhook resource for creating GitHub webhooks for CodeBuild projects #4473

Conversation

joestump commented May 7, 2018

gthole commented May 18, 2018

bflad commented May 24, 2018

bflad left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bflad commented May 25, 2018 • edited Loading

bflad commented May 25, 2018

gthole commented May 25, 2018

joestump commented May 29, 2018

bflad commented May 31, 2018

salvianreynaldi commented May 31, 2018

ghost commented Apr 5, 2020

bflad commented May 25, 2018 •

edited

Loading