Get latest cloudflare-go and update paginations

[tamas-jozsa]

Updating cloudflare-go to v0.46.0 and use paginated APIs for filter, firewall-rule and rate-limit resources.

Furthermore, I had to fix the following test tf resources to make go test work:

• cloudflare_access_application_simple_account/test.tf
• cloudflare_access_application_with_cors_account/test.tf
• cloudflare_access_rule_account/test.tf

Related issue cloudflare/terraform-provider-cloudflare#1819 about:

• cloudflare_account_member/test.tf

I'm also a bit puzzled, why go vet is failing during test action.

@jacobbednarz I'd like to ask your guidance on the failing terraform validate step for certificate_authority resource. I see that the test resource doesn't reflect the tf resource: https://registry.terraform.io/providers/cloudflare/cloudflare/latest/docs/resources/certificate_pack

../cloudflare_certificate_pack/test.tf:

resource "cloudflare_certificate_pack" "terraform_managed_resource" {
  hosts   = ["example.com", "*.example.com", "www.example.com"]
  type    = "dedicated_custom"
  zone_id = "0da42c8d2132a9ddaf714f9e7c920711"
}

And I also can't align the new tf resource to the actual API response at: https://api.cloudflare.com/client/v4/zones/1fa53a4819fe128dd98be7df45ec7cc6/ssl/certificate_packs?status=all

Furthermore, it looks like the filter and firewall_rules APIs with the pagination updates are calling public API endpoints twice. This PR is aiming to fix that: cloudflare/cloudflare-go#1012

[jacobbednarz]

for future reference, there is automation that handles bumping versions (we shouldn't do it manually). it helps with things like OS dependencies and any version mismatches, see #445 for an example.

as you've done the work here, we'll push forward with this one.

[jacobbednarz]

the last failing validate step looks to be a conflict in the provider and output where not all fields are being exported. will need to dig in to the provider to see why that is.

[tamas-jozsa]

the last failing validate step looks to be a conflict in the provider and output where not all fields are being exported. will need to dig in to the provider to see why that is.

There are some inconsistencies here. The example response published by the doc seems to be outdated: https://api.cloudflare.com/#certificate-packs-list-certificate-packs

I run a query against https://api.cloudflare.com/client/v4/zones/{zone_id}/ssl/certificate_packs
And I see new fields like validity_days, validation_method, but I'm not seeing certificate_authority.

I wonder what should be my next action with this situation.

[jacobbednarz]

there are a couple of things here to address this.

the first is that testdata/cloudflare/cloudflare_certificate_pack.yaml should be deleted as it is now deprecated and in it's place, testdata/cloudflare/cloudflare_certificate_pack_acm.yaml should be added. this can be done in https://github.com/cloudflare/cf-terraforming/blob/master/internal/app/cf-terraforming/cmd/generate_test.go#L93

And I see new fields like validity_days, validation_method, but I'm not seeing certificate_authority.

the issue looks to be that we are marshaling []CertificatePack from https://github.com/cloudflare/cf-terraforming/blob/master/internal/app/cf-terraforming/cmd/generate.go#L299, however, we're the new responses for ACM are wanting []AdvancedCertificatePack (hence the missing fields). to fix this, i've raised cloudflare/cloudflare-go#1032 to clean up the structs and return types which will address this failure.

alternatively, you could try manually mapping the values using something like https://github.com/tidwall/gjson however that seems really error prone instead of fixing this correctly in the underlying Go library.

[tamas-jozsa]

I'm in no rush merging this, so I'll just wait for cloudflare/cloudflare-go#1032 :)
I'm also thinking waiting for the next cloudflare-go version bump here in cf-tf, so I can avoid the problems with double endpoint calls during paginations.

[tamas-jozsa]

@jacobbednarz I updated cloudflare-go to 47, and it looks like one of the test cloudflare ruleset (no configuration) is killed by rate limit. Error is pointing here: https://github.com/cloudflare/cloudflare-go/blob/master/cloudflare.go#L255

Not sure though if this was the case in cloudflare-go 46 as well. I will look into this more later.

[jacobbednarz]

if you track this down, let me know and we can cut a hotfix in cloudflare-go if needed. this was fine in the test suite but yet to run the full acceptance test suite in terraform.

[tamas-jozsa]

@jacobbednarz so looking a bit deeper into I found that cf-tf test is failing because resp is nil here: cloudflare/cloudflare-go@8563c7a and that actually uncovered a missing mock:

https://github.com/cloudflare/cf-terraforming/blob/master/testdata/cloudflare/cloudflare_ruleset_zone_no_configuration.yaml

We have mocked the API call: https://api.cloudflare.com/client/v4/zones/0da42c8d2132a9ddaf714f9e7c920711/rulesets
But don't have a mock for https://api.cloudflare.com/client/v4/zones/0da42c8d2132a9ddaf714f9e7c920711/rulesets/0800fc577294c34e0b28ad2839435945

I'm curious how should that look like?

[jacobbednarz]

fwiw, both cloudflare-go and the entire terraform provider test suites aren't seeing issues so it looks like we're doing something in this library (possibly VCR related as you mentioned above).

will have a look into this one for you.

[jacobbednarz]