DMS Endpoint: Apply extra_connection_attributes to all engine types in DMS

[microamp]

Changes proposed in this pull request:

• For certain DMS endpoint engines, there is a dedicated set of the engine-specific settings. s3_settings for s3 and mongodb_settings for mongodb to name a few. However, there exist engine-specific attributes that can only be changed via extra_connection_attributes. While this appears to be an inconsistent behaviour of the corresponding AWS API, we have to make extra_connection_attributes available to all engine types. One example is addColumnName for s3. It's currently not being handled under s3_settings hence must be specified in extra_connection_attributes if you want a non-default behaviour (in this case, false).

Output from acceptance testing:

$ make testacc TESTARGS='-run=TestAccAwsDmsEndpointS3'

==> Checking that code complies with gofmt requirements...
TF_ACC=1 go test ./... -v -run=TestAccAwsDmsEndpointS3 -timeout 120m
?      github.com/terraform-providers/terraform-provider-aws  [no test files]
=== RUN   TestAccAwsDmsEndpointS3
--- PASS: TestAccAwsDmsEndpointS3 (81.30s)
PASS
ok     github.com/terraform-providers/terraform-provider-aws/aws      81.309s

[microamp]

Hi. Do we have any updates on this by any chance? At present, any non-default extra_connection_attributes for s3 and mongodb are not going through when the task is first created due to this bug. Something I should've mentioned earlier. 😺

[stephencoe]

I have also come across this issue recently, currently the only way to get the extra_connection_attributes for services is to trigger an update and double apply 👎 It would be good to see this merged soon

[stephencoe]

While it's not related to extra_connection_attributes I also think the kms_key_arn is in the wrong place here.

Its handled in the state outside of the default block and the docs describe this as the encryption key used by the replication instance and connection information so it shouldn't be service specific.

[microamp]

Thanks for pointing that out. I'll take a closer look at it soon.

[microamp]

@bflad @stephencoe Rebased and placed two commits on top. Please review/test.

[ghost]

I have just been caught out by this. I had set extra_connection_attributes = "addColumnName=true" Set this about a month ago, about to write a lambda to leverage the column headers and they aren't to be found in the S3 files.

Any update on when this will be merged?

[a4c-ricardo-chaves]

I have the same issue.

[ramziyassine]

I ran into the same issue will this be something mergeable in Terraform 12?

[elijah-ward]

Same issue 😪. I'm wondering since this has been open for so long, have people found a workaround? I'm wanting to set DataFormat: parquet on an s3 target endpoint in addition to some of the other extra params mentioned above.

[anGie44]

Hi @microamp, thank you for you submission and apologies for the wait! given the changes in the codebase since the time of your submission we'll be moving forward with #16827 to address the extra_connection_attributes change to make it applicable to all engine types.

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!