Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

r/firehose_delivery_stream - add support for elasticsearch cluster_endpoint + validations #12484

Merged
merged 8 commits into from
Aug 31, 2020
Merged

r/firehose_delivery_stream - add support for elasticsearch cluster_endpoint + validations #12484

merged 8 commits into from
Aug 31, 2020

Conversation

DrFaust92
Copy link
Collaborator

@DrFaust92 DrFaust92 commented Mar 19, 2020 •
edited
Loading

Community Note

  • Please vote on this pull request by adding a 👍 reaction to the original pull request comment to help the community and maintainers prioritize this request
  • Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for pull request followers and do not help prioritize the request

Closes #10623
Relates #13624

Release note for CHANGELOG:

resource_aws_firehose_delivery_stream: add support for `cluster_endpoint` in `elasticserach_configuration`

resource_aws_firehose_delivery_stream: plan time validations for `bucket_arn`, `buffer_size`, `buffer_interval`, `compression_format`, `role_arn`

Output from acceptance testing:

$ make testacc TESTARGS='-run=TestAccAWSKinesisFirehoseDeliveryStream_'
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_basic (183.65s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_s3basic (154.23s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_s3basicWithSSE (559.93s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_s3basicWithTags (336.71s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_s3KinesisStreamSource (181.64s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_s3WithCloudwatchLogging (179.94s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_s3ConfigUpdates (301.67s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3basic (195.71s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_DataFormatConversionConfiguration_Enabled (328.87s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_ExternalUpdate (243.77s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_DataFormatConversionConfiguration_Deserializer_Update (223.10s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_DataFormatConversionConfiguration_HiveJsonSerDe_Empty (173.16s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_DataFormatConversionConfiguration_HiveJsonSerDe_Empty (173.16s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_DataFormatConversionConfiguration_OrcSerDe_Empty (224.00s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_DataFormatConversionConfiguration_ParquetSerDe_Empty (202.04s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_DataFormatConversionConfiguration_Serializer_Update (287.97s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_ErrorOutputPrefix (253.96s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3KmsKeyArn (243.57s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3Updates (377.56s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_RedshiftConfigUpdates (564.50s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_SplunkConfigUpdates (305.58s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_missingProcessingConfiguration (188.65s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_ElasticsearchConfigUpdates (1117.34s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_ElasticsearchConfigEndpointUpdates (1352.41s)

@DrFaust92 DrFaust92 requested a review from a team March 19, 2020 22:46
@ghost ghost added needs-triage Waiting for first response or review from a maintainer. size/XL Managed by automation to categorize the size of a PR. service/firehose Issues and PRs that pertain to the firehose service. tests PRs: expanded test coverage. Issues: expanded coverage, enhancements to test infrastructure. documentation Introduces or discusses updates to documentation. labels Mar 19, 2020
@DrFaust92 DrFaust92 changed the title [WIP]r/firehose_delivery_stream - add support for elasticsearch cluster_endpoint + validations r/firehose_delivery_stream - add support for elasticsearch cluster_endpoint + validations Mar 20, 2020
@ewbankkit
Copy link
Contributor

ewbankkit commented Mar 20, 2020 •
edited
Loading

Acceptance tests:

$ make testacc TEST=./aws/ TESTARGS='-run=TestAccAWSKinesisFirehoseDeliveryStream_'
==> Checking that code complies with gofmt requirements...
TF_ACC=1 go test ./aws/ -v -count 1 -parallel 20 -run=TestAccAWSKinesisFirehoseDeliveryStream_ -timeout 120m
=== RUN   TestAccAWSKinesisFirehoseDeliveryStream_basic
=== PAUSE TestAccAWSKinesisFirehoseDeliveryStream_basic
=== RUN   TestAccAWSKinesisFirehoseDeliveryStream_s3basic
=== PAUSE TestAccAWSKinesisFirehoseDeliveryStream_s3basic
=== RUN   TestAccAWSKinesisFirehoseDeliveryStream_s3basicWithSSE
=== PAUSE TestAccAWSKinesisFirehoseDeliveryStream_s3basicWithSSE
=== RUN   TestAccAWSKinesisFirehoseDeliveryStream_s3basicWithTags
=== PAUSE TestAccAWSKinesisFirehoseDeliveryStream_s3basicWithTags
=== RUN   TestAccAWSKinesisFirehoseDeliveryStream_s3KinesisStreamSource
=== PAUSE TestAccAWSKinesisFirehoseDeliveryStream_s3KinesisStreamSource
=== RUN   TestAccAWSKinesisFirehoseDeliveryStream_s3WithCloudwatchLogging
=== PAUSE TestAccAWSKinesisFirehoseDeliveryStream_s3WithCloudwatchLogging
=== RUN   TestAccAWSKinesisFirehoseDeliveryStream_s3ConfigUpdates
=== PAUSE TestAccAWSKinesisFirehoseDeliveryStream_s3ConfigUpdates
=== RUN   TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3basic
=== PAUSE TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3basic
=== RUN   TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_DataFormatConversionConfiguration_Enabled
=== PAUSE TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_DataFormatConversionConfiguration_Enabled
=== RUN   TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_ExternalUpdate
=== PAUSE TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_ExternalUpdate
=== RUN   TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_DataFormatConversionConfiguration_Deserializer_Update
=== PAUSE TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_DataFormatConversionConfiguration_Deserializer_Update
=== RUN   TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_DataFormatConversionConfiguration_HiveJsonSerDe_Empty
=== PAUSE TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_DataFormatConversionConfiguration_HiveJsonSerDe_Empty
=== RUN   TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_DataFormatConversionConfiguration_OpenXJsonSerDe_Empty
=== PAUSE TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_DataFormatConversionConfiguration_OpenXJsonSerDe_Empty
=== RUN   TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_DataFormatConversionConfiguration_OrcSerDe_Empty
=== PAUSE TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_DataFormatConversionConfiguration_OrcSerDe_Empty
=== RUN   TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_DataFormatConversionConfiguration_ParquetSerDe_Empty
=== PAUSE TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_DataFormatConversionConfiguration_ParquetSerDe_Empty
=== RUN   TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_DataFormatConversionConfiguration_Serializer_Update
=== PAUSE TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_DataFormatConversionConfiguration_Serializer_Update
=== RUN   TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_ErrorOutputPrefix
=== PAUSE TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_ErrorOutputPrefix
=== RUN   TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3KmsKeyArn
=== PAUSE TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3KmsKeyArn
=== RUN   TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3Updates
=== PAUSE TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3Updates
=== RUN   TestAccAWSKinesisFirehoseDeliveryStream_RedshiftConfigUpdates
=== PAUSE TestAccAWSKinesisFirehoseDeliveryStream_RedshiftConfigUpdates
=== RUN   TestAccAWSKinesisFirehoseDeliveryStream_SplunkConfigUpdates
=== PAUSE TestAccAWSKinesisFirehoseDeliveryStream_SplunkConfigUpdates
=== RUN   TestAccAWSKinesisFirehoseDeliveryStream_ElasticsearchConfigUpdates
=== PAUSE TestAccAWSKinesisFirehoseDeliveryStream_ElasticsearchConfigUpdates
=== RUN   TestAccAWSKinesisFirehoseDeliveryStream_ElasticsearchConfigEndpointUpdates
=== PAUSE TestAccAWSKinesisFirehoseDeliveryStream_ElasticsearchConfigEndpointUpdates
=== RUN   TestAccAWSKinesisFirehoseDeliveryStream_missingProcessingConfiguration
=== PAUSE TestAccAWSKinesisFirehoseDeliveryStream_missingProcessingConfiguration
=== CONT  TestAccAWSKinesisFirehoseDeliveryStream_basic
=== CONT  TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_DataFormatConversionConfiguration_OrcSerDe_Empty
=== CONT  TestAccAWSKinesisFirehoseDeliveryStream_missingProcessingConfiguration
=== CONT  TestAccAWSKinesisFirehoseDeliveryStream_ElasticsearchConfigEndpointUpdates
=== CONT  TestAccAWSKinesisFirehoseDeliveryStream_ElasticsearchConfigUpdates
=== CONT  TestAccAWSKinesisFirehoseDeliveryStream_SplunkConfigUpdates
=== CONT  TestAccAWSKinesisFirehoseDeliveryStream_RedshiftConfigUpdates
=== CONT  TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3Updates
=== CONT  TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3KmsKeyArn
=== CONT  TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_ErrorOutputPrefix
=== CONT  TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_DataFormatConversionConfiguration_Serializer_Update
=== CONT  TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_DataFormatConversionConfiguration_ParquetSerDe_Empty
=== CONT  TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3basic
=== CONT  TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_DataFormatConversionConfiguration_OpenXJsonSerDe_Empty
=== CONT  TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_DataFormatConversionConfiguration_HiveJsonSerDe_Empty
=== CONT  TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_DataFormatConversionConfiguration_Deserializer_Update
=== CONT  TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_ExternalUpdate
=== CONT  TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_DataFormatConversionConfiguration_Enabled
=== CONT  TestAccAWSKinesisFirehoseDeliveryStream_s3KinesisStreamSource
=== CONT  TestAccAWSKinesisFirehoseDeliveryStream_s3ConfigUpdates
--- FAIL: TestAccAWSKinesisFirehoseDeliveryStream_RedshiftConfigUpdates (46.29s)
    testing.go:654: Step 0 error: errors during apply:
        
        Error: InvalidVPCNetworkStateFault: Cannot create a publicly accessible cluster because customer VPC vpc-b6fbb6cf has no internet gateway attached.
        	status code: 400, request id: 84136a5b-6ad3-11ea-b9ec-87dad191ba4d
        
          on /tmp/tf-test290254966/main.tf line 74:
          (source code not available)
        
        
=== CONT  TestAccAWSKinesisFirehoseDeliveryStream_s3WithCloudwatchLogging
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_s3KinesisStreamSource (144.19s)
=== CONT  TestAccAWSKinesisFirehoseDeliveryStream_s3basicWithSSE
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_DataFormatConversionConfiguration_OpenXJsonSerDe_Empty (144.68s)
=== CONT  TestAccAWSKinesisFirehoseDeliveryStream_s3basicWithTags
--- FAIL: TestAccAWSKinesisFirehoseDeliveryStream_s3WithCloudwatchLogging (106.39s)
    testing.go:654: Step 0 error: Check failed: Check 2/2 error: Bad Stream name: tf-acc-test-8589700275110392786
=== CONT  TestAccAWSKinesisFirehoseDeliveryStream_s3basic
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_DataFormatConversionConfiguration_OrcSerDe_Empty (153.39s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_DataFormatConversionConfiguration_HiveJsonSerDe_Empty (153.54s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_DataFormatConversionConfiguration_ParquetSerDe_Empty (156.32s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3basic (167.80s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_missingProcessingConfiguration (168.77s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_basic (198.25s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_DataFormatConversionConfiguration_Serializer_Update (198.35s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3KmsKeyArn (200.16s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_SplunkConfigUpdates (210.96s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_ExternalUpdate (214.11s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_DataFormatConversionConfiguration_Deserializer_Update (220.42s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_ErrorOutputPrefix (232.71s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3Updates (255.25s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_DataFormatConversionConfiguration_Enabled (258.90s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_s3basic (118.01s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_s3ConfigUpdates (292.28s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_s3basicWithTags (210.00s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_s3basicWithSSE (318.50s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_ElasticsearchConfigUpdates (825.32s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_ElasticsearchConfigEndpointUpdates (835.93s)
FAIL
FAIL	github.com/terraform-providers/terraform-provider-aws/aws	836.004s
FAIL
GNUmakefile:26: recipe for target 'testacc' failed
make: *** [testacc] Error 1

TestAccAWSKinesisFirehoseDeliveryStream_RedshiftConfigUpdates fails because the default VPC in the test region doesn't have an IGW attached - Probably too esoteric to add a PreCheck for.
TestAccAWSKinesisFirehoseDeliveryStream_s3WithCloudwatchLogging fails because testAccCheckAWSKinesisFirehoseDeliveryStreamAttributes is checking just for the stream name prefix of terraform-kinesis-firehose.

@DrFaust92 DrFaust92 force-pushed the r/firehost-stream-clusterendpoint branch from 9ea342c to e9800ac Compare April 25, 2020 19:18
@DrFaust92
Copy link
Collaborator Author

Fixed failing TestAccAWSKinesisFirehoseDeliveryStream_s3WithCloudwatchLogging test + add prefix to enums

@DrFaust92 DrFaust92 added the enhancement Requests to existing resources that expand the functionality or scope. label May 22, 2020
@williamsbdev
Copy link

Great job @DrFaust92! I just ran into this bug and would love to see this get merged since you've put all the effort into this resource/PR. Thank you!

Any idea what else is needed to get this merge? Do we need to reach out to someone?

@MeMan-MasterOfTheUniverse
Copy link

Very nice! Following up on this as well. Any ETA on when this would/could be released?

@DrFaust92 DrFaust92 force-pushed the r/firehost-stream-clusterendpoint branch from 3ae1356 to 34caea3 Compare June 24, 2020 15:13
@DrFaust92 DrFaust92 removed the needs-triage Waiting for first response or review from a maintainer. label Aug 4, 2020
@rsclarke-vgw
Copy link

This is blocking those using the new WAFv2 resources and logging cross account to Elasticsearch.

@DrFaust92 DrFaust92 force-pushed the r/firehost-stream-clusterendpoint branch from 34caea3 to 00d5715 Compare August 24, 2020 13:37
@ewbankkit
Copy link
Contributor

Verified acceptance tests:

$ make testacc TEST=./aws/ TESTARGS='-run=TestAccAWSKinesisFirehoseDeliveryStream_' ACCTEST_PARALLELISM=2
==> Checking that code complies with gofmt requirements...
TF_ACC=1 go test ./aws -v -count 1 -parallel 2 -run=TestAccAWSKinesisFirehoseDeliveryStream_ -timeout 120m
=== RUN   TestAccAWSKinesisFirehoseDeliveryStream_basic
=== PAUSE TestAccAWSKinesisFirehoseDeliveryStream_basic
=== RUN   TestAccAWSKinesisFirehoseDeliveryStream_disappears
=== PAUSE TestAccAWSKinesisFirehoseDeliveryStream_disappears
=== RUN   TestAccAWSKinesisFirehoseDeliveryStream_s3basic
=== PAUSE TestAccAWSKinesisFirehoseDeliveryStream_s3basic
=== RUN   TestAccAWSKinesisFirehoseDeliveryStream_s3basicWithSSE
=== PAUSE TestAccAWSKinesisFirehoseDeliveryStream_s3basicWithSSE
=== RUN   TestAccAWSKinesisFirehoseDeliveryStream_s3basicWithTags
=== PAUSE TestAccAWSKinesisFirehoseDeliveryStream_s3basicWithTags
=== RUN   TestAccAWSKinesisFirehoseDeliveryStream_s3KinesisStreamSource
=== PAUSE TestAccAWSKinesisFirehoseDeliveryStream_s3KinesisStreamSource
=== RUN   TestAccAWSKinesisFirehoseDeliveryStream_s3WithCloudwatchLogging
=== PAUSE TestAccAWSKinesisFirehoseDeliveryStream_s3WithCloudwatchLogging
=== RUN   TestAccAWSKinesisFirehoseDeliveryStream_s3ConfigUpdates
=== PAUSE TestAccAWSKinesisFirehoseDeliveryStream_s3ConfigUpdates
=== RUN   TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3basic
=== PAUSE TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3basic
=== RUN   TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_DataFormatConversionConfiguration_Enabled
=== PAUSE TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_DataFormatConversionConfiguration_Enabled
=== RUN   TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_ExternalUpdate
=== PAUSE TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_ExternalUpdate
=== RUN   TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_DataFormatConversionConfiguration_Deserializer_Update
=== PAUSE TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_DataFormatConversionConfiguration_Deserializer_Update
=== RUN   TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_DataFormatConversionConfiguration_HiveJsonSerDe_Empty
=== PAUSE TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_DataFormatConversionConfiguration_HiveJsonSerDe_Empty
=== RUN   TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_DataFormatConversionConfiguration_OpenXJsonSerDe_Empty
=== PAUSE TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_DataFormatConversionConfiguration_OpenXJsonSerDe_Empty
=== RUN   TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_DataFormatConversionConfiguration_OrcSerDe_Empty
=== PAUSE TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_DataFormatConversionConfiguration_OrcSerDe_Empty
=== RUN   TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_DataFormatConversionConfiguration_ParquetSerDe_Empty
=== PAUSE TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_DataFormatConversionConfiguration_ParquetSerDe_Empty
=== RUN   TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_DataFormatConversionConfiguration_Serializer_Update
=== PAUSE TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_DataFormatConversionConfiguration_Serializer_Update
=== RUN   TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_ErrorOutputPrefix
=== PAUSE TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_ErrorOutputPrefix
=== RUN   TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_ProcessingConfiguration_Empty
=== PAUSE TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_ProcessingConfiguration_Empty
=== RUN   TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3KmsKeyArn
=== PAUSE TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3KmsKeyArn
=== RUN   TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3Updates
=== PAUSE TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3Updates
=== RUN   TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_KinesisStreamSource
=== PAUSE TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_KinesisStreamSource
=== RUN   TestAccAWSKinesisFirehoseDeliveryStream_RedshiftConfigUpdates
=== PAUSE TestAccAWSKinesisFirehoseDeliveryStream_RedshiftConfigUpdates
=== RUN   TestAccAWSKinesisFirehoseDeliveryStream_SplunkConfigUpdates
=== PAUSE TestAccAWSKinesisFirehoseDeliveryStream_SplunkConfigUpdates
=== RUN   TestAccAWSKinesisFirehoseDeliveryStream_ElasticsearchConfigUpdates
=== PAUSE TestAccAWSKinesisFirehoseDeliveryStream_ElasticsearchConfigUpdates
=== RUN   TestAccAWSKinesisFirehoseDeliveryStream_ElasticsearchConfigEndpointUpdates
=== PAUSE TestAccAWSKinesisFirehoseDeliveryStream_ElasticsearchConfigEndpointUpdates
=== RUN   TestAccAWSKinesisFirehoseDeliveryStream_missingProcessingConfiguration
=== PAUSE TestAccAWSKinesisFirehoseDeliveryStream_missingProcessingConfiguration
=== CONT  TestAccAWSKinesisFirehoseDeliveryStream_basic
=== CONT  TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_DataFormatConversionConfiguration_OrcSerDe_Empty
=== CONT  TestAccAWSKinesisFirehoseDeliveryStream_basic
    testing_new_import_state.go:74: failed to import: terraform failed: exit status 1
        
        stderr:
        
        Error: Expected ID in format of arn:PARTITION:firehose:REGION:ACCOUNTID:deliverystream/NAME and provided: just-a-name
        
        
        
--- FAIL: TestAccAWSKinesisFirehoseDeliveryStream_basic (86.95s)
=== CONT  TestAccAWSKinesisFirehoseDeliveryStream_missingProcessingConfiguration
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_DataFormatConversionConfiguration_OrcSerDe_Empty (96.70s)
=== CONT  TestAccAWSKinesisFirehoseDeliveryStream_ElasticsearchConfigEndpointUpdates
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_missingProcessingConfiguration (83.69s)
=== CONT  TestAccAWSKinesisFirehoseDeliveryStream_ElasticsearchConfigUpdates
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_ElasticsearchConfigEndpointUpdates (811.02s)
=== CONT  TestAccAWSKinesisFirehoseDeliveryStream_SplunkConfigUpdates
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_ElasticsearchConfigUpdates (842.17s)
=== CONT  TestAccAWSKinesisFirehoseDeliveryStream_RedshiftConfigUpdates
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_SplunkConfigUpdates (140.11s)
=== CONT  TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_KinesisStreamSource
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_KinesisStreamSource (100.45s)
=== CONT  TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3Updates
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3Updates (155.36s)
=== CONT  TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3KmsKeyArn
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3KmsKeyArn (101.70s)
=== CONT  TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_ProcessingConfiguration_Empty
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_ProcessingConfiguration_Empty (89.48s)
=== CONT  TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_ErrorOutputPrefix
=== CONT  TestAccAWSKinesisFirehoseDeliveryStream_RedshiftConfigUpdates
    testing_new.go:22: WARNING: destroy failed, so remote objects may still exist and be subject to billing
    testing_new.go:22: failed to destroy: terraform failed: exit status 1
        
        stderr:
        
        Error: InvalidClusterSubnetGroupStateFault: Cannot delete the subnet group 'tf-acc-test-2969345511223302005' because at least one Cluster: deleted-305110 is still using it.
        	status code: 400, request id: bda4ddb1-413c-417e-89aa-fec69891188f
        
        
        
--- FAIL: TestAccAWSKinesisFirehoseDeliveryStream_RedshiftConfigUpdates (507.63s)
=== CONT  TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_DataFormatConversionConfiguration_Serializer_Update
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_ErrorOutputPrefix (132.42s)
=== CONT  TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_DataFormatConversionConfiguration_ParquetSerDe_Empty
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_DataFormatConversionConfiguration_Serializer_Update (131.20s)
=== CONT  TestAccAWSKinesisFirehoseDeliveryStream_s3ConfigUpdates
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_DataFormatConversionConfiguration_ParquetSerDe_Empty (89.71s)
=== CONT  TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_DataFormatConversionConfiguration_OpenXJsonSerDe_Empty
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_s3ConfigUpdates (160.89s)
=== CONT  TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_DataFormatConversionConfiguration_HiveJsonSerDe_Empty
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_DataFormatConversionConfiguration_OpenXJsonSerDe_Empty (97.53s)
=== CONT  TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_DataFormatConversionConfiguration_Deserializer_Update
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_DataFormatConversionConfiguration_HiveJsonSerDe_Empty (93.39s)
=== CONT  TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_ExternalUpdate
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_DataFormatConversionConfiguration_Deserializer_Update (131.58s)
=== CONT  TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_DataFormatConversionConfiguration_Enabled
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_ExternalUpdate (104.05s)
=== CONT  TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3basic
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_DataFormatConversionConfiguration_Enabled (160.93s)
=== CONT  TestAccAWSKinesisFirehoseDeliveryStream_s3basicWithTags
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3basic (102.40s)
=== CONT  TestAccAWSKinesisFirehoseDeliveryStream_s3WithCloudwatchLogging
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_s3WithCloudwatchLogging (83.62s)
=== CONT  TestAccAWSKinesisFirehoseDeliveryStream_s3KinesisStreamSource
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_s3basicWithTags (146.31s)
=== CONT  TestAccAWSKinesisFirehoseDeliveryStream_s3basic
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_s3KinesisStreamSource (97.17s)
=== CONT  TestAccAWSKinesisFirehoseDeliveryStream_s3basicWithSSE
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_s3basic (85.82s)
=== CONT  TestAccAWSKinesisFirehoseDeliveryStream_disappears
    resource_aws_kinesis_firehose_delivery_stream_test.go:141: [INFO] Got non-empty plan, as expected
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_disappears (84.85s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_s3basicWithSSE (231.36s)
FAIL
FAIL	github.com/terraform-providers/terraform-provider-aws/aws	2524.575s
FAIL
GNUmakefile:27: recipe for target 'testacc' failed
make: *** [testacc] Error 1

The first error is caused by #14801.
The second seems to be an eventual consistency issue when deleting a Redshift Subnet Group and is unrelated to these changes.

ewbankkit
ewbankkit approved these changes Aug 24, 2020
View reviewed changes
Copy link
Contributor

@ewbankkit ewbankkit left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

@bflad
Copy link
Contributor

bflad commented Aug 31, 2020

Ready for rebase

@DrFaust92 DrFaust92 force-pushed the r/firehost-stream-clusterendpoint branch from b7e891c to 4ba2886 Compare August 31, 2020 12:30
@DrFaust92
Copy link
Collaborator Author

Rebased

@ewbankkit ewbankkit mentioned this pull request Aug 31, 2020
Closed
bflad
bflad approved these changes Aug 31, 2020
View reviewed changes
Copy link
Contributor

@bflad bflad left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good 🚀

Output from acceptance testing (failure related to #14801, which is pending SDK release):

--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_DataFormatConversionConfiguration_HiveJsonSerDe_Empty (117.70s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_s3KinesisStreamSource (120.98s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_s3basic (138.97s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_ExternalUpdate (139.32s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_DataFormatConversionConfiguration_ParquetSerDe_Empty (147.64s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_s3WithCloudwatchLogging (151.49s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_disappears (163.31s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_ErrorOutputPrefix (168.98s)
--- FAIL: TestAccAWSKinesisFirehoseDeliveryStream_basic (170.08s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3KmsKeyArn (178.69s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_ProcessingConfiguration_Empty (182.19s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_DataFormatConversionConfiguration_OpenXJsonSerDe_Empty (182.93s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_DataFormatConversionConfiguration_OrcSerDe_Empty (183.27s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3basic (188.71s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_DataFormatConversionConfiguration_Deserializer_Update (193.04s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_DataFormatConversionConfiguration_Serializer_Update (196.71s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_DataFormatConversionConfiguration_Enabled (196.89s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_s3basicWithTags (205.97s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_KinesisStreamSource (99.58s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_s3basicWithSSE (248.87s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_s3ConfigUpdates (263.69s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3Updates (169.93s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_SplunkConfigUpdates (161.74s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_missingProcessingConfiguration (133.82s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_RedshiftConfigUpdates (512.06s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_ElasticsearchConfigUpdates (753.17s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_ElasticsearchConfigEndpointUpdates (753.32s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_ElasticsearchWithVpcConfigUpdates (1437.39s)

@bflad bflad added this to the v3.5.0 milestone Aug 31, 2020
@bflad bflad merged commit 1e9c2ae into hashicorp:master Aug 31, 2020
bflad added a commit that referenced this pull request Aug 31, 2020
@bflad
Update CHANGELOG for #12484
d3fb310
@ghost
Copy link

ghost commented Sep 3, 2020

This has been released in version 3.5.0 of the Terraform AWS provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template for triage. Thanks!

@ghost
Copy link

ghost commented Sep 30, 2020

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!

@ghost ghost locked as resolved and limited conversation to collaborators Sep 30, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Reviewers

@bflad bflad bflad approved these changes

@ewbankkit ewbankkit ewbankkit approved these changes

Assignees
No one assigned
Labels
documentation Introduces or discusses updates to documentation. enhancement Requests to existing resources that expand the functionality or scope. service/firehose Issues and PRs that pertain to the firehose service. size/XL Managed by automation to categorize the size of a PR. tests PRs: expanded test coverage. Issues: expanded coverage, enhancements to test infrastructure.
Projects
None yet
Milestone
v3.5.0
Development

Successfully merging this pull request may close these issues.

aws_kinesis_firehose_delivery_stream to support ClusterEndpoint for ElasticSearch
6 participants
@DrFaust92 @ewbankkit @williamsbdev @MeMan-MasterOfTheUniverse @rsclarke-vgw @bflad