Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

docs: Clarify kinesis_firehose_delivery_stream dynamic partitioning requirements #29093

Merged
merged 6 commits into from
Feb 10, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .changelog/29093.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
```release-note:enhancement
resource/aws_firehose_delivery_stream: Add `ForceNew` to `dynamic_partitioning_configuration` attribute
```
2 changes: 2 additions & 0 deletions internal/service/firehose/delivery_stream.go
Original file line number Diff line number Diff line change
Expand Up @@ -65,12 +65,14 @@ func dynamicPartitioningConfigurationSchema() *schema.Schema {
Type: schema.TypeList,
MaxItems: 1,
Optional: true,
ForceNew: true,
Elem: &schema.Resource{
Schema: map[string]*schema.Schema{
"enabled": {
Type: schema.TypeBool,
Optional: true,
Default: false,
ForceNew: true,
},
"retry_duration": {
Type: schema.TypeInt,
Expand Down
52 changes: 52 additions & 0 deletions internal/service/firehose/delivery_stream_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -974,6 +974,38 @@ func TestAccFirehoseDeliveryStream_extendedS3DynamicPartitioning(t *testing.T) {
})
}

func TestAccFirehoseDeliveryStream_extendedS3DynamicPartitioningUpdate(t *testing.T) {
ctx := acctest.Context(t)
var stream firehose.DeliveryStreamDescription
rName := sdkacctest.RandomWithPrefix(acctest.ResourcePrefix)
resourceName := "aws_kinesis_firehose_delivery_stream.test"

resource.ParallelTest(t, resource.TestCase{
PreCheck: func() { acctest.PreCheck(t) },
ErrorCheck: acctest.ErrorCheck(t, firehose.EndpointsID),
ProtoV5ProviderFactories: acctest.ProtoV5ProviderFactories,
CheckDestroy: testAccCheckDeliveryStreamDestroy_ExtendedS3(ctx),
Steps: []resource.TestStep{
{
Config: testAccDeliveryStreamConfig_extendedS3DynamicPartitioningBasic(rName),
Check: resource.ComposeTestCheckFunc(
testAccCheckDeliveryStreamExists(ctx, resourceName, &stream),
testAccCheckDeliveryStreamAttributes(&stream, nil, nil, nil, nil, nil, nil),
),
},
{
Config: testAccDeliveryStreamConfig_extendedS3DynamicPartitioning(rName),
Check: resource.ComposeTestCheckFunc(
testAccCheckDeliveryStreamExists(ctx, resourceName, &stream),
testAccCheckDeliveryStreamAttributes(&stream, nil, nil, nil, nil, nil, nil),
resource.TestCheckResourceAttr(resourceName, "extended_s3_configuration.0.processing_configuration.#", "1"),
resource.TestCheckResourceAttr(resourceName, "extended_s3_configuration.0.dynamic_partitioning_configuration.#", "1"),
),
},
},
})
}

func TestAccFirehoseDeliveryStream_extendedS3Updates(t *testing.T) {
ctx := acctest.Context(t)
var stream firehose.DeliveryStreamDescription
Expand Down Expand Up @@ -3022,6 +3054,26 @@ resource "aws_kinesis_firehose_delivery_stream" "test" {
`, rName))
}

func testAccDeliveryStreamConfig_extendedS3DynamicPartitioningBasic(rName string) string {
return acctest.ConfigCompose(
testAccLambdaBasicConfig(rName),
testAccDeliveryStreamBaseConfig(rName),
fmt.Sprintf(`
resource "aws_kinesis_firehose_delivery_stream" "test" {
depends_on = [aws_iam_role_policy.firehose]
name = %[1]q
destination = "extended_s3"

extended_s3_configuration {
role_arn = aws_iam_role.firehose.arn
bucket_arn = aws_s3_bucket.bucket.arn
error_output_prefix = "prefix1"
buffer_size = 64
}
}
`, rName))
}

func testAccDeliveryStreamConfig_extendedS3UpdatesInitial(rName string) string {
return acctest.ConfigCompose(
testAccLambdaBasicConfig(rName),
Expand Down
17 changes: 13 additions & 4 deletions website/docs/r/kinesis_firehose_delivery_stream.html.markdown
Original file line number Diff line number Diff line change
Expand Up @@ -110,12 +110,17 @@ resource "aws_kinesis_firehose_delivery_stream" "extended_s3_stream" {
role_arn = aws_iam_role.firehose_role.arn
bucket_arn = aws_s3_bucket.bucket.arn

buffer_size = 64

# https://docs.aws.amazon.com/firehose/latest/dev/dynamic-partitioning.html
dynamic_partitioning_configuration {
enabled = "true"
}

# Example prefix using partitionKeyFromQuery, applicable to JQ processor
prefix = "data/customer_id=!{partitionKeyFromQuery:customer_id}/year=!{timestamp:yyyy}/month=!{timestamp:MM}/day=!{timestamp:dd}/hour=!{timestamp:HH}/"
error_output_prefix = "errors/year=!{timestamp:yyyy}/month=!{timestamp:MM}/day=!{timestamp:dd}/hour=!{timestamp:HH}/!{firehose:error-output-type}/"

# https://docs.aws.amazon.com/firehose/latest/dev/dynamic-partitioning.html
buffer_size = 64
processing_configuration {
enabled = "true"

Expand Down Expand Up @@ -483,7 +488,7 @@ The `extended_s3_configuration` object supports the same fields from `s3_configu
* `processing_configuration` - (Optional) The data processing configuration. More details are given below.
* `s3_backup_mode` - (Optional) The Amazon S3 backup mode. Valid values are `Disabled` and `Enabled`. Default value is `Disabled`.
* `s3_backup_configuration` - (Optional) The configuration for backup in Amazon S3. Required if `s3_backup_mode` is `Enabled`. Supports the same fields as `s3_configuration` object.
* `dynamic_partitioning_configuration` - (Optional) The configuration for dynamic partitioning. See [Dynamic Partitioning Configuration](#dynamic_partitioning_configuration) below for more details.
* `dynamic_partitioning_configuration` - (Optional) The configuration for dynamic partitioning. See [Dynamic Partitioning Configuration](#dynamic_partitioning_configuration) below for more details. Required when using dynamic partitioning.

The `redshift_configuration` object supports the following:

Expand Down Expand Up @@ -687,9 +692,13 @@ resource "aws_kinesis_firehose_delivery_stream" "example" {

#### dynamic_partitioning_configuration

* `enabled` - (Optional) Enables or disables [dynamic partitioning](https://docs.aws.amazon.com/firehose/latest/dev/dynamic-partitioning.html). Defaults to `false`.
Required when using [dynamic partitioning](https://docs.aws.amazon.com/firehose/latest/dev/dynamic-partitioning.html).

* `enabled` - (Optional) Enables or disables dynamic partitioning. Defaults to `false`.
* `retry_duration` - (Optional) Total amount of seconds Firehose spends on retries. Valid values between 0 and 7200. Default is 300.

~> **NOTE:** You can enable dynamic partitioning only when you create a new delivery stream. Once you enable dynamic partitioning on a delivery stream, it cannot be disabled on this delivery stream. Therefore, Terraform will recreate the resource whenever dynamic partitioning is enabled or disabled.

## Attributes Reference

In addition to all arguments above, the following attributes are exported:
Expand Down