Terraform attempts to re-create MSK Connect Custom Plugins

[DeGuitard]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform CLI and Terraform AWS Provider Version

Terraform v1.1.4
on linux_amd64
+ provider registry.terraform.io/hashicorp/aws v3.72.0
+ provider registry.terraform.io/hashicorp/tls v3.1.0

Affected Resource(s)

• aws_mskconnect_custom_plugin

Terraform Configuration Files

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 3"
    }
  }
}

provider "aws" {
  region = "eu-west-3"
}

resource "aws_s3_bucket" "kafka-connect-plugins" {
  bucket = "kafka-connect-plugins"
}

resource "aws_s3_bucket_object" "kafka-connect-s3" {
  bucket = aws_s3_bucket.kafka-connect-plugins.id
  key    = "kafka-connect-s3-3.0.1.zip"
  source = "kafka-connect-s3-3.0.1.zip"
}

resource "aws_mskconnect_custom_plugin" "s3" {
  name         = "s3"
  content_type = "ZIP"
  location {
    s3 {
      bucket_arn = aws_s3_bucket.kafka-connect-plugins.arn
      file_key   = aws_s3_bucket_object.kafka-connect-s3.key
    }
  }
}

Debug Output

Available on request.

Expected Behavior

Running apply, destroy and apply again should not generate any error.

Actual Behavior

The first terraform apply works fine.

The terraform destroy says it destroyed the MSK Custom Plugin, except it's not the case since it's impossible (AWS does not support this operation).

The next terraform apply attempts to re-create the MSK Custom Plugin but fails with this error:

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # module.kafka-connect.aws_mskconnect_custom_plugin.s3 will be created
  + resource "aws_mskconnect_custom_plugin" "s3" {
      + arn             = (known after apply)
      + content_type    = "ZIP"
      + id              = (known after apply)
      + latest_revision = (known after apply)
      + name            = "s3"
      + state           = (known after apply)

      + location {
          + s3 {
              + bucket_arn = "arn:aws:s3:::kafka-connect-plugins"
              + file_key   = "kafka-connect-s3-3.0.1.zip"
            }
        }
    }

Plan: 1 to add, 0 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

module.kafka-connect.aws_mskconnect_custom_plugin.s3: Creating...
╷
│ Error: error creating MSK Connect Custom Plugin (s3): ConflictException: Invalid parameter name: A resource with this name already exists.
│
│   with module.kafka-connect.aws_mskconnect_custom_plugin.s3,
│   on modules/kafka-connect/plugins.tf line 17, in resource "aws_mskconnect_custom_plugin" "s3":
│   17: resource "aws_mskconnect_custom_plugin" "s3" {
│
╵

Steps to Reproduce

1. terraform apply
2. terraform destroy
3. terraform apply

If you manually import the MSK Connect Custom Plugin with terraform import before step 3, this solves the issue.

Important Factoids

As of now AWS doesn't support MSK Connect Custom Plugin deletion.

[DeGuitard]

It seems AWS now supports MSK Connect Custom Plugin deletion. The option shows up both in the CLI and the UI. This should make things easier.

[1968mnelson]

I just tried testing this using latest v4.6 provider and behavior is the SAME. TF THINKS its destroyed but indeed does NOT destroy. We are actually constrained now to <v4 provider for the time being until we work through all of the ddeprecations. Is my only option boto3 OR CloudFormation? Looking at CloudFormation, there only appears to be a resource for the connector...
Hesitant to have such a mixed bag of ways to deploy this service
Just to re-iterate...it DOES delete using the boto3 CLI...im jsut referring to Terraform

[ewbankkit]

Kafka Connect Custom Plugin deletion available with AWS SDK v1.43.9.

[github-actions]

This functionality has been released in v4.8.0 of the Terraform AWS Provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template. Thank you!

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.