Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug]: nil pointer dereference when removing cloudwatch deployment monitor from aws_ecs_service #29349

Closed
justin-watkinson-electric opened this issue Feb 10, 2023 · 9 comments · Fixed by #31683
Labels
bug Addresses a defect in current functionality. crash Results from or addresses a Terraform crash or kernel panic. service/ecs Issues and PRs that pertain to the ecs service.

Comments

@justin-watkinson-electric
Copy link

justin-watkinson-electric commented Feb 10, 2023

Terraform Core Version

1.1.7

AWS Provider Version

4.54.0

Affected Resource(s)

aws_ecs_service

Expected Behavior

CloudWatch Deployment Alarm configuration should be removed upon apply

Actual Behavior

Received error and apply fails

Relevant Error/Panic Output Snippet

│ The plugin encountered an error, and failed to respond to the
│ plugin.(*GRPCProvider).ApplyResourceChange call. The plugin
│ logs may contain more details.

Stack trace from the terraform-provider-aws_v4.54.0_x5 plugin:

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x2 addr=0x0 pc=0x107ddc320]

goroutine 406 [running]:
github.com/hashicorp/terraform-provider-aws/internal/service/ecs.resourceServiceUpdate({0x10d10b740?, 0x1400276d650}, 0x14004a83b00, {0x10cfca360?, 0x14001845000})
        github.com/hashicorp/terraform-provider-aws/internal/service/ecs/service.go:871 +0xb80
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*Resource).update(0x10d10b740?, {0x10d10b740?, 0x1400276d650?}, 0xd?, {0x10cfca360?, 0x14001845000?})
        github.com/hashicorp/terraform-plugin-sdk/v2@v2.24.1/helper/schema/resource.go:736 +0x64
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*Resource).Apply(0x14001166d20, {0x10d10b740, 0x1400276d650}, 0x1400328c340, 0x14004a83980, {0x10cfca360, 0x14001845000})
        github.com/hashicorp/terraform-plugin-sdk/v2@v2.24.1/helper/schema/resource.go:847 +0x67c
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*GRPCProviderServer).ApplyResourceChange(0x1400000ca08, {0x10d10b740?, 0x1400276cb40?}, 0x14005b676d0)
        github.com/hashicorp/terraform-plugin-sdk/v2@v2.24.1/helper/schema/grpc_provider.go:1021 +0xb70
github.com/hashicorp/terraform-plugin-mux/tf5muxserver.muxServer.ApplyResourceChange({0x140031a4210, 0x140031a4270, {0x1400493b4e0, 0x2, 0x2}, {0x0, 0x0, 0x0}, {0x0, 0x0, ...}, ...}, ...)
        github.com/hashicorp/terraform-plugin-mux@v0.8.0/tf5muxserver/mux_server_ApplyResourceChange.go:27 +0xdc
github.com/hashicorp/terraform-plugin-go/tfprotov5/tf5server.(*server).ApplyResourceChange(0x14002507720, {0x10d10b740?, 0x1400386fd10?}, 0x14002478230)
        github.com/hashicorp/terraform-plugin-go@v0.14.3/tfprotov5/tf5server/server.go:818 +0x3b8
github.com/hashicorp/terraform-plugin-go/tfprotov5/internal/tfplugin5._Provider_ApplyResourceChange_Handler({0x10cce7900?, 0x14002507720}, {0x10d10b740, 0x1400386fd10}, 0x140024781c0, 0x0)
        github.com/hashicorp/terraform-plugin-go@v0.14.3/tfprotov5/internal/tfplugin5/tfplugin5_grpc.pb.go:385 +0x170
google.golang.org/grpc.(*Server).processUnaryRPC(0x1400420dc20, {0x10d11a600, 0x1400026c680}, 0x140042e6900, 0x14004a36270, 0x113660ac0, 0x0)
        google.golang.org/grpc@v1.53.0/server.go:1336 +0xb7c
google.golang.org/grpc.(*Server).handleStream(0x1400420dc20, {0x10d11a600, 0x1400026c680}, 0x140042e6900, 0x0)
        google.golang.org/grpc@v1.53.0/server.go:1704 +0x82c
google.golang.org/grpc.(*Server).serveStreams.func1.2()
        google.golang.org/grpc@v1.53.0/server.go:965 +0x84
created by google.golang.org/grpc.(*Server).serveStreams.func1
        google.golang.org/grpc@v1.53.0/server.go:963 +0x290

Error: The terraform-provider-aws_v4.54.0_x5 plugin crashed!

This is always indicative of a bug within the plugin. It would be immensely
helpful if you could report the crash with the plugin's maintainers so that it
can be fixed. The output above should help diagnose the issue.

Terraform Configuration Files

# partially redacted

resource "aws_ecs_service" "this" {
  name            = "foo-test"
  cluster         = var.cluster_id
  task_definition = aws_ecs_task_definition.this.arn

  load_balancer {
    target_group_arn = module.alb.target_group_arns[0]
    container_name   = "my-container"
    container_port   = 8000
  }

  network_configuration {
    subnets          = ["subnet-00000000"]
    security_groups  = ["sg-00000000"]
    assign_public_ip = false
  }

  capacity_provider_strategy {
    base              = 20
    capacity_provider = "FARGATE"
    weight            = 50
  }

   alarms {
     enable   = true
     rollback = true
     alarm_names = [
       "test-alarm"
     ]
   }

  desired_count = 2

  deployment_maximum_percent         = 200
  deployment_minimum_healthy_percent = 50

  wait_for_steady_state = true
}

Steps to Reproduce

Terraform plan/apply - nothing special

Debug Output

No response

Panic Output

No response

Important Factoids

Nothing special. Interested in providing a fix, would be a first-timer.

References

No response

Would you like to implement a fix?

None

@justin-watkinson-electric justin-watkinson-electric added bug Addresses a defect in current functionality. needs-triage Waiting for first response or review from a maintainer. labels Feb 10, 2023
@github-actions
Copy link

Community Note

Voting for Prioritization

  • Please vote on this issue by adding a 👍 reaction to the original post to help the community and maintainers prioritize this request.
  • Please see our prioritization guide for information on how we prioritize.
  • Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.

Volunteering to Work on This Issue

  • If you are interested in working on this issue, please leave a comment.
  • If this would be your first contribution, please review the contribution guide.

@github-actions github-actions bot added crash Results from or addresses a Terraform crash or kernel panic. service/ecs Issues and PRs that pertain to the ecs service. labels Feb 10, 2023
@justin-watkinson-electric
Copy link
Author

justin-watkinson-electric commented Feb 10, 2023

Not finding a good workaround. I did a state rm to remove the entire resource and re-import it, and the terraform plan still shows it's trying to remove the alarm configuration. A CLI aws ecs describe-services doesn't have any mention of the Cloudwatch deployment alarm configuration. Removing it from both UI and state file also doesn't seem to be working.

Using Terraform to taint then re-create the ECS service seems to have resolved this one.

@nosovj
Copy link

nosovj commented Mar 22, 2023

Running into the same issue

│ The plugin encountered an error, and failed to respond to the plugin.(*GRPCProvider).ApplyResourceChange call. The plugin logs may contain more details.
╵
Releasing state lock. This may take a few moments...

Stack trace from the terraform-provider-aws_v4.59.0_x5 plugin:

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x9df1a63]

goroutine 379 [running]:
github.com/hashicorp/terraform-provider-aws/internal/service/ecs.resourceServiceUpdate({0xf2ed180?, 0xc004f0b080}, 0xc0048e4180, {0xde16860?, 0xc0003c3400})
        github.com/hashicorp/terraform-provider-aws/internal/service/ecs/service.go:872 +0xd63
github.com/hashicorp/terraform-provider-aws/internal/provider.wrappedUpdateContextFunc.func1({0xf2ed180?, 0xc004f0b080?}, 0x0?, {0xde16860?, 0xc0003c3400?})
        github.com/hashicorp/terraform-provider-aws/internal/provider/provider.go:818 +0x3e
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*Resource).update(0xf2ed180?, {0xf2ed180?, 0xc004f0b080?}, 0xd?, {0xde16860?, 0xc0003c3400?})
        github.com/hashicorp/terraform-plugin-sdk/v2@v2.25.0/helper/schema/resource.go:736 +0x87
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*Resource).Apply(0xc000f1bdc0, {0xf2ed180, 0xc004f0b080}, 0xc004662a90, 0xc0048e4000, {0xde16860, 0xc0003c3400})
        github.com/hashicorp/terraform-plugin-sdk/v2@v2.25.0/helper/schema/resource.go:847 +0x83a
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*GRPCProviderServer).ApplyResourceChange(0xc000010690, {0xf2ed180?, 0xc004f0af60?}, 0xc002269d60)
        github.com/hashicorp/terraform-plugin-sdk/v2@v2.25.0/helper/schema/grpc_provider.go:1021 +0xe8d
github.com/hashicorp/terraform-plugin-mux/tf5muxserver.muxServer.ApplyResourceChange({0xc002289110, 0xc002289380, {0xc004c9cfe0, 0x2, 0x2}, {0x0, 0x0, 0x0}, {0x0, 0x0, ...}, ...}, ...)
        github.com/hashicorp/terraform-plugin-mux@v0.9.0/tf5muxserver/mux_server_ApplyResourceChange.go:27 +0x102
github.com/hashicorp/terraform-plugin-go/tfprotov5/tf5server.(*server).ApplyResourceChange(0xc002256aa0, {0xf2ed180?, 0xc004f0a210?}, 0xc00459d960)
        github.com/hashicorp/terraform-plugin-go@v0.14.3/tfprotov5/tf5server/server.go:818 +0x574
github.com/hashicorp/terraform-plugin-go/tfprotov5/internal/tfplugin5._Provider_ApplyResourceChange_Handler({0xdb25e00?, 0xc002256aa0}, {0xf2ed180, 0xc004f0a210}, 0xc00459d8f0, 0x0)
        github.com/hashicorp/terraform-plugin-go@v0.14.3/tfprotov5/internal/tfplugin5/tfplugin5_grpc.pb.go:385 +0x170
google.golang.org/grpc.(*Server).processUnaryRPC(0xc0044a5860, {0xf2fc2a0, 0xc000de6680}, 0xc0045fd200, 0xc004d71590, 0x15a976e0, 0x0)
        google.golang.org/grpc@v1.53.0/server.go:1336 +0xd23
google.golang.org/grpc.(*Server).handleStream(0xc0044a5860, {0xf2fc2a0, 0xc000de6680}, 0xc0045fd200, 0x0)
        google.golang.org/grpc@v1.53.0/server.go:1704 +0xa2f
google.golang.org/grpc.(*Server).serveStreams.func1.2()
        google.golang.org/grpc@v1.53.0/server.go:965 +0x98
created by google.golang.org/grpc.(*Server).serveStreams.func1
        google.golang.org/grpc@v1.53.0/server.go:963 +0x28a

Error: The terraform-provider-aws_v4.59.0_x5 plugin crashed!

This is always indicative of a bug within the plugin. It would be immensely
helpful if you could report the crash with the plugin's maintainers so that it
can be fixed. The output above should help diagnose the issue.

Here is the metric alarm that I'm attempting to create:

resource "aws_cloudwatch_metric_alarm" "unhealthy_hosts" {
  for_each            = { for entry in var.target_group_list : entry.suffix_arn => entry.suffix_arn }
  alarm_name          = "UnhealthyHosts-${var.load_balancer_suffix_arn}-${each.key}"
  comparison_operator = "GreaterThanOrEqualToThreshold"
  evaluation_periods  = 2
  metric_name         = "UnHealthyHostCount"
  namespace           = "AWS/ApplicationELB"
  period              = 60
  statistic           = "Sum"
  threshold           = 1
  alarm_description   = "Number of unhealthy hosts in Target Group"
  actions_enabled     = "true"
  dimensions = {
    TargetGroup  = each.key
    LoadBalancer = var.load_balancer_suffix_arn
  }
}

And attempting to add it to the following aws_ecs_service resource:

resource "aws_ecs_service" "service" {

  ...

  # This throws an exception
  alarms {
    enable      = false
    rollback    = true
    alarm_names = [for alarm in aws_cloudwatch_metric_alarm.unhealthy_hosts : alarm.alarm_name]
  }
}

@Valentinolucas
Copy link

Same here...

Stack trace from the terraform-provider-aws_v4.61.0_x5 plugin:

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x92ed003]

goroutine 351 [running]:
github.com/hashicorp/terraform-provider-aws/internal/service/ecs.resourceServiceUpdate({0xe8838a0?, 0xc0055dcff0}, 0xc005294a80, {0xd384cc0?, 0xc0003b8800})
        github.com/hashicorp/terraform-provider-aws/internal/service/ecs/service.go:872 +0xd63
github.com/hashicorp/terraform-provider-aws/internal/provider.interceptedHandler[...].func1(0x0?, {0xd384cc0?, 0xc0003b8800?})
        github.com/hashicorp/terraform-provider-aws/internal/provider/intercept.go:94 +0x175
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*Resource).update(0xe8838a0?, {0xe8838a0?, 0xc0052b3260?}, 0xd?, {0xd384cc0?, 0xc0003b8800?})
        github.com/hashicorp/terraform-plugin-sdk/v2@v2.26.1/helper/schema/resource.go:736 +0x87
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*Resource).Apply(0xc000e500e0, {0xe8838a0, 0xc0052b3260}, 0xc005293930, 0xc005294900, {0xd384cc0, 0xc0003b8800})
        github.com/hashicorp/terraform-plugin-sdk/v2@v2.26.1/helper/schema/resource.go:847 +0x83a
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*GRPCProviderServer).ApplyResourceChange(0xc00165db30, {0xe8838a0?, 0xc0052b3140?}, 0xc0036f2370)
        github.com/hashicorp/terraform-plugin-sdk/v2@v2.26.1/helper/schema/grpc_provider.go:1021 +0xe8d
github.com/hashicorp/terraform-plugin-mux/tf5muxserver.muxServer.ApplyResourceChange({0xc002ca5c50, 0xc002cc6030, {0xc004cfc2a0, 0x2, 0x2}, {0x0, 0x0, 0x0}, {0x0, 0x0, ...}, ...}, ...)
        github.com/hashicorp/terraform-plugin-mux@v0.9.0/tf5muxserver/mux_server_ApplyResourceChange.go:27 +0x102
github.com/hashicorp/terraform-plugin-go/tfprotov5/tf5server.(*server).ApplyResourceChange(0xc0022fe140, {0xe8838a0?, 0xc0052b2600?}, 0xc004c9fb90)
        github.com/hashicorp/terraform-plugin-go@v0.14.3/tfprotov5/tf5server/server.go:818 +0x574
github.com/hashicorp/terraform-plugin-go/tfprotov5/internal/tfplugin5._Provider_ApplyResourceChange_Handler({0xd0899e0?, 0xc0022fe140}, {0xe8838a0, 0xc0052b2600}, 0xc004c9fb20, 0x0)
        github.com/hashicorp/terraform-plugin-go@v0.14.3/tfprotov5/internal/tfplugin5/tfplugin5_grpc.pb.go:385 +0x170
google.golang.org/grpc.(*Server).processUnaryRPC(0xc004493c20, {0xe892fa0, 0xc0031181a0}, 0xc0052bc480, 0xc004d21ad0, 0x150d5e00, 0x0)
        google.golang.org/grpc@v1.53.0/server.go:1336 +0xd23
google.golang.org/grpc.(*Server).handleStream(0xc004493c20, {0xe892fa0, 0xc0031181a0}, 0xc0052bc480, 0x0)
        google.golang.org/grpc@v1.53.0/server.go:1704 +0xa2f
google.golang.org/grpc.(*Server).serveStreams.func1.2()
        google.golang.org/grpc@v1.53.0/server.go:965 +0x98
created by google.golang.org/grpc.(*Server).serveStreams.func1
        google.golang.org/grpc@v1.53.0/server.go:963 +0x28a

Error: The terraform-provider-aws_v4.61.0_x5 plugin crashed!

This is always indicative of a bug within the plugin. It would be immensely
helpful if you could report the crash with the plugin's maintainers so that it
can be fixed. The output above should help diagnose the issue.

╷
│ Error: Plugin did not respond
│ 
│   with module.application.module.ecs.aws_ecs_service.this,
│   on .terraform/modules/application.ecs/main.tf line 61, in resource "aws_ecs_service" "this":
│   61: resource "aws_ecs_service" "this" {
│ 
│ The plugin encountered an error, and failed to respond to the plugin.(*GRPCProvider).ApplyResourceChange call. The plugin logs may contain more details. 

When using the following aws_ecs_service resource:

resource "aws_ecs_service" "this" {
 
  ...

  deployment_circuit_breaker {
    enable   = true
    rollback = true
  }
  alarms {
    enable   = true
    rollback = true
    alarm_names = [
      "test-alarm"
    ]
  }

@moritzzimmer
Copy link

We get the same error, even when using v4.49.0 where this feature was introduced

Stack trace from the terraform-provider-aws_v4.49.0_x5 plugin:

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x98ec2e3]

goroutine 48 [running]:
github.com/hashicorp/terraform-provider-aws/internal/service/ecs.resourceServiceUpdate(0xc005aacd80, {0xd66d640?, 0xc0006e8400})
        github.com/hashicorp/terraform-provider-aws/internal/service/ecs/service.go:866 +0xa83
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*Resource).update(0xe9ad9e0?, {0xe9ad9e0?, 0xc0032edce0?}, 0xd?, {0xd66d640?, 0xc0006e8400?})
        github.com/hashicorp/terraform-plugin-sdk/v2@v2.24.1/helper/schema/resource.go:729 +0x178
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*Resource).Apply(0xc001353a40, {0xe9ad9e0, 0xc0032edce0}, 0xc000c9ed00, 0xc005aaca80, {0xd66d640, 0xc0006e8400})
        github.com/hashicorp/terraform-plugin-sdk/v2@v2.24.1/helper/schema/resource.go:847 +0x83a
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*GRPCProviderServer).ApplyResourceChange(0xc000204330, {0xe9ad9e0?, 0xc0032edbc0?}, 0xc005ce78b0)
        github.com/hashicorp/terraform-plugin-sdk/v2@v2.24.1/helper/schema/grpc_provider.go:1021 +0xe8d
github.com/hashicorp/terraform-plugin-mux/tf5muxserver.muxServer.ApplyResourceChange({0xc0032afe00, 0xc0032afe60, {0xc00492ef40, 0x2, 0x2}, {0x0, 0x0, 0x0}, {0x0, 0x0, ...}, ...}, ...)
        github.com/hashicorp/terraform-plugin-mux@v0.8.0/tf5muxserver/mux_server_ApplyResourceChange.go:27 +0x102
github.com/hashicorp/terraform-plugin-go/tfprotov5/tf5server.(*server).ApplyResourceChange(0xc00208c460, {0xe9ad9e0?, 0xc0032ec900?}, 0xc000a093b0)
        github.com/hashicorp/terraform-plugin-go@v0.14.2/tfprotov5/tf5server/server.go:818 +0x574
github.com/hashicorp/terraform-plugin-go/tfprotov5/internal/tfplugin5._Provider_ApplyResourceChange_Handler({0xd390b40?, 0xc00208c460}, {0xe9ad9e0, 0xc0032ec900}, 0xc000a09340, 0x0)
        github.com/hashicorp/terraform-plugin-go@v0.14.2/tfprotov5/internal/tfplugin5/tfplugin5_grpc.pb.go:385 +0x170
google.golang.org/grpc.(*Server).processUnaryRPC(0xc004187c20, {0xe9bc3c0, 0xc0001e8680}, 0xc00431d320, 0xc00497ddd0, 0x14ea12c0, 0x0)
        google.golang.org/grpc@v1.51.0/server.go:1340 +0xd23
google.golang.org/grpc.(*Server).handleStream(0xc004187c20, {0xe9bc3c0, 0xc0001e8680}, 0xc00431d320, 0x0)
        google.golang.org/grpc@v1.51.0/server.go:1713 +0xa2f
google.golang.org/grpc.(*Server).serveStreams.func1.2()
        google.golang.org/grpc@v1.51.0/server.go:965 +0x98
created by google.golang.org/grpc.(*Server).serveStreams.func1
        google.golang.org/grpc@v1.51.0/server.go:963 +0x28a

Error: The terraform-provider-aws_v4.49.0_x5 plugin crashed!

This is always indicative of a bug within the plugin. It would be immensely
helpful if you could report the crash with the plugin's maintainers so that it
can be fixed. The output above should help diagnose the issue.

@justin-watkinson-electric
Copy link
Author

I recently re-tested this with the 4.62.0 AWS provider version and I am no longer able to reproduce the issue. Adding/removing the circuit breaker logic no longer results in the nil pointer.

@Valentinolucas
Copy link

I recently re-tested this with the 4.62.0 AWS provider version and I am no longer able to reproduce the issue. Adding/removing the circuit breaker logic no longer results in the nil pointer.

Hi! My problem is with the alarms argument not with the circuit breaker. When applying for the first time everything is okay and the config gets added to the service but when removing the alarms argument and re applying (to disable the feature) I'm getting the error. I'm using 4.62. I also see the error when trying to add alarms to existing ECS services that were created without any alarm configuration, and the only way to sort it is by recreating the service which is not convenient. Any clues/recommendations? Thanks 🙏

@saefty
Copy link

saefty commented Apr 20, 2023

Pay attention: If anyhow the tf provider pulls a configured deployment circuit breaker into the tf state it will not be possible to apply any changes to the service before it is not deleted from the state..

In our case we needed to manually use the aws cli to set the values to null. The ECS UI only hide the configs.

ecs update-service --deployment-configuration "deploymentCircuitBreaker={enable=false,rollback=false},maximumPercent=200,minimumHealthyPercent=100,alarms={alarmNames=[],enable=false,rollback=false}" --cluster k8 --service [REDACTED] --region [REDACTED]

@justinretzolk justinretzolk removed the needs-triage Waiting for first response or review from a maintainer. label Apr 20, 2023
@github-actions
Copy link

github-actions bot commented Jul 1, 2023

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Jul 1, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
bug Addresses a defect in current functionality. crash Results from or addresses a Terraform crash or kernel panic. service/ecs Issues and PRs that pertain to the ecs service.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

6 participants