Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Export private IPs/ENIs of internal NLBs as attributes on the resource #3007

Closed
IainSteers opened this issue Jan 16, 2018 · 9 comments
Closed
Labels
enhancement Requests to existing resources that expand the functionality or scope. service/elbv2 Issues and PRs that pertain to the elbv2 service.

Comments

@IainSteers
Copy link
Contributor

IainSteers commented Jan 16, 2018

We've found ourselves using internal NLBs for several services which need IP addressability, and are finding it annoyingly tedious to automate provisioning and registration via the aws terraform module.

For internal NLBs were there isn't any EIPs the entry-point to the NLB is either the DNS A record and/or going via the AWS GUI/API to manually grab the private IPs of the NLB's network interfaces.

The aws_network_interface data-source doesn't seem to help here because you need to know the ENI id first.

We lock down our security groups and don't want to just randomly open entire vpc cidrs for the sake of health checks so need the private ips for that as well as services where DNS isn't available/usable.

Terraform Version

v0.10.8
Latest released aws provider

Affected Resource(s)

  • aws_lb

Terraform Configuration Files

resource "aws_lb_target_group" "nlb_tcp" {
  name     = "test-nlb-tcp"
  port     = 6356
  protocol = "TCP"
  vpc_id   = "vpc-abc134"

  health_check {
    protocol = "TCP"
    port     = "5588"

    healthy_threshold   = 3
    unhealthy_threshold = 3
    interval            = 10
  }

  lifecycle {
    create_before_destroy = true
  }
}

resource "aws_lb_listener" "test" {
  load_balancer_arn = "${aws_lb.test_nlb.arn}"

  port     = "516"
  protocol = "TCP"

  default_action {
    target_group_arn = "${aws_lb_target_group.nlb_tcp.arn}"
    type             = "forward"
  }

  lifecycle {
    create_before_destroy = true
  }
}

resource "aws_lb" "test_nlb" {
  name     = "${var.mod_name}-nlb"
  internal = true

  load_balancer_type = "network"

  subnets = [
    "subnet-abc1234",
    "subnet-xyz1234",
  ]
}

Is there any possibility of either getting the ENI ids or private IPs exported as attributes of the lb resource?

@bflad
Copy link
Contributor

bflad commented Jan 16, 2018

Probably worth noting there is already a PR (pending discussion) of exporting the private IPs of aws_lb here: #2901

@bflad bflad added enhancement Requests to existing resources that expand the functionality or scope. service/elbv2 Issues and PRs that pertain to the elbv2 service. labels Jan 16, 2018
@saxonww
Copy link

saxonww commented Apr 18, 2018

We ran into this earlier today, and there is an existing workaround using aws_network_interface data source:

variable "vpc_id" {
  description = "The AWS VPC ID"
}

variable "subnets" {
  type = "list"
  description = "A list of subnet IDs in the above VPC"
}

resource "aws_lb" "example_lb" {
  load_balancer_type = "network"
  name               = "example-lb"
  internal           = true
  subnets            = ["${var.subnets}"]
}

data "aws_network_interface" "example_lb" {
  count = "${length(var.subnets)}"

  filter = {
    name   = "description"
    values = ["ELB ${aws_lb.example_lb.arn_suffix}"]
  }

  filter = {
    name   = "subnet-id"
    values = ["${element(var.subnets, count.index)}"]
  }
}

resource "aws_security_group" "lb_sg" {
  vpc_id = "${var.vpc_id}"

  ingress {
    from_port   = 0
    to_port     = 0
    protocol    = "tcp"
    cidr_blocks = ["${formatlist("%s/32",flatten(data.aws_network_interface.example_lb.*.private_ips))}"]
    description = "Permit connection from load balancer"
  }
}

@IainSteers
Copy link
Contributor Author

@saxonww yes that's the obvious work-around.

However, it's hardly efficient nor first-class and gets crufty quickly.

@saxonww
Copy link

saxonww commented Apr 18, 2018

@IainSteers I agree, I mostly put it here so that other people encountering this issue could find it. Your original submission read (to me) like you were saying the data source would not work; sorry if I misunderstood.

@omerfsen
Copy link

omerfsen commented Sep 20, 2019

variable "vpc_id" {
  description = "The AWS VPC ID"
}

variable "subnets" {
  type = "list"
  description = "A list of subnet IDs in the above VPC"
}

resource "aws_lb" "example_lb" {
  load_balancer_type = "network"
  name               = "example-lb"
  internal           = true
  subnets            = ["${var.subnets}"]
}

data "aws_network_interface" "example_lb" {
  count = "${length(var.subnets)}"

  filter = {
    name   = "description"
    values = ["ELB ${aws_lb.example_lb.arn_suffix}"]
  }

  filter = {
    name   = "subnet-id"
    values = ["${element(var.subnets, count.index)}"]
  }
}

resource "aws_security_group" "lb_sg" {
  vpc_id = "${var.vpc_id}"

  ingress {
    from_port   = 0
    to_port     = 0
    protocol    = "tcp"
    cidr_blocks = ["${formatlist("%s/32",flatten(data.aws_network_interface.example_lb.*.private_ips))}"]
    description = "Permit connection from load balancer"
  }
}

This solution does not work as if you use NLB SGs you have created (Yes NLBs can't have SG as of yet) as source it only adds Assets that is using this SG is added namely

When you specify a security group as a source, it allows traffic from the instances associated with the security group. It does not add the rules from the source security group.

http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-network-security.html#security-group-rules

So instead of creating an SG for NLB you just add following to your target SG that you do health check against

  cidr_blocks = ["${formatlist("%s/32",flatten(data.aws_network_interface.privatelink.*.private_ips))}"]

So sample one would be:

# Add NLB Internal IPs to Instance's SGs (We allow ALL traffic from NLB not just the port/s that are healthchecked) 
resource "aws_security_group_rule" "target_service_sg" {
  type                      = "ingress"
  from_port                 = 0
  to_port                   = 0
  protocol                  = "-1"
  cidr_blocks = ["${formatlist("%s/32",flatten(data.aws_network_interface.privatelink.*.private_ips))}"]
  security_group_id         = "${var.target_service_sg_id}"
}

@nbouscal
Copy link

nbouscal commented Nov 5, 2019

For anyone still dealing with this issue while the PR linked above is in purgatory, to maybe save you a bit of time rewriting the aws_network_interface solution for 0.12, here's an updated example corresponding to what I'm now using in 0.12.13:

data "aws_network_interface" "example_lb" {
  for_each = var.subnets

  filter {
    name   = "description"
    values = ["ELB ${aws_lb.example_lb.arn_suffix}"]
  }

  filter {
    name   = "subnet-id"
    values = [each.value]
  }
}

resource "aws_security_group" "lb_sg" {
  vpc_id = var.vpc_id

  ingress {
    from_port   = 0
    to_port     = 0
    protocol    = "tcp"
    cidr_blocks = formatlist("%s/32", [for eni in data.aws_network_interface.example_lb : eni.private_ip])
    description = "Permit connection from load balancer"
  }
}

(I'm not an HCL expert, this is just what I got to work, feel free to comment if it could be done better.)

@80kk
Copy link

80kk commented Jan 24, 2020

Any ideas how to use that for target group attachment which expects IPs rather than eni IDs?

@maryelizbeth
Copy link
Contributor

Hi Y’all,

Per the conversation on #2901, we’ll work with the author of #11404 to address any changes that may be needed to merge the PR and will merge it once the work is complete.

As a result, we will close this issue in order to focus conversation on #11404.

If you feel that #11404 does not adequately address your workflow, please open a new GitHub Issue describing the gap. If you have concerns regarding the design of the upstream API, please reach out to AWS.

We apologize for letting this linger without response and will work to merge #11404 in an upcoming release.

Once that is merged, it will close #11403 & #11044.

@ghost
Copy link

ghost commented May 23, 2020

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!

@ghost ghost locked and limited conversation to collaborators May 23, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
enhancement Requests to existing resources that expand the functionality or scope. service/elbv2 Issues and PRs that pertain to the elbv2 service.
Projects
None yet
Development

No branches or pull requests

7 participants