Options to deploy Nebari to existing AWS VPC/subnets?

[mwengren]

We have some particular networking/security requirements that compel us to use a pre-existing VPC and associated public/private subnets in our AWS account. Similarly, we can't create new InternetGateway (IGW) in our public subnet, we must use an existing networking setup for outgoing traffic out of AWS to our network or for any public traffic.

Is there a way to configure such a deployment by editing the nebari-config.yaml file appropriately? Essentially to pass IDs for existing AWS resources (VPC, subnet, etc) for Nebari to deploy components to? I can't find documentation for this type of deployment scenario in the Nebari docs, so I assume it would require some manual editing of Nebari internals beyond just the options provided in nebari-config?

TIA!

[viniciusdc]

Hi @mwengren, Nebari supports existing AWS subnets, by passing the associated security group ID and existing subnet IDs in the nebari-config.yaml. However, I am not sure if Nebari even creates an IGW at all as part of its deployment process, so I guess it would default for any configuration already present in your infrastructure.

Here's an example of how that can be set in the config:

amazon_web_services:
  terraform_overrides:
    existing_subnet_ids: ["subnet-05b2b1f41f0b1d8a6", "subnet-017efc3309fbca2da"]
    existing_security_group_id: "sg-0e2c865bdd8824b1a"
  region: ...
  <other_attributes>: ...

I am c.c. @aktech @Adam-D-Lewis as both have more experience with AWS deployments, and Adam worked on adding support for this

[mwengren]

Thanks @viniciusdc Do you know if it's possible to deploy the Load Balancer in one subnet (our public subnet) and the k8s cluster and other resources like EFS in another (private) subnet?

We have a small public subnet (/27) and a larger private subnet (/23). Networking is provided between the two subnets via pre-configured Route Table.

[mwengren]

@viniciusdc When I add the terraform_overrides YAML I get a validation error running nebari deploy:

ValidationError: 1 validation error for ConfigSchema
amazon_web_services.terraform_overrides
  Extra inputs are not permitted [type=extra_forbidden, input_value={'existing_subnet_ids': [... 'sg-xxxxxxxxxxxxxx'}, input_type=CommentedMap]
    For further information visit https://errors.pydantic.dev/2.4/v/extra_forbidden

Do I need a custom validation configuration as well?

[aktech]

The syntax is as following:

amazon_web_services:
  existing_subnet_ids:
    - subnet-xxxxxxxxxxxx
    - subnet-yyyyyyyyyyyy
  existing_security_group_id: sg-kkkkkkkkkkkk

[mwengren]

Thanks @aktech, I modified the formatting of my config file, but I am still getting the same validation error.

Config section:

amazon_web_services:
  terraform_overrides:
       existing_subnet_ids:
           - subnet-xxxxxxxxxxxxxxxx
           - subnet-yyyyyyyyyyyyyyy
        existing_security_group_id: sg-zzzzzzzzzzzzzzz

Output:

ValidationError: 1 validation error for ConfigSchema
amazon_web_services.terraform_overrides
  Extra inputs are not permitted [type=extra_forbidden, input_value={'existing_subnet_ids': [...net-xxxxxxxxxxxxxx']}, input_type=CommentedMap]
    For further information visit https://errors.pydantic.dev/2.4/v/extra_forbidden

Is it required to include two subnets? What if I would like to deploy Nebari to only a single subnet?

More accurately, I have a public/private subnet configuration (see details above), but I don't know if this is an architecture that's supported by Nebari 'out of the box' or not? We need the end-user access point (Load Balancer?) to be in our public subnet, and the k8s cluster in the private subnet, as we only have 16 IP addresses available in our public subnet.

I have several other questions if you can assist:

What does passing an existing security group do? I have a security group that manages permissions to our public subnet, so I planned to try passing that in hoping Nebari would adopt the same permissions rather than create a new permission set that is more open than I want.

Are these terraform overrides documented anywhere or can you point me to the right section of the code to poke around to see what other override options are available, if any?

[mwengren]

I figured out the answer to the nebari-config.yaml question above, however I'm hitting an issue in the deployment process. This is the YAML config I used to override and use my existing subnets (same as above but without the terraform_overrides node:

existing_subnet_ids:
           - subnet-xxxxxxxxxxxxxxxx
           - subnet-yyyyyyyyyyyyyyy
existing_security_group_id: sg-zzzzzzzzzzzzzzz

Adding these resulted in the Nebari deploy starting.

Is the purpose of the Security Group documented anywhere? I found this part of the TF code that both the existing_subnet_ids and existing_security_group_id parameters are needed in order for networking override to occur. But how do I know what my existing_security_group_id needs to include for Nebari to deploy successfully?

Asking because it's not clear to me why my general k8s node group failed to initialize. The error I got was:

[terraform]: │ Error: waiting for EKS Node Group (nebari-dev-dev:general) create: unexpected state 'CREATE_FAILED', wanted target 'ACTIVE'. last error: i-08575544df69eef5c: NodeCreationFailure: Instances failed to join the kubernetes cluster

Looking in the AWS console it appears the desired size of the general node group is 1, so presumably it failed when the instance failed to join, which must be security group or other permission-related issue?

Thanks for any advice!

[mwengren]

I've tried redeploying a few times but each time the node instance (which is created successfully as m52xlarge type per nebari-config.yml) isn't able to join the general k8s cluster. I also get the following message at the top of the nebari-dev-dev cluster page in the console:

Your current IAM principal doesn’t have access to Kubernetes objects on this cluster.
This may be due to the current user or role not having Kubernetes RBAC permissions to describe cluster resources or not having an entry in the cluster’s auth config map

It's odd because I have AdministratorAccess role for my account which appears to have full access as far as I can tell for IAM permissions, so I'm not sure where to go to troubleshoot further. Are there other common reasons why the instance might not be able to connect to the node group?

Thanks in advance!

[viniciusdc]

Hi @mwengren,

Sorry for the delay; it's been a busy week. I recommend doubling the AWS CloudTrail regarding the permission issue to identify which request is failing. This should help pinpoint why it's not joining the cluster and determine if permissions are missing. As you mentioned, there are some customizations, such as the exposure of the Internet Gateway, which Nebari does not support out-of-the-box. We're currently working on an updated version of the documentation that will address these steps.

Also, could you give me an overview of your current infrastructure, mainly how the networking is set up? This would help us trying to reproduce the infra on our end. I know you've provided some details in the previous discussion, but if there are any missing points there, just let me know; once we can reproduce a somewhat similar infrastructure, we can expand the available configurations.