Machine is created with amiFamily but is not bootstrapped when using amiSelector

[JRemitz]

Description

Observed Behavior: Karpenter is configured correctly, using the amiFamily: Bottlerocket. However we wanted to test recent changes before the upgrade to Bottlerocket v1.15. To do that we wanted to filter for the lower version via the amiFamily: Custom setting and filter the AMI. In doing so, the node appears to not bootstrap and attach to the cluster despite a new EC2 machine coming online and registering as a machine CRD.

AWSNodeTemplate Working - Latest Bottlerocket:

spec:
  amiFamily: Bottlerocket
...

AWSNodeTemplate Working - Latest Bottlerocket:

spec:
  amiFamily: Custom
  amiSelector:
    aws::owners: "amazon"
    aws::name: "bottlerocket-aws-k8s-1.24-*-v1.14*"
...

Expected Behavior: The older Bottlerocket node (v1.14.3) would be selected and come online.

Reproduction Steps (Please include YAML):

I can change the filter to match the same AMI for v1.15 and it also fails when not using the BottlerocketamiFamily

apiVersion: karpenter.k8s.aws/v1alpha1
kind: AWSNodeTemplate
metadata:
  name: aws
spec:
  #amiFamily: Bottlerocket  # Working 
  amiFamily: Custom
  amiSelector:
    aws::owners: "amazon"
    aws::name: "bottlerocket-aws-k8s-1.24-*-v1.14*"
  instanceProfile: <instance-profile>
  subnetSelector:
    karpenter.sh/discovery: <cluster-name>
  securityGroupSelector:
    karpenter.sh/discovery: <cluster-name>
  blockDeviceMappings:
    - deviceName: /dev/xvda
      ebs:
        volumeType: gp3
        volumeSize: 2Gi
        deleteOnTermination: true
        encrypted: true
    - deviceName: /dev/xvdb
      ebs:
        volumeType: gp3
        volumeSize: 100Gi 
        deleteOnTermination: true
        encrypted: true
  tags:
    "billing:eks:cluster_name": <cluster-name>
    "billing:service": kubernetes
    "billing:team": infrastructure
    env: staging
    managed_by: karpenter
    aws-node-termination-handler/managed: "true"
  userData: |
    [settings.host-containers]
    admin.enabled = false
    control.enabled = true

Versions:

• Chart Version: v0.29.2
• Kubernetes Version (kubectl version): v1.24.16

[JRemitz]

Aha, user-data is empty. Is there a default mechanism to set this? I didn't see anything in the guide indicated it would be cleared out if not using the amiFamily setting.

Karpenter will automatically merge settings to ensure successful bootstrap including cluster-name, api-server and cluster-certificate. Any labels and taints that need to be set based on pod requirements will also be specified in the final merged UserData.

Missing in the Custom AMI config

[settings.host-containers]
[settings.host-containers.admin]
enabled = false

[settings.host-containers.control]
enabled = true

[settings.kubernetes]

[tzneal]

I put up a PR to clarify in the docs, for Custom AMI families there is no merging of UserData.

[JRemitz]

Ah thanks for that. Could there be? Is there a good reason not to or at least allow the option to override?