elastic agent connecting to wrong fleet address

[Raboo]

Bug Report

What did you do?
I'm running fleet-server and elastic-agent.

What did you expect to see?
I expect elastic-agent to connect to the fleet server

What did you see instead? Under which circumstances?
elastic-agent is trying to connect to fleet at address fleet-server-agent-http.elastic0.svc, but the real address is fleet-server0-agent-http.elastic0.svc

Logs looke like this

WARN [transport] transport/tcp.go:52 DNS lookup failure "fleet-server-agent-http.elastic0.svc": lookup fleet-server-agent-http.elastic0.svc on 10.43.0.10:53: no such host
ERROR fleet/fleet_gateway.go:180 failed to dispatch actions, error: fail to communicate with updated API client hosts: Get "https://fleet-server-agent-http.elastic0.svc:8220/api/status?": lookup fleet-server-agent-http.elastic0.svc on 10.43.0.10:53: no such host 

When executing into a running container, the address in the configuration files is correct

[root@elastic-agent0-agent-jr4z8 elastic-agent]# cat fleet-setup.yml
fleet:
  ca: /mnt/elastic-internal/fleetserver-association/elastic0/fleet-server0/certs/ca.crt
  enroll: true
  url: https://fleet-server0-agent-http.elastic0.svc:8220
kibana:
  fleet:
    password: 
    setup: true
    username: elastic0-elastic-agent0-agent-kb-user

[root@elastic-agent0-agent-jr4z8 elastic-agent]# cat fleet.yml
agent:
  id: 
  monitoring.http:
    enabled: false
    host: ""
    port: 6791
fleet:
  enabled: true
  access_api_key: 
  protocol: https
  host: fleet-server0-agent-http.elastic0.svc:8220
  ssl:
    verification_mode: full
    certificate_authorities:
    - /mnt/elastic-internal/fleetserver-association/elastic0/fleet-server0/certs/ca.crt
    renegotiation: never
  timeout: 10m0s
  reporting:
    threshold: 10000
    check_frequency_sec: 30
  agent:
    id: ""

Environment

• ECK version: 1.7.0
• elastic stack version: 7.14.0
• Kubernetes information:
  • On premise
  • Kubernetes distribution: Rancher v2.5.8
  • Kubernetes Version: v1.20.6

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.3", GitCommit:"ca643a4d1f7bfe34773c74f79527be4afd95bf39", GitTreeState:"clean", BuildDate:"2021-07-15T20:58:09Z", GoVersion:"go1.16.5", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.6", GitCommit:"8a62859e515889f07e3e3be6a1080413f17cf2c3", GitTreeState:"clean", BuildDate:"2021-04-15T03:19:55Z", GoVersion:"go1.15.10", Compiler:"gc", Platform:"linux/amd64"}

• Resource definition:

I have a little different yaml manifest because another bug #4790

apiVersion: agent.k8s.elastic.co/v1alpha1
kind: Agent
metadata:
  name: elastic-agent0
  namespace: elastic0
spec:
  version: 7.14.0
  kibanaRef:
    name: kibana0
  fleetServerRef:
    name: fleet-server0
  mode: fleet
  daemonSet:
    podTemplate:
      spec:
        serviceAccountName: elastic-agent
        automountServiceAccountToken: true
        securityContext:
          runAsUser: 0
        containers:
        - name: agent
          volumeMounts: 
          - mountPath: /usr/share/elastic-agent/fleet-setup-base.yml
            name: fleet-setup-config # this volume is added by ECK, we only need to change the mount
            subPath: fleet-setup.yml
          command:
          - /usr/bin/env
          - bash
          - -c
          - |
            #!/usr/bin/env bash
            set -e
            grep -v "ca: /mnt/elastic-internal/.*\(elasticsearch\|kibana\).*" fleet-setup-base.yml | grep -v "host: .*" > fleet-setup.yml
            /usr/bin/tini -- /usr/local/bin/docker-entrypoint -e
          env:
            - name: KIBANA_FLEET_HOST
              value: https://kibana0.k8s.mydomain.net:443

[david-kow]

Hey @Raboo, can you check Kibana UI (Fleet > Fleet Settings) and see what URL is there? The fleet-setup.yml is only responsible for the setup part. After setup (enrolment and initial connection to Fleet Server) is done, Fleet Server passes the hosts that are specified in Kibana to all Elastic Agents to connect to. It seems to me that trying to connect to those hosts fails.

If that happens to be your issue, please note that right now Kibana won't use updated values of xpack.fleet.agent.* config settings after they've been set once. You have to update them via UI.

Please let me know if this is indeed the issue you were hitting, if not I'd be glad to investigate further.

[Raboo]

@david-kow that worked, thanks! I don't know how the hostname fleet-server-agent-http.elastic0.svc snuck into that setting.