DNS using container name fails for containers on user-defined rootless cni network with a pod

[usury]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

DNS using container name fails for rootless containers on user-defined rootless cni network WITHIN a pod. The same succeeds for similar rootless containers NOT WITHIN a pod.

Steps to reproduce the issue:

    normaluser@containerhost $> podman rm -fa       # fresh set of examples
    normaluser@containerhost $> podman network create myCNI
    normaluser@containerhost $> podman network inspect myCNI | grep -i subnet
`                "subnet": "10.89.0.0/24"`
    normaluser@containerhost $> podman pod create --name myPod
    normaluser@containerhost $> podman run --name myNginx2 --network myCNI --pod myPod -d nginx:alpine
    normaluser@containerhost $> podman run --name myNginx3 --network myCNI --pod myPod -d nginx:alpine

    normaluser@containerhost $> podman inspect myNginx2 | grep -i ipaddress
`                    "IPAddress": "10.89.0.2",`
    normaluser@containerhost $> podman inspect myNginx3 | grep -i ipaddress
`                    "IPAddress": "10.89.0.3",`

    normaluser@containerhost $> podman exec myNginx2 ping myNginx3
        ping: bad address 'myNginx3'
    normaluser@containerhost $> podman exec myNginx3 ping myNginx2
        ping: bad address 'myNginx2'

Describe the results you received:
DNS lookup based on container name fails within pod

Describe the results you expected:
DNS lookup works for containers within a pod the way it works for containers not within pod

Additional information you deem important (e.g. issue happens only occasionally):
happens consistently and reproducably

Output of podman version:

        Version:      2.1.1
        API Version:  2.0.0
        Go Version:   go1.14
        Built:        Wed Dec 31 16:00:00 1969
        OS/Arch:      linux/amd64

Output of podman info --debug:

host:
  arch: amd64
  buildahVersion: 1.16.1
  cgroupManager: cgroupfs
  cgroupVersion: v1
  conmon:
    package: 'conmon: /usr/libexec/podman/conmon'
    path: /usr/libexec/podman/conmon
    version: 'conmon version 2.0.20, commit: '
  cpus: 4
  distribution:
    distribution: debian
    version: "10"
  eventLogger: journald
  hostname: arachne
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 2000
      size: 1
    - container_id: 1
      host_id: 100001
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 2000
      size: 1
    - container_id: 1
      host_id: 100001
      size: 65536
  kernel: 4.19.0-10-amd64
  linkmode: dynamic
  memFree: 369057792
  memTotal: 2091732992
  ociRuntime:
    name: runc
    package: 'runc: /usr/sbin/runc'
    path: /usr/sbin/runc
    version: |-
      runc version 1.0.0~rc6+dfsg1
      commit: 1.0.0~rc6+dfsg1-3
      spec: 1.0.1
  os: linux
  remoteSocket:
    path: /run/user/2000/podman/podman.sock
  rootless: true
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: 'slirp4netns: /usr/bin/slirp4netns'
    version: |-
      slirp4netns version 1.1.4
      commit: unknown
      libslirp: 4.3.1-git
      SLIRP_CONFIG_VERSION_MAX: 3
  swapFree: 746057728
  swapTotal: 1073737728
  uptime: 184h 14m 26.29s (Approximately 7.67 days)
registries:
  search:
  - docker.io
  - quay.io
version:
  APIVersion: 2.0.0
  Built: 0
  BuiltTime: Wed Dec 31 16:00:00 1969
  GitCommit: ""
  GoVersion: go1.14
  OsArch: linux/amd64
  Version: 2.1.1

Package info (e.g. output of rpm -q podman or apt list podman):

podman/unknown,now 2.1.1~2 amd64 [installed]
podman/unknown 2.1.1~2 arm64
podman/unknown 2.1.1~2 armhf
podman/unknown 2.1.1~2 ppc64el

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide?

Yes

Additional environment details (AWS, VirtualBox, physical, etc.):
Container host is a VirtualBox VM running on Fedora 32
podman packages installed from OpenSuse repo
$> cat "/etc/apt/sources.list.d/devel:kubic:libcontainers:stable.list"

## "Kubic" repo from "OpenSuse" for "podman" packages since they aren't in Debian 10 (buster) repos
deb https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/Debian_10/ /

$> cat /etc/os-release
PRETTY_NAME="Debian GNU/Linux 10 (buster)"

[usury]

Working DNS using container names - containers on user-defined rootless cni network NOT in a pod
This is not a problem. Rather, this is how a user would expect it to work if the same containers were within a pod.

    normaluser@containerhost $> podman rm -fa       # fresh set of examples
    normaluser@containerhost $> podman network create myCNI

    normaluser@containerhost $> podman run --name myNginx2 --network myCNI -d nginx:alpine
    normaluser@containerhost $> podman run --name myNginx3 --network myCNI -d nginx:alpine

    normaluser@containerhost $>  podman inspect myNginx2 | grep -i ipaddress
                    "IPAddress": "10.89.0.2",
    normaluser@containerhost $>  podman inspect myNginx3 | grep -i ipaddress
                    "IPAddress": "10.89.0.3",

    normaluser@containerhost $> podman exec myNginx2 ping -c1 myNginx3
        PING myNginx3 (10.89.0.3): 56 data bytes
        64 bytes from 10.89.0.3: seq=0 ttl=64 time=0.156 ms

    normaluser@containerhost $>  podman exec myNginx3 ping -c1 myNginx2
        PING myNginx2 (10.89.0.2): 56 data bytes
        64 bytes from 10.89.0.2: seq=0 ttl=64 time=0.267 ms

[mheon]

Dupe of #8040

[Luap99]

No this is not a duplicate.

The problem is that we always use the podname for the dns entry if the container is in a pod. In this case we do not join the pod network namespace and thus should use the container name for the dns entry.

[Luap99]

This happens with both rootful and rootless cni. I can fix this.

[mheon]

Good catch - thanks!

…

On Fri, Oct 30, 2020, 09:43 Luap99 ***@***.***> wrote: This happens with both rootful and rootless cni. I can fix this. — You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub <#8194 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AB3AOCG4O7ACNBHLEUWZL53SNK7H3ANCNFSM4TEQI5HA> .

[usury]

I know the documentation says rootless containers within pods need to access each other's services from localhost:port within the pod, or publish their services to a host port and access each other's services via containerhost:published_port.

However, if pod containers don't need to share "net" namespace (as is already the case when containers within a pod belong to a user-defined CNI network), and each container within a pod can have its own hostname assignment (not the pod name as it is now) courtesy of current bug fix, then shouldn't the example in the second block (below) be possible if container-name-based DNS resolution is made to work within a pod?

For example:

• Run multiples web server containers on port 80 within a pod that could access each other by container-name
• Or a reverse proxy and multiple web servers all on containers within a pod, each listening on their own port 80, only publishing the port of the reverse proxy (the use case I'm interested in)

Containers within pod created with default set of "--share ipc,net,pid,uts"

normaluser@containerhost $> podman rm -fa        # fresh set of examples
normaluser@containerhost $> podman pod create --name myPod
normaluser@containerhost $> podman run --name myNginx1 --pod myPod -d nginx:alpine
normaluser@containerhost $> podman run --name myNginx2 --pod myPod -d nginx:alpine
    ## launches but dies immediately
    ## "myNginx1" already listens on port 80 and "net" namespace is shared so "myNginx2" fails
    
normaluser@containerhost $> podman exec myNginx1 ping myNginx2
    ping: bad address 'myNginx2'

normaluser@containerhost $> podman exec myNginx1 hostname
    myPod

The above behavior makes sense. According to podman documentation, by default containers within a pod share the name network namespace and must access each other's services via localhost.

Containers within pod NOT SHARING "net"

normaluser@containerhost $> podman rm -fa        # fresh set of examples
normaluser@containerhost $> podman pod create --name myPod --share ipc,pid,uts      # no "net"
normaluser@containerhost $> podman run --name myNginx1 --pod myPod -d nginx:alpine
normaluser@containerhost $> podman run --name myNginx2 --pod myPod -d nginx:alpine
    ## launches successfully - each container has it's own port 80 since "net" namespace not shared
    
normaluser@containerhost $> podman exec myNginx1 ping myNginx2
    ping: bad address 'myNginx2'

normaluser@containerhost $> podman exec myNginx1 hostname
    myPod
normaluser@containerhost $> podman exec myNginx2 hostname
    myPod

normaluser@containerhost $> podman exec myNginx1 curl --head http://myNginx2
    curl: (6) Could not resolve host: myNginx2

These containers behave exactly the same way regarding container-name-based DNS lookup and hostname assignment as containers in user-defined rootless cni networks.

If the current bug fix addresses hostname assignment to containers within a pod that belong to a user-defined cni network, and if those containers do not share "net" namespace, then perhaps the current bug fix can also makes the second block, above, work properly (since the above containers also do not share "net" namespace).

If such containers within a pod can successfully reach each other via container name, then the section of the documentation explaining how containers within a rootless pod reach each other's services also needs revision.