Regression: CoreDNS in version 1.8.0 fails to resolve domain (works in 1.6.9)

[lhotrifork]

Environmental Info:
K3s Version:

k3s version v1.20.4+k3s1 (838a906a)
go version go1.15.8

Node(s) CPU architecture, OS, and Version:

uname -a
Linux hostname 4.15.0-147-generic #151-Ubuntu SMP Fri Jun 18 19:21:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

Cluster Configuration:
1 server acting as both control plane and worker node.

Describe the bug:
When using k3s with CoreDNS 1.8.0 for a specific domain (which is my corporate http proxy) returns 'rflags' in a different way then when using CoreDNS 1.6.9. In practices this means that OpenID Connect using OpenResty as a reverse-proxy fails. Downgrading to 1.6.9 fixes the issue.

I've narrowed it down to the following difference (which may be a wild goose-chase).
For 1.6.9 CoreDNS logs the following way for the domain:

[INFO] 10.42.0.220:44598 - 59764 "A IN proxy.mydomain.com. udp 32 false 512" NOERROR qr,aa,rd,ra 336 0.000129347s

For 1.8.0 it logs:

[INFO] 10.42.0.228:33081 - 63992 "A IN proxy.mydomain.com. udp 32 false 512" NOERROR qr,rd,ra,ad 336 0.000702693s

The difference seems to be in the 'rflags'.

As this is my proxy server a lot of requests also goes through it, and for those request there's also a difference between 1.6.9 and 1.8.0.
1.6.9:

[INFO] 10.42.0.213:36482 - 19579 "A IN proxy.mydomain.com.flux-system.svc.cluster.local. udp 62 false 512" NXDOMAIN qr,aa,rd 155 0.000166089s
[INFO] 10.42.0.213:35226 - 61848 "AAAA IN proxy.mydomain.com.flux-system.svc.cluster.local. udp 62 false 512" NXDOMAIN qr,aa,rd 155 0.003467135s
[INFO] 10.42.0.213:37512 - 34923 "A IN proxy.mydomain.com.svc.cluster.local. udp 50 false 512" NXDOMAIN qr,aa,rd 143 0.000105847s
[INFO] 10.42.0.213:59737 - 16377 "AAAA IN proxy.mydomain.com.svc.cluster.local. udp 50 false 512" NXDOMAIN qr,aa,rd 143 0.000061375s
[INFO] 10.42.0.213:55805 - 65513 "AAAA IN proxy.mydomain.com.cluster.local. udp 46 false 512" NXDOMAIN qr,aa,rd 139 0.000271467s
[INFO] 10.42.0.213:44383 - 8234 "A IN proxy.mydomain.com.cluster.local. udp 46 false 512" NXDOMAIN qr,aa,rd 139 0.000194574s
[INFO] 10.42.0.213:35004 - 18276 "A IN proxy.mydomain.com.my.subdomain.mydomain.com. udp 53 false 512" NXDOMAIN qr,rd,ra 141 0.000447348s

1.8.0:

[INFO] 10.42.0.251:41582 - 30418 "AAAA IN proxy.mydomain.com. udp 32 false 512" NOERROR qr,aa,rd,ra 348 0.000044463s
[INFO] 10.42.0.251:51639 - 8933 "A IN proxy.mydomain.com.kube-system.svc.cluster.local. udp 62 false 512" NXDOMAIN qr,aa,rd 155 0.000155996s
[INFO] 10.42.0.251:46513 - 14889 "A IN proxy.mydomain.com.svc.cluster.local. udp 50 false 512" NXDOMAIN qr,aa,rd 143 0.00009106s
[INFO] 10.42.0.251:56726 - 44362 "A IN proxy.mydomain.com.cluster.local. udp 46 false 512" NXDOMAIN qr,aa,rd 139 0.000055118s
[INFO] 10.42.0.251:52852 - 37567 "A IN proxy.mydomain.com.my.subdomain.mydomain.com. udp 53 false 512" NXDOMAIN qr,aa,rd,ra 141 0.000026808s
[INFO] 10.42.0.251:55715 - 2845 "A IN proxy.mydomain.com.netic.dk. udp 41 false 512" NXDOMAIN qr,aa,rd,ra 117 0.000024033s
[INFO] 10.42.0.251:54047 - 17602 "A IN proxy.mydomain.com. udp 32 false 512" NOERROR qr,aa,rd,ra 336 0.00005718s
[INFO] 10.42.0.251:41358 - 4578 "AAAA IN proxy.mydomain.com. udp 32 false 512" NOERROR qr,aa,rd,ra 348 0.000046303s

Steps To Reproduce:

• Installed K3s:

export HTTP_PROXY=proxy.mydomain.com:3128
export HTTPS_PROXY=proxy.mydomain.com:3128
curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=v1.20.4+k3s1 sh -s - --docker

Additional context / logs:

CoreDNS configuration:

    .:53 {
        log
        errors
        health
        ready
        kubernetes cluster.local in-addr.arpa ip6.arpa {
          pods insecure
          fallthrough in-addr.arpa ip6.arpa
        }
        hosts /etc/coredns/NodeHosts {
          ttl 60
          reload 15s
          fallthrough
        }
        prometheus :9153
        forward . /etc/resolv.conf
        cache 30
        loop
        reload
        loadbalance
    }

/ect/resolv.conf on the host:

nameserver 127.0.0.53
options edns0
search my.subdomain.mydomain.com mydomain.com

I was conflicted if I should report this bug for k3s or for CoreDNS directly. Please let me know if this is not a k3s bug as such.

[brandond]

The issue should probably be reported upstream to coredns, but K3s 1.20 will be updating to 1.8.3 in #3541 so you might try with that. You could also try a k3s 1.21 release, as that's already on 1.8.3.

[lhotrifork]

Thanks for the suggestion!
I've opened a bug in upstream CoreDNS: coredns/coredns#4735