CoreDNS resolution failure for external hostnames with "overflow unpacking uint32" #3305
Description
First of all forgive me if this is not the right place to post. I'm using coredns (in a k8s/rancher installation). Rancher uses its own fork of coredns, but looking at their repository it seems to be the same code of coredns itself.
Here's my original issue: rancher/rke/issues/1662 but I think that I've posted it in the wrong place.
Their rke tool install CoreDNS-1.3.1 with this configuration
.:53 {
errors
health
kubernetes cluster.local in-addr.arpa ip6.arpa {
pods insecure
upstream
fallthrough in-addr.arpa ip6.arpa
ttl 30
}
prometheus :9153
forward . '/etc/resolv.conf'
cache 30
loop
reload
loadbalance
}
Log start:
.:53
2019-09-24T13:54:37.187Z [INFO] CoreDNS-1.3.1
2019-09-24T13:54:37.187Z [INFO] linux/amd64, go1.11.4, 6b56a9c
CoreDNS-1.3.1
linux/amd64, go1.11.4, 6b56a9c
I've installed a rancher cluster using RKE with 3 nodes on KVM, until now everything worked well, but starting today I have an (intermittent) issue in DNS names resolution in my pods. It happens only for some hosts and randomly.
When it happens dns resolution (for external names) in pods is not working and I get some nasty errors in coredns pod, here are some examples:
2019-09-24T12:46:25.111Z [INFO] plugin/reload: Running configuration MD5 = 45cd9f91917cc54711e243e0d08537a7
2019-09-24T12:47:27.474Z [ERROR] plugin/errors: 2 security.ubuntu.com. A: dns: overflow unpacking uint32
2019-09-24T12:47:32.475Z [ERROR] plugin/errors: 2 security.ubuntu.com. A: dns: overflow unpacking uint32
2019-09-24T12:47:37.476Z [ERROR] plugin/errors: 2 security.ubuntu.com. A: dns: overflow unpacking uint32
2019-09-24T13:12:39.537Z [ERROR] plugin/errors: 2 registry.npmjs.org. A: dns: overflow unpacking uint32
2019-09-24T13:12:39.549Z [ERROR] plugin/errors: 2 registry.npmjs.org. AAAA: dns: overflow unpacking uint16
2019-09-24T13:12:44.539Z [ERROR] plugin/errors: 2 registry.npmjs.org. AAAA: dns: overflow unpacking uint16
2019-09-24T13:12:44.543Z [ERROR] plugin/errors: 2 registry.npmjs.org. A: dns: overflow unpacking uint32
I've enabled coredns logs directive in configmap to have verbose logging and I get this:
2019-09-24T13:13:56.246Z [INFO] 10.42.0.230:56169 - 21605 "A IN registry.npmjs.org. udp 36 false 512" SERVFAIL qr,rd 36 5.003233501s
2019-09-24T13:13:56.246Z [ERROR] plugin/errors: 0 registry.npmjs.org. A: dns: overflow unpacking uint32
2019-09-24T13:13:56.251Z [INFO] 10.42.0.230:56169 - 4205 "AAAA IN registry.npmjs.org. udp 36 false 512" SERVFAIL qr,rd 36 5.008031338s
2019-09-24T13:13:56.251Z [ERROR] plugin/errors: 0 registry.npmjs.org. AAAA: dns: overflow unpacking uint16
Finally I also changed this directive in Corefile
forward . "/etc/resolv.conf"
# replaced with
forward . 1.1.1.1
# also tried with
forward . 8.8.8.8
Nothing changes.
I can find nothing relevant by googling for message "overflow unpacking uint32", just some code fragment where is triggered.
My coredns deplyoment uses rancher/coredns-coredns:1.3.1 as image.
What can be?