I recently spun up a couple hosts as Kubernetes workers and interestingly, one of them worked fine but the other would not resolve kube-dns queries. On the node itself, running this command would fail:
nslookup kubernetes 10.43.0.10
In kube-proxy logs I found this:
E0209 04:40:18.259923 1168 proxier.go:1571] Failed to execute iptables-restore: exit status 2 (iptables-restore v1.8.4 (legacy): Couldn't load target `KUBE-MARK-DROP':No such file or directory Error occurred at line: 131 Try `iptables-restore -h' or 'iptables-restore --help' for more information. )
I found this article:
Specifically these commands:
sudo sysctl net.bridge.bridge-nf-call-iptables=1 sudo update-alternatives --set iptables /usr/sbin/iptables-legacy
Running that seemed to fix my problems, but after a reboot it didn’t work, and running the commands again had no effect.
(Un)helpfully, there is a GitHub issue for this, which says the problem is fixed in a later version.
I ended up just blasting away the node and making sure to bring up a new one with a different name. Unsatisfying, I know.