Kubernetesで名前解決できなくなって困った事がったのでメモ

kubernetes v1.24.0

# kubeadm version
kubeadm version: &version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.0", GitCommit:"4ce5a8954017644c5420bae81d72b09b735c21f0", GitTreeState:"clean", BuildDate:"2022-05-03T13:44:24Z", GoVersion:"go1.18.1", Compiler:"gc", Platform:"linux/amd64"}

cni0がすでにあるというエラーでした。

# kubectl get pod --all-namespaces 
NAMESPACE     NAME                                       READY   STATUS              RESTARTS   AGE
kube-system   coredns-6d4b75cb6d-9hjwp                   0/1     ContainerCreating   0          25s
kube-system   coredns-6d4b75cb6d-lz4dl                   0/1     ContainerCreating   0          25s
kube-system   etcd-linstor-master02                      1/1     Running             1          39s
kube-system   kube-apiserver-linstor-master02            1/1     Running             1          38s
kube-system   kube-controller-manager-linstor-master02   1/1     Running             1          38s
kube-system   kube-proxy-4ffvr                           1/1     Running             0          11s
kube-system   kube-proxy-r62b5                           1/1     Running             0          10s
kube-system   kube-proxy-rj92f                           0/1     Pending             0          6s
kube-system   kube-proxy-rsthb                           1/1     Running             0          25s
kube-system   kube-scheduler-linstor-master02            1/1     Running             1          39s

# kubectl describe pod coredns-6d4b75cb6d-lz4dl -n kube-system
Events:
  Type     Reason                  Age   From               Message
  ----     ------                  ----  ----               -------
  Normal   Scheduled               59s   default-scheduler  Successfully assigned kube-system/coredns-6d4b75cb6d-lz4dl to linstor-master02
  Warning  FailedCreatePodSandBox  58s   kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "5b729fceffd1ec702b1040dd856f6d6375c5eaa3bad4d0012a1a88dda710314d": plugin type="flannel" failed (add): failed to delegate add: failed to set bridge addr: "cni0" already has an IP address different from 10.224.0.1/24
  Warning  FailedCreatePodSandBox  45s   kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "10304b386803230d68d2cf54df60df31efac23070567d1962088a68ce89fdcab": plugin type="flannel" failed (add): failed to delegate add: failed to set bridge addr: "cni0" already has an IP address different from 10.224.0.1/24
  Warning  FailedCreatePodSandBox  35s   kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "7907fdfa62306f9e79d4b11f70bf59b4db2fc8fc4455c1d1c079c187e1a8ace2": plugin type="flannel" failed (add): failed to delegate add: failed to set bridge addr: "cni0" already has an IP address different from 10.224.0.1/24
  Warning  FailedCreatePodSandBox  23s   kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "f75551eb1886794291b837561974d8758e57e00669748d9f8960e1285fca76e8": plugin type="flannel" failed (add): failed to delegate add: failed to set bridge addr: "cni0" already has an IP address different from 10.224.0.1/24
  Warning  FailedCreatePodSandBox  8s    kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "cf343dbc894e0913f8901776a48d81c0ddf91ea3dfa2832582c08f0789c27f48": plugin type="flannel" failed (add): failed to delegate add: failed to set bridge addr: "cni0" already has an IP address different from 10.224.0.1/24

全ノードで、resetしてから、cni0とflannel.1を削除した後、initし直したらcorednsも上がり治りました。

kubeadm reset

ip link delete cni0 ; ip link delete flannel.1; \
rm -rfv /etc/cni ; rm -rvf /var/lib/cni/ ; rm -rvf /var/run/flannel ; \
rm -rfv /etc/containerd/config.toml ; \
systemctl restart containerd ;

kubeadm init

DNSは起動してて、kubernetesホストからは名前解決できるのに、Podからだけ名前解決できない。

# kubectl get pod --all-namespaces 
NAMESPACE     NAME                                       READY   STATUS    RESTARTS   AGE
default       alpine-test                                1/1     Running   0          14m
kube-system   coredns-6d4b75cb6d-6mmj7                   1/1     Running   0          18m
kube-system   coredns-6d4b75cb6d-zt2nq                   1/1     Running   0          18m


# kubectl exec -it alpine-test -- sh
/ # nslookup fl8.jp 
;; connection timed out; no servers could be reached

/ # ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 56 data bytes
64 bytes from 8.8.8.8: seq=0 ttl=117 time=2.225 ms
64 bytes from 8.8.8.8: seq=1 ttl=117 time=1.767 ms
64 bytes from 8.8.8.8: seq=2 ttl=117 time=1.667 ms

flannelインストールができてない。

flannelインストールしてあげれば名前解決できるようになった。

# kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

# kubectl exec -it alpine-test -- sh
/ # nslookup fl8.jp 
Server:		10.96.0.10
Address:	10.96.0.10:53


Name:	fl8.jp
Address: 182.48.51.190

corednsが上がらない

# kubectl describe pod coredns-76c94cf8bb-654df -n kube-system
Events:
  Type     Reason     Age                   From               Message
  ----     ------     ----                  ----               -------
  Normal   Scheduled  3m41s                 default-scheduler  Successfully assigned kube-system/coredns-76c94cf8bb-654df to linstor-node06
  Normal   Pulling    3m41s                 kubelet            Pulling image "k8s.gcr.io/coredns/coredns:v1.8.4"
  Normal   Pulled     3m38s                 kubelet            Successfully pulled image "k8s.gcr.io/coredns/coredns:v1.8.4" in 3.089546945s
  Normal   Created    3m38s                 kubelet            Created container coredns
  Normal   Started    3m37s                 kubelet            Started container coredns
  Warning  Unhealthy  51s (x21 over 3m37s)  kubelet            Readiness probe failed: HTTP probe failed with statuscode: 503

これもエラー1と同じで、kubeadm reset して作成すれば、治った。

kubeadm reset

ip link delete cni0 ; ip link delete flannel.1; \
rm -rfv /etc/cni ; rm -rvf /var/lib/cni/ ; rm -rvf /var/run/flannel ; \
rm -rfv /etc/containerd/config.toml ; \
systemctl restart containerd ;

kubeadm init