Kubernetesで名前解決できなくなって困った事がったのでメモ
kubernetes v1.24.0
# kubeadm version
kubeadm version: &version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.0", GitCommit:"4ce5a8954017644c5420bae81d72b09b735c21f0", GitTreeState:"clean", BuildDate:"2022-05-03T13:44:24Z", GoVersion:"go1.18.1", Compiler:"gc", Platform:"linux/amd64"}
cni0がすでにあるというエラーでした。
# kubectl get pod --all-namespaces NAMESPACE NAME READY STATUS RESTARTS AGE kube-system coredns-6d4b75cb6d-9hjwp 0/1 ContainerCreating 0 25s kube-system coredns-6d4b75cb6d-lz4dl 0/1 ContainerCreating 0 25s kube-system etcd-linstor-master02 1/1 Running 1 39s kube-system kube-apiserver-linstor-master02 1/1 Running 1 38s kube-system kube-controller-manager-linstor-master02 1/1 Running 1 38s kube-system kube-proxy-4ffvr 1/1 Running 0 11s kube-system kube-proxy-r62b5 1/1 Running 0 10s kube-system kube-proxy-rj92f 0/1 Pending 0 6s kube-system kube-proxy-rsthb 1/1 Running 0 25s kube-system kube-scheduler-linstor-master02 1/1 Running 1 39s # kubectl describe pod coredns-6d4b75cb6d-lz4dl -n kube-system Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 59s default-scheduler Successfully assigned kube-system/coredns-6d4b75cb6d-lz4dl to linstor-master02 Warning FailedCreatePodSandBox 58s kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "5b729fceffd1ec702b1040dd856f6d6375c5eaa3bad4d0012a1a88dda710314d": plugin type="flannel" failed (add): failed to delegate add: failed to set bridge addr: "cni0" already has an IP address different from 10.224.0.1/24 Warning FailedCreatePodSandBox 45s kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "10304b386803230d68d2cf54df60df31efac23070567d1962088a68ce89fdcab": plugin type="flannel" failed (add): failed to delegate add: failed to set bridge addr: "cni0" already has an IP address different from 10.224.0.1/24 Warning FailedCreatePodSandBox 35s kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "7907fdfa62306f9e79d4b11f70bf59b4db2fc8fc4455c1d1c079c187e1a8ace2": plugin type="flannel" failed (add): failed to delegate add: failed to set bridge addr: "cni0" already has an IP address different from 10.224.0.1/24 Warning FailedCreatePodSandBox 23s kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "f75551eb1886794291b837561974d8758e57e00669748d9f8960e1285fca76e8": plugin type="flannel" failed (add): failed to delegate add: failed to set bridge addr: "cni0" already has an IP address different from 10.224.0.1/24 Warning FailedCreatePodSandBox 8s kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "cf343dbc894e0913f8901776a48d81c0ddf91ea3dfa2832582c08f0789c27f48": plugin type="flannel" failed (add): failed to delegate add: failed to set bridge addr: "cni0" already has an IP address different from 10.224.0.1/24
全ノードで、resetしてから、cni0とflannel.1を削除した後、initし直したらcorednsも上がり治りました。
kubeadm reset ip link delete cni0 ; ip link delete flannel.1; \ rm -rfv /etc/cni ; rm -rvf /var/lib/cni/ ; rm -rvf /var/run/flannel ; \ rm -rfv /etc/containerd/config.toml ; \ systemctl restart containerd ; kubeadm init
DNSは起動してて、kubernetesホストからは名前解決できるのに、Podからだけ名前解決できない。
# kubectl get pod --all-namespaces NAMESPACE NAME READY STATUS RESTARTS AGE default alpine-test 1/1 Running 0 14m kube-system coredns-6d4b75cb6d-6mmj7 1/1 Running 0 18m kube-system coredns-6d4b75cb6d-zt2nq 1/1 Running 0 18m # kubectl exec -it alpine-test -- sh / # nslookup fl8.jp ;; connection timed out; no servers could be reached / # ping 8.8.8.8 PING 8.8.8.8 (8.8.8.8): 56 data bytes 64 bytes from 8.8.8.8: seq=0 ttl=117 time=2.225 ms 64 bytes from 8.8.8.8: seq=1 ttl=117 time=1.767 ms 64 bytes from 8.8.8.8: seq=2 ttl=117 time=1.667 ms
flannelインストールができてない。
flannelインストールしてあげれば名前解決できるようになった。
# kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml # kubectl exec -it alpine-test -- sh / # nslookup fl8.jp Server: 10.96.0.10 Address: 10.96.0.10:53 Name: fl8.jp Address: 182.48.51.190
corednsが上がらない
# kubectl describe pod coredns-76c94cf8bb-654df -n kube-system Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 3m41s default-scheduler Successfully assigned kube-system/coredns-76c94cf8bb-654df to linstor-node06 Normal Pulling 3m41s kubelet Pulling image "k8s.gcr.io/coredns/coredns:v1.8.4" Normal Pulled 3m38s kubelet Successfully pulled image "k8s.gcr.io/coredns/coredns:v1.8.4" in 3.089546945s Normal Created 3m38s kubelet Created container coredns Normal Started 3m37s kubelet Started container coredns Warning Unhealthy 51s (x21 over 3m37s) kubelet Readiness probe failed: HTTP probe failed with statuscode: 503
これもエラー1と同じで、kubeadm reset して作成すれば、治った。
kubeadm reset ip link delete cni0 ; ip link delete flannel.1; \ rm -rfv /etc/cni ; rm -rvf /var/lib/cni/ ; rm -rvf /var/run/flannel ; \ rm -rfv /etc/containerd/config.toml ; \ systemctl restart containerd ; kubeadm init