====== 2022.05.26 Kubernetes Podで名前解決できない ======
Kubernetesで名前解決できなくなって困った事がったのでメモ
===== 環境 =====
kubernetes v1.24.0
# kubeadm version
kubeadm version: &version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.0", GitCommit:"4ce5a8954017644c5420bae81d72b09b735c21f0", GitTreeState:"clean", BuildDate:"2022-05-03T13:44:24Z", GoVersion:"go1.18.1", Compiler:"gc", Platform:"linux/amd64"}
===== エラー1 =====
cni0がすでにあるというエラーでした。
# kubectl get pod --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system coredns-6d4b75cb6d-9hjwp 0/1 ContainerCreating 0 25s
kube-system coredns-6d4b75cb6d-lz4dl 0/1 ContainerCreating 0 25s
kube-system etcd-linstor-master02 1/1 Running 1 39s
kube-system kube-apiserver-linstor-master02 1/1 Running 1 38s
kube-system kube-controller-manager-linstor-master02 1/1 Running 1 38s
kube-system kube-proxy-4ffvr 1/1 Running 0 11s
kube-system kube-proxy-r62b5 1/1 Running 0 10s
kube-system kube-proxy-rj92f 0/1 Pending 0 6s
kube-system kube-proxy-rsthb 1/1 Running 0 25s
kube-system kube-scheduler-linstor-master02 1/1 Running 1 39s
# kubectl describe pod coredns-6d4b75cb6d-lz4dl -n kube-system
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 59s default-scheduler Successfully assigned kube-system/coredns-6d4b75cb6d-lz4dl to linstor-master02
Warning FailedCreatePodSandBox 58s kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "5b729fceffd1ec702b1040dd856f6d6375c5eaa3bad4d0012a1a88dda710314d": plugin type="flannel" failed (add): failed to delegate add: failed to set bridge addr: "cni0" already has an IP address different from 10.224.0.1/24
Warning FailedCreatePodSandBox 45s kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "10304b386803230d68d2cf54df60df31efac23070567d1962088a68ce89fdcab": plugin type="flannel" failed (add): failed to delegate add: failed to set bridge addr: "cni0" already has an IP address different from 10.224.0.1/24
Warning FailedCreatePodSandBox 35s kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "7907fdfa62306f9e79d4b11f70bf59b4db2fc8fc4455c1d1c079c187e1a8ace2": plugin type="flannel" failed (add): failed to delegate add: failed to set bridge addr: "cni0" already has an IP address different from 10.224.0.1/24
Warning FailedCreatePodSandBox 23s kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "f75551eb1886794291b837561974d8758e57e00669748d9f8960e1285fca76e8": plugin type="flannel" failed (add): failed to delegate add: failed to set bridge addr: "cni0" already has an IP address different from 10.224.0.1/24
Warning FailedCreatePodSandBox 8s kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "cf343dbc894e0913f8901776a48d81c0ddf91ea3dfa2832582c08f0789c27f48": plugin type="flannel" failed (add): failed to delegate add: failed to set bridge addr: "cni0" already has an IP address different from 10.224.0.1/24
==== 対応 ====
全ノードで、resetしてから、cni0とflannel.1を削除した後、initし直したらcorednsも上がり治りました。
kubeadm reset
ip link delete cni0 ; ip link delete flannel.1; \
rm -rfv /etc/cni ; rm -rvf /var/lib/cni/ ; rm -rvf /var/run/flannel ; \
rm -rfv /etc/containerd/config.toml ; \
systemctl restart containerd ;
kubeadm init
===== エラー2 =====
DNSは起動してて、kubernetesホストからは名前解決できるのに、Podからだけ名前解決できない。
# kubectl get pod --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
default alpine-test 1/1 Running 0 14m
kube-system coredns-6d4b75cb6d-6mmj7 1/1 Running 0 18m
kube-system coredns-6d4b75cb6d-zt2nq 1/1 Running 0 18m
# kubectl exec -it alpine-test -- sh
/ # nslookup fl8.jp
;; connection timed out; no servers could be reached
/ # ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 56 data bytes
64 bytes from 8.8.8.8: seq=0 ttl=117 time=2.225 ms
64 bytes from 8.8.8.8: seq=1 ttl=117 time=1.767 ms
64 bytes from 8.8.8.8: seq=2 ttl=117 time=1.667 ms
==== 対応 ====
[[06_virtualization:05_container:12_kubernetes#6.flannelインストール|flannelインストール]]ができてない。
flannelインストールしてあげれば名前解決できるようになった。
# kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
# kubectl exec -it alpine-test -- sh
/ # nslookup fl8.jp
Server: 10.96.0.10
Address: 10.96.0.10:53
Name: fl8.jp
Address: 182.48.51.190
===== エラー3 =====
corednsが上がらない
# kubectl describe pod coredns-76c94cf8bb-654df -n kube-system
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 3m41s default-scheduler Successfully assigned kube-system/coredns-76c94cf8bb-654df to linstor-node06
Normal Pulling 3m41s kubelet Pulling image "k8s.gcr.io/coredns/coredns:v1.8.4"
Normal Pulled 3m38s kubelet Successfully pulled image "k8s.gcr.io/coredns/coredns:v1.8.4" in 3.089546945s
Normal Created 3m38s kubelet Created container coredns
Normal Started 3m37s kubelet Started container coredns
Warning Unhealthy 51s (x21 over 3m37s) kubelet Readiness probe failed: HTTP probe failed with statuscode: 503
==== 対応 ====
これもエラー1と同じで、kubeadm reset して作成すれば、治った。
kubeadm reset
ip link delete cni0 ; ip link delete flannel.1; \
rm -rfv /etc/cni ; rm -rvf /var/lib/cni/ ; rm -rvf /var/run/flannel ; \
rm -rfv /etc/containerd/config.toml ; \
systemctl restart containerd ;
kubeadm init
{{tag>日記 kubernetes}}