====== 16 Kubernetes + DRBD Linstor ====== Kubernetes用の冗長化ストレージとして、Linstorを用意してみる。 LinstorはLINBITが開発した、DRBDを管理する管理DBを持つlinstor-controllerとlinstor-satelliteというデータディスクを扱うプログラムで構成されてます。 [[https://github.com/LINBIT/linstor-server]] {{:06_virtualization:05_container:linstor.png|}} ====== 1.環境 ====== 今回はKubernetesのworkerノード3台で、Linstor ^ name ^ IP ^ Disk ^ DRBD用 ^ |node1|172.16.0.65|/dev/sda|/dev/sdb| |node2|172.16.0.252|/dev/sda|/dev/sdb| |node3|172.16.0.234|/dev/sda|/dev/sdb| ※sdaは/用、sdbはlinstor用 ====== 2. DRBDインストール ====== add-apt-repository ppa:linbit/linbit-drbd9-stack apt-get update && apt install drbd-utils drbd-dkms lvm2 modprobe drbd ===== DRBD確認 ===== # cat /proc/drbd version: 9.1.6 (api:2/proto:110-121) GIT-hash: f85adeb71f16a0aead1e875665e2fb68852d94eb build by root@worker01, 2022-04-15 21:43:08 Transports (api:17): tcp (9.1.6) ====== 3. LINSTORインストール ====== 今回は3台でlinstor-controllerを冗長化するので、3台とも全部インストール apt install linstor-controller linstor-satellite linstor-client ===== linstor-controllerはDisable ===== systemctl disable linstor-controller ===== 自動補完 ===== # source /etc/bash_completion.d/linstor # または # source /usr/share/bash_completion/completions/linstor ====== 4. Linstor初期化 ====== 1度node1で、linstor-controllerを上げて初期設定を行います。 node1# systemctl start linstor-controller linstor node create node1 172.16.0.65 linstor node create node2 172.16.0.252 linstor node create node3 172.16.0.234 ===== 確認 ===== node1# linstor node list ╭────────────────────────────────────────────────────────╮ ┊ Node ┊ NodeType ┊ Addresses ┊ State ┊ ╞════════════════════════════════════════════════════════╡ ┊ node1 ┊ SATELLITE ┊ 172.16.0.65:3366 (PLAIN) ┊ Online ┊ ┊ node2 ┊ SATELLITE ┊ 172.16.0.252:3366 (PLAIN) ┊ Online ┊ ┊ node3 ┊ SATELLITE ┊ 172.16.0.234:3366 (PLAIN) ┊ Online ┊ ╰────────────────────────────────────────────────────────╯ ====== 5.Storage pool 設定 ====== こちらも、linstor-controllerを起動しているnode1で実行する。 sdbでLVMを用意してlinstorのストレージプールに利用する。 pvcreate /dev/sdb vgcreate vg /dev/sdb lvcreate -l 100%FREE --thinpool vg/lvmthinpool linstor storage-pool create lvmthin node1 linstor-pool vg/lvmthinpool linstor storage-pool create lvmthin node2 linstor-pool vg/lvmthinpool linstor storage-pool create lvmthin node3 linstor-pool vg/lvmthinpool ===== storage-pool確認 ===== node1# linstor storage-pool list -p +-----------------------------------------------------------------------------------------------------------------------------+ | StoragePool | Node | Driver | PoolName | FreeCapacity | TotalCapacity | CanSnapshots | State | SharedName | |=============================================================================================================================| | DfltDisklessStorPool | node1 | DISKLESS | | | | False | Ok | | | DfltDisklessStorPool | node2 | DISKLESS | | | | False | Ok | | | DfltDisklessStorPool | node3 | DISKLESS | | | | False | Ok | | | linstor-pool | node1 | LVM_THIN | vg/lvmthinpool | 19.90 GiB | 19.96 GiB | True | Ok | | | linstor-pool | node2 | LVM_THIN | vg/lvmthinpool | 19.90 GiB | 19.96 GiB | True | Ok | | | linstor-pool | node3 | LVM_THIN | vg/lvmthinpool | 19.90 GiB | 19.96 GiB | True | Ok | | +-----------------------------------------------------------------------------------------------------------------------------+ ===== 1つVolumeを作ってみる ===== === TestVol01を作成 === 流れは - リソース定義を作成 - ボリューム定義を作成 - リソースを作成 linstor resource-definition create TestVol01 linstor volume-definition create TestVol01 1G linstor resource create TestVol01 -s linstor-pool --auto-place 3 たとえば、node2,node3にリソースを作成して、node1はディスクレスにする場合 linstor resource create node2 node3 TestVol01 -s linstor-pool linstor resource create node1 TestVol01 --diskless === 確認 === # linstor volume list -r TestVol01 +----------------------------------------------------------------------------------------------------+ | Node | Resource | StoragePool | VolNr | MinorNr | DeviceName | Allocated | InUse | State | |====================================================================================================| | node1 | TestVol01 | linstor-pool | 0 | 1002 | /dev/drbd1002 | 315 KiB | Unused | UpToDate | | node2 | TestVol01 | linstor-pool | 0 | 1002 | /dev/drbd1002 | 315 KiB | Unused | UpToDate | | node3 | TestVol01 | linstor-pool | 0 | 1002 | /dev/drbd1002 | 315 KiB | Unused | UpToDate | +----------------------------------------------------------------------------------------------------+ === ディスクを使ってみる === mkfs.xfs /dev/drbd1002 mount /dev/drbd1002 /mnt/ # df /mnt Filesystem 1K-blocks Used Available Use% Mounted on /dev/drbd1002 1042168 40604 1001564 4% /mnt === マウントしているnode1のInUseがInUseに変更して、ディスクを利用している事が分かる === # linstor volume list -r TestVol01 | sed 's/^/ /g' +----------------------------------------------------------------------------------------------------+ | Node | Resource | StoragePool | VolNr | MinorNr | DeviceName | Allocated | InUse | State | |====================================================================================================| | node1 | TestVol01 | linstor-pool | 0 | 1002 | /dev/drbd1002 | 11.00 MiB | InUse | UpToDate | | node2 | TestVol01 | linstor-pool | 0 | 1002 | /dev/drbd1002 | 11.00 MiB | Unused | UpToDate | | node3 | TestVol01 | linstor-pool | 0 | 1002 | /dev/drbd1002 | 11.00 MiB | Unused | UpToDate | +----------------------------------------------------------------------------------------------------+ === 削除 === umount /mnt linstor volume-definition delete TestVol01 0 linstor resource-definition delete TestVol01 ====== 6.冗長化構成 ====== linstorの情報は、/var/lib/linstorに入っているので、これをどのノードでも使えるように冗長化する。 drbd-reactor で/var/lib/linstorをマウントしてないと、linstor-controllerが起動しないようにする仕組みです。 これで、3つのノードどれかで、linstore-controllerが起動して、その他のlinstore-controllerは起動しないという設定が可能となります。 ===== linstor_dbボリュームの用意 ===== linstor resource-definition create linstor_db linstor resource-definition drbd-options --on-no-quorum=io-error linstor_db linstor resource-definition drbd-options --auto-promote=no linstor_db linstor volume-definition create linstor_db 200M linstor resource create linstor_db -s linstor-pool --auto-place 3 ==== もしlinstor-controllerが動いている場合は、disableにして停止 ==== systemctl disable --now linstor-controller ==== linstore_dbマウントするsystemd用意 ==== cat << EOF > /etc/systemd/system/var-lib-linstor.mount [Unit] Description=Filesystem for the LINSTOR controller [Mount] # you can use the minor like /dev/drbdX or the udev symlink What=/dev/drbd/by-res/linstor_db/0 Where=/var/lib/linstor EOF ==== linstor_dbをマウントしてlinstor-controller起動 ==== mv /var/lib/linstor{,.orig} mkdir /var/lib/linstor chattr +i /var/lib/linstor # only if on LINSTOR >= 1.14.0 drbdadm primary linstor_db mkfs.ext4 /dev/drbd/by-res/linstor_db/0 systemctl start var-lib-linstor.mount cp -r /var/lib/linstor.orig/* /var/lib/linstor systemctl start linstor-controller ===== drbd-reactor用意 ===== ※下記の手順は全てのノードで実行する。 ==== install ==== apt install drbd-reactor ==== 起動順のファイル作成 ==== cat << EOF > /etc/drbd-reactor.d/linstor_db.toml [[promoter]] id = "linstor_db" [promoter.resources.linstor_db] start = ["var-lib-linstor.mount", "linstor-controller.service"] EOF ==== 一度drbd-reactor再起動 ==== systemctl restart drbd-reactor systemctl enable drbd-reactor ===== linstor-satellite修正 ===== linstor_dbボリュームに変更を加えないように設定変更 # systemctl edit linstor-satellite [Service] Environment=LS_KEEP_RES=linstor_db ===== linstor-client.conf修正 ===== どのノードでlinstor-controllerが動いていても良いように、linstor-client.confを修正しておく。 # cat /etc/linstor/linstor-client.conf [global] controllers=node1,node2,node3 ===== 確認 ===== linstor_dbがPrimaryのノードでしか、linstor-controllerが起動しない事を確認 # linstor controller which linstor://node2 # drbdadm status linstor_db role:Secondary disk:UpToDate worker02 role:Primary peer-disk:UpToDate worker03 role:Secondary peer-disk:UpToDate node1# systemctl start linstor-controller.service A dependency job for linstor-controller.service failed. See 'journalctl -xe' for details. node2# systemctl stop linstor-controller.service node2# umount /var/lib/linstor node2# # drbdadm secondary linstor_db node1# drbdadm primary linstor_db node1# # systemctl start linstor-controller.service # linstor controller which linstor://node1 ===== 確認（正常な動作） ===== # cat /proc/drbd version: 9.0.32-1 (api:2/proto:86-121) # linstor --version linstor 1.13.0 # drbd-reactor --version drbd-reactor 0.6.0 # linstor controller which linstor://node02 ## ノード再起動や、drbd-reactorを停止した状態で、linstor-controllerが停止 node02:~# systemctl stop drbd-reactor.service node02:~# systemctl stop linstor-controller.service ## 即時に別のノードで、linstor-controllerが起動 node01:~# linstor controller which linstor://linstor-node01 ====== 7.Kubernetsから接続 ====== ==== 大前提条件 ==== Kubernetes helm インストール時下記の３つがlinstorインストールの前提の条件です。 - kubernetes controller からPodのIPに通信できる事 - Pod内から外部へ通信可能 - Pod内で名前解決可能 root@linstor-master:~# kubectl get pod alpine-test -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES alpine-test 1/1 Running 0 73s 10.224.1.2 linstor-node04 root@linstor-master:~# ping 10.224.1.2 -c 3 PING 10.224.1.2 (10.224.1.2) 56(84) bytes of data. 64 bytes from 10.224.1.2: icmp_seq=1 ttl=63 time=0.959 ms 64 bytes from 10.224.1.2: icmp_seq=2 ttl=63 time=0.679 ms 64 bytes from 10.224.1.2: icmp_seq=3 ttl=63 time=0.506 ms --- 10.224.1.2 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2019ms rtt min/avg/max/mdev = 0.506/0.714/0.959/0.186 ms root@linstor-master:~# kubectl exec -it alpine-test -- ping fl8.jp -c 3 PING fl8.jp (182.48.51.190): 56 data bytes 64 bytes from 182.48.51.190: seq=0 ttl=53 time=8.568 ms 64 bytes from 182.48.51.190: seq=1 ttl=53 time=8.715 ms 64 bytes from 182.48.51.190: seq=2 ttl=53 time=8.532 ms --- fl8.jp ping statistics --- 3 packets transmitted, 3 packets received, 0% packet loss round-trip min/avg/max = 8.532/8.605/8.715 ms ==== secret登録 ==== kubectl create secret docker-registry drbdiocred --docker-server=drbd.io \ --docker-username=[User] --docker-password=[Pass] ==== helm repository登録 ==== helm repo add linstor https://charts.linstor.io # helm repo list NAME URL linstor https://charts.linstor.io ==== linstor-opインストール ==== helm install linstor-op linstor/linstor \ --set operator.controller.enabled=false \ --set etcd.persistentVolume.enabled=false \ --set operator.etcd.enabled=false \ --set controllerEndpoint=http://172.16.0.65:3370 ==== 確認 ==== ちゃんとpodが起動してればOK # kubectl get pod NAME READY STATUS RESTARTS AGE alpine-test 1/1 Running 0 9h linstor-op-csi-controller-76d5bcfcb9-z2mrw 6/6 Running 0 9h linstor-op-csi-node-54wf6 3/3 Running 0 9h linstor-op-csi-node-9fcn9 3/3 Running 0 9h linstor-op-csi-node-rw4h5 3/3 Running 0 9h linstor-op-etcd-0 1/1 Running 0 9h linstor-op-ha-controller-6f9784f965-bqqcp 1/1 Running 0 9h linstor-op-ns-node-74zjb 0/2 Init:CrashLoopBackOff 111 (3m4s ago) 9h linstor-op-ns-node-ldc5q 0/2 Init:CrashLoopBackOff 111 (3m51s ago) 9h linstor-op-ns-node-m7hcm 0/2 Init:CrashLoopBackOff 111 (3m28s ago) 9h linstor-op-operator-7b9b9b685d-mnc8z 1/1 Running 0 9h ubuntu 1/1 Running 0 8h ubuntu-7c969d5f7d-lq96g 1/1 Running 0 8h ==== StorageClass作成 ==== Kubernetes から利用する為に、StorageClass作成 apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: # The name used to identify this StorageClass. name: linstor-basic-storage-class # The name used to match this StorageClass with a provisioner. # linstor.csi.linbit.com is the name that the LINSTOR CSI plug-in uses to identify itself provisioner: linstor.csi.linbit.com volumeBindingMode: WaitForFirstConsumer parameters: # LINSTOR will provision volumes from the drbdpool storage pool configured # On the satellite nodes in the LINSTOR cluster specified in the plug-in's deployment storagePool: "linstor-pool" resourceGroup: "linstor-basic-storage-class" # Setting a fstype is required for "fsGroup" permissions to work correctly. # Currently supported: xfs/ext4 csi.storage.k8s.io/fstype: xfs linstor.csi.linbit.com/autoPlace: "3" ==== PVC作成 ==== kind: PersistentVolumeClaim apiVersion: v1 metadata: name: linstor-volume-01 spec: storageClassName: linstor-basic-storage-class accessModes: - ReadWriteOnce resources: requests: storage: 3Gi === PVC作成した時点 === PVC作成した時点ではまだPVは作成してない。 # kubectl apply -f pvc.yaml persistentvolumeclaim/linstor-volume-01 created # kubectl get pvc NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE linstor-volume-01 Pending linstor-basic-storage-class 4s # kubectl get pv No resources found # linstor volume list -p +------------------------------------------------------------------------------------------+ | Node | Resource | StoragePool | VolNr | MinorNr | DeviceName | Allocated | InUse | State | |==========================================================================================| +------------------------------------------------------------------------------------------+ ==== PodからPV利用 ==== apiVersion: v1 kind: Pod metadata: name: ubuntu spec: containers: - image: ubuntu name: ubuntu command: [ "/bin/bash", "-c", "--" ] args: [ "while true; do sleep 30; done;" ] volumeMounts: - name: linstor-volume mountPath: /data ports: - containerPort: 80 volumes: - name: linstor-volume persistentVolumeClaim: claimName: "linstor-volume-01" === Pod作成すると、PVも作成される === kubectl apply -f ubutnu.yaml # kubectl get pv NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE pvc-c83e52e0-f8c7-447c-a3f2-8f01daaeb4c2 3Gi RWO Delete Bound default/linstor-volume-01 linstor-basic-storage-class 82s # kubectl get pvc NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE linstor-volume-01 Bound pvc-c83e52e0-f8c7-447c-a3f2-8f01daaeb4c2 3Gi RWO linstor-basic-storage-class 4m54s # linstor volume list -p +--------------------------------------------------------------------------------------------------------------------------------------------+ | Node | Resource | StoragePool | VolNr | MinorNr | DeviceName | Allocated | InUse | State | |============================================================================================================================================| | linstor-node04 | pvc-c83e52e0-f8c7-447c-a3f2-8f01daaeb4c2 | linstor-pool | 0 | 1000 | /dev/drbd1000 | 5.23 MiB | Unused | UpToDate | | linstor-node05 | pvc-c83e52e0-f8c7-447c-a3f2-8f01daaeb4c2 | linstor-pool | 0 | 1000 | /dev/drbd1000 | 4.61 MiB | Unused | UpToDate | | linstor-node06 | pvc-c83e52e0-f8c7-447c-a3f2-8f01daaeb4c2 | linstor-pool | 0 | 1000 | /dev/drbd1000 | 11.38 MiB | Unused | UpToDate | +--------------------------------------------------------------------------------------------------------------------------------------------+ === PVC削除 === PVCを削除すると、ちゃんとlinstorで削除されている。 # kubectl delete pod ubuntu pod "ubuntu" deleted # kubectl get pvc NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE linstor-volume-01 Bound pvc-06ac4732-f3f4-46fe-a0b6-6f58a9895cb2 3Gi RWO linstor-basic-storage-class 29m # kubectl delete pvc linstor-volume-01 persistentvolumeclaim "linstor-volume-01" deleted # linstor volume list -p +------------------------------------------------------------------------------------------+ | Node | Resource | StoragePool | VolNr | MinorNr | DeviceName | Allocated | InUse | State | |==========================================================================================| +------------------------------------------------------------------------------------------+ {{tag>Kuberneted DRBD}}