目次
16 Kubernetes + DRBD Linstor
Kubernetes用の冗長化ストレージとして、Linstorを用意してみる。
LinstorはLINBITが開発した、DRBDを管理する管理DBを持つlinstor-controllerとlinstor-satelliteというデータディスクを扱うプログラムで構成されてます。
https://github.com/LINBIT/linstor-server
1.環境
今回はKubernetesのworkerノード3台で、Linstor
| name | IP | Disk | DRBD用 |
|---|---|---|---|
| node1 | 172.16.0.65 | /dev/sda | /dev/sdb |
| node2 | 172.16.0.252 | /dev/sda | /dev/sdb |
| node3 | 172.16.0.234 | /dev/sda | /dev/sdb |
※sdaは/用、sdbはlinstor用
2. DRBDインストール
add-apt-repository ppa:linbit/linbit-drbd9-stack apt-get update && apt install drbd-utils drbd-dkms lvm2 modprobe drbd
DRBD確認
# cat /proc/drbd version: 9.1.6 (api:2/proto:110-121) GIT-hash: f85adeb71f16a0aead1e875665e2fb68852d94eb build by root@worker01, 2022-04-15 21:43:08 Transports (api:17): tcp (9.1.6)
3. LINSTORインストール
今回は3台でlinstor-controllerを冗長化するので、3台とも全部インストール
apt install linstor-controller linstor-satellite linstor-client
linstor-controllerはDisable
systemctl disable linstor-controller
自動補完
# source /etc/bash_completion.d/linstor # または # source /usr/share/bash_completion/completions/linstor
4. Linstor初期化
1度node1で、linstor-controllerを上げて初期設定を行います。
node1# systemctl start linstor-controller linstor node create node1 172.16.0.65 linstor node create node2 172.16.0.252 linstor node create node3 172.16.0.234
確認
node1# linstor node list ╭────────────────────────────────────────────────────────╮ ┊ Node ┊ NodeType ┊ Addresses ┊ State ┊ ╞════════════════════════════════════════════════════════╡ ┊ node1 ┊ SATELLITE ┊ 172.16.0.65:3366 (PLAIN) ┊ Online ┊ ┊ node2 ┊ SATELLITE ┊ 172.16.0.252:3366 (PLAIN) ┊ Online ┊ ┊ node3 ┊ SATELLITE ┊ 172.16.0.234:3366 (PLAIN) ┊ Online ┊ ╰────────────────────────────────────────────────────────╯
5.Storage pool 設定
こちらも、linstor-controllerを起動しているnode1で実行する。
sdbでLVMを用意してlinstorのストレージプールに利用する。
pvcreate /dev/sdb vgcreate vg /dev/sdb lvcreate -l 100%FREE --thinpool vg/lvmthinpool linstor storage-pool create lvmthin node1 linstor-pool vg/lvmthinpool linstor storage-pool create lvmthin node2 linstor-pool vg/lvmthinpool linstor storage-pool create lvmthin node3 linstor-pool vg/lvmthinpool
storage-pool確認
node1# linstor storage-pool list -p +-----------------------------------------------------------------------------------------------------------------------------+ | StoragePool | Node | Driver | PoolName | FreeCapacity | TotalCapacity | CanSnapshots | State | SharedName | |=============================================================================================================================| | DfltDisklessStorPool | node1 | DISKLESS | | | | False | Ok | | | DfltDisklessStorPool | node2 | DISKLESS | | | | False | Ok | | | DfltDisklessStorPool | node3 | DISKLESS | | | | False | Ok | | | linstor-pool | node1 | LVM_THIN | vg/lvmthinpool | 19.90 GiB | 19.96 GiB | True | Ok | | | linstor-pool | node2 | LVM_THIN | vg/lvmthinpool | 19.90 GiB | 19.96 GiB | True | Ok | | | linstor-pool | node3 | LVM_THIN | vg/lvmthinpool | 19.90 GiB | 19.96 GiB | True | Ok | | +-----------------------------------------------------------------------------------------------------------------------------+
1つVolumeを作ってみる
TestVol01を作成
流れは
- リソース定義を作成
- ボリューム定義を作成
- リソースを作成
linstor resource-definition create TestVol01 linstor volume-definition create TestVol01 1G linstor resource create TestVol01 -s linstor-pool --auto-place 3
たとえば、node2,node3にリソースを作成して、node1はディスクレスにする場合
linstor resource create node2 node3 TestVol01 -s linstor-pool linstor resource create node1 TestVol01 --diskless
確認
# linstor volume list -r TestVol01 +----------------------------------------------------------------------------------------------------+ | Node | Resource | StoragePool | VolNr | MinorNr | DeviceName | Allocated | InUse | State | |====================================================================================================| | node1 | TestVol01 | linstor-pool | 0 | 1002 | /dev/drbd1002 | 315 KiB | Unused | UpToDate | | node2 | TestVol01 | linstor-pool | 0 | 1002 | /dev/drbd1002 | 315 KiB | Unused | UpToDate | | node3 | TestVol01 | linstor-pool | 0 | 1002 | /dev/drbd1002 | 315 KiB | Unused | UpToDate | +----------------------------------------------------------------------------------------------------+
ディスクを使ってみる
mkfs.xfs /dev/drbd1002 mount /dev/drbd1002 /mnt/ # df /mnt Filesystem 1K-blocks Used Available Use% Mounted on /dev/drbd1002 1042168 40604 1001564 4% /mnt
マウントしているnode1のInUseがInUseに変更して、ディスクを利用している事が分かる
# linstor volume list -r TestVol01 | sed 's/^/ /g' +----------------------------------------------------------------------------------------------------+ | Node | Resource | StoragePool | VolNr | MinorNr | DeviceName | Allocated | InUse | State | |====================================================================================================| | node1 | TestVol01 | linstor-pool | 0 | 1002 | /dev/drbd1002 | 11.00 MiB | InUse | UpToDate | | node2 | TestVol01 | linstor-pool | 0 | 1002 | /dev/drbd1002 | 11.00 MiB | Unused | UpToDate | | node3 | TestVol01 | linstor-pool | 0 | 1002 | /dev/drbd1002 | 11.00 MiB | Unused | UpToDate | +----------------------------------------------------------------------------------------------------+
削除
umount /mnt linstor volume-definition delete TestVol01 0 linstor resource-definition delete TestVol01
6.冗長化構成
linstorの情報は、/var/lib/linstorに入っているので、これをどのノードでも使えるように冗長化する。
drbd-reactor で/var/lib/linstorをマウントしてないと、linstor-controllerが起動しないようにする仕組みです。
これで、3つのノードどれかで、linstore-controllerが起動して、その他のlinstore-controllerは起動しないという設定が可能となります。
linstor_dbボリュームの用意
linstor resource-definition create linstor_db linstor resource-definition drbd-options --on-no-quorum=io-error linstor_db linstor resource-definition drbd-options --auto-promote=no linstor_db linstor volume-definition create linstor_db 200M linstor resource create linstor_db -s linstor-pool --auto-place 3
もしlinstor-controllerが動いている場合は、disableにして停止
systemctl disable --now linstor-controller
linstore_dbマウントするsystemd用意
cat << EOF > /etc/systemd/system/var-lib-linstor.mount [Unit] Description=Filesystem for the LINSTOR controller [Mount] # you can use the minor like /dev/drbdX or the udev symlink What=/dev/drbd/by-res/linstor_db/0 Where=/var/lib/linstor EOF
linstor_dbをマウントしてlinstor-controller起動
mv /var/lib/linstor{,.orig}
mkdir /var/lib/linstor
chattr +i /var/lib/linstor # only if on LINSTOR >= 1.14.0
drbdadm primary linstor_db
mkfs.ext4 /dev/drbd/by-res/linstor_db/0
systemctl start var-lib-linstor.mount
cp -r /var/lib/linstor.orig/* /var/lib/linstor
systemctl start linstor-controller
drbd-reactor用意
※下記の手順は全てのノードで実行する。
install
apt install drbd-reactor
起動順のファイル作成
cat << EOF > /etc/drbd-reactor.d/linstor_db.toml [[promoter]] id = "linstor_db" [promoter.resources.linstor_db] start = ["var-lib-linstor.mount", "linstor-controller.service"] EOF
一度drbd-reactor再起動
systemctl restart drbd-reactor systemctl enable drbd-reactor
linstor-satellite修正
linstor_dbボリュームに変更を加えないように設定変更
# systemctl edit linstor-satellite [Service] Environment=LS_KEEP_RES=linstor_db
linstor-client.conf修正
どのノードでlinstor-controllerが動いていても良いように、linstor-client.confを修正しておく。
# cat /etc/linstor/linstor-client.conf [global] controllers=node1,node2,node3
確認
linstor_dbがPrimaryのノードでしか、linstor-controllerが起動しない事を確認
# linstor controller which
linstor://node2
# drbdadm status
linstor_db role:Secondary
disk:UpToDate
worker02 role:Primary
peer-disk:UpToDate
worker03 role:Secondary
peer-disk:UpToDate
node1# systemctl start linstor-controller.service
A dependency job for linstor-controller.service failed. See 'journalctl -xe' for details.
node2# systemctl stop linstor-controller.service
node2# umount /var/lib/linstor
node2# # drbdadm secondary linstor_db
node1# drbdadm primary linstor_db
node1# # systemctl start linstor-controller.service
# linstor controller which
linstor://node1
確認（正常な動作）
# cat /proc/drbd version: 9.0.32-1 (api:2/proto:86-121) # linstor --version linstor 1.13.0 # drbd-reactor --version drbd-reactor 0.6.0
# linstor controller which linstor://node02 ## ノード再起動や、drbd-reactorを停止した状態で、linstor-controllerが停止 node02:~# systemctl stop drbd-reactor.service node02:~# systemctl stop linstor-controller.service ## 即時に別のノードで、linstor-controllerが起動 node01:~# linstor controller which linstor://linstor-node01
7.Kubernetsから接続
大前提条件
Kubernetes helm インストール時下記の３つがlinstorインストールの前提の条件です。
- kubernetes controller からPodのIPに通信できる事
- Pod内から外部へ通信可能
- Pod内で名前解決可能
root@linstor-master:~# kubectl get pod alpine-test -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES alpine-test 1/1 Running 0 73s 10.224.1.2 linstor-node04 <none> <none> root@linstor-master:~# ping 10.224.1.2 -c 3 PING 10.224.1.2 (10.224.1.2) 56(84) bytes of data. 64 bytes from 10.224.1.2: icmp_seq=1 ttl=63 time=0.959 ms 64 bytes from 10.224.1.2: icmp_seq=2 ttl=63 time=0.679 ms 64 bytes from 10.224.1.2: icmp_seq=3 ttl=63 time=0.506 ms --- 10.224.1.2 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2019ms rtt min/avg/max/mdev = 0.506/0.714/0.959/0.186 ms root@linstor-master:~# kubectl exec -it alpine-test -- ping fl8.jp -c 3 PING fl8.jp (182.48.51.190): 56 data bytes 64 bytes from 182.48.51.190: seq=0 ttl=53 time=8.568 ms 64 bytes from 182.48.51.190: seq=1 ttl=53 time=8.715 ms 64 bytes from 182.48.51.190: seq=2 ttl=53 time=8.532 ms --- fl8.jp ping statistics --- 3 packets transmitted, 3 packets received, 0% packet loss round-trip min/avg/max = 8.532/8.605/8.715 ms
secret登録
kubectl create secret docker-registry drbdiocred --docker-server=drbd.io \ --docker-username=[User] --docker-password=[Pass]
helm repository登録
helm repo add linstor https://charts.linstor.io # helm repo list NAME URL linstor https://charts.linstor.io
linstor-opインストール
helm install linstor-op linstor/linstor \ --set operator.controller.enabled=false \ --set etcd.persistentVolume.enabled=false \ --set operator.etcd.enabled=false \ --set controllerEndpoint=http://172.16.0.65:3370
確認
ちゃんとpodが起動してればOK
# kubectl get pod NAME READY STATUS RESTARTS AGE alpine-test 1/1 Running 0 9h linstor-op-csi-controller-76d5bcfcb9-z2mrw 6/6 Running 0 9h linstor-op-csi-node-54wf6 3/3 Running 0 9h linstor-op-csi-node-9fcn9 3/3 Running 0 9h linstor-op-csi-node-rw4h5 3/3 Running 0 9h linstor-op-etcd-0 1/1 Running 0 9h linstor-op-ha-controller-6f9784f965-bqqcp 1/1 Running 0 9h linstor-op-ns-node-74zjb 0/2 Init:CrashLoopBackOff 111 (3m4s ago) 9h linstor-op-ns-node-ldc5q 0/2 Init:CrashLoopBackOff 111 (3m51s ago) 9h linstor-op-ns-node-m7hcm 0/2 Init:CrashLoopBackOff 111 (3m28s ago) 9h linstor-op-operator-7b9b9b685d-mnc8z 1/1 Running 0 9h ubuntu 1/1 Running 0 8h ubuntu-7c969d5f7d-lq96g 1/1 Running 0 8h
StorageClass作成
Kubernetes から利用する為に、StorageClass作成
StorageClass.yaml
apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: # The name used to identify this StorageClass. name: linstor-basic-storage-class # The name used to match this StorageClass with a provisioner. # linstor.csi.linbit.com is the name that the LINSTOR CSI plug-in uses to identify itself provisioner: linstor.csi.linbit.com volumeBindingMode: WaitForFirstConsumer parameters: # LINSTOR will provision volumes from the drbdpool storage pool configured # On the satellite nodes in the LINSTOR cluster specified in the plug-in's deployment storagePool: "linstor-pool" resourceGroup: "linstor-basic-storage-class" # Setting a fstype is required for "fsGroup" permissions to work correctly. # Currently supported: xfs/ext4 csi.storage.k8s.io/fstype: xfs linstor.csi.linbit.com/autoPlace: "3"
PVC作成
pvc.yaml
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: linstor-volume-01
spec:
storageClassName: linstor-basic-storage-class
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 3Gi
PVC作成した時点
PVC作成した時点ではまだPVは作成してない。
# kubectl apply -f pvc.yaml persistentvolumeclaim/linstor-volume-01 created # kubectl get pvc NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE linstor-volume-01 Pending linstor-basic-storage-class 4s # kubectl get pv No resources found # linstor volume list -p +------------------------------------------------------------------------------------------+ | Node | Resource | StoragePool | VolNr | MinorNr | DeviceName | Allocated | InUse | State | |==========================================================================================| +------------------------------------------------------------------------------------------+
PodからPV利用
ubutnu.yaml
apiVersion: v1
kind: Pod
metadata:
name: ubuntu
spec:
containers:
- image: ubuntu
name: ubuntu
command: [ "/bin/bash", "-c", "--" ]
args: [ "while true; do sleep 30; done;" ]
volumeMounts:
- name: linstor-volume
mountPath: /data
ports:
- containerPort: 80
volumes:
- name: linstor-volume
persistentVolumeClaim:
claimName: "linstor-volume-01"
Pod作成すると、PVも作成される
kubectl apply -f ubutnu.yaml # kubectl get pv NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE pvc-c83e52e0-f8c7-447c-a3f2-8f01daaeb4c2 3Gi RWO Delete Bound default/linstor-volume-01 linstor-basic-storage-class 82s # kubectl get pvc NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE linstor-volume-01 Bound pvc-c83e52e0-f8c7-447c-a3f2-8f01daaeb4c2 3Gi RWO linstor-basic-storage-class 4m54s # linstor volume list -p +--------------------------------------------------------------------------------------------------------------------------------------------+ | Node | Resource | StoragePool | VolNr | MinorNr | DeviceName | Allocated | InUse | State | |============================================================================================================================================| | linstor-node04 | pvc-c83e52e0-f8c7-447c-a3f2-8f01daaeb4c2 | linstor-pool | 0 | 1000 | /dev/drbd1000 | 5.23 MiB | Unused | UpToDate | | linstor-node05 | pvc-c83e52e0-f8c7-447c-a3f2-8f01daaeb4c2 | linstor-pool | 0 | 1000 | /dev/drbd1000 | 4.61 MiB | Unused | UpToDate | | linstor-node06 | pvc-c83e52e0-f8c7-447c-a3f2-8f01daaeb4c2 | linstor-pool | 0 | 1000 | /dev/drbd1000 | 11.38 MiB | Unused | UpToDate | +--------------------------------------------------------------------------------------------------------------------------------------------+
PVC削除
PVCを削除すると、ちゃんとlinstorで削除されている。
# kubectl delete pod ubuntu pod "ubuntu" deleted # kubectl get pvc NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE linstor-volume-01 Bound pvc-06ac4732-f3f4-46fe-a0b6-6f58a9895cb2 3Gi RWO linstor-basic-storage-class 29m # kubectl delete pvc linstor-volume-01 persistentvolumeclaim "linstor-volume-01" deleted # linstor volume list -p +------------------------------------------------------------------------------------------+ | Node | Resource | StoragePool | VolNr | MinorNr | DeviceName | Allocated | InUse | State | |==========================================================================================| +------------------------------------------------------------------------------------------+