目次

16 Kubernetes + DRBD Linstor

Kubernetes用の冗長化ストレージとして、Linstorを用意してみる。

LinstorはLINBITが開発した、DRBDを管理する管理DBを持つlinstor-controllerとlinstor-satelliteというデータディスクを扱うプログラムで構成されてます。

https://github.com/LINBIT/linstor-server

1.環境

今回はKubernetesのworkerノード3台で、Linstor

name IP Disk DRBD用
node1172.16.0.65/dev/sda/dev/sdb
node2172.16.0.252/dev/sda/dev/sdb
node3172.16.0.234/dev/sda/dev/sdb

※sdaは/用、sdbはlinstor用

2. DRBDインストール

add-apt-repository ppa:linbit/linbit-drbd9-stack
apt-get update && apt install drbd-utils drbd-dkms lvm2
modprobe drbd

DRBD確認

# cat /proc/drbd
version: 9.1.6 (api:2/proto:110-121)
GIT-hash: f85adeb71f16a0aead1e875665e2fb68852d94eb build by root@worker01, 2022-04-15 21:43:08
Transports (api:17): tcp (9.1.6)

3. LINSTORインストール

今回は3台でlinstor-controllerを冗長化するので、3台とも全部インストール

apt install linstor-controller linstor-satellite linstor-client

linstor-controllerはDisable

systemctl disable linstor-controller

自動補完

# source /etc/bash_completion.d/linstor # または
# source /usr/share/bash_completion/completions/linstor

4. Linstor初期化

1度node1で、linstor-controllerを上げて初期設定を行います。

node1# systemctl start linstor-controller

linstor node create node1 172.16.0.65
linstor node create node2 172.16.0.252
linstor node create node3 172.16.0.234

確認

node1# linstor node list 
╭────────────────────────────────────────────────────────╮
┊ Node  ┊ NodeType  ┊ Addresses                 ┊ State  ┊
╞════════════════════════════════════════════════════════╡
┊ node1 ┊ SATELLITE ┊ 172.16.0.65:3366 (PLAIN)  ┊ Online ┊
┊ node2 ┊ SATELLITE ┊ 172.16.0.252:3366 (PLAIN) ┊ Online ┊
┊ node3 ┊ SATELLITE ┊ 172.16.0.234:3366 (PLAIN) ┊ Online ┊
╰────────────────────────────────────────────────────────╯

5.Storage pool 設定

こちらも、linstor-controllerを起動しているnode1で実行する。

sdbでLVMを用意してlinstorのストレージプールに利用する。

pvcreate /dev/sdb
vgcreate vg /dev/sdb
lvcreate -l 100%FREE --thinpool vg/lvmthinpool

linstor storage-pool create lvmthin node1 linstor-pool vg/lvmthinpool
linstor storage-pool create lvmthin node2 linstor-pool vg/lvmthinpool
linstor storage-pool create lvmthin node3 linstor-pool vg/lvmthinpool

storage-pool確認

node1# linstor storage-pool list -p 
+-----------------------------------------------------------------------------------------------------------------------------+
| StoragePool          | Node  | Driver   | PoolName       | FreeCapacity | TotalCapacity | CanSnapshots | State | SharedName |
|=============================================================================================================================|
| DfltDisklessStorPool | node1 | DISKLESS |                |              |               | False        | Ok    |            |
| DfltDisklessStorPool | node2 | DISKLESS |                |              |               | False        | Ok    |            |
| DfltDisklessStorPool | node3 | DISKLESS |                |              |               | False        | Ok    |            |
| linstor-pool         | node1 | LVM_THIN | vg/lvmthinpool |    19.90 GiB |     19.96 GiB | True         | Ok    |            |
| linstor-pool         | node2 | LVM_THIN | vg/lvmthinpool |    19.90 GiB |     19.96 GiB | True         | Ok    |            |
| linstor-pool         | node3 | LVM_THIN | vg/lvmthinpool |    19.90 GiB |     19.96 GiB | True         | Ok    |            |
+-----------------------------------------------------------------------------------------------------------------------------+

1つVolumeを作ってみる

TestVol01を作成

流れは

  1. リソース定義を作成
  2. ボリューム定義を作成
  3. リソースを作成
linstor resource-definition create TestVol01
linstor volume-definition create TestVol01 1G
linstor resource create TestVol01 -s linstor-pool --auto-place 3

たとえば、node2,node3にリソースを作成して、node1はディスクレスにする場合

linstor resource create node2 node3 TestVol01 -s linstor-pool
linstor resource create node1 TestVol01 --diskless

確認

# linstor volume list -r TestVol01
+----------------------------------------------------------------------------------------------------+
| Node  | Resource  | StoragePool  | VolNr | MinorNr | DeviceName    | Allocated | InUse  |    State |
|====================================================================================================|
| node1 | TestVol01 | linstor-pool |     0 |    1002 | /dev/drbd1002 |   315 KiB | Unused | UpToDate |
| node2 | TestVol01 | linstor-pool |     0 |    1002 | /dev/drbd1002 |   315 KiB | Unused | UpToDate |
| node3 | TestVol01 | linstor-pool |     0 |    1002 | /dev/drbd1002 |   315 KiB | Unused | UpToDate |
+----------------------------------------------------------------------------------------------------+

ディスクを使ってみる

mkfs.xfs /dev/drbd1002
mount /dev/drbd1002 /mnt/
# df /mnt
Filesystem     1K-blocks  Used Available Use% Mounted on
/dev/drbd1002    1042168 40604   1001564   4% /mnt

マウントしているnode1のInUseがInUseに変更して、ディスクを利用している事が分かる

# linstor volume list -r TestVol01 | sed 's/^/  /g'
+----------------------------------------------------------------------------------------------------+
| Node  | Resource  | StoragePool  | VolNr | MinorNr | DeviceName    | Allocated | InUse  |    State |
|====================================================================================================|
| node1 | TestVol01 | linstor-pool |     0 |    1002 | /dev/drbd1002 | 11.00 MiB | InUse  | UpToDate |
| node2 | TestVol01 | linstor-pool |     0 |    1002 | /dev/drbd1002 | 11.00 MiB | Unused | UpToDate |
| node3 | TestVol01 | linstor-pool |     0 |    1002 | /dev/drbd1002 | 11.00 MiB | Unused | UpToDate |
+----------------------------------------------------------------------------------------------------+

削除

umount /mnt
linstor volume-definition delete TestVol01 0
linstor resource-definition delete TestVol01

6.冗長化構成

linstorの情報は、/var/lib/linstorに入っているので、これをどのノードでも使えるように冗長化する。

drbd-reactor で/var/lib/linstorをマウントしてないと、linstor-controllerが起動しないようにする仕組みです。

これで、3つのノードどれかで、linstore-controllerが起動して、その他のlinstore-controllerは起動しないという設定が可能となります。

linstor_dbボリュームの用意

linstor resource-definition create linstor_db
linstor resource-definition drbd-options --on-no-quorum=io-error linstor_db
linstor resource-definition drbd-options --auto-promote=no linstor_db
linstor volume-definition create linstor_db 200M
linstor resource create linstor_db -s linstor-pool --auto-place 3

もしlinstor-controllerが動いている場合は、disableにして停止

systemctl disable --now linstor-controller

linstore_dbマウントするsystemd用意

cat << EOF > /etc/systemd/system/var-lib-linstor.mount
[Unit]
Description=Filesystem for the LINSTOR controller

[Mount]
# you can use the minor like /dev/drbdX or the udev symlink
What=/dev/drbd/by-res/linstor_db/0
Where=/var/lib/linstor
EOF

linstor_dbをマウントしてlinstor-controller起動

 mv /var/lib/linstor{,.orig}
 mkdir /var/lib/linstor
 chattr +i /var/lib/linstor  # only if on LINSTOR >= 1.14.0
 drbdadm primary linstor_db
 mkfs.ext4 /dev/drbd/by-res/linstor_db/0
 systemctl start var-lib-linstor.mount
 cp -r /var/lib/linstor.orig/* /var/lib/linstor
 systemctl start linstor-controller

drbd-reactor用意

※下記の手順は全てのノードで実行する。

install

apt install drbd-reactor

起動順のファイル作成

cat << EOF > /etc/drbd-reactor.d/linstor_db.toml
[[promoter]]
id = "linstor_db"
[promoter.resources.linstor_db]
start = ["var-lib-linstor.mount", "linstor-controller.service"]
EOF

一度drbd-reactor再起動

systemctl restart drbd-reactor
systemctl enable drbd-reactor

linstor-satellite修正

linstor_dbボリュームに変更を加えないように設定変更

# systemctl edit linstor-satellite
[Service]
Environment=LS_KEEP_RES=linstor_db

linstor-client.conf修正

どのノードでlinstor-controllerが動いていても良いように、linstor-client.confを修正しておく。

# cat /etc/linstor/linstor-client.conf
[global]
controllers=node1,node2,node3

確認

linstor_dbがPrimaryのノードでしか、linstor-controllerが起動しない事を確認

# linstor controller which 
linstor://node2

# drbdadm status
linstor_db role:Secondary
  disk:UpToDate
  worker02 role:Primary
    peer-disk:UpToDate
  worker03 role:Secondary
    peer-disk:UpToDate

node1# systemctl start linstor-controller.service 
A dependency job for linstor-controller.service failed. See 'journalctl -xe' for details.

node2# systemctl stop linstor-controller.service
node2# umount /var/lib/linstor
node2# # drbdadm secondary linstor_db

node1# drbdadm primary linstor_db
node1# # systemctl start linstor-controller.service

# linstor controller which 
linstor://node1

確認(正常な動作)

# cat /proc/drbd 
version: 9.0.32-1 (api:2/proto:86-121)
# linstor --version
linstor 1.13.0
# drbd-reactor --version
drbd-reactor 0.6.0
# linstor controller which 
linstor://node02

## ノード再起動や、drbd-reactorを停止した状態で、linstor-controllerが停止
node02:~# systemctl stop drbd-reactor.service
node02:~# systemctl stop linstor-controller.service

## 即時に別のノードで、linstor-controllerが起動
node01:~# linstor controller which 
linstor://linstor-node01

7.Kubernetsから接続

大前提条件

Kubernetes helm インストール時下記の3つがlinstorインストールの前提の条件です。

  1. kubernetes controller からPodのIPに通信できる事
  2. Pod内から外部へ通信可能
  3. Pod内で名前解決可能
root@linstor-master:~# kubectl get pod alpine-test -o wide
NAME          READY   STATUS    RESTARTS   AGE   IP           NODE             NOMINATED NODE   READINESS GATES
alpine-test   1/1     Running   0          73s   10.224.1.2   linstor-node04   <none>           <none>
root@linstor-master:~# ping 10.224.1.2 -c 3
PING 10.224.1.2 (10.224.1.2) 56(84) bytes of data.
64 bytes from 10.224.1.2: icmp_seq=1 ttl=63 time=0.959 ms
64 bytes from 10.224.1.2: icmp_seq=2 ttl=63 time=0.679 ms
64 bytes from 10.224.1.2: icmp_seq=3 ttl=63 time=0.506 ms

--- 10.224.1.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2019ms
rtt min/avg/max/mdev = 0.506/0.714/0.959/0.186 ms

root@linstor-master:~# kubectl exec -it alpine-test -- ping fl8.jp -c 3
PING fl8.jp (182.48.51.190): 56 data bytes
64 bytes from 182.48.51.190: seq=0 ttl=53 time=8.568 ms
64 bytes from 182.48.51.190: seq=1 ttl=53 time=8.715 ms
64 bytes from 182.48.51.190: seq=2 ttl=53 time=8.532 ms

--- fl8.jp ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 8.532/8.605/8.715 ms

secret登録

kubectl create secret docker-registry drbdiocred --docker-server=drbd.io \
  --docker-username=[User] --docker-password=[Pass]

helm repository登録

helm repo add linstor https://charts.linstor.io
# helm repo list
NAME   	URL
linstor	https://charts.linstor.io

linstor-opインストール

helm install linstor-op linstor/linstor \
--set operator.controller.enabled=false \
--set etcd.persistentVolume.enabled=false \
--set operator.etcd.enabled=false \
--set controllerEndpoint=http://172.16.0.65:3370

確認

ちゃんとpodが起動してればOK

# kubectl get pod
NAME                                         READY   STATUS                  RESTARTS          AGE
alpine-test                                  1/1     Running                 0                 9h
linstor-op-csi-controller-76d5bcfcb9-z2mrw   6/6     Running                 0                 9h
linstor-op-csi-node-54wf6                    3/3     Running                 0                 9h
linstor-op-csi-node-9fcn9                    3/3     Running                 0                 9h
linstor-op-csi-node-rw4h5                    3/3     Running                 0                 9h
linstor-op-etcd-0                            1/1     Running                 0                 9h
linstor-op-ha-controller-6f9784f965-bqqcp    1/1     Running                 0                 9h
linstor-op-ns-node-74zjb                     0/2     Init:CrashLoopBackOff   111 (3m4s ago)    9h
linstor-op-ns-node-ldc5q                     0/2     Init:CrashLoopBackOff   111 (3m51s ago)   9h
linstor-op-ns-node-m7hcm                     0/2     Init:CrashLoopBackOff   111 (3m28s ago)   9h
linstor-op-operator-7b9b9b685d-mnc8z         1/1     Running                 0                 9h
ubuntu                                       1/1     Running                 0                 8h
ubuntu-7c969d5f7d-lq96g                      1/1     Running                 0                 8h

StorageClass作成

Kubernetes から利用する為に、StorageClass作成

StorageClass.yaml

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  # The name used to identify this StorageClass.
  name: linstor-basic-storage-class
  # The name used to match this StorageClass with a provisioner.
  # linstor.csi.linbit.com is the name that the LINSTOR CSI plug-in uses to identify itself
provisioner: linstor.csi.linbit.com
volumeBindingMode: WaitForFirstConsumer
parameters:
  # LINSTOR will provision volumes from the drbdpool storage pool configured
  # On the satellite nodes in the LINSTOR cluster specified in the plug-in's deployment
  storagePool: "linstor-pool"
  resourceGroup: "linstor-basic-storage-class"
  # Setting a fstype is required for "fsGroup" permissions to work correctly.
  # Currently supported: xfs/ext4
  csi.storage.k8s.io/fstype: xfs
  linstor.csi.linbit.com/autoPlace: "3"

PVC作成

pvc.yaml

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: linstor-volume-01
spec:
  storageClassName: linstor-basic-storage-class
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 3Gi

PVC作成した時点

PVC作成した時点ではまだPVは作成してない。

# kubectl apply -f pvc.yaml 
persistentvolumeclaim/linstor-volume-01 created

# kubectl get pvc
NAME                STATUS    VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS                  AGE
linstor-volume-01   Pending                                      linstor-basic-storage-class   4s

# kubectl get pv
No resources found

# linstor volume list -p
+------------------------------------------------------------------------------------------+
| Node | Resource | StoragePool | VolNr | MinorNr | DeviceName | Allocated | InUse | State |
|==========================================================================================|
+------------------------------------------------------------------------------------------+

PodからPV利用

ubutnu.yaml

apiVersion: v1
kind: Pod
metadata:
  name: ubuntu
spec:
  containers:
  - image: ubuntu
    name: ubuntu
    command: [ "/bin/bash", "-c", "--" ]
    args: [ "while true; do sleep 30; done;" ]
    volumeMounts:
    - name: linstor-volume
      mountPath: /data
    ports:
    - containerPort: 80
  volumes:
  - name: linstor-volume
    persistentVolumeClaim:
      claimName: "linstor-volume-01"

Pod作成すると、PVも作成される

kubectl apply -f ubutnu.yaml

# kubectl get pv
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                       STORAGECLASS                  REASON   AGE
pvc-c83e52e0-f8c7-447c-a3f2-8f01daaeb4c2   3Gi        RWO            Delete           Bound    default/linstor-volume-01   linstor-basic-storage-class            82s

# kubectl get pvc
NAME                STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS                  AGE
linstor-volume-01   Bound    pvc-c83e52e0-f8c7-447c-a3f2-8f01daaeb4c2   3Gi        RWO            linstor-basic-storage-class   4m54s

# linstor volume list -p
+--------------------------------------------------------------------------------------------------------------------------------------------+
| Node           | Resource                                 | StoragePool  | VolNr | MinorNr | DeviceName    | Allocated | InUse  |    State |
|============================================================================================================================================|
| linstor-node04 | pvc-c83e52e0-f8c7-447c-a3f2-8f01daaeb4c2 | linstor-pool |     0 |    1000 | /dev/drbd1000 |  5.23 MiB | Unused | UpToDate |
| linstor-node05 | pvc-c83e52e0-f8c7-447c-a3f2-8f01daaeb4c2 | linstor-pool |     0 |    1000 | /dev/drbd1000 |  4.61 MiB | Unused | UpToDate |
| linstor-node06 | pvc-c83e52e0-f8c7-447c-a3f2-8f01daaeb4c2 | linstor-pool |     0 |    1000 | /dev/drbd1000 | 11.38 MiB | Unused | UpToDate |
+--------------------------------------------------------------------------------------------------------------------------------------------+

PVC削除

PVCを削除すると、ちゃんとlinstorで削除されている。

# kubectl delete pod ubuntu 
pod "ubuntu" deleted

# kubectl get pvc
NAME                STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS                  AGE
linstor-volume-01   Bound    pvc-06ac4732-f3f4-46fe-a0b6-6f58a9895cb2   3Gi        RWO            linstor-basic-storage-class   29m

# kubectl delete pvc linstor-volume-01 
persistentvolumeclaim "linstor-volume-01" deleted

# linstor volume list -p
+------------------------------------------------------------------------------------------+
| Node | Resource | StoragePool | VolNr | MinorNr | DeviceName | Allocated | InUse | State |
|==========================================================================================|
+------------------------------------------------------------------------------------------+