Kubernetes用の冗長化ストレージとして、Linstorを用意してみる。

LinstorはLINBITが開発した、DRBDを管理する管理DBを持つlinstor-controllerとlinstor-satelliteというデータディスクを扱うプログラムで構成されてます。

https://github.com/LINBIT/linstor-server

今回はKubernetesのworkerノード3台で、Linstor

※sdaは/用、sdbはlinstor用

add-apt-repository ppa:linbit/linbit-drbd9-stack
apt-get update && apt install drbd-utils drbd-dkms lvm2
modprobe drbd

# cat /proc/drbd
version: 9.1.6 (api:2/proto:110-121)
GIT-hash: f85adeb71f16a0aead1e875665e2fb68852d94eb build by root@worker01, 2022-04-15 21:43:08
Transports (api:17): tcp (9.1.6)

今回は3台でlinstor-controllerを冗長化するので、3台とも全部インストール

apt install linstor-controller linstor-satellite linstor-client

systemctl disable linstor-controller

1度node1で、linstor-controllerを上げて初期設定を行います。

node1# systemctl start linstor-controller

linstor node create node1 172.16.0.65
linstor node create node2 172.16.0.252
linstor node create node3 172.16.0.234

node1# linstor node list 
╭────────────────────────────────────────────────────────╮
┊ Node  ┊ NodeType  ┊ Addresses                 ┊ State  ┊
╞════════════════════════════════════════════════════════╡
┊ node1 ┊ SATELLITE ┊ 172.16.0.65:3366 (PLAIN)  ┊ Online ┊
┊ node2 ┊ SATELLITE ┊ 172.16.0.252:3366 (PLAIN) ┊ Online ┊
┊ node3 ┊ SATELLITE ┊ 172.16.0.234:3366 (PLAIN) ┊ Online ┊
╰────────────────────────────────────────────────────────╯

こちらも、linstor-controllerを起動しているnode1で実行する。

sdbでLVMを用意してlinstorのストレージプールに利用する。

pvcreate /dev/sdb
vgcreate vg /dev/sdb
lvcreate -l 100%FREE --thinpool vg/lvmthinpool

linstor storage-pool create lvmthin node1 linstor-pool vg/lvmthinpool
linstor storage-pool create lvmthin node2 linstor-pool vg/lvmthinpool
linstor storage-pool create lvmthin node3 linstor-pool vg/lvmthinpool

node1# linstor storage-pool list -p 
+-----------------------------------------------------------------------------------------------------------------------------+
| StoragePool          | Node  | Driver   | PoolName       | FreeCapacity | TotalCapacity | CanSnapshots | State | SharedName |
|=============================================================================================================================|
| DfltDisklessStorPool | node1 | DISKLESS |                |              |               | False        | Ok    |            |
| DfltDisklessStorPool | node2 | DISKLESS |                |              |               | False        | Ok    |            |
| DfltDisklessStorPool | node3 | DISKLESS |                |              |               | False        | Ok    |            |
| linstor-pool         | node1 | LVM_THIN | vg/lvmthinpool |    19.90 GiB |     19.96 GiB | True         | Ok    |            |
| linstor-pool         | node2 | LVM_THIN | vg/lvmthinpool |    19.90 GiB |     19.96 GiB | True         | Ok    |            |
| linstor-pool         | node3 | LVM_THIN | vg/lvmthinpool |    19.90 GiB |     19.96 GiB | True         | Ok    |            |
+-----------------------------------------------------------------------------------------------------------------------------+

流れは

1. リソース定義を作成
2. ボリューム定義を作成
3. リソースを作成

linstor resource-definition create TestVol01
linstor volume-definition create TestVol01 1G
linstor resource create TestVol01 -s linstor-pool --auto-place 3

たとえば、node2,node3にリソースを作成して、node1はディスクレスにする場合

linstor resource create node2 node3 TestVol01 -s linstor-pool
linstor resource create node1 TestVol01 --diskless

# linstor volume list -r TestVol01
+----------------------------------------------------------------------------------------------------+
| Node  | Resource  | StoragePool  | VolNr | MinorNr | DeviceName    | Allocated | InUse  |    State |
|====================================================================================================|
| node1 | TestVol01 | linstor-pool |     0 |    1002 | /dev/drbd1002 |   315 KiB | Unused | UpToDate |
| node2 | TestVol01 | linstor-pool |     0 |    1002 | /dev/drbd1002 |   315 KiB | Unused | UpToDate |
| node3 | TestVol01 | linstor-pool |     0 |    1002 | /dev/drbd1002 |   315 KiB | Unused | UpToDate |
+----------------------------------------------------------------------------------------------------+

mkfs.xfs /dev/drbd1002
mount /dev/drbd1002 /mnt/
# df /mnt
Filesystem     1K-blocks  Used Available Use% Mounted on
/dev/drbd1002    1042168 40604   1001564   4% /mnt

# linstor volume list -r TestVol01 | sed 's/^/  /g'
+----------------------------------------------------------------------------------------------------+
| Node  | Resource  | StoragePool  | VolNr | MinorNr | DeviceName    | Allocated | InUse  |    State |
|====================================================================================================|
| node1 | TestVol01 | linstor-pool |     0 |    1002 | /dev/drbd1002 | 11.00 MiB | InUse  | UpToDate |
| node2 | TestVol01 | linstor-pool |     0 |    1002 | /dev/drbd1002 | 11.00 MiB | Unused | UpToDate |
| node3 | TestVol01 | linstor-pool |     0 |    1002 | /dev/drbd1002 | 11.00 MiB | Unused | UpToDate |
+----------------------------------------------------------------------------------------------------+

umount /mnt
linstor volume-definition delete TestVol01 0
linstor resource-definition delete TestVol01

linstorの情報は、/var/lib/linstorに入っているので、これをどのノードでも使えるように冗長化する。

drbd-reactor で/var/lib/linstorをマウントしてないと、linstor-controllerが起動しないようにする仕組みです。

これで、3つのノードどれかで、linstore-controllerが起動して、その他のlinstore-controllerは起動しないという設定が可能となります。

linstor resource-definition create linstor_db
linstor resource-definition drbd-options --on-no-quorum=io-error linstor_db
linstor resource-definition drbd-options --auto-promote=no linstor_db
linstor volume-definition create linstor_db 200M
linstor resource create linstor_db -s pool1 --auto-place 3

systemctl disable --now linstor-controller

cat << EOF > /etc/systemd/system/var-lib-linstor.mount
[Unit]
Description=Filesystem for the LINSTOR controller

[Mount]
# you can use the minor like /dev/drbdX or the udev symlink
What=/dev/drbd/by-res/linstor_db/0
Where=/var/lib/linstor
EOF

 mv /var/lib/linstor{,.orig}
 mkdir /var/lib/linstor
 chattr +i /var/lib/linstor  # only if on LINSTOR >= 1.14.0
 drbdadm primary linstor_db
 mkfs.ext4 /dev/drbd/by-res/linstor_db/0
 systemctl start var-lib-linstor.mount
 cp -r /var/lib/linstor.orig/* /var/lib/linstor
 systemctl start linstor-controller

※下記の手順は全てのノードで実行する。

apt install drbd-reactor

cat << EOF > /etc/drbd-reactor.d/linstor_db.toml
[[promoter]]
id = "linstor_db"
[promoter.resources.linstor_db]
start = ["var-lib-linstor.mount", "linstor-controller.service"]
EOF

systemctl restart drbd-reactor
systemctl enable drbd-reactor

linstor_dbボリュームに変更を加えないように設定変更

# systemctl edit linstor-satellite
[Service]
Environment=LS_KEEP_RES=linstor_db

どのノードでlinstor-controllerが動いていても良いように、linstor-client.confを修正しておく。

# cat /etc/linstor/linstor-client.conf
[global]
controllers=node1,node2,node3

linstor_dbがPrimaryのノードでしか、linstor-controllerが起動しない事を確認

# linstor controller which 
linstor://node2

# drbdadm status
linstor_db role:Secondary
  disk:UpToDate
  worker02 role:Primary
    peer-disk:UpToDate
  worker03 role:Secondary
    peer-disk:UpToDate

node1# systemctl start linstor-controller.service 
A dependency job for linstor-controller.service failed. See 'journalctl -xe' for details.

node2# systemctl stop linstor-controller.service
node2# umount /var/lib/linstor
node2# # drbdadm secondary linstor_db

node1# drbdadm primary linstor_db
node1# # systemctl start linstor-controller.service

# linstor controller which 
linstor://node1