- 日本語 (ja)
- English (en)
最近の更新
- 2024.01.18 SSL Error in Rails created
- 2024.01.18 RailsでSSLエラー 以前のリビジョンを復元 (2024/01/18 07:06)
- 31 CentOS5 TLS1.2 created
- 2024.04.12 MySQL BIT Field created
最近の更新
文書の過去の版を表示しています。
Kubernetes用の冗長化ストレージとして、Linstorを用意してみる。
LinstorはLINBITが開発した、DRBDを管理する管理DBを持つlinstor-controllerとlinstor-satelliteというデータディスクを扱うプログラムで構成されてます。
今回はKubernetesのworkerノード3台で、Linstor
name | IP | Disk | DRBD用 |
---|---|---|---|
node1 | 172.16.0.65 | /dev/sda | /dev/sdb |
node2 | 172.16.0.252 | /dev/sda | /dev/sdb |
node3 | 172.16.0.234 | /dev/sda | /dev/sdb |
※sdaは/用、sdbはlinstor用
add-apt-repository ppa:linbit/linbit-drbd9-stack apt-get update && apt install drbd-utils drbd-dkms lvm2 modprobe drbd
# cat /proc/drbd version: 9.1.6 (api:2/proto:110-121) GIT-hash: f85adeb71f16a0aead1e875665e2fb68852d94eb build by root@worker01, 2022-04-15 21:43:08 Transports (api:17): tcp (9.1.6)
今回は3台でlinstor-controllerを冗長化するので、3台とも全部インストール
apt install linstor-controller linstor-satellite linstor-client
systemctl disable linstor-controller
# source /etc/bash_completion.d/linstor # または # source /usr/share/bash_completion/completions/linstor
1度node1で、linstor-controllerを上げて初期設定を行います。
node1# systemctl start linstor-controller linstor node create node1 172.16.0.65 linstor node create node2 172.16.0.252 linstor node create node3 172.16.0.234
node1# linstor node list ╭────────────────────────────────────────────────────────╮ ┊ Node ┊ NodeType ┊ Addresses ┊ State ┊ ╞════════════════════════════════════════════════════════╡ ┊ node1 ┊ SATELLITE ┊ 172.16.0.65:3366 (PLAIN) ┊ Online ┊ ┊ node2 ┊ SATELLITE ┊ 172.16.0.252:3366 (PLAIN) ┊ Online ┊ ┊ node3 ┊ SATELLITE ┊ 172.16.0.234:3366 (PLAIN) ┊ Online ┊ ╰────────────────────────────────────────────────────────╯
こちらも、linstor-controllerを起動しているnode1で実行する。
sdbでLVMを用意してlinstorのストレージプールに利用する。
pvcreate /dev/sdb vgcreate vg /dev/sdb lvcreate -l 100%FREE --thinpool vg/lvmthinpool linstor storage-pool create lvmthin node1 linstor-pool vg/lvmthinpool linstor storage-pool create lvmthin node2 linstor-pool vg/lvmthinpool linstor storage-pool create lvmthin node3 linstor-pool vg/lvmthinpool
node1# linstor storage-pool list -p +-----------------------------------------------------------------------------------------------------------------------------+ | StoragePool | Node | Driver | PoolName | FreeCapacity | TotalCapacity | CanSnapshots | State | SharedName | |=============================================================================================================================| | DfltDisklessStorPool | node1 | DISKLESS | | | | False | Ok | | | DfltDisklessStorPool | node2 | DISKLESS | | | | False | Ok | | | DfltDisklessStorPool | node3 | DISKLESS | | | | False | Ok | | | linstor-pool | node1 | LVM_THIN | vg/lvmthinpool | 19.90 GiB | 19.96 GiB | True | Ok | | | linstor-pool | node2 | LVM_THIN | vg/lvmthinpool | 19.90 GiB | 19.96 GiB | True | Ok | | | linstor-pool | node3 | LVM_THIN | vg/lvmthinpool | 19.90 GiB | 19.96 GiB | True | Ok | | +-----------------------------------------------------------------------------------------------------------------------------+
流れは
linstor resource-definition create TestVol01 linstor volume-definition create TestVol01 1G linstor resource create TestVol01 -s linstor-pool --auto-place 3
たとえば、node2,node3にリソースを作成して、node1はディスクレスにする場合
linstor resource create node2 node3 TestVol01 -s linstor-pool linstor resource create node1 TestVol01 --diskless
# linstor volume list -r TestVol01 +----------------------------------------------------------------------------------------------------+ | Node | Resource | StoragePool | VolNr | MinorNr | DeviceName | Allocated | InUse | State | |====================================================================================================| | node1 | TestVol01 | linstor-pool | 0 | 1002 | /dev/drbd1002 | 315 KiB | Unused | UpToDate | | node2 | TestVol01 | linstor-pool | 0 | 1002 | /dev/drbd1002 | 315 KiB | Unused | UpToDate | | node3 | TestVol01 | linstor-pool | 0 | 1002 | /dev/drbd1002 | 315 KiB | Unused | UpToDate | +----------------------------------------------------------------------------------------------------+
mkfs.xfs /dev/drbd1002 mount /dev/drbd1002 /mnt/ # df /mnt Filesystem 1K-blocks Used Available Use% Mounted on /dev/drbd1002 1042168 40604 1001564 4% /mnt
# linstor volume list -r TestVol01 | sed 's/^/ /g' +----------------------------------------------------------------------------------------------------+ | Node | Resource | StoragePool | VolNr | MinorNr | DeviceName | Allocated | InUse | State | |====================================================================================================| | node1 | TestVol01 | linstor-pool | 0 | 1002 | /dev/drbd1002 | 11.00 MiB | InUse | UpToDate | | node2 | TestVol01 | linstor-pool | 0 | 1002 | /dev/drbd1002 | 11.00 MiB | Unused | UpToDate | | node3 | TestVol01 | linstor-pool | 0 | 1002 | /dev/drbd1002 | 11.00 MiB | Unused | UpToDate | +----------------------------------------------------------------------------------------------------+
umount /mnt linstor volume-definition delete TestVol01 0 linstor resource-definition delete TestVol01
linstorの情報は、/var/lib/linstorに入っているので、これをどのノードでも使えるように冗長化する。
drbd-reactor で/var/lib/linstorをマウントしてないと、linstor-controllerが起動しないようにする仕組みです。
これで、3つのノードどれかで、linstore-controllerが起動して、その他のlinstore-controllerは起動しないという設定が可能となります。
linstor resource-definition create linstor_db linstor resource-definition drbd-options --on-no-quorum=io-error linstor_db linstor resource-definition drbd-options --auto-promote=no linstor_db linstor volume-definition create linstor_db 200M linstor resource create linstor_db -s linstor-pool --auto-place 3
systemctl disable --now linstor-controller
cat << EOF > /etc/systemd/system/var-lib-linstor.mount [Unit] Description=Filesystem for the LINSTOR controller [Mount] # you can use the minor like /dev/drbdX or the udev symlink What=/dev/drbd/by-res/linstor_db/0 Where=/var/lib/linstor EOF
mv /var/lib/linstor{,.orig} mkdir /var/lib/linstor chattr +i /var/lib/linstor # only if on LINSTOR >= 1.14.0 drbdadm primary linstor_db mkfs.ext4 /dev/drbd/by-res/linstor_db/0 systemctl start var-lib-linstor.mount cp -r /var/lib/linstor.orig/* /var/lib/linstor systemctl start linstor-controller
※下記の手順は全てのノードで実行する。
apt install drbd-reactor
cat << EOF > /etc/drbd-reactor.d/linstor_db.toml [[promoter]] id = "linstor_db" [promoter.resources.linstor_db] start = ["var-lib-linstor.mount", "linstor-controller.service"] EOF
systemctl restart drbd-reactor systemctl enable drbd-reactor
linstor_dbボリュームに変更を加えないように設定変更
# systemctl edit linstor-satellite [Service] Environment=LS_KEEP_RES=linstor_db
どのノードでlinstor-controllerが動いていても良いように、linstor-client.confを修正しておく。
# cat /etc/linstor/linstor-client.conf [global] controllers=node1,node2,node3
linstor_dbがPrimaryのノードでしか、linstor-controllerが起動しない事を確認
# linstor controller which linstor://node2 # drbdadm status linstor_db role:Secondary disk:UpToDate worker02 role:Primary peer-disk:UpToDate worker03 role:Secondary peer-disk:UpToDate node1# systemctl start linstor-controller.service A dependency job for linstor-controller.service failed. See 'journalctl -xe' for details. node2# systemctl stop linstor-controller.service node2# umount /var/lib/linstor node2# # drbdadm secondary linstor_db node1# drbdadm primary linstor_db node1# # systemctl start linstor-controller.service # linstor controller which linstor://node1
# cat /proc/drbd version: 9.0.32-1 (api:2/proto:86-121) # linstor --version linstor 1.13.0 # drbd-reactor --version drbd-reactor 0.6.0
# linstor controller which linstor://node02 ## ノード再起動や、drbd-reactorを停止した状態で、linstor-controllerが停止 node02:~# systemctl stop drbd-reactor.service node02:~# systemctl stop linstor-controller.service ## 即時に別のノードで、linstor-controllerが起動 node01:~# linstor controller which linstor://linstor-node01