差分

このページの2つのバージョン間の差分を表示します。

--- 06_virtualization:05_container:16_kubernetes_linstor [2022/04/17 06:28] – matsui
+++ 06_virtualization:05_container:16_kubernetes_linstor [2022/07/07 23:06] (現在) – [3. Linstorインストール] matsui
@@ 行 35: / 行 35: @@
 </code>
-====== 3. Linstorインストール ======
+====== 3. LINSTORインストール ======
 今回は3台でlinstor-controllerを冗長化するので、3台とも全部インストール
@@ 行 45: / 行 45: @@
   systemctl disable linstor-controller
+===== 自動補完 =====
+  # source /etc/bash_completion.d/linstor # または
+  # source /usr/share/bash_completion/completions/linstor
 ====== 4. Linstor初期化 ======
@@ 行 85: / 行 90: @@
 </code>
-===== 確認 =====
+===== storage-pool確認 =====
@@ 行 99: / 行 104: @@
   | linstor-pool         | node3 | LVM_THIN | vg/lvmthinpool |    19.90 GiB |     19.96 GiB | True         | Ok    |            |
   +-----------------------------------------------------------------------------------------------------------------------------+
+===== 1つVolumeを作ってみる =====
+=== TestVol01を作成 ===
+流れは
+  - リソース定義を作成
+  - ボリューム定義を作成
+  - リソースを作成
+  linstor resource-definition create TestVol01
+  linstor volume-definition create TestVol01 1G
+  linstor resource create TestVol01 -s linstor-pool --auto-place 3
+たとえば、node2,node3にリソースを作成して、node1はディスクレスにする場合
+  linstor resource create node2 node3 TestVol01 -s linstor-pool
+  linstor resource create node1 TestVol01 --diskless
+=== 確認 ===
+  # linstor volume list -r TestVol01
+  +----------------------------------------------------------------------------------------------------+
+  | Node  | Resource  | StoragePool  | VolNr | MinorNr | DeviceName    | Allocated | InUse  |    State |
+  |====================================================================================================|
+  | node1 | TestVol01 | linstor-pool |     0 |    1002 | /dev/drbd1002 |   315 KiB | Unused | UpToDate |
+  | node2 | TestVol01 | linstor-pool |     0 |    1002 | /dev/drbd1002 |   315 KiB | Unused | UpToDate |
+  | node3 | TestVol01 | linstor-pool |     0 |    1002 | /dev/drbd1002 |   315 KiB | Unused | UpToDate |
+  +----------------------------------------------------------------------------------------------------+
+=== ディスクを使ってみる ===
+  mkfs.xfs /dev/drbd1002
+  mount /dev/drbd1002 /mnt/
+  # df /mnt
+  Filesystem     1K-blocks  Used Available Use% Mounted on
+  /dev/drbd1002    1042168 40604   1001564   4% /mnt
+=== マウントしているnode1のInUseがInUseに変更して、ディスクを利用している事が分かる ===
+  # linstor volume list -r TestVol01 | sed 's/^/  /g'
+  +----------------------------------------------------------------------------------------------------+
+  | Node  | Resource  | StoragePool  | VolNr | MinorNr | DeviceName    | Allocated | InUse  |    State |
+  |====================================================================================================|
+  | node1 | TestVol01 | linstor-pool |     0 |    1002 | /dev/drbd1002 | 11.00 MiB | InUse  | UpToDate |
+  | node2 | TestVol01 | linstor-pool |     0 |    1002 | /dev/drbd1002 | 11.00 MiB | Unused | UpToDate |
+  | node3 | TestVol01 | linstor-pool |     0 |    1002 | /dev/drbd1002 | 11.00 MiB | Unused | UpToDate |
+  +----------------------------------------------------------------------------------------------------+
+=== 削除 ===
+  umount /mnt
+  linstor volume-definition delete TestVol01 0
+  linstor resource-definition delete TestVol01
+====== 6.冗長化構成 ======
+linstorの情報は、/var/lib/linstorに入っているので、これをどのノードでも使えるように冗長化する。
+drbd-reactor で/var/lib/linstorをマウントしてないと、linstor-controllerが起動しないようにする仕組みです。
+これで、3つのノードどれかで、linstore-controllerが起動して、その他のlinstore-controllerは起動しないという設定が可能となります。
+===== linstor_dbボリュームの用意 =====
+  linstor resource-definition create linstor_db
+  linstor resource-definition drbd-options --on-no-quorum=io-error linstor_db
+  linstor resource-definition drbd-options --auto-promote=no linstor_db
+  linstor volume-definition create linstor_db 200M
+  linstor resource create linstor_db -s linstor-pool --auto-place 3
+==== もしlinstor-controllerが動いている場合は、disableにして停止 ====
+  systemctl disable --now linstor-controller
+==== linstore_dbマウントするsystemd用意 ====
+<code>
+cat << EOF > /etc/systemd/system/var-lib-linstor.mount
+[Unit]
+Description=Filesystem for the LINSTOR controller
+[Mount]
+# you can use the minor like /dev/drbdX or the udev symlink
+What=/dev/drbd/by-res/linstor_db/0
+Where=/var/lib/linstor
+EOF
+</code>
+==== linstor_dbをマウントしてlinstor-controller起動 ====
+<code>
+ mv /var/lib/linstor{,.orig}
+ mkdir /var/lib/linstor
+ chattr +i /var/lib/linstor  # only if on LINSTOR >= 1.14.0
+ drbdadm primary linstor_db
+ mkfs.ext4 /dev/drbd/by-res/linstor_db/0
+ systemctl start var-lib-linstor.mount
+ cp -r /var/lib/linstor.orig/* /var/lib/linstor
+ systemctl start linstor-controller
+</code>
+===== drbd-reactor用意 =====
+※下記の手順は全てのノードで実行する。
+==== install ====
+  apt install drbd-reactor
+==== 起動順のファイル作成 ====
+<code>
+cat << EOF > /etc/drbd-reactor.d/linstor_db.toml
+[[promoter]]
+id = "linstor_db"
+[promoter.resources.linstor_db]
+start = ["var-lib-linstor.mount", "linstor-controller.service"]
+EOF
+</code>
+==== 一度drbd-reactor再起動 ====
+  systemctl restart drbd-reactor
+  systemctl enable drbd-reactor
+===== linstor-satellite修正 =====
+linstor_dbボリュームに変更を加えないように設定変更
+<code>
+# systemctl edit linstor-satellite
+[Service]
+Environment=LS_KEEP_RES=linstor_db
+</code>
+===== linstor-client.conf修正 =====
+どのノードでlinstor-controllerが動いていても良いように、linstor-client.confを修正しておく。
+<code>
+# cat /etc/linstor/linstor-client.conf
+[global]
+controllers=node1,node2,node3
+</code>
+===== 確認 =====
+linstor_dbがPrimaryのノードでしか、linstor-controllerが起動しない事を確認
+<code>
+# linstor controller which
+linstor://node2
+# drbdadm status
+linstor_db role:Secondary
+  disk:UpToDate
+  worker02 role:Primary
+    peer-disk:UpToDate
+  worker03 role:Secondary
+    peer-disk:UpToDate
+node1# systemctl start linstor-controller.service
+A dependency job for linstor-controller.service failed. See 'journalctl -xe' for details.
+node2# systemctl stop linstor-controller.service
+node2# umount /var/lib/linstor
+node2# # drbdadm secondary linstor_db
+node1# drbdadm primary linstor_db
+node1# # systemctl start linstor-controller.service
+# linstor controller which
+linstor://node1
+</code>
+===== 確認（正常な動作） =====
+  # cat /proc/drbd
+  version: 9.0.32-1 (api:2/proto:86-121)
+  # linstor --version
+  linstor 1.13.0
+  # drbd-reactor --version
+  drbd-reactor 0.6.0
+<code>
+# linstor controller which
+linstor://node02
+## ノード再起動や、drbd-reactorを停止した状態で、linstor-controllerが停止
+node02:~# systemctl stop drbd-reactor.service
+node02:~# systemctl stop linstor-controller.service
+## 即時に別のノードで、linstor-controllerが起動
+node01:~# linstor controller which
+linstor://linstor-node01
+</code>
+====== 7.Kubernetsから接続 ======
+==== 大前提条件 ====
+Kubernetes helm インストール時下記の３つがlinstorインストールの前提の条件です。
+  - kubernetes controller からPodのIPに通信できる事
+  - Pod内から外部へ通信可能
+  - Pod内で名前解決可能
+<code>
+root@linstor-master:~# kubectl get pod alpine-test -o wide
+NAME          READY   STATUS    RESTARTS   AGE   IP           NODE             NOMINATED NODE   READINESS GATES
+alpine-test   1/1     Running   0          73s   10.224.1.2   linstor-node04   <none>           <none>
+root@linstor-master:~# ping 10.224.1.2 -c 3
+PING 10.224.1.2 (10.224.1.2) 56(84) bytes of data.
+bytes from 10.224.1.2: icmp_seq=1 ttl=63 time=0.959 ms
+bytes from 10.224.1.2: icmp_seq=2 ttl=63 time=0.679 ms
+bytes from 10.224.1.2: icmp_seq=3 ttl=63 time=0.506 ms
+--- 10.224.1.2 ping statistics ---
+packets transmitted, 3 received, 0% packet loss, time 2019ms
+rtt min/avg/max/mdev = 0.506/0.714/0.959/0.186 ms
+root@linstor-master:~# kubectl exec -it alpine-test -- ping fl8.jp -c 3
+PING fl8.jp (182.48.51.190): 56 data bytes
+bytes from 182.48.51.190: seq=0 ttl=53 time=8.568 ms
+bytes from 182.48.51.190: seq=1 ttl=53 time=8.715 ms
+bytes from 182.48.51.190: seq=2 ttl=53 time=8.532 ms
+--- fl8.jp ping statistics ---
+packets transmitted, 3 packets received, 0% packet loss
+round-trip min/avg/max = 8.532/8.605/8.715 ms
+</code>
+==== secret登録 ====
+<code>
+kubectl create secret docker-registry drbdiocred --docker-server=drbd.io \
+  --docker-username=[User] --docker-password=[Pass]
+</code>
+==== helm repository登録 ====
+<code>
+helm repo add linstor https://charts.linstor.io
+# helm repo list
+NAME   	URL
+linstor	https://charts.linstor.io
+</code>
+==== linstor-opインストール ====
+<code>
+helm install linstor-op linstor/linstor \
+--set operator.controller.enabled=false \
+--set etcd.persistentVolume.enabled=false \
+--set operator.etcd.enabled=false \
+--set controllerEndpoint=http://172.16.0.65:3370
+</code>
+==== 確認 ====
+ちゃんとpodが起動してればOK
+<code>
+# kubectl get pod
+NAME                                         READY   STATUS                  RESTARTS          AGE
+alpine-test                                  1/1     Running                 0                 9h
+linstor-op-csi-controller-76d5bcfcb9-z2mrw   6/6     Running                 0                 9h
+linstor-op-csi-node-54wf6                    3/3     Running                 0                 9h
+linstor-op-csi-node-9fcn9                    3/3     Running                 0                 9h
+linstor-op-csi-node-rw4h5                    3/3     Running                 0                 9h
+linstor-op-etcd-0                            1/1     Running                 0                 9h
+linstor-op-ha-controller-6f9784f965-bqqcp    1/1     Running                 0                 9h
+linstor-op-ns-node-74zjb                     0/2     Init:CrashLoopBackOff   111 (3m4s ago)    9h
+linstor-op-ns-node-ldc5q                     0/2     Init:CrashLoopBackOff   111 (3m51s ago)   9h
+linstor-op-ns-node-m7hcm                     0/2     Init:CrashLoopBackOff   111 (3m28s ago)   9h
+linstor-op-operator-7b9b9b685d-mnc8z         1/1     Running                 0                 9h
+ubuntu                                       1/1     Running                 0                 8h
+ubuntu-7c969d5f7d-lq96g                      1/1     Running                 0                 8h
+</code>
+==== StorageClass作成 ====
+Kubernetes から利用する為に、StorageClass作成
+<code|StorageClass.yaml>
+apiVersion: storage.k8s.io/v1
+kind: StorageClass
+metadata:
+  # The name used to identify this StorageClass.
+  name: linstor-basic-storage-class
+  # The name used to match this StorageClass with a provisioner.
+  # linstor.csi.linbit.com is the name that the LINSTOR CSI plug-in uses to identify itself
+provisioner: linstor.csi.linbit.com
+volumeBindingMode: WaitForFirstConsumer
+parameters:
+  # LINSTOR will provision volumes from the drbdpool storage pool configured
+  # On the satellite nodes in the LINSTOR cluster specified in the plug-in's deployment
+  storagePool: "linstor-pool"
+  resourceGroup: "linstor-basic-storage-class"
+  # Setting a fstype is required for "fsGroup" permissions to work correctly.
+  # Currently supported: xfs/ext4
+  csi.storage.k8s.io/fstype: xfs
+  linstor.csi.linbit.com/autoPlace: "3"
+</code>
+==== PVC作成 ====
+<code|pvc.yaml>
+kind: PersistentVolumeClaim
+apiVersion: v1
+metadata:
+  name: linstor-volume-01
+spec:
+  storageClassName: linstor-basic-storage-class
+  accessModes:
+    - ReadWriteOnce
+  resources:
+    requests:
+      storage: 3Gi
+</code>
+=== PVC作成した時点 ===
+PVC作成した時点ではまだPVは作成してない。
+<code>
+# kubectl apply -f pvc.yaml
+persistentvolumeclaim/linstor-volume-01 created
+# kubectl get pvc
+NAME                STATUS    VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS                  AGE
+linstor-volume-01   Pending                                      linstor-basic-storage-class   4s
+# kubectl get pv
+No resources found
+# linstor volume list -p
++------------------------------------------------------------------------------------------+
+| Node | Resource | StoragePool | VolNr | MinorNr | DeviceName | Allocated | InUse | State |
+|==========================================================================================|
++------------------------------------------------------------------------------------------+
+</code>
+==== PodからPV利用 ====
+<code|ubutnu.yaml>
+apiVersion: v1
+kind: Pod
+metadata:
+  name: ubuntu
+spec:
+  containers:
+  - image: ubuntu
+    name: ubuntu
+    command: [ "/bin/bash", "-c", "--" ]
+    args: [ "while true; do sleep 30; done;" ]
+    volumeMounts:
+    - name: linstor-volume
+      mountPath: /data
+    ports:
+    - containerPort: 80
+  volumes:
+  - name: linstor-volume
+    persistentVolumeClaim:
+      claimName: "linstor-volume-01"
+</code>
+=== Pod作成すると、PVも作成される ===
+<code>
+kubectl apply -f ubutnu.yaml
+# kubectl get pv
+NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                       STORAGECLASS                  REASON   AGE
+pvc-c83e52e0-f8c7-447c-a3f2-8f01daaeb4c2   3Gi        RWO            Delete           Bound    default/linstor-volume-01   linstor-basic-storage-class            82s
+# kubectl get pvc
+NAME                STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS                  AGE
+linstor-volume-01   Bound    pvc-c83e52e0-f8c7-447c-a3f2-8f01daaeb4c2   3Gi        RWO            linstor-basic-storage-class   4m54s
+# linstor volume list -p
++--------------------------------------------------------------------------------------------------------------------------------------------+
+| Node           | Resource                                 | StoragePool  | VolNr | MinorNr | DeviceName    | Allocated | InUse  |    State |
+|============================================================================================================================================|
+| linstor-node04 | pvc-c83e52e0-f8c7-447c-a3f2-8f01daaeb4c2 | linstor-pool |     0 |    1000 | /dev/drbd1000 |  5.23 MiB | Unused | UpToDate |
+| linstor-node05 | pvc-c83e52e0-f8c7-447c-a3f2-8f01daaeb4c2 | linstor-pool |     0 |    1000 | /dev/drbd1000 |  4.61 MiB | Unused | UpToDate |
+| linstor-node06 | pvc-c83e52e0-f8c7-447c-a3f2-8f01daaeb4c2 | linstor-pool |     0 |    1000 | /dev/drbd1000 | 11.38 MiB | Unused | UpToDate |
++--------------------------------------------------------------------------------------------------------------------------------------------+
+</code>
+=== PVC削除 ===
+PVCを削除すると、ちゃんとlinstorで削除されている。
+<code>
+# kubectl delete pod ubuntu
+pod "ubuntu" deleted
+# kubectl get pvc
+NAME                STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS                  AGE
+linstor-volume-01   Bound    pvc-06ac4732-f3f4-46fe-a0b6-6f58a9895cb2   3Gi        RWO            linstor-basic-storage-class   29m
+# kubectl delete pvc linstor-volume-01
+persistentvolumeclaim "linstor-volume-01" deleted
+# linstor volume list -p
++------------------------------------------------------------------------------------------+
+| Node | Resource | StoragePool | VolNr | MinorNr | DeviceName | Allocated | InUse | State |
+|==========================================================================================|
++------------------------------------------------------------------------------------------+
+</code>
 {{tag>Kuberneted DRBD}}

fl8 Wiki

ユーザ用ツール

サイト用ツール

差分

ページ用ツール