====== 22 Ceph create-volume でエラー ======
===== エラー内容 =====
# ceph-volume lvm prepare --bluestore --data ssd/data --block.wal optane/waldb --block.db optane/rocksdb
--> RuntimeError: skipping ssd/data, it is already prepared
[root@cephdev001 my-cluster]# lvremove -f /dev/ssd/data
Logical volume "data" successfully removed
[root@cephdev001 my-cluster]# lvcreate -n data -l 100%Free ssd
Logical volume "data" created.
[root@cephdev001 my-cluster]# ceph-volume lvm prepare --bluestore --data ssd/data --block.wal optane/waldb --block.db optane/rocksdb
Running command: /usr/bin/ceph-authtool --gen-print-key
Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 2dd353ac-dcb8-48a8-88cc-cdf96b6ecfcf
Running command: /usr/bin/ceph-authtool --gen-print-key
Running command: /usr/bin/mount -t tmpfs tmpfs /var/lib/ceph/osd/ceph-0
Running command: /usr/bin/chown -h ceph:ceph /dev/ssd/data
Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2
Running command: /usr/bin/ln -s /dev/ssd/data /var/lib/ceph/osd/ceph-0/block
Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring mon getmap -o /var/lib/ceph/osd/ceph-0/activate.monmap
stderr: 2020-09-02 18:30:09.744 7f8fd77c0700 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.bootstrap-osd.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2020-09-02 18:30:09.744 7f8fd77c0700 -1 AuthRegistry(0x7f8fd00656b8) no keyring found at /etc/ceph/ceph.client.bootstrap-osd.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,, disabling cephx
stderr: got monmap epoch 1
Running command: /usr/bin/ceph-authtool /var/lib/ceph/osd/ceph-0/keyring --create-keyring --name osd.0 --add-key AQAgZk9fqxUgKRAA8gJEbwtnuVb91YJHVZgm/Q==
stdout: creating /var/lib/ceph/osd/ceph-0/keyring
added entity osd.0 auth(key=AQAgZk9fqxUgKRAA8gJEbwtnuVb91YJHVZgm/Q==)
Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0/keyring
Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0/
Running command: /usr/bin/chown -h ceph:ceph /dev/optane/waldb
Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3
Running command: /usr/bin/chown -h ceph:ceph /dev/optane/rocksdb
Running command: /usr/bin/chown -R ceph:ceph /dev/dm-5
Running command: /usr/bin/ceph-osd --cluster ceph --osd-objectstore bluestore --mkfs -i 0 --monmap /var/lib/ceph/osd/ceph-0/activate.monmap --keyfile - --bluestore-block-wal-path /dev/optane/waldb --bluestore-block-db-path /dev/optane/rocksdb --osd-data /var/lib/ceph/osd/ceph-0/ --osd-uuid 2dd353ac-dcb8-48a8-88cc-cdf96b6ecfcf --setuser ceph --setgroup ceph
stderr: 2020-09-02 18:30:10.301 7f600913da80 -1 bluestore(/var/lib/ceph/osd/ceph-0/) _read_fsid unparsable uuid
stderr: 2020-09-02 18:30:10.303 7f600913da80 -1 rocksdb: Invalid argument: Can't parse option compaction_threads
stderr: 2020-09-02 18:30:10.558 7f600913da80 -1 bluestore(/var/lib/ceph/osd/ceph-0/) mkfs failed, (22) Invalid argument
stderr: 2020-09-02 18:30:10.558 7f600913da80 -1 OSD::mkfs: ObjectStore::mkfs failed with error (22) Invalid argument
stderr: /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/gigantic/release/14.2.11/rpm/el7/BUILD/ceph-14.2.11/src/os/bluestore/BlueStore.cc: In function 'virtual BlueStore::~BlueStore()' thread 7f600913da80 time 2020-09-02 18:30:10.559517
stderr: /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/gigantic/release/14.2.11/rpm/el7/BUILD/ceph-14.2.11/src/os/bluestore/BlueStore.cc: 3944: FAILED ceph_assert(bluefs == __null)
stderr: ceph version 14.2.11 (f7fdb2f52131f54b891a2ec99d8205561242cdaf) nautilus (stable)
stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14a) [0x56366177c0e5]
stderr: 2: (()+0x4d72ad) [0x56366177c2ad]
stderr: 3: (BlueStore::~BlueStore()+0x7dd) [0x563661cf15fd]
stderr: 4: (BlueStore::~BlueStore()+0x9) [0x563661cf1639]
stderr: 5: (OSD::mkfs(CephContext*, ObjectStore*, uuid_d, int)+0x898) [0x563661839d28]
stderr: 6: (main()+0x1501) [0x5636617e3481]
stderr: 7: (__libc_start_main()+0xf5) [0x7f6004f69555]
stderr: 8: (()+0x573c85) [0x563661818c85]
stderr: *** Caught signal (Aborted) **
stderr: in thread 7f600913da80 thread_name:ceph-osd
stderr: 2020-09-02 18:30:10.567 7f600913da80 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/gigantic/release/14.2.11/rpm/el7/BUILD/ceph-14.2.11/src/os/bluestore/BlueStore.cc: In function 'virtual BlueStore::~BlueStore()' thread 7f600913da80 time 2020-09-02 18:30:10.559517
stderr: /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/gigantic/release/14.2.11/rpm/el7/BUILD/ceph-14.2.11/src/os/bluestore/BlueStore.cc: 3944: FAILED ceph_assert(bluefs == __null)
stderr: ceph version 14.2.11 (f7fdb2f52131f54b891a2ec99d8205561242cdaf) nautilus (stable)
stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14a) [0x56366177c0e5]
stderr: 2: (()+0x4d72ad) [0x56366177c2ad]
stderr: 3: (BlueStore::~BlueStore()+0x7dd) [0x563661cf15fd]
stderr: 4: (BlueStore::~BlueStore()+0x9) [0x563661cf1639]
stderr: 5: (OSD::mkfs(CephContext*, ObjectStore*, uuid_d, int)+0x898) [0x563661839d28]
stderr: 6: (main()+0x1501) [0x5636617e3481]
stderr: 7: (__libc_start_main()+0xf5) [0x7f6004f69555]
stderr: 8: (()+0x573c85) [0x563661818c85]
stderr: ceph version 14.2.11 (f7fdb2f52131f54b891a2ec99d8205561242cdaf) nautilus (stable)
stderr: 1: (()+0xf630) [0x7f6006189630]
stderr: 2: (gsignal()+0x37) [0x7f6004f7d387]
stderr: 3: (abort()+0x148) [0x7f6004f7ea78]
stderr: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x199) [0x56366177c134]
stderr: 5: (()+0x4d72ad) [0x56366177c2ad]
stderr: 6: (BlueStore::~BlueStore()+0x7dd) [0x563661cf15fd]
stderr: 7: (BlueStore::~BlueStore()+0x9) [0x563661cf1639]
stderr: 8: (OSD::mkfs(CephContext*, ObjectStore*, uuid_d, int)+0x898) [0x563661839d28]
stderr: 9: (main()+0x1501) [0x5636617e3481]
stderr: 10: (__libc_start_main()+0xf5) [0x7f6004f69555]
stderr: 11: (()+0x573c85) [0x563661818c85]
stderr: 2020-09-02 18:30:10.574 7f600913da80 -1 *** Caught signal (Aborted) **
stderr: in thread 7f600913da80 thread_name:ceph-osd
stderr: ceph version 14.2.11 (f7fdb2f52131f54b891a2ec99d8205561242cdaf) nautilus (stable)
stderr: 1: (()+0xf630) [0x7f6006189630]
stderr: 2: (gsignal()+0x37) [0x7f6004f7d387]
stderr: 3: (abort()+0x148) [0x7f6004f7ea78]
stderr: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x199) [0x56366177c134]
stderr: 5: (()+0x4d72ad) [0x56366177c2ad]
stderr: 6: (BlueStore::~BlueStore()+0x7dd) [0x563661cf15fd]
stderr: 7: (BlueStore::~BlueStore()+0x9) [0x563661cf1639]
stderr: 8: (OSD::mkfs(CephContext*, ObjectStore*, uuid_d, int)+0x898) [0x563661839d28]
stderr: 9: (main()+0x1501) [0x5636617e3481]
stderr: 10: (__libc_start_main()+0xf5) [0x7f6004f69555]
stderr: 11: (()+0x573c85) [0x563661818c85]
stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this.
stderr: -34> 2020-09-02 18:30:10.301 7f600913da80 -1 bluestore(/var/lib/ceph/osd/ceph-0/) _read_fsid unparsable uuid
stderr: -5> 2020-09-02 18:30:10.303 7f600913da80 -1 rocksdb: Invalid argument: Can't parse option compaction_threads
stderr: -3> 2020-09-02 18:30:10.558 7f600913da80 -1 bluestore(/var/lib/ceph/osd/ceph-0/) mkfs failed, (22) Invalid argument
stderr: -2> 2020-09-02 18:30:10.558 7f600913da80 -1 OSD::mkfs: ObjectStore::mkfs failed with error (22) Invalid argument
stderr: -1> 2020-09-02 18:30:10.567 7f600913da80 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/gigantic/release/14.2.11/rpm/el7/BUILD/ceph-14.2.11/src/os/bluestore/BlueStore.cc: In function 'virtual BlueStore::~BlueStore()' thread 7f600913da80 time 2020-09-02 18:30:10.559517
stderr: /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/gigantic/release/14.2.11/rpm/el7/BUILD/ceph-14.2.11/src/os/bluestore/BlueStore.cc: 3944: FAILED ceph_assert(bluefs == __null)
stderr: ceph version 14.2.11 (f7fdb2f52131f54b891a2ec99d8205561242cdaf) nautilus (stable)
stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14a) [0x56366177c0e5]
stderr: 2: (()+0x4d72ad) [0x56366177c2ad]
stderr: 3: (BlueStore::~BlueStore()+0x7dd) [0x563661cf15fd]
stderr: 4: (BlueStore::~BlueStore()+0x9) [0x563661cf1639]
stderr: 5: (OSD::mkfs(CephContext*, ObjectStore*, uuid_d, int)+0x898) [0x563661839d28]
stderr: 6: (main()+0x1501) [0x5636617e3481]
stderr: 7: (__libc_start_main()+0xf5) [0x7f6004f69555]
stderr: 8: (()+0x573c85) [0x563661818c85]
stderr: 0> 2020-09-02 18:30:10.574 7f600913da80 -1 *** Caught signal (Aborted) **
stderr: in thread 7f600913da80 thread_name:ceph-osd
stderr: ceph version 14.2.11 (f7fdb2f52131f54b891a2ec99d8205561242cdaf) nautilus (stable)
stderr: 1: (()+0xf630) [0x7f6006189630]
stderr: 2: (gsignal()+0x37) [0x7f6004f7d387]
stderr: 3: (abort()+0x148) [0x7f6004f7ea78]
stderr: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x199) [0x56366177c134]
stderr: 5: (()+0x4d72ad) [0x56366177c2ad]
stderr: 6: (BlueStore::~BlueStore()+0x7dd) [0x563661cf15fd]
stderr: 7: (BlueStore::~BlueStore()+0x9) [0x563661cf1639]
stderr: 8: (OSD::mkfs(CephContext*, ObjectStore*, uuid_d, int)+0x898) [0x563661839d28]
stderr: 9: (main()+0x1501) [0x5636617e3481]
stderr: 10: (__libc_start_main()+0xf5) [0x7f6004f69555]
stderr: 11: (()+0x573c85) [0x563661818c85]
stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this.
stderr: -34> 2020-09-02 18:30:10.301 7f600913da80 -1 bluestore(/var/lib/ceph/osd/ceph-0/) _read_fsid unparsable uuid
stderr: -5> 2020-09-02 18:30:10.303 7f600913da80 -1 rocksdb: Invalid argument: Can't parse option compaction_threads
stderr: -3> 2020-09-02 18:30:10.558 7f600913da80 -1 bluestore(/var/lib/ceph/osd/ceph-0/) mkfs failed, (22) Invalid argument
stderr: -2> 2020-09-02 18:30:10.558 7f600913da80 -1 OSD::mkfs: ObjectStore::mkfs failed with error (22) Invalid argument
stderr: -1> 2020-09-02 18:30:10.567 7f600913da80 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/gigantic/release/14.2.11/rpm/el7/BUILD/ceph-14.2.11/src/os/bluestore/BlueStore.cc: In function 'virtual BlueStore::~BlueStore()' thread 7f600913da80 time 2020-09-02 18:30:10.559517
stderr: /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/gigantic/release/14.2.11/rpm/el7/BUILD/ceph-14.2.11/src/os/bluestore/BlueStore.cc: 3944: FAILED ceph_assert(bluefs == __null)
stderr: ceph version 14.2.11 (f7fdb2f52131f54b891a2ec99d8205561242cdaf) nautilus (stable)
stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14a) [0x56366177c0e5]
stderr: 2: (()+0x4d72ad) [0x56366177c2ad]
stderr: 3: (BlueStore::~BlueStore()+0x7dd) [0x563661cf15fd]
stderr: 4: (BlueStore::~BlueStore()+0x9) [0x563661cf1639]
stderr: 5: (OSD::mkfs(CephContext*, ObjectStore*, uuid_d, int)+0x898) [0x563661839d28]
stderr: 6: (main()+0x1501) [0x5636617e3481]
stderr: 7: (__libc_start_main()+0xf5) [0x7f6004f69555]
stderr: 8: (()+0x573c85) [0x563661818c85]
stderr: 0> 2020-09-02 18:30:10.574 7f600913da80 -1 *** Caught signal (Aborted) **
stderr: in thread 7f600913da80 thread_name:ceph-osd
stderr: ceph version 14.2.11 (f7fdb2f52131f54b891a2ec99d8205561242cdaf) nautilus (stable)
stderr: 1: (()+0xf630) [0x7f6006189630]
stderr: 2: (gsignal()+0x37) [0x7f6004f7d387]
stderr: 3: (abort()+0x148) [0x7f6004f7ea78]
stderr: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x199) [0x56366177c134]
stderr: 5: (()+0x4d72ad) [0x56366177c2ad]
stderr: 6: (BlueStore::~BlueStore()+0x7dd) [0x563661cf15fd]
stderr: 7: (BlueStore::~BlueStore()+0x9) [0x563661cf1639]
stderr: 8: (OSD::mkfs(CephContext*, ObjectStore*, uuid_d, int)+0x898) [0x563661839d28]
stderr: 9: (main()+0x1501) [0x5636617e3481]
stderr: 10: (__libc_start_main()+0xf5) [0x7f6004f69555]
stderr: 11: (()+0x573c85) [0x563661818c85]
stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this.
--> Was unable to complete a new OSD, will rollback changes
Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it
stderr: 2020-09-02 18:30:10.730 7f4ea281a700 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.bootstrap-osd.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2020-09-02 18:30:10.730 7f4ea281a700 -1 AuthRegistry(0x7f4e9c0656b8) no keyring found at /etc/ceph/ceph.client.bootstrap-osd.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,, disabling cephx
stderr: purged osd.0
--> RuntimeError: Command failed with exit code 250: /usr/bin/ceph-osd --cluster ceph --osd-objectstore bluestore --mkfs -i 0 --monmap /var/lib/ceph/osd/ceph-0/activate.monmap --keyfile - --bluestore-block-wal-path /dev/optane/waldb --bluestore-block-db-path /dev/optane/rocksdb --osd-data /var/lib/ceph/osd/ceph-0/ --osd-uuid 2dd353ac-dcb8-48a8-88cc-cdf96b6ecfcf --setuser ceph --setgroup ceph
===== 対応 =====
これすごく分かりづらいのですけど、bluestore_rocksdb_options の行を1行にまとめて書くと失敗してしまいます。
同じ内容でもカンマ区切りで改行してやるとcreateに成功します。
==== NG ====
bluestore_rocksdb_options = compression=kNoCompression,max_write_buffer_number=32,min_write_buffer_number_to_merge=2,recycle_log_file_num=32,compaction_style=kCompactionStyleLevel,write_buffer_size=67108864,target_file_size_base=67108864,max_background_compactions=31,level0_file_num_compaction_trigger=8,level0_slowdown_writes_trigger=32,level0_stop_writes_trigger=64,max_bytes_for_level_base=536870912,compaction_threads=32,max_bytes_for_level_multiplier=8,flusher_threads=8,compaction_readahead_size=2MB
==== OK ====
bluestore_rocksdb_options =
compression=kNoCompression,
max_write_buffer_number=32,
min_write_buffer_number_to_merge=2,
recycle_log_file_num=32,
compaction_style=kCompactionStyleLevel,
write_buffer_size=67108864,
target_file_size_base=67108864,
max_background_compactions=31,
level0_file_num_compaction_trigger=8,
level0_slowdown_writes_trigger=32,
level0_stop_writes_trigger=64,
max_bytes_for_level_base=536870912,
compaction_threads=32,
max_bytes_for_level_multiplier=8,
flusher_threads=8,
compaction_readahead_size=2MB
{{tag>ceph}}