fl8 Wiki

この文書は読取専用です。文書のソースを閲覧することは可能ですが、変更はできません。もし変更したい場合は管理者に連絡してください。
====== 22 Ceph create-volume でエラー ======


===== エラー内容 =====

<code>
# ceph-volume lvm prepare --bluestore --data ssd/data --block.wal optane/waldb --block.db optane/rocksdb
-->  RuntimeError: skipping ssd/data, it is already prepared
[root@cephdev001 my-cluster]# lvremove -f /dev/ssd/data 
  Logical volume "data" successfully removed
[root@cephdev001 my-cluster]# lvcreate -n data -l 100%Free ssd
  Logical volume "data" created.
[root@cephdev001 my-cluster]# ceph-volume lvm prepare --bluestore --data ssd/data --block.wal optane/waldb --block.db optane/rocksdb
Running command: /usr/bin/ceph-authtool --gen-print-key
Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 2dd353ac-dcb8-48a8-88cc-cdf96b6ecfcf
Running command: /usr/bin/ceph-authtool --gen-print-key
Running command: /usr/bin/mount -t tmpfs tmpfs /var/lib/ceph/osd/ceph-0
Running command: /usr/bin/chown -h ceph:ceph /dev/ssd/data
Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2
Running command: /usr/bin/ln -s /dev/ssd/data /var/lib/ceph/osd/ceph-0/block
Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring mon getmap -o /var/lib/ceph/osd/ceph-0/activate.monmap
 stderr: 2020-09-02 18:30:09.744 7f8fd77c0700 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.bootstrap-osd.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2020-09-02 18:30:09.744 7f8fd77c0700 -1 AuthRegistry(0x7f8fd00656b8) no keyring found at /etc/ceph/ceph.client.bootstrap-osd.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,, disabling cephx
 stderr: got monmap epoch 1
Running command: /usr/bin/ceph-authtool /var/lib/ceph/osd/ceph-0/keyring --create-keyring --name osd.0 --add-key AQAgZk9fqxUgKRAA8gJEbwtnuVb91YJHVZgm/Q==
 stdout: creating /var/lib/ceph/osd/ceph-0/keyring
added entity osd.0 auth(key=AQAgZk9fqxUgKRAA8gJEbwtnuVb91YJHVZgm/Q==)
Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0/keyring
Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0/
Running command: /usr/bin/chown -h ceph:ceph /dev/optane/waldb
Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3
Running command: /usr/bin/chown -h ceph:ceph /dev/optane/rocksdb
Running command: /usr/bin/chown -R ceph:ceph /dev/dm-5
Running command: /usr/bin/ceph-osd --cluster ceph --osd-objectstore bluestore --mkfs -i 0 --monmap /var/lib/ceph/osd/ceph-0/activate.monmap --keyfile - --bluestore-block-wal-path /dev/optane/waldb --bluestore-block-db-path /dev/optane/rocksdb --osd-data /var/lib/ceph/osd/ceph-0/ --osd-uuid 2dd353ac-dcb8-48a8-88cc-cdf96b6ecfcf --setuser ceph --setgroup ceph
 stderr: 2020-09-02 18:30:10.301 7f600913da80 -1 bluestore(/var/lib/ceph/osd/ceph-0/) _read_fsid unparsable uuid
 stderr: 2020-09-02 18:30:10.303 7f600913da80 -1 rocksdb: Invalid argument: Can't parse option compaction_threads
 stderr: 2020-09-02 18:30:10.558 7f600913da80 -1 bluestore(/var/lib/ceph/osd/ceph-0/) mkfs failed, (22) Invalid argument
 stderr: 2020-09-02 18:30:10.558 7f600913da80 -1 OSD::mkfs: ObjectStore::mkfs failed with error (22) Invalid argument
 stderr: /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/gigantic/release/14.2.11/rpm/el7/BUILD/ceph-14.2.11/src/os/bluestore/BlueStore.cc: In function 'virtual BlueStore::~BlueStore()' thread 7f600913da80 time 2020-09-02 18:30:10.559517
 stderr: /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/gigantic/release/14.2.11/rpm/el7/BUILD/ceph-14.2.11/src/os/bluestore/BlueStore.cc: 3944: FAILED ceph_assert(bluefs == __null)
 stderr: ceph version 14.2.11 (f7fdb2f52131f54b891a2ec99d8205561242cdaf) nautilus (stable)
 stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14a) [0x56366177c0e5]
 stderr: 2: (()+0x4d72ad) [0x56366177c2ad]
 stderr: 3: (BlueStore::~BlueStore()+0x7dd) [0x563661cf15fd]
 stderr: 4: (BlueStore::~BlueStore()+0x9) [0x563661cf1639]
 stderr: 5: (OSD::mkfs(CephContext*, ObjectStore*, uuid_d, int)+0x898) [0x563661839d28]
 stderr: 6: (main()+0x1501) [0x5636617e3481]
 stderr: 7: (__libc_start_main()+0xf5) [0x7f6004f69555]
 stderr: 8: (()+0x573c85) [0x563661818c85]
 stderr: *** Caught signal (Aborted) **
 stderr: in thread 7f600913da80 thread_name:ceph-osd
 stderr: 2020-09-02 18:30:10.567 7f600913da80 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/gigantic/release/14.2.11/rpm/el7/BUILD/ceph-14.2.11/src/os/bluestore/BlueStore.cc: In function 'virtual BlueStore::~BlueStore()' thread 7f600913da80 time 2020-09-02 18:30:10.559517
 stderr: /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/gigantic/release/14.2.11/rpm/el7/BUILD/ceph-14.2.11/src/os/bluestore/BlueStore.cc: 3944: FAILED ceph_assert(bluefs == __null)
 stderr: ceph version 14.2.11 (f7fdb2f52131f54b891a2ec99d8205561242cdaf) nautilus (stable)
 stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14a) [0x56366177c0e5]
 stderr: 2: (()+0x4d72ad) [0x56366177c2ad]
 stderr: 3: (BlueStore::~BlueStore()+0x7dd) [0x563661cf15fd]
 stderr: 4: (BlueStore::~BlueStore()+0x9) [0x563661cf1639]
 stderr: 5: (OSD::mkfs(CephContext*, ObjectStore*, uuid_d, int)+0x898) [0x563661839d28]
 stderr: 6: (main()+0x1501) [0x5636617e3481]
 stderr: 7: (__libc_start_main()+0xf5) [0x7f6004f69555]
 stderr: 8: (()+0x573c85) [0x563661818c85]
 stderr: ceph version 14.2.11 (f7fdb2f52131f54b891a2ec99d8205561242cdaf) nautilus (stable)
 stderr: 1: (()+0xf630) [0x7f6006189630]
 stderr: 2: (gsignal()+0x37) [0x7f6004f7d387]
 stderr: 3: (abort()+0x148) [0x7f6004f7ea78]
 stderr: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x199) [0x56366177c134]
 stderr: 5: (()+0x4d72ad) [0x56366177c2ad]
 stderr: 6: (BlueStore::~BlueStore()+0x7dd) [0x563661cf15fd]
 stderr: 7: (BlueStore::~BlueStore()+0x9) [0x563661cf1639]
 stderr: 8: (OSD::mkfs(CephContext*, ObjectStore*, uuid_d, int)+0x898) [0x563661839d28]
 stderr: 9: (main()+0x1501) [0x5636617e3481]
 stderr: 10: (__libc_start_main()+0xf5) [0x7f6004f69555]
 stderr: 11: (()+0x573c85) [0x563661818c85]
 stderr: 2020-09-02 18:30:10.574 7f600913da80 -1 *** Caught signal (Aborted) **
 stderr: in thread 7f600913da80 thread_name:ceph-osd
 stderr: ceph version 14.2.11 (f7fdb2f52131f54b891a2ec99d8205561242cdaf) nautilus (stable)
 stderr: 1: (()+0xf630) [0x7f6006189630]
 stderr: 2: (gsignal()+0x37) [0x7f6004f7d387]
 stderr: 3: (abort()+0x148) [0x7f6004f7ea78]
 stderr: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x199) [0x56366177c134]
 stderr: 5: (()+0x4d72ad) [0x56366177c2ad]
 stderr: 6: (BlueStore::~BlueStore()+0x7dd) [0x563661cf15fd]
 stderr: 7: (BlueStore::~BlueStore()+0x9) [0x563661cf1639]
 stderr: 8: (OSD::mkfs(CephContext*, ObjectStore*, uuid_d, int)+0x898) [0x563661839d28]
 stderr: 9: (main()+0x1501) [0x5636617e3481]
 stderr: 10: (__libc_start_main()+0xf5) [0x7f6004f69555]
 stderr: 11: (()+0x573c85) [0x563661818c85]
 stderr: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
 stderr: -34> 2020-09-02 18:30:10.301 7f600913da80 -1 bluestore(/var/lib/ceph/osd/ceph-0/) _read_fsid unparsable uuid
 stderr: -5> 2020-09-02 18:30:10.303 7f600913da80 -1 rocksdb: Invalid argument: Can't parse option compaction_threads
 stderr: -3> 2020-09-02 18:30:10.558 7f600913da80 -1 bluestore(/var/lib/ceph/osd/ceph-0/) mkfs failed, (22) Invalid argument
 stderr: -2> 2020-09-02 18:30:10.558 7f600913da80 -1 OSD::mkfs: ObjectStore::mkfs failed with error (22) Invalid argument
 stderr: -1> 2020-09-02 18:30:10.567 7f600913da80 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/gigantic/release/14.2.11/rpm/el7/BUILD/ceph-14.2.11/src/os/bluestore/BlueStore.cc: In function 'virtual BlueStore::~BlueStore()' thread 7f600913da80 time 2020-09-02 18:30:10.559517
 stderr: /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/gigantic/release/14.2.11/rpm/el7/BUILD/ceph-14.2.11/src/os/bluestore/BlueStore.cc: 3944: FAILED ceph_assert(bluefs == __null)
 stderr: ceph version 14.2.11 (f7fdb2f52131f54b891a2ec99d8205561242cdaf) nautilus (stable)
 stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14a) [0x56366177c0e5]
 stderr: 2: (()+0x4d72ad) [0x56366177c2ad]
 stderr: 3: (BlueStore::~BlueStore()+0x7dd) [0x563661cf15fd]
 stderr: 4: (BlueStore::~BlueStore()+0x9) [0x563661cf1639]
 stderr: 5: (OSD::mkfs(CephContext*, ObjectStore*, uuid_d, int)+0x898) [0x563661839d28]
 stderr: 6: (main()+0x1501) [0x5636617e3481]
 stderr: 7: (__libc_start_main()+0xf5) [0x7f6004f69555]
 stderr: 8: (()+0x573c85) [0x563661818c85]
 stderr: 0> 2020-09-02 18:30:10.574 7f600913da80 -1 *** Caught signal (Aborted) **
 stderr: in thread 7f600913da80 thread_name:ceph-osd
 stderr: ceph version 14.2.11 (f7fdb2f52131f54b891a2ec99d8205561242cdaf) nautilus (stable)
 stderr: 1: (()+0xf630) [0x7f6006189630]
 stderr: 2: (gsignal()+0x37) [0x7f6004f7d387]
 stderr: 3: (abort()+0x148) [0x7f6004f7ea78]
 stderr: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x199) [0x56366177c134]
 stderr: 5: (()+0x4d72ad) [0x56366177c2ad]
 stderr: 6: (BlueStore::~BlueStore()+0x7dd) [0x563661cf15fd]
 stderr: 7: (BlueStore::~BlueStore()+0x9) [0x563661cf1639]
 stderr: 8: (OSD::mkfs(CephContext*, ObjectStore*, uuid_d, int)+0x898) [0x563661839d28]
 stderr: 9: (main()+0x1501) [0x5636617e3481]
 stderr: 10: (__libc_start_main()+0xf5) [0x7f6004f69555]
 stderr: 11: (()+0x573c85) [0x563661818c85]
 stderr: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
 stderr: -34> 2020-09-02 18:30:10.301 7f600913da80 -1 bluestore(/var/lib/ceph/osd/ceph-0/) _read_fsid unparsable uuid
 stderr: -5> 2020-09-02 18:30:10.303 7f600913da80 -1 rocksdb: Invalid argument: Can't parse option compaction_threads
 stderr: -3> 2020-09-02 18:30:10.558 7f600913da80 -1 bluestore(/var/lib/ceph/osd/ceph-0/) mkfs failed, (22) Invalid argument
 stderr: -2> 2020-09-02 18:30:10.558 7f600913da80 -1 OSD::mkfs: ObjectStore::mkfs failed with error (22) Invalid argument
 stderr: -1> 2020-09-02 18:30:10.567 7f600913da80 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/gigantic/release/14.2.11/rpm/el7/BUILD/ceph-14.2.11/src/os/bluestore/BlueStore.cc: In function 'virtual BlueStore::~BlueStore()' thread 7f600913da80 time 2020-09-02 18:30:10.559517
 stderr: /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/gigantic/release/14.2.11/rpm/el7/BUILD/ceph-14.2.11/src/os/bluestore/BlueStore.cc: 3944: FAILED ceph_assert(bluefs == __null)
 stderr: ceph version 14.2.11 (f7fdb2f52131f54b891a2ec99d8205561242cdaf) nautilus (stable)
 stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14a) [0x56366177c0e5]
 stderr: 2: (()+0x4d72ad) [0x56366177c2ad]
 stderr: 3: (BlueStore::~BlueStore()+0x7dd) [0x563661cf15fd]
 stderr: 4: (BlueStore::~BlueStore()+0x9) [0x563661cf1639]
 stderr: 5: (OSD::mkfs(CephContext*, ObjectStore*, uuid_d, int)+0x898) [0x563661839d28]
 stderr: 6: (main()+0x1501) [0x5636617e3481]
 stderr: 7: (__libc_start_main()+0xf5) [0x7f6004f69555]
 stderr: 8: (()+0x573c85) [0x563661818c85]
 stderr: 0> 2020-09-02 18:30:10.574 7f600913da80 -1 *** Caught signal (Aborted) **
 stderr: in thread 7f600913da80 thread_name:ceph-osd
 stderr: ceph version 14.2.11 (f7fdb2f52131f54b891a2ec99d8205561242cdaf) nautilus (stable)
 stderr: 1: (()+0xf630) [0x7f6006189630]
 stderr: 2: (gsignal()+0x37) [0x7f6004f7d387]
 stderr: 3: (abort()+0x148) [0x7f6004f7ea78]
 stderr: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x199) [0x56366177c134]
 stderr: 5: (()+0x4d72ad) [0x56366177c2ad]
 stderr: 6: (BlueStore::~BlueStore()+0x7dd) [0x563661cf15fd]
 stderr: 7: (BlueStore::~BlueStore()+0x9) [0x563661cf1639]
 stderr: 8: (OSD::mkfs(CephContext*, ObjectStore*, uuid_d, int)+0x898) [0x563661839d28]
 stderr: 9: (main()+0x1501) [0x5636617e3481]
 stderr: 10: (__libc_start_main()+0xf5) [0x7f6004f69555]
 stderr: 11: (()+0x573c85) [0x563661818c85]
 stderr: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
--> Was unable to complete a new OSD, will rollback changes
Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it
 stderr: 2020-09-02 18:30:10.730 7f4ea281a700 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.bootstrap-osd.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2020-09-02 18:30:10.730 7f4ea281a700 -1 AuthRegistry(0x7f4e9c0656b8) no keyring found at /etc/ceph/ceph.client.bootstrap-osd.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,, disabling cephx
 stderr: purged osd.0
-->  RuntimeError: Command failed with exit code 250: /usr/bin/ceph-osd --cluster ceph --osd-objectstore bluestore --mkfs -i 0 --monmap /var/lib/ceph/osd/ceph-0/activate.monmap --keyfile - --bluestore-block-wal-path /dev/optane/waldb --bluestore-block-db-path /dev/optane/rocksdb --osd-data /var/lib/ceph/osd/ceph-0/ --osd-uuid 2dd353ac-dcb8-48a8-88cc-cdf96b6ecfcf --setuser ceph --setgroup ceph
</code>


===== 対応 =====

これすごく分かりづらいのですけど、bluestore_rocksdb_options の行を1行にまとめて書くと失敗してしまいます。
同じ内容でもカンマ区切りで改行してやるとcreateに成功します。

==== NG ====

<code|/etc/ceph/ceph.conf>
bluestore_rocksdb_options = compression=kNoCompression,max_write_buffer_number=32,min_write_buffer_number_to_merge=2,recycle_log_file_num=32,compaction_style=kCompactionStyleLevel,write_buffer_size=67108864,target_file_size_base=67108864,max_background_compactions=31,level0_file_num_compaction_trigger=8,level0_slowdown_writes_trigger=32,level0_stop_writes_trigger=64,max_bytes_for_level_base=536870912,compaction_threads=32,max_bytes_for_level_multiplier=8,flusher_threads=8,compaction_readahead_size=2MB
</code>

==== OK ====

<code| /etc/ceph/ceph.conf>
bluestore_rocksdb_options = 
compression=kNoCompression, 
max_write_buffer_number=32, 
min_write_buffer_number_to_merge=2, 
recycle_log_file_num=32, 
compaction_style=kCompactionStyleLevel,
write_buffer_size=67108864, 
target_file_size_base=67108864,
max_background_compactions=31,
level0_file_num_compaction_trigger=8,
level0_slowdown_writes_trigger=32,
level0_stop_writes_trigger=64,
max_bytes_for_level_base=536870912,
compaction_threads=32,
max_bytes_for_level_multiplier=8,
flusher_threads=8,
compaction_readahead_size=2MB
</code>


{{tag>ceph}}
fl8 Wiki

ユーザ用ツール

サイト用ツール

ページ用ツール
