搜索到
96
篇与
Ceph
的结果
-
Luminous下删除和新建OSD的正确姿势 如何正确删除OSDL版本开始极大的降低了对运维操作复杂度，新增了很多命令去确保数据安全，很多新手在删除OSD的时候很容易忽视了集群PGs的状态最终导致数据丢失，因此官方加入以下几个命令：ceph osd ok-to-stop: Checks whether it looks like PGs will remaineven if the specified OSD(s) are stopped.ceph osd safe-to-destroy: Checks whether it is safe to destroy an OSD. This does various checks to ensure there is no data on the OSD(s), nounfound objects, stuck peering, and so forth.用户在删除OSD之前运行这些命令，通过命令返回的内容，就可以判断删除操作是否能够确保数据安全。另外在删除OSD的时候，官方也提供了2种类型的操作，一种是使用ceph osd destroy去替换故障磁盘，一种是彻底删除OSD，具体说明如下：ceph osd destroy: zap info about an OSD but keep it's ID in place (with'destroyed' flag) so that it can be recreated with a replacement device.ceph osd purge: zap everything about an OSD, including the ID下面用真实案例来告诉大家如何删除一个OSD-0，删除前，运行前面提到的ok-to-stop和safe-to-destroy命令，根据返回的结果来决定是否能够执行删除OSD操作。[root@node1 ~]# ceph osd ok-to-stop osd.0OSD(s) 0 are ok to stop without reducing availability, provided there are no other concurrent failures or interventions. 88 PGs are likely to be degraded (but remain available) as a result.[root@node1 ~]# ceph osd safe-to-destroy osd.0Error EBUSY: OSD(s) 0 have 176 pgs currently mapped to them删除之前最好确认OSD对应的状态和数据信息：确认OSD的STATUS等信息[root@node1 ~]# ceph osd treeID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 180.81212 root default -3 60.27071 host node1 0 hdd 5.45799 osd.0 up 1.00000 1.00000确认osd-0的Crush信息[root@node1 ~]# ceph osd crush ls osd.0osd.0获取osd-0相关的cephx信息[root@node1 ~]# ceph auth get osd.0exported keyring for osd.0[osd.0]key = AQBxTtJc7DubHRAAxVPEIuLT4oAj+YClXS/2zg== caps mgr = "allow profile osd" caps mon = "allow profile osd" caps osd = "allow *" 如果是替换现有OSD-0的磁盘，则执行destroy操作，你会发现只有OSD-0的keyring被删除。[root@node1 ~]# systemctl stop ceph-osd@0[root@node1 ~]# ceph osd destroy osd.0 --yes-i-really-mean-itdestroyed osd.0[root@node1 ~]# ceph osd treeID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF-1 0 root default 0 hdd 0 osd.0 destroyed 1.00000 1.00000 #状态变为destroyed[root@node1 ~]# ceph osd crush ls osd.0osd.0[root@node1 ~]# ceph auth get osd.0Error ENOENT: failed to find osd.0 in keyring如果是彻底删除OSD-0，则执行purge操作，你会发现所有OSD-0关联的信息都被删除。[root@node1 ~]# systemctl stop ceph-osd@0[root@node1 ~]# ceph osd purge osd.0 --yes-i-really-mean-itpurged osd.0[root@node1 ~]# ceph osd treeID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF-1 0 root default[root@node1 ~]# ceph osd crush ls osd.0Error ENOENT: node 'osd.0' does not exist[root@node1 ~]# ceph auth get osd.0Error ENOENT: failed to find osd.0 in keyring最后如果你需要清理原来OSD-0对应数据盘的数据，可以执行以下命令[root@node1 ~]# ceph-volume lvm zap /dev/sdb --destroy--> Zapping: /dev/sdb--> Unmounting /var/lib/ceph/osd/ceph-0Running command: umount -v /var/lib/ceph/osd/ceph-0 stderr: umount: /var/lib/ceph/osd/ceph-0 (tmpfs) 已卸载--> Destroying volume group ceph-3d2c442b-c663-48d5-bb18-098cb5f307fa because --destroy was givenRunning command: vgremove -v -f ceph-3d2c442b-c663-48d5-bb18-098cb5f307fa stderr: Removing ceph--3d2c442b--c663--48d5--bb18--098cb5f307fa-osd--block--7449a599--585a--4caf--8452--9b5facab3df3 (253:2) stderr: Archiving volume group "ceph-3d2c442b-c663-48d5-bb18-098cb5f307fa" metadata (seqno 21). stderr: Releasing logical volume "osd-block-7449a599-585a-4caf-8452-9b5facab3df3" stderr: Creating volume group backup "/etc/lvm/backup/ceph-3d2c442b-c663-48d5-bb18-098cb5f307fa" (seqno 22). stdout: Logical volume "osd-block-7449a599-585a-4caf-8452-9b5facab3df3" successfully removed stderr: Removing physical volume "/dev/sdb" from volume group "ceph-3d2c442b-c663-48d5-bb18-098cb5f307fa" stdout: Volume group "ceph-3d2c442b-c663-48d5-bb18-098cb5f307fa" successfully removed--> Destroying physical volume /dev/sdb because --destroy was givenRunning command: pvremove -v -f /dev/sdb stderr: Wiping internal VG cacheWiping cache of LVM-capable devicesstdout: Labels on physical volume "/dev/sdb" successfully wiped.Running command: wipefs --all /dev/sdbRunning command: dd if=/dev/zero of=/dev/sdb bs=1M count=10--> Zapping successful for: /dev/sdb新增OSD的注意事项在使用ceph-volume的时候如果你需要将db和wal放置到独立的SSD分区上，那么你需要提前手工进行分区（ceph-volume后续会提供自动分区方案，目前需要手工）,以建立OSD-1的wal和db为例。使用sgdisk新建分区，并指定分区的partuuid以及label标签sgdisk --new=0:0:+100M --change-name=1:osd-1-wal --partition-guid=1:4fbd7e29-9d25-41b8-afd0-062c0ceff051 --mbrtogpt -- /dev/sdcsgdisk --new=0:0:+100M --change-name=2:osd-1-db --partition-guid=2:4fbd7e29-9d25-41b8-afd0打上label标签的好处就是方便运维，一目了然就知道分区的具体作用。效果如下[root@demo cephuser]# blkid/dev/sda1: UUID="a1e4eaa2-fd83-44e9-937e-a1b360c1c707" TYPE="xfs"/dev/sda2: UUID="UYtArS-hwDi-nNkc-N78D-6EZX-cIXW-PSpbgq" TYPE="LVM2_member"/dev/mapper/centos-root: UUID="7065f959-f667-4253-8a90-d49afcf19a29" TYPE="xfs"/dev/mapper/centos-swap: UUID="bbdef107-d9f9-4470-860e-9d2f55821c4c" TYPE="swap"/dev/sdc1: PARTLABEL="osd-1-wal" PARTUUID="4fbd7e29-9d25-41b8-afd0-062c0ceff051"/dev/sdc2: PARTLABEL="osd-1-db" PARTUUID="4fbd7e29-9d25-41b8-afd0-062c0ceff052"在使用ceph-volume初始化osd的时候，最好使用partuuid去进行初始化，这样在LV的tag字段ceph.wal_device和ceph.db_device会按你的partuuid进行设置[root@node1 ~]# ceph-volume lvm create --bluestore --data /dev/sdb --block.db /dev/disk/by-partuuid/4fbd7e29-9d25-41b8-afd0-062c0ceff052 --block.wal /dev/disk/by-partuuid/4fbd7e29-9d25-41b8-afd0-062c0ceff051[root@demo cephuser]# /usr/sbin/lvs --noheadings --readonly --separator=" " -o lv_tags,lv_path,lv_name,vg_name,lv_uuid /dev/centos/root root centos fq7hUc-BtCA-AeIk-VvBD-Mxad-7uix-ZtK4I6 /dev/centos/swap swap centos r7oGD8-Tf15-v4ir-eOR3-LeDg-qqrf-gblNWIceph.block_device=/dev/ceph-e6f9ba96-7323-4a2f-b854-f343d088eb8d/osd-block-9fe04363-fb0b-498d-9如果你使用/dev/sdc1一类名称进行初始化，那么如果发生重启，则可能会出现对应的ceph.wal_device和ceph.db_device发生变化，虽然现在OSD在挂载的时候会按uuid来查找对应的设备（使用get_osd_device_path方法，具体见下面代码），但是为了确保tag数据的一致性，还是推荐使用partuuid方式。[root@node1 ~]# ceph-volume lvm create --bluestore --data /dev/sdb --block.db /dev/sdc1 --block.wal /dev/sdc2[root@node1 ~]# /usr/sbin/lvs --noheadings --readonly --separator=" " -o lv_tags,lv_path,lv_name,vg_name,lv_uuid /dev/centos/root root centos fq7hUc-BtCA-AeIk-VvBD-Mxad-7uix-ZtK4I6 /dev/centos/swap swap centos r7oGD8-Tf15-v4ir-eOR3-LeDg-qqrf-gblNWIceph.block_device=/dev/ceph-2850c361-dc1c-434b-9689-f73798bb514e/osd-block-471be0c0-905b-4836-a5uuid获取db和wal路径的源码示例如下ceph-12.2.5/src/ceph-volume/ceph_volume/devices/lvm/activate.py激活osd的时候调用下面的方法def activate_bluestore(lvs, no_systemd=False):..... db_device_path = get_osd_device_path(osd_lv, lvs, 'db', dmcrypt_secret=dmcrypt_secret) wal_device_path = get_osd_device_path(osd_lv, lvs, 'wal', dmcrypt_secret=dmcrypt_secret) 获取最终存储设备路径def get_osd_device_path(osd_lv, lvs, device_type, dmcrypt_secret=None):""" ``device_type`` can be one of ``db``, ``wal`` or ``block`` so that we can query ``lvs`` (a ``Volumes`` object) and fallback to querying the uuid if that is not present. Return a path if possible, failing to do that a ``None``, since some of these devices are optional """ osd_lv = lvs.get(lv_tags={'ceph.type': 'block'}) is_encrypted = osd_lv.tags.get('ceph.encrypted', '0') == '1' logger.debug('Found block device (%s) with encryption: %s', osd_lv.name, is_encrypted) uuid_tag = 'ceph.%s_uuid' % device_type device_uuid = osd_lv.tags.get(uuid_tag) #根据lv tag的ceph.db_uuid或者ceph.wal_uuid字段来获取最终的设备路径 if not device_uuid: return None device_lv = lvs.get(lv_uuid=device_uuid) if device_lv: if is_encrypted: encryption_utils.luks_open(dmcrypt_secret, device_lv.lv_path, device_uuid) return '/dev/mapper/%s' % device_uuid return device_lv.lv_path else: # this could be a regular device, so query it with blkid physical_device = disk.get_device_from_partuuid(device_uuid) if physical_device and is_encrypted: encryption_utils.luks_open(dmcrypt_secret, physical_device, device_uuid) return '/dev/mapper/%s' % device_uuid return physical_device or None return None -
-
查看池属性 [root@node1 ~]# ceph osd pool get images allsize: 3min_size: 2pg_num: 64pgp_num: 64crush_rule: ssdhashpspool: truenodelete: falsenopgchange: falsenosizechange: falsewrite_fadvise_dontneed: falsenoscrub: falsenodeep-scrub: falseuse_gmt_hitset: 1auid: 0fast_read: 0[root@node1 ~]# ceph osd pool get images sizesize: 3
-
-
查看 ceph pool 中一个镜像的信息 [root@node1 ~]# rbd info -p images --image a56330e7-79d7-4639-a68f-366ac344bfe2rbd image 'a56330e7-79d7-4639-a68f-366ac344bfe2':size 1.3 GiB in 169 objects order 23 (8 MiB objects) id: 6ce065344cca block_name_prefix: rbd_data.6ce065344cca format: 2 features: layering, exclusive-lock, object-map, fast-diff, deep-flatten op_features: flags: create_timestamp: Wed May 8 17:43:27 2019
院长正在为您加急渲染，请耐心等待......