Ceph故障修复记录

Ceph 故障修复

集群状况：

Cluster：

root@gigabyte:~# ceph -s
    cluster de3627bb-c748-4623-8cb1-b88c646ff5d5
     health HEALTH_WARN 42 pgs degraded; 1 pgs down; 1 pgs peering; 17 pgs recovering; 25 pgs recovery_wait; 359 pgs stale; 42 pgs stuck degraded; 1 pgs stuck inactive; 359 pgs stuck stale; 244 pgs stuck unclean; recovery 6665/163785 objects degraded (4.069%); 6665/163785 unfound (4.069%); 2/23 in osds are down
     monmap e1: 1 mons at {erxdl=10.16.180.28:6789/0}, election epoch 2, quorum 0 erxdl
     mdsmap e654: 1/1/1 up {0=irlhy=up:active}
     osdmap e218: 23 osds: 21 up, 23 in
      pgmap v8943: 7936 pgs, 16 pools, 639 GB data, 159 kobjects
            105 GB used, 76827 GB / 76933 GB avail
            6665/163785 objects degraded (4.069%); 6665/163785 unfound (4.069%)
                 359 stale+active+clean
                 201 active+remapped
                7333 active+clean
                  25 active+recovery_wait+degraded
                  17 active+recovering+degraded
                   1 down+peering

OSD：

root@test:/data# ceph osd tree
# id    weight  type name   up/down reweight
-3  82.29   pool 4T
-4  82.29       host test1
0   0           osd.0   up      1   
8   3.578           osd.8   up      1   
17  3.578           osd.17  up      1   
5   3.578           osd.5   up      1       
...
...
16  3.578           osd.16  up      1   
10  3.578           osd.10  up      1   
15  3.578           osd.15  up      1   
13  0           osd.13  down    1   

由上可知：osd.0和osd.13已经被集群剔除，并且权重变为了0

报错日志：

报错1：

2017-12-06 10:18:05.180802 7f4809ae07c0 -1 osd.13 234 set_disk_tp_priority(22) Invalid argument: osd_disk_thread_ioprio_class is  but only the following values are allowed: idle, be or rt
2017-12-06 10:10:44.974634 7f4cbcfff700 -1 os/FileStore.cc: In function 'virtual int FileStore::read(coll_t, const ghobject_t&, uint64_t, size_t, ceph::bufferlist&, bool)' thread 7f4cbcfff700
2017-12-06 10:10:44.972299 os/FileStore.cc: 2851: FAILED assert(allow_eio || !m_filestore_fail_eio || got != -5)

报错2:

2017-12-06 10:32:56.602065 7f576071b7c0  0 genericfilestorebackend(/ceph/osd.13) detect_features: FIEMAP ioctl is supported and appears to work
2017-12-06 10:32:56.602095 7f576071b7c0  0 genericfilestorebackend(/ceph/osd.13) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option
2017-12-06 10:32:56.620337 7f576071b7c0  0 genericfilestorebackend(/ceph/osd.13) detect_features: syncfs(2) syscall fully supported (by glibc and kernel)
2017-12-06 10:32:56.739219 7f576071b7c0  0 filestore(/ceph/osd.13) limited size xattrs

报错3：

2017-12-06 10:16:32.442815 7f06cddf2700 -1 journal aio to 1398235136~434176 got (5) Input/output error
2017-12-06 10:16:32.443787 7f06cddf2700 -1 os/FileJournal.cc: In function 'void FileJournal::write_finish_thread_entry()' thread 7f06cddf2700
2017-12-06 10:16:32.442867 os/FileJournal.cc: 1383: FAILED assert(0 == "unexpected aio error")

报错4：

2017-12-06 00:52:33.814785 7fb24be877c0 -1 filestore(/ceph/osd.13) FileStore::mount: unable to access basedir '/ceph/osd.0': (30) Read-only file system
2017-12-06 00:52:33.814801 7fb24be877c0 -1 osd.13 0 OSD:init: unable to mount object store
2017-12-06 00:52:33.814806 7fb24be877c0 -1 ^[[0;31m ** ERROR: osd init failed: (30) Read-only file system

解决方案

首先我们肯定是要尝试把OSD起来嘛，所以要做的就是先给osd加权重，接着加入集群，最后再启动。

ceph osd crush add osd.0 3.578 host=test1
ceph osd in osd.0
service ceph start osd.0
ceph osd crush add osd.13 3.578 host=test1
ceph osd in osd.13
service ceph start osd.13

这时候发现OSD还是起不来，我们就去看osd.0和osd.13的log，也就会发现以上一系列的报错日志，错误较多；

filesystem出现较多limited size xattrs这一行引起了我的注意，由于此环境osd用的文件系统是ext4，而ext4对存储xattr的大小有限制，使得OSD信息不能安全的保存。

所以在ceph中如果osd采用ext4文件系统时，需要在配置项里面加入相关配置实现用omap来存储xattr，而xfs文件系统由于对xattr的存储是足够的，所以不存在这个问题。

所以解决这个问题有三个方案：

方案1

更改文件系统，将ext4改成xfs

方案2

文件系统还是采用ext4，配置让Ceph filestore中的omap存储xattr，在/etc/ceph/ceph.conf中global section或osd section中插入一行以下配置：

filestore xattr use omap = true

方案3

限制对象的长度大小，同样是修改ceph.conf，在global section或者osd section中加入以下配置：

osd max object name len = 256 
osd max object namespace len = 64 

然后再次重启以下对应的osd服务就OK了！

总结

在Ceph中如果用ext4文件系统的话，一定要注意配置将xattr存在omap中。