ceph-Monitor clock skew detected

2017-06-13 15:00:09

ceph异常警告:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
[root@admin ~]# ceph -s
    cluster d6d92de4-2a08-4bd6-a749-6c104c88fc40
     health HEALTH_WARN
            clock skew detected on mon.node2
            Monitor clock skew detected 
     monmap e1: 3 mons at {node1=192.168.138.141:6789/0,node2=192.168.138.142:6789/0,node3=192.168.138.143:6789/0}
            election epoch 46, quorum 0,1,2 node1,node2,node3
     osdmap e218: 9 osds: 9 up, 9 in
            flags sortbitwise,require_jewel_osds
      pgmap v1007: 256 pgs, 1 pools, 4 bytes data, 1 objects
            345 MB used, 18377 GB / 18378 GB avail
                 256 active+clean
				 			 
[root@admin ~]# ceph health detail
HEALTH_WARN clock skew detected on mon.node2, mon.node3; Monitor clock skew detected 
mon.node2 addr 192.168.138.142:6789/0 clock skew 0.434161s > max 0.05s (latency 0.00740637s)
mon.node3 addr 192.168.138.143:6789/0 clock skew 0.687451s > max 0.05s (latency 0.00722567s)

解决方法:

  • 方法一:配置ntp server
    本来以为配置ntp server了,时间应该就一致了,原来ceph默认容忍的时间偏差不到1秒,随意只能用本地的ntp server了
    下面来说说配置ntp server(If Iptables is running, allow NTP port. NTP uses 123/UDP.)
    chrony_server

    1.install chrony

    1
    yum -y install chrony

    2.config chrony

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    [root@admin ~]# cat /etc/chrony.conf 
    # These servers were defined in the installation:
    server 3.centos.pool.ntp.org iburst
    server 0.centos.pool.ntp.org iburst
    server cn.pool.ntp.org iburst
    server 1.centos.pool.ntp.org iburst
    server 2.centos.pool.ntp.org iburst
    # Serve time even if not synchronized to any NTP server.
    local stratum 10
    ....
    allow 192.168/16

    3.run chrony

    1
    2
    systemctl enable chronyd
    systemctl start chronyd

    4.timedatectl

    1
    2
    3
    4
    ## 查看时间
    timedatectl
    ## 开启ntp时间同步
    timedatectl set-ntp true

    5.chronyc

    1
    2
    3
    4
    5
    6
    7
    8
    ## 查看ntp_servers状态
    chronyc sources -v
    ## 查看ntp_sync状态
    chronyc sourcestats -v
    ## 查看ntp_servers 是否在线
    chronyc activity -v
    ## 查看ntp时间详细信息
    chronyc tracking -v

    chrony_client
    1.install chrony

    1
    yum -y install chrony

    2.config chrony

    1
    2
    3
    [root@node3 ~]# cat /etc/chrony.conf
    # These servers were defined in the installation:
    server 192.168.138.140 iburst

    3.run chrony

    1
    2
    systemctl enable chronyd
    systemctl start chronyd

    4.timedatectl

    1
    2
    3
    4
    ## 查看时间
    timedatectl
    ## 开启ntp时间同步
    timedatectl set-ntp true

    5.chronyc

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    ## 查看ntp_servers状态
    chronyc sources -v
    ## 查看ntp_sync状态
    chronyc sourcestats -v
    ## 查看ntp_servers 是否在线
    chronyc activity -v
    ## 查看ntp时间详细信息
    chronyc tracking -v

    [root@node3 ~]# chronyc tracking
    Reference ID    : 192.168.138.140 (192.168.138.140)
    Stratum         : 5
    Ref time (UTC)  : Tue Jun 13 05:52:48 2017
    System time     : 0.000494327 seconds fast of NTP time
    Last offset     : +0.000704829 seconds
    RMS offset      : 0.001765276 seconds
    Frequency       : 71.667 ppm slow
    Residual freq   : +0.269 ppm
    Skew            : 6.979 ppm
    Root delay      : 0.324675 seconds
    Root dispersion : 0.039305 seconds
    Update interval : 64.8 seconds
    Leap status     : Normal

    备注:chrony手动校时

    1
    chronyc -a makestep

  • 方法二:调整ceph参数避免
    1.在admin结点上，修改ceph.conf，添加：

    1
    2
    mon_clock_drift_allowed = 5
    mon_clock_drift_warn_backoff = 30

    详细参数参考这里

    2.执行下面命令，node1等是monitor结点的名称

    1
    ceph-deploy --overwrite-conf admin node1 node2 node3

    3.重启monitor

    1
    2
    3
    systemctl restart ceph-mon@node1
    systemctl restart ceph-mon@node2
    systemctl restart ceph-mon@node3

    4.验证

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    [root@admin ~]# ceph -s
        cluster d6d92de4-2a08-4bd6-a749-6c104c88fc40
         health HEALTH_OK
         monmap e1: 3 mons at {node1=192.168.138.141:6789/0,node2=192.168.138.142:6789/0,node3=192.168.138.143:6789/0}
                election epoch 48, quorum 0,1,2 node1,node2,node3
         osdmap e218: 9 osds: 9 up, 9 in
                flags sortbitwise,require_jewel_osds
          pgmap v1007: 256 pgs, 1 pools, 4 bytes data, 1 objects
                345 MB used, 18377 GB / 18378 GB avail
                     256 active+clean

ref
chrony时间同步
chronyd 使用
Chrony : Configure NTP Server
ceph: HEALTH_WARN: Monitor clock skew detected

您的鼓励是我写作最大的动力

俗话说，投资效率是最好的投资。 如果您感觉我的文章质量不错，读后收获很大，预计能为您提高 10% 的工作效率，不妨小额捐助我一下，让我有动力继续写出更多好文章。