Prometheus 监控方案

2017-07-04 16:01:05
Prometheus 安装

centos6的安装参考这里,以下以centos7为例介绍:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
#For centos7
cd /usr/local
wget https://github.com/prometheus/prometheus/releases/download/v2.2.1/prometheus-2.2.1.linux-amd64.tar.gz
tar -zxvf prometheus-2.2.1.linux-amd64.tar.gz
ln -s prometheus-2.2.1.linux-amd64 prometheus
groupadd prometheus
useradd -g prometheus -s /sbin/nologin prometheus
chown prometheus:prometheus /usr/local/prometheus-2.2.1.linux-amd64/ -R


# Create Systemd script
#  /etc/systemd/system/prometheus.service
[Unit]
Description=Prometheus Server
Documentation=https://prometheus.io/docs/introduction/overview/
After=network-online.target
[Service]
User=prometheus
Restart=on-failure
ExecStart=/usr/local/prometheus \
  -config.file=/usr/local/prometheus/prometheus.yml \
  -storage.local.path=/usr/local/prometheus/data
[Install]
WantedBy=multi-user.target


[root@localhost ~]# ln -s /usr/local/prometheus/prometheus.yml /etc
[root@localhost ~]# systemctl daemon-reload
[root@localhost ~]# systemctl enable prometheus
[root@localhost ~]# systemctl start prometheus

可以通过http://localhost:9090/metrics访问啦

另外，还可以通过以下方法安装，只是版本不是最新的,packagecloud源已不能yum安装

1
2
3
4
cd /etc/yum.repos.d/
wget https://download.opensuse.org/repositories/home:justlest:prometheus/CentOS_7/home:justlest:prometheus.repo
yum install prometheus -y
systemctl start prometheus

grafana 安装

grafana安装方法参考这里,这里选择yum安装

1
2
3
[root@localhost ~]# sudo yum install https://s3-us-west-2.amazonaws.com/grafana-releases/release/grafana-5.1.2-1.x86_64.rpm
[root@localhost ~]# systemctl start grafana-server.service
[root@localhost ~]# sudo systemctl enable grafana-server.service

可以通过http://localhost:3000/metrics访问啦,默认密码admin/admin
如果单独的导入模板,可以忽略以下步骤

1
2
3
4
5
6
7
8
9
10
11
#uncomment config file on /etc/grafana/grafana.ini 
[dashboards.json]
enabled = true
path = /var/lib/grafana/dashboards

#install dashboards
git clone https://github.com/percona/grafana-dashboards.git
cp -r grafana-dashboards/dashboards /var/lib/grafana/

#Restart Grafana
systemctl restart grafana-server.service

这里需要注意的是,如果你的grafana数据源名字不是Prometheus,请注意导入数据库模板时重新关联数据源,否则会包模板初始化失败
添加数据源和模板参考这里

Configuring Prometheus

  • linux node监控配置
    首先,配置好prometheus.repo,如果有防火墙开放9100端口.

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    # curl -Lo /etc/yum.repos.d/_copr_ibotty-prometheus-exporters.repo https://copr.fedorainfracloud.org/coprs/ibotty/prometheus-exporters/repo/epel-7/ibotty-prometheus-exporters-epel-7.repo

    #install node_exporter
    [root@localhost ~]# yum install node_exporter -y
    [root@localhost ~]# systemctl start node_exporter

    #prometheus server configure,add to /etc/prometheus/prometheus.yml

      - job_name: 'linux'

        static_configs:
          - targets: ['192.168.1.106:9100']
            labels:
              instance: 'dev_106'

  • mysql node 监控配置,如果有防火墙开放9104端口.

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    #install mysqld_exporter
    [root@localhost ~]# yum install mysqld_exporter -y
    [root@localhost ~]# systemctl start mysqld-exporter

    #prometheus server configure,add to /etc/prometheus/prometheus.yml

      - job_name: 'mysql'

        static_configs:
          - targets: ['192.168.1.106:9104']
            labels:
              instance: 'dev_106_db' 

    #mysqld_exporter需要连接到Mysql，创建用户并赋予所需的权限： 

    mysql> create user monitor@localhost identified by 'monitor' with max_user_connections 3;
    mysql> grant process,replication client,select on *.* to monitor@localhost;

    #mysqld_exporter默认会读取~/.my.cnf文件
    #my.cnf file for MySQL exporter should be as follows: 
    cat << EOF > .my.cnf
    [client]
    user=monitor
    password=monitor
    host=192.168.1.106
    EOF

    grafana中mysql模板参考这里

  • 网络监控,提供 http、dns、tcp、icmp（ping）的监控,如果有防火墙开放9115端口.

    1
    2
    3
    # install blackbox_exporter 
    [root@localhost ~]# yum install blackbox_exporter -y
    [root@localhost ~]# systemctl start blackbox_exporter

    1.Ping 应用

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    #prometheus server configure,add to /etc/prometheus/prometheus.yml

      - job_name: 'ping_all'
        scrape_interval: 5s
        metrics_path: /probe
        params:
          module: [icmp]  #ping
        static_configs:
          - targets: ['219.150.32.132', '219.148.204.66']
            labels:
              group: '一线城市-电信网络监控'
          - targets: ['218.8.251.163', '218.107.51.1']
            labels:
              group: '一线城市-联通网络监控'
        relabel_configs:
          - source_labels: [__address__]
            regex: (.*)(:80)?
            target_label: __param_target
            replacement: ${1}
          - source_labels: [__param_target]
            regex: (.*)
            target_label: ping
            replacement: ${1}
          - source_labels: []
            regex: .*
            target_label: __address__
            replacement: 127.0.0.1:9115  # Blackbox exporter.

    在 grafana中增加 Data Sources 选 prometheus,然后按照grafana的文档新定制一个面板
    ROW中指标选probe_duration_seconds{job=”ping_all”}
    2.检测ssl 证书失效

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    #prometheus server configure,add to /etc/prometheus/prometheus.yml

    rule_files:
      - ssl_expiry.rules
    scrape_configs:
      - job_name: 'blackbox'
        metrics_path: /probe
        params:
          module: [http_2xx]  # Look for a HTTP 200 response.
        static_configs:
          - targets:
            - example.com  # Target to probe
        relabel_configs:
          - source_labels: [__address__]
            regex: (.*?)(:80)?
            target_label: __param_target
            replacement: https://${1}
          - source_labels: [__param_target]
            target_label: instance
          - target_label: __address__
            replacement: 127.0.0.1:9115  # Blackbox exporter.

    cat << 'EOF' > ssl_expiry.rules
    ALERT SSLCertExpiringSoon
     IF probe_ssl_earliest_cert_expiry{job="blackbox"} - time() < 86400 * 30
     FOR 10m
    EOF

    可以通过 http://localhost:9090/alerts访问,失效前30天将收到告警.

  • snmp_exporter 监控配置,如果有防火墙开放9116端口.
    安装snmp_exporter,参考这里,下面为安装脚本

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    version=v0.4.0
    file=snmp_exporter-0.4.0.linux-amd64
     
    wget https://github.com/prometheus/snmp_exporter/releases/download/$version/$file.tar.gz \
      -O /tmp/$file.tar.gz
    cd /tmp
    tar xvf /tmp/$file.tar.gz
    cp /tmp/$file/snmp_exporter /usr/local/bin/snmp_exporter
     
    tee /usr/lib/systemd/system/snmp_exporter.service << EOS
    [Unit]
    Description=SNMP Exporter
    [Service]
    ExecStart=/usr/local/bin/snmp_exporter -config.file /etc/prometheus/snmp.yml
    [Install]
    WantedBy=default.target
    EOS
     
    systemctl daemon-reload
    systemctl enable snmp_exporter
    systemctl start snmp_exporter 
    cd -

    我们可以通过 http://localhost:9116来验证是否允许,接下来我们进行prometheus配置

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    #prometheus server configure,add to /etc/prometheus/prometheus.yml	
    	scrape_configs:
      - job_name: 'snmp'
        static_configs:
          - targets:
            - 192.168.1.2  # SNMP device.
        metrics_path: /snmp
        params:
          module: [default]
        relabel_configs:
          - source_labels: [__address__]
            target_label: __param_target
          - source_labels: [__param_target]
            target_label: instance
          - target_label: __address__
            replacement: 127.0.0.1:9116  # SNMP exporter.

    稍等一会儿,我们就可以通过 http://localhost:9090/consoles/snmp.html 查看接口统计

  • redis_exporter 监控配置,如果有防火墙开放9121端口.
    安装 redis_exporter ,具体参考这里

    1
    2
    3
    4
    5
    6
    7
    8
    9
    $ go get
    $ cd $GOPATH/src/github.com/oliver006/redis_exporter
    $ go build
    $ ./redis_exporter <flags>

    #prometheus server configure,add to /etc/prometheus/prometheus.yml
    - job_name: redis_exporter
      static_configs:
      - targets: ['localhost:9121']

    grafana中模板redis模板参考这里

  • nginx_exporter 监控配置
    1.通过nginx-vts-exporter 监控,安装参考这里,grafana模板参考这里
    2.通过nginx-lua-prometheus 监控,安装参考这里,grafana模板参考这里
    3.通过nginx-exporter 监控,安装参考这里,未找到相应grafana模板,不推荐
    4.监控Nginx流量的扩展程序,安装参考这里
    5.通过日志监控,参考这里

  • ceph_exporter 监控配置
    1.通过ceph_exporter 监控,安装参考这里,grafana模板参考这里

  • gluster_exporter 监控配置
    1.通过gluster_exporter 监控,安装参考这里

  • JMX Exporter 监控配置
    1.通过JMX Exporter 监控,安装参考这里

  • Docker 监控配置

  1. 通过docker-compose安装, 参考这里
  2. 通过cadvisor ,参考这里

ref
prometheus
用Prometheus进行网络质量ping监控Grafana进行监控数据展示
Get alerted before your SSL certificates expire
Prometheus Monitoring for Java Developers
EXPORTERS AND INTEGRATIONS
How To Add a Prometheus Dashboard to Grafana
Prometheus 监控 Nginx 流量
NGINX Performance Metrics with Prometheus
Can’t build current code version
ldap配置系列三：grafana集成ldap

您的鼓励是我写作最大的动力

俗话说，投资效率是最好的投资。 如果您感觉我的文章质量不错，读后收获很大，预计能为您提高 10% 的工作效率，不妨小额捐助我一下，让我有动力继续写出更多好文章。