prometheus-监控docker服务器

时间:2022-12-29 10:07:26

1. prometheus-监控docker服务器

  • prometheus-监控docker服务器

    • cAdvisor(Container Advisor):用于收集正在运行的容器资源使用和性能信息。

    • 项目地址:https://github.com/google/cadvisor

    • docker部署cAdvisor示例:

      docker run -d \
      --volume=/:/rootfs:ro \
      --volume=/var/run:/var/run:ro \
      --volume=/sys:/sys:ro \
      --volume=/var/lib/docker/:/var/lib/docker:ro \
      --volume=/dev/disk/:/dev/disk:ro \
      --publish=8080:8080 \
      --detach=true \
      --name=cadvisor \
      google/cadvisor:latest
      
  • 案例: 监控docker服务器

    • 监控执行命令

      docker run -d \
      --volume=/:/rootfs:ro \
      --volume=/var/run:/var/run:ro \
      --volume=/sys:/sys:ro \
      --volume=/var/lib/docker/:/var/lib/docker:ro \
      --volume=/dev/disk/:/dev/disk:ro \
      --publish=8080:8080 \
      --detach=true \
      --name=cadvisor \
      google/cadvisor:latest
      
    • 执行创建监控docker服务

      [root@VM-0-17-centos ~]# docker run -d \
      > --volume=/:/rootfs:ro \
      > --volume=/var/run:/var/run:ro \
      > --volume=/sys:/sys:ro \
      > --volume=/var/lib/docker/:/var/lib/docker:ro \
      > --volume=/dev/disk/:/dev/disk:ro \
      > --publish=8080:8080 \
      > --detach=true \
      > --name=cadvisor \
      > google/cadvisor:latest
      Unable to find image 'google/cadvisor:latest' locally
      latest: Pulling from google/cadvisor
      ff3a5c916c92: Pull complete 
      44a45bb65cdf: Pull complete 
      0bbe1a2fe2a6: Pull complete 
      Digest: sha256:815386ebbe9a3490f38785ab11bda34ec8dacf4634af77b8912832d4f85dca04
      Status: Downloaded newer image for google/cadvisor:latest
      78d6d7db3b715f5800346cd592575a4b7be5e644e198dbf95160e64c3545fa53
      
    • 进行数据访问http://ip:8080
      prometheus-监控docker服务器

    • 配置prometheus添加服务

      [root@prometheus ~]# cd /opt/monitor/
      [root@prometheus monitor]# ll
      total 23072
      drwxr-xr-x 2 3434 3434       93 Jun  7 14:39 alertmanager
      -rw-r--r-- 1 root root 23624308 May 11 04:11 alertmanager-0.22.0-rc.1.linux-amd64.tar.gz
      drwxr-xr-x 8 root root      157 Jun  6 17:18 grafana
      drwxr-xr-x 5 3434 3434      145 Jun  7 17:07 prometheus
      [root@prometheus monitor]# cd prometheus/
      [root@prometheus prometheus]# ll
      total 167980
      drwxr-xr-x 2 3434 3434       38 Mar 17 04:20 console_libraries
      drwxr-xr-x 2 3434 3434      173 Mar 17 04:20 consoles
      -rw-r--r-- 1 3434 3434    11357 Mar 17 04:20 LICENSE
      -rw-r--r-- 1 3434 3434     3420 Mar 17 04:20 NOTICE
      -rwxr-xr-x 1 3434 3434 91044140 Mar 17 02:10 prometheus
      -rw-r--r-- 1 3434 3434     1043 Jun  7 17:07 prometheus.yml
      -rwxr-xr-x 1 3434 3434 80944687 Mar 17 02:12 promtool
      drwxr-xr-x 2 root root       22 Jun  7 14:43 rules
      [root@prometheus prometheus]# vim prometheus.yml 
      [root@prometheus prometheus]# cat prometheus.yml 
      # my global config
      global:
        scrape_interval:     15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
        evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
        # scrape_timeout is set to the global default (10s).
      
      # Alertmanager configuration
      alerting:
        alertmanagers:
        - static_configs:
          - targets:
             - 127.0.0.1:9093
      
      # Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
      rule_files:
        - "rules/*.yml"
        # - "second_rules.yml"
      
      # A scrape configuration containing exactly one endpoint to scrape:
      # Here it's Prometheus itself.
      scrape_configs:
        # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
        - job_name: 'prometheus'
      
          # metrics_path defaults to '/metrics'
          # scheme defaults to 'http'.
      
          static_configs:
          - targets: ['127.0.0.1:9090']
        - job_name: 'linux server'
          static_configs:
            - targets: ['121.4.78.187:9100']
              labels:
                prod: 'web1'
        - job_name: 'docker server'
          static_configs:
            - targets: ['121.4.63.211:8080']
              labels:
                prod: 'web2'
      
      
    • 重新加载prometheus服务

      [root@prometheus prometheus]# /bin/systemctl restart prometheus
      [root@prometheus prometheus]# /bin/systemctl status prometheus
      ● prometheus.service - prometheus
         Loaded: loaded (/usr/lib/systemd/system/prometheus.service; enabled; vendor preset: disabled)
         Active: active (running) since Mon 2021-06-07 22:53:36 CST; 4s ago
       Main PID: 14647 (prometheus)
         CGroup: /system.slice/prometheus.service
                 └─14647 /opt/monitor/prometheus/prometheus --config.file=/opt/monitor/prometheus/prometheus.yml
      
      Jun 07 22:53:36 prometheus prometheus[14647]: level=info ts=2021-06-07T14:53:36.914Z caller=head.go:740 component=tsdb msg="WAL segment loaded" segment=1...egment=19
      Jun 07 22:53:36 prometheus prometheus[14647]: level=info ts=2021-06-07T14:53:36.958Z caller=head.go:740 component=tsdb msg="WAL segment loaded" segment=1...egment=19
      Jun 07 22:53:36 prometheus prometheus[14647]: level=info ts=2021-06-07T14:53:36.990Z caller=head.go:740 component=tsdb msg="WAL segment loaded" segment=1...egment=19
      Jun 07 22:53:36 prometheus prometheus[14647]: level=info ts=2021-06-07T14:53:36.990Z caller=head.go:740 component=tsdb msg="WAL segment loaded" segment=1...egment=19
      Jun 07 22:53:36 prometheus prometheus[14647]: level=info ts=2021-06-07T14:53:36.990Z caller=head.go:745 component=tsdb msg="WAL replay completed" checkpo....353439ms
      Jun 07 22:53:36 prometheus prometheus[14647]: level=info ts=2021-06-07T14:53:36.993Z caller=main.go:799 fs_type=XFS_SUPER_MAGIC
      Jun 07 22:53:36 prometheus prometheus[14647]: level=info ts=2021-06-07T14:53:36.993Z caller=main.go:802 msg="TSDB started"
      Jun 07 22:53:36 prometheus prometheus[14647]: level=info ts=2021-06-07T14:53:36.993Z caller=main.go:928 msg="Loading configuration file" filename=/opt/mo...theus.yml
      Jun 07 22:53:36 prometheus prometheus[14647]: level=info ts=2021-06-07T14:53:36.996Z caller=main.go:959 msg="Completed loading of configuration file" filename=/op…µs
      Jun 07 22:53:36 prometheus prometheus[14647]: level=info ts=2021-06-07T14:53:36.996Z caller=main.go:751 msg="Server is ready to receive web requests."
      Hint: Some lines were ellipsized, use -l to show in full.
      
    • 浏览器验证prometheus配置文件是否生成
      prometheus-监控docker服务器

    • 使用grafana进行监控docker服务数据展示
      prometheus-监控docker服务器

      • 导入监控docker的仪表盘,ID为193
        prometheus-监控docker服务器

      • 填写名称,选择数据源
        prometheus-监控docker服务器

      • 发现已有监控数据了
        prometheus-监控docker服务器

    • 监控docker服务添加一个导航栏
      prometheus-监控docker服务器
      prometheus-监控docker服务器
      prometheus-监控docker服务器
      prometheus-监控docker服务器
      prometheus-监控docker服务器

      点击save dashboard保存

      prometheus-监控docker服务器

      发现有了导航点击数据没有变化
      我们需要修改图表信息

      prometheus-监控docker服务器

      每张图片添加如上信息

      prometheus-监控docker服务器
      发现修改之后,就有了变化了