先上一个架构图
Flink App ： 通过report 将数据发出去
Pushgateway : Prometheus 生态中一个重要工具
Prometheus : 一套开源的系统监控报警框架 （Prometheus 入门与实践）
Grafana： 一个跨平台的开源的度量分析和可视化工具，可以通过将采集的数据查询然后可视化的展示，并及时通知（可视化工具Grafana：简介及安装）
Node_exporter : 跟Pushgateway一样是Prometheus 的组件，采集到主机的运行指标如CPU, 内存，磁盘等信息
以下安装，大部分参考博客： https://www.cnblogs.com/xiao987334176/p/9930517.html#autoid-0-0-0
1、docker pull 镜像
docker pull prom/node-exporter
docker pull prom/pushgateway
docker pull prom/prometheus
docker pull grafana/grafana
查看下载的镜像
$ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
prom/prometheus latest d5b9d7ed160a weeks ago 138MB
grafana/grafana latest a6e14b4109af weeks ago 253MB
prom/pushgateway latest 20e6dcae675f weeks ago .2MB
prom/node-exporter latest e5a616e4b9cf months ago .9MB
2、编辑prometheus.yml 、创建 Grafana 数据存储目录
$ mkdir /opt/grafana-storage # grafana 数据存储目录 $ cat /opt/prometheus/prometheus.yml # prometheus 配置
global:
scrape_interval: 60s
evaluation_interval: 60s scrape_configs:
- job_name: prometheus
static_configs:
- targets: ['localhost:9090']
labels:
instance: prometheus - job_name: linux
static_configs:
- targets: ['venn:9100']
labels:
instance: localhost
- job_name: 'pushgateway'
static_configs:
- targets: ['venn:9091']
labels:
instance: 'pushgateway'
3、启动各个组件
docker run -d -p : --name=grafana -v /opt/grafana-storage:/var/lib/grafana grafana/grafana
docker run -d -p : -v "/proc:/host/proc:ro" -v "/sys:/host/sys:ro" -v "/:/rootfs:ro" --net="host" prom/node-exporter
docker run -d -p : -v /opt/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml prom/prometheus
docker run -d -p : prom/pushgateway
查看docker进程
$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
4a689cf48e10 prom/pushgateway "/bin/pushgateway" days ago Up days 0.0.0.0:->/tcp infallible_goldstine
fcc40433bf75 grafana/grafana "/run.sh" days ago Up days 0.0.0.0:->/tcp grafana
8ba942d0cf35 prom/prometheus "/bin/prometheus --c…" days ago Up days 0.0.0.0:->/tcp quizzical_colden
b84b0f4be2b2 prom/node-exporter "/bin/node_exporter" days ago Up days fervent_poitras
查看端口
$ netstat -apn | grep -E '9091|3000|9090|9100'
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
tcp 172.17.0.1: 172.17.0.4: ESTABLISHED -
tcp6 ::: :::* LISTEN -
tcp6 ::: :::* LISTEN -
tcp6 ::: :::* LISTEN -
tcp6 ::: :::* LISTEN -
tcp6 192.168.229.129: 192.168.229.128: TIME_WAIT -
tcp6 192.168.229.129: 192.168.229.128: TIME_WAIT -
tcp6 192.168.229.129: 192.168.229.128: TIME_WAIT -
tcp6 192.168.229.129: 192.168.229.128: TIME_WAIT -
tcp6 192.168.229.129: 192.168.229.128: TIME_WAIT -
tcp6 192.168.229.129: 192.168.229.128: TIME_WAIT -
tcp6 192.168.229.129: 192.168.229.128: TIME_WAIT -
tcp6 192.168.229.128: 192.168.229.1: ESTABLISHED -
tcp6 192.168.229.129: 192.168.229.128: TIME_WAIT -
tcp6 192.168.229.129: 192.168.229.128: TIME_WAIT -
tcp6 192.168.229.128: 172.17.0.2: ESTABLISHED -
tcp6 192.168.229.129: 192.168.229.128: TIME_WAIT -
tcp6 192.168.229.128: 172.17.0.2: ESTABLISHED -
tcp6 192.168.229.129: 192.168.229.128: TIME_WAIT -
tcp6 192.168.229.129: 192.168.229.128: ESTABLISHED /java
tcp6 192.168.229.129: 192.168.229.128: TIME_WAIT -
tcp6 192.168.229.129: 192.168.229.128: TIME_WAIT -
4、查看组件页面
node_exporter: ip:9100/metrics
查看 prometheus： ip:9090/targets
如果state 不是 UP 的，等一会就起来了
查看Grafana：
默认用户名密码 ： amin/admin
此处不再赘述，配置数据源、创建系统负载监控参考博客：https://www.cnblogs.com/xiao987334176/p/9930517.html#autoid-0-0-0
5、配置Flink report ：
在Flink 配置文件 flink-conf.yml 中添加如下内容：
##metrics
metrics.reporter.promgateway.class: org.apache.flink.metrics.prometheus.PrometheusPushGatewayReporter
metrics.reporter.promgateway.host: venn
metrics.reporter.promgateway.port:
metrics.reporter.promgateway.jobName: myJob
metrics.reporter.promgateway.randomJobNameSuffix: true
metrics.reporter.promgateway.deleteOnShutdown: false
启动一个任务（上一篇博客的案例迟到数据处理）：
flink run -m yarn-cluster -ynm LateDataProcess -yn -c com.venn.stream.api.sideoutput.lateDataProcess.LateDataProcess jar/flinkDemo-1.0.jar
查看任务webUI：
PS：任务已经跑了一段时间了
6、Grafana 中配置Flink监控
由于上面一句配置好Flink report、 pushgateway、prometheus，并且在Grafana中已经添加了prometheus 数据源，所以Grafana中会自动获取到 flink job的metrics 。
Grafana 首页，点击New dashboard，创建一个新的dashboard
选中之后，即会出现对应的监控指标
至此，Flink 的metrics 的指标展示在Grafana 中了
flink 指标对应的指标名比较长，可以在Legend 中配置显示内容，在{{key}} 将key换成对应需要展示的字段即可，如： {{job_name}},{{operator_name}}
对应显示如下：
保存，搞定