ELK监控系统nginx / mysql慢日志

时间：2024-08-05 17:05:38

ELK监控系统nginx / mysql慢日志

elasticsearch
logstash
kibana

ELK监控系统nginx日志

1.环境准备

centos6.8_64 mini

IP:192.168.10.78

tar包：

logstash-2.4.0.tar.gz

elasticsearch-2.4.0.tar.gz

kibana-4.6.1-linux-x86_64.tar.gz

JDK环境

elasticsearch logstach kibana java
V2.4 V2.4 V4.6.1 V1.8.0_111

nginx采用yum安装即可，安装后启动此服务。

2.logstash简单配置编写

logstash目录下新建测试文件test.conf 其内动如下

  1. input { 
  2. file { 
  3. type => "syslog" 
  4. tags => ["log"] 
  5. path => ["/var/log/messages","/log/*.log"] 
  6. start_position => beginning 
  7. ignore_older => 0 
  8. } 
  9. file { 
  10. type => "nginx_log" 
  11. tags => ["nginx"] 
  12. path => ["/var/log/nginx/access.log"] 
  13. start_position => beginning 
  14. ignore_older => 0 
  15. } 
  16. } 
  19. output 
  20. { 
  22. elasticsearch { 
  23. hosts => [ "192.168.10.78:9200" ] 
  24. } 
  26. } 

编写好后测试运行：

  1. [root@localhost logstash]# ./bin/logstash -f test.conf  
  2. Settings: Default pipeline workers: 2 
  3. Pipeline main started 

此时logstash已经运行，我们通过浏览器访问web：192.168.10.68

ELK监控系统nginx / mysql慢日志

同时在终端下重启网卡服务，用以查看系统日志是否有输出至kibana

打开kibana查看日志输出

ELK监控系统nginx / mysql慢日志

测试kibana.png

可以看见有网卡日志和web访问日志。

接下来监控mysql日志

yum install mysql mysql-server -y

安装完直接在kibana就看见安装mysql的日志输出结果了

Time    type    tags    message

November 10th 2016, 10:18:13.193    syslog  log Nov 10 10:18:12 localhost yum[11445]: Installed: mysql-server-5.1.73-7.el6.x86_64

November 10th 2016, 10:18:13.057    syslog  log Nov 10 10:18:12 localhost yum[11445]: Installed: mysql-server-5.1.73-7.el6.x86_64

November 10th 2016, 10:18:10.187    syslog  log Nov 10 10:18:09 localhost yum[11445]: Installed: mysql-5.1.73-7.el6.x86_64

November 10th 2016, 10:18:10.185    syslog  log Nov 10 10:18:09 localhost yum[11445]: Installed: perl-DBD-MySQL-4.013-3.el6.x86_64

November 10th 2016, 10:18:10.183    syslog  log Nov 10 10:18:09 localhost yum[11445]: Installed: perl-DBI-1.609-4.el6.x86_64

November 10th 2016, 10:18:10.051    syslog  log Nov 10 10:18:09 localhost yum[11445]: Installed: mysql-5.1.73-7.el6.x86_64

November 10th 2016, 10:18:10.050    syslog  log Nov 10 10:18:09 localhost yum[11445]: Installed: perl-DBD-MySQL-4.013-3.el6.x86_64

November 10th 2016, 10:18:10.048    syslog  log Nov 10 10:18:09 localhost yum[11445]: Installed: perl-DBI-1.609-4.el6.x86_64

接下来修改test.conf配置文件，添加对mysql满日志的监控

1.yum install mysql mysql-server

2.service network restart

3.mysql慢日志开启:

(此处参考http://blog.itpub.net/29500582/viewspace-1432985/)

Mysql 启动慢查询日志 （不用重启）

  1. 查看mysql系统参数

mysql> show variables like "%slow%";

+---------------------------+-------------------------------+

| Variable_name | Value |

+---------------------------+-------------------------------+

| log_slow_admin_statements | OFF |

| log_slow_slave_statements | OFF |

| slow_launch_time | 2 |

| slow_query_log | OFF |

| slow_query_log_file | /mysql/data/nagiosdb-slow.log |

+---------------------------+-------------------------------+

5 rows in set (0.00 sec)

slow_query_log： off关闭状态 on开启状态

slow_launch_time 默认超过2s为慢查询

slow_query_log_file 慢查询日志存放地点

这三个参数，在不同的mysql版本中，不太一样，不过都可以通过 show variables like "%slow%" 查看出来

  1. 运行如下命令即可运行慢查询日志

mysql> set global slow_query_log=ON;

Query OK, 0 rows affected (0.03 sec)

mysql> set global slow_launch_time=5;

Query OK, 0 rows affected (0.00 sec)

mysql> show variables like "%slow%";

+---------------------------+-------------------------------+

| Variable_name | Value |

+---------------------------+-------------------------------+

| log_slow_admin_statements | OFF |

| log_slow_slave_statements | OFF |

| slow_launch_time | 5 |

| slow_query_log | ON |

| slow_query_log_file | /mysql/data/nagiosdb-slow.log |

+---------------------------+-------------------------------+

5 rows in set (0.00 sec)

mysql 5.1.6版本起，slow_query_log 和 slow_launch_time 支持写文件或写数据库表两种方式，并且日志的开启，输出方式的修改，都可以在global级别动态修改。

只需简单通过set global slow_query_log=ON;即可开启慢查询，而不需要重启数据库！

  1. 可以直接写到配置文件中 my.cnf

slow_query_log_file=/mysql/log/nagiosdb-slow.log

slow_launch_time=5

可以完成配置！！

！！！根据上述配置开启慢日志后，查询本机日志名称及目录，不要搞错。

在test.conf配置文件添加如下：

  1. # this is mysql log 
  2. file { 
  3. type => "mysql_log" 
  4. tags => ["mysql"] 
  5. path => ["/var/log/mysql/mysqld.log"] 
  6. start_position => beginning 
  7. ignore_older => 0 
  8. } 
  9. # this is mysql-slow log 
  10. file { 
  11. type => "mysql_slow" 
  12. tags => ["mysql-slow"] 
  13. path => ["/var/run/mysqld/mysqld-slow.log"] 
  14. start_position => beginning 
  15. ignore_older => 0 
  16. } 

配置完后启动logstash

[root@localhost logstash]# ./bin/logstash -f test.conf

确认慢查询已开启

mysql> show variables like "%slow%";

+---------------------+---------------------------------+

| Variable_name       | Value                           |

+---------------------+---------------------------------+

| log_slow_queries    | ON                              |

| slow_launch_time    | 2                               |

| slow_query_log      | ON                              |

| slow_query_log_file | /var/run/mysqld/mysqld-slow.log |

+---------------------+---------------------------------+

4 rows in set (0.00 sec)

mysql>

输入测试命令，并查看kibana是否有mysql-low数据输出；

mysql> select sleep(6);

+----------+

| sleep(6) |

+----------+

|        0 |

+----------+

1 row in set (5.99 sec)

mysql>

慢数据输出正常：

ELK监控系统nginx / mysql慢日志

!!!目前所有日志数据均未作规范化输出处理，只是简单测试了elk的基本功能，后续将升入研究学习elk.后面会继续做好笔记供自己参考学习。

相关文章