1. input-file收集日志信息

 [yun@mini04 config]$ pwd

 /app/logstash/config

 [yun@mini04 config]$ cat file.conf

 input{

   file{

     path => ["/var/log/messages", "/var/log/secure"]

     type => "system-log"

     start_position => "beginning"

   }

 }

 filter{

 }

 output{

   # es有3台，随便指定一台即可  也可以是多台如 ["127.0.0.1:9200","127.0.0.2:9200"]

   elasticsearch {

      hosts => ["mini01:9200", "mini02:9200", "mini03:9200"]

      index => "system-log-%{+YYYY.MM}"

   }

 }

 ##################################################

 [root@mini04 ~]# /app/logstash/bin/logstash -f /app/logstash/config/file.conf   # 启动  此处需要root用户启动才行，否则没有权限

 …………

1.1. 浏览器访问

ELK-logstash-6.3.2-常用配置

2. input-if判断【日志多点收集】

为了方便，我把logstatsh部署到了mini03上

本节作用：收集java日志【日志收集得有些缺陷，不方便查看，需要改进配置】

 [yun@mini03 config]$ pwd

 /app/logstash/config

 [yun@mini03 config]$ cat file2.conf

 input{

   file{

     path => ["/var/log/messages", "/var/log/secure"]

     type => "system-log"

     start_position => "beginning"

   }

   file{

     path => ["/app/es-data/logs/zhang-es.log"]

     type => "es-log"

     start_position => "beginning"

   }

 }

 filter{

 }

 output{

   # es有3台，随便指定一台即可  也可以是多台如 ["127.0.0.1:9200","127.0.0.2:9200"]

   if [type=] == "system-log" {

     elasticsearch {

        hosts => ["mini01:9200", "mini02:9200", "mini03:9200"]

        index => "system-log-%{+YYYY.MM}"

     }

   }

   if [type] == "es-log" {

     elasticsearch {

        hosts => ["mini01:9200", "mini02:9200", "mini03:9200"]

        index => "es-log-%{+YYYY.MM}"

     }

   }

 }

 ##################################

 [root@mini04 ~]# /app/logstash/bin/logstash -f /app/logstash/config/file2.conf   # 启动  此处需要root用户启动才行，否则没有权限

 …………

浏览器访问

 http://mini01:9100/       # head访问

ELK-logstash-6.3.2-常用配置

 http://mini01:5601      # kibana 访问

ELK-logstash-6.3.2-常用配置

缺点：

　　java应用的日志有报错等，这样直接收集那么不方便查看看

3. codec-multiline——多行合并收集【可用于java程序日志收集】

3.1. 命令行输入输出测试

多行合并，以 [ 开头作为匹配

 # 配置文件

 [yun@mini03 config]$ pwd

 /app/logstash/config

 [yun@mini03 config]$ cat codec_test.conf

 # 输入配置说明：

 # pattern => "^\["   匹配 [ 开头的行；

 # negate => "true" 表示如果不能匹配则放在一起；

 # what => "previous"  如果是"previous"表示，任何不以 [ 开头的行都应该与前面的行合并。

 #      如果为"next" 表示, 任何以 [ 结尾的行都应该与以下行合并。

 input{

   stdin{

     codec => multiline {

       pattern => "^\["

       negate => "true"

       what => "previous"

     }

   }

 }

 filter{

 }

 output{

   stdout{

     codec => rubydebug

   }

 }

 # 执行

 [yun@mini03 config]$ /app/logstash/bin/logstash -f /app/logstash/config/codec_test.conf   # 执行

 ………………

 [

 {

           "host" => "mini03",

        "message" => "1111\n222\n333",

       "@version" => "",

           "tags" => [

         [] "multiline"

     ],

     "@timestamp" => --25T06::.486Z

 }

 [

 {

           "host" => "mini03",

        "message" => "[444\n555\n666\n8888",

       "@version" => "",

           "tags" => [

         [] "multiline"

     ],

     "@timestamp" => --25T06::.319Z

 }

3.2. 重新收集ES日志

3.2.1. 在ES上删除之前收集的mini03 ES日志

停止mini03上的logstash程序

ELK-logstash-6.3.2-常用配置

3.2.2. 删除logstash的标记

插件通过在一个名为sincedb的单独文件并记录每个文件中当前的位置来跟踪当前位置。这样就可以停止并重新启动Logstash，并让它在结束的地方继续运行，而不会遗漏在log出来时添加到文件中的行。

 # 查找标记文件

 [yun@mini03 logstash]$ pwd

 /app/logstash

 [yun@mini03 logstash]$ find . -type f | grep 'sincedb'

 ./data/plugins/inputs/file/.sincedb_1fb922e15ccea4ac0d028d33639ba3ea

 ./data/plugins/inputs/file/.sincedb_56a0ba191c6aa2202fcdc058933e33b0

 ##### mini03 es的日志信息

 [yun@mini03 logs]$ pwd

 /app/es-data/logs

 [yun@mini03 logs]$ ll -i zhang-es.log

  -rw-rw-r--  yun yun  Aug  : zhang-es.log  # 第一列为es的inode信息

 ##### logstash sincedb 的文件信息

 [yun@mini03 file]$ pwd

 /app/logstash/data/plugins/inputs/file

 [yun@mini03 file]$ ll -a

 total

 drwxr-xr-x  yun yun  Aug  : .

 drwxr-xr-x  yun yun   Aug  : ..

 -rw-r--r--  yun yun   Aug  : .sincedb_1fb922e15ccea4ac0d028d33639ba3ea

 -rw-r--r--  yun yun   Aug  : .sincedb_56a0ba191c6aa2202fcdc058933e33b0

 [yun@mini03 file]$ cat .sincedb_56a0ba191c6aa2202fcdc058933e33b0 

 [yun@mini03 file]$ rm -f .sincedb_56a0ba191c6aa2202fcdc058933e33b0  # 删除es的sincedb文件

说明：其中 33588216为对应es日志的inode信息，所以删除 .sincedb_56a0ba191c6aa2202fcdc058933e33b0 文件，那么再次采集es日志时，就会从新开始采集

3.2.3. logstash配置并启动

 [yun@mini03 config]$ pwd

 /app/logstash/config

 [yun@mini03 config]$ cat codec.conf

 input{

   file{

     path => ["/var/log/messages", "/var/log/secure"]

     type => "system-log"

     start_position => "beginning"

   }

   file{

     path => ["/app/es-data/logs/zhang-es.log"]

     type => "es-log"

     start_position => "beginning"

     codec => multiline {

       pattern => "^\["

       negate => "true"

       what => "previous"

     }

   }

 }

 filter{

 }

 output{

   # es有3台，随便指定一台即可  也可以是多台如 ["127.0.0.1:9200","127.0.0.2:9200"]

   if [type=] == "system-log" {

     elasticsearch {

        hosts => ["mini01:9200", "mini02:9200", "mini03:9200"]

        index => "system-log-%{+YYYY.MM}"

     }

   }

   if [type] == "es-log" {

     elasticsearch {

        hosts => ["mini01:9200", "mini02:9200", "mini03:9200"]

        index => "es-log-%{+YYYY.MM}"

     }

   }

 }

 #### 使用root权限启动，因为该配置中有 "/var/log/messages", "/var/log/secure" 日志收集

 [root@mini03 ~]# /app/logstash/bin/logstash -f /app/logstash/config/codec.conf &

3.2.4. 浏览器通过kibana查看

通过kibana查询，得知此次收集的日志确实符合我们的浏览习惯。

ELK-logstash-6.3.2-常用配置

4. codec-json【收集Nginx访问日志】

需要将Nginx的访问日志改为json格式

4.1. nginx 部分日志配置

在mini03 yum安装Nginx

[root@mini03 ~]# vim /etc/nginx/nginx.conf

# For more information on configuration, see:

#   * Official English Documentation: http://nginx.org/en/docs/

#   * Official Russian Documentation: http://nginx.org/ru/docs/

user nginx;

worker_processes auto;

error_log /var/log/nginx/error.log;

pid /run/nginx.pid;

# Load dynamic modules. See /usr/share/nginx/README.dynamic.

include /usr/share/nginx/modules/*.conf;

events {

    worker_connections 1024;

}

http {

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '

                      '$status $body_bytes_sent "$http_referer" '

                      '"$http_user_agent" "$http_x_forwarded_for"';

    # 新增配置，上面的配置没有被引用，所以可以不管

    # 切记：不要换行★★★★★

    log_format  access_log_json  '{"user_ip":"$http_x_real_ip","lan_ip":"$remote_addr","log_time":"$time_iso8601","user_req":"$request","http_code":"$status","body_bytes_sent":"$body_bytes_sent","req_time":"$request_time","user_ua":"$http_user_agent"}';

    # access_log  /var/log/nginx/access.log  main;  # 注释

    access_log  /var/log/nginx/access_log_json.log  access_log_json;  # 新增

4.2. logstash配置并启动

 [yun@mini03 config]$ pwd

 /app/logstash/config

 [yun@mini03 config]$ cat codec_json.conf

 input{

   file{

     path => ["/var/log/nginx/access_log_json.log"]

     type => "nginx-access-log"

     codec => json

   }

 }

 filter{

 }

 output{

   # es有3台，随便指定一台即可  也可以是多台如 ["127.0.0.1:9200","127.0.0.2:9200"]

   elasticsearch {

     hosts => ["mini01:9200", "mini02:9200", "mini03:9200"]

     index => "nginx-access-log-%{+YYYY.MM.dd}"

   }

 }

 ##### 需要root权限，因为Nginx是yum安装的  访问日志在/var/log/nginx/access_log_json.log中

 [root@mini03 ~]# /app/logstash/bin/logstash -f /app/logstash/config/codec_json.conf &

4.3. 浏览器访问Nginx

访问方式如下：

通过浏览器访问 http://mini03/

 http://mini03/32t23t23t/ee    # 可以得到404状态码

在mini01、mini02、mini03 通过如下命令访问

 # 需要安装软件

 yum -y install httpd-tools

 # 访问命令如下

 ab -n10 -c  http://mini03/

 ab -n10 -c  http://mini03/aa/bbb/ccc    # 为了得到404 状态码

4.4. 信息查看

通过head查看

ELK-logstash-6.3.2-常用配置

通过kibana查看

ELK-logstash-6.3.2-常用配置

5. input-rsyslog日志收集

要求：收集mini01、mini02、mini03的rsyslog日志

5.1. rsyslog日志收集测试

logstash配置

 [yun@mini03 config]$ pwd

 /app/logstash/config

 [yun@mini03 config]$ cat rsyslog_test.conf

 input{

   syslog{

     type => "system-rsyslog"

     port =>

   }

 }

 filter{

 }

 output{

   stdout{

     codec => rubydebug

   }

 }

 ##### 使用root用户，不然有权限限制

 [root@mini03 ~]# /app/logstash/bin/logstash -f /app/logstash/config/rsyslog_test.conf

mini01、mini02、mini03配置修改

 [root@mini01 ~]# tail -n5 /etc/rsyslog.conf   # mini01、mini02、mini03

 # remote host is: name/ip:port, e.g. 192.168.0.1:, port optional

 #*.* @@remote-host:

 # 下面要添加的配置

 *.* @@172.16.1.13:

 # ### end of the forwarding rule ###

 [root@mini01 ~]# systemctl restart rsyslog.service  # 重启rsyslog

在mini03的logstash上，可见有rsyslog刷过来。

5.2. rsyslog收集到ES配置

其中mini01、mini02、mini03上的配置已经按上面修改，因此不用改变。

logstash配置

 [yun@mini03 config]$ pwd

 /app/logstash/config

 [yun@mini03 config]$ cat rsyslog.conf

 input{

   syslog{

     type => "system-rsyslog"

     port =>

   }

 }

 filter{

 }

 output{

   # es有3台，随便指定一台即可  也可以是多台如 ["127.0.0.1:9200","127.0.0.2:9200"]

   elasticsearch {

     hosts => ["mini01:9200", "mini02:9200", "mini03:9200"]

     index => "system-rsyslog-%{+YYYY.MM}"

   }

 }

 ##### 使用root用户，不然有权限限制

 [root@mini03 ~]# /app/logstash/bin/logstash -f /app/logstash/config/rsyslog.conf &

5.3. 浏览器查看

通过head查看

ELK-logstash-6.3.2-常用配置

通过kibana查看

ELK-logstash-6.3.2-常用配置

6. input-tcp收集

这次只做测试，就不收集到ES了。

6.1. logstash配置

 [yun@mini03 config]$ pwd

 /app/logstash/config

 [yun@mini03 config]$ cat tcp_test.conf

 input{

   tcp {

     port =>

     mode => "server"

     type => "tcp_test"

   }

 }

 filter{

 }

 output{

   stdout{

     codec => rubydebug

   }

 }

 ##########################

 [yun@mini03 ~]$ /app/logstash/bin/logstash -f /app/logstash/config/tcp_test.conf  # 可以使用普通用户

6.2. 在mini02测试

 [yun@mini02 ~]$ echo "" | nc mini03

 [yun@mini02 ~]$ echo "testinfo" | nc mini03

 [yun@mini02 ~]$ nc mini03  < /etc/resolv.conf

 [yun@mini02 ~]$ echo "myinfo" > /dev/tcp/mini03/

在mini03上可见，命令行有logstash的信息输出

7. filter-Grok

生产环境几乎不用

原因：

1、grok是非常影响性能的

2、不灵活

最佳实践：做到分离，各司其职

 logstash => redis/kafka => logstash/python => ES

7.1. 查看grok位置和文件

 [yun@mini03 patterns]$ pwd

 /app/logstash/vendor/bundle/jruby/2.3./gems/logstash-patterns-core-4.1./patterns

 [yun@mini03 patterns]$ ll

 total

 -rw-r--r--  yun yun   Jul  : aws

 -rw-r--r--  yun yun   Jul  : bacula

 -rw-r--r--  yun yun    Jul  : bind

 -rw-r--r--  yun yun   Jul  : bro

 -rw-r--r--  yun yun    Jul  : exim

 -rw-r--r--  yun yun  Jul  : firewalls

 -rw-r--r--  yun yun   Jul  : grok-patterns

 -rw-r--r--  yun yun   Jul  : haproxy

 -rw-r--r--  yun yun    Jul  : httpd

 -rw-r--r--  yun yun   Jul  : java

 -rw-r--r--  yun yun   Jul  : junos

 -rw-r--r--  yun yun   Jul  : linux-syslog

 -rw-r--r--  yun yun     Jul  : maven

 -rw-r--r--  yun yun     Jul  : mcollective

 -rw-r--r--  yun yun    Jul  : mcollective-patterns

 -rw-r--r--  yun yun    Jul  : mongodb

 -rw-r--r--  yun yun   Jul  : nagios

 -rw-r--r--  yun yun    Jul  : postgresql

 -rw-r--r--  yun yun    Jul  : rails

 -rw-r--r--  yun yun    Jul  : redis

 -rw-r--r--  yun yun    Jul  : ruby

 -rw-r--r--  yun yun    Jul  : squid

7.2. 命令行测试

 [yun@mini03 config]$ pwd

 /app/logstash/config

 [yun@mini03 config]$

 [yun@mini03 config]$ cat filter-grok_test.conf

 input{

   stdin{}

 }

 filter{

   grok {

     match => { "message" => "%{IP:client} %{WORD:method} %{URIPATHPARAM:request} %{NUMBER:bytes} %{NUMBER:duration}" }

   }

 }

 output{

   stdout{

     codec => rubydebug

   }

 }

 #######################################

 [yun@mini03 ~]$ /app/logstash/bin/logstash -f /app/logstash/config/filter-grok_test.conf  # 使用普通用户

 ……………………

 # 输入如下一行字符串

 55.3.244.1 GET /index.html  0.043

 {

       "@version" => "",

           "host" => "mini03",

          "bytes" => "",

        "message" => "55.3.244.1 GET /index.html 15824 0.043",

         "client" => "55.3.244.1",

       "duration" => "0.043",

        "request" => "/index.html",

     "@timestamp" => --28T13::.910Z,

         "method" => "GET"

 }

7.3. httpd日志收集命令行测试

[yun@mini03 config]$ pwd

/app/logstash/config

[yun@mini03 config]$ cat filter-grok_httpd-test.conf

input{

  file{

    path => ["/var/log/httpd/access_log"]

    type => "httpd-access-log"

    start_position => "beginning"

  }

}

filter{

  grok {

    match => { "message" => "%{HTTPD_COMBINEDLOG}" }

  }

}

output{

  stdout{

    codec => rubydebug

  }

}

################# 使用root用户，涉及权限问题

[root@mini03 ~]# /app/logstash/bin/logstash -f /app/logstash/config/filter-grok_httpd-test.conf

……………………

# 可见httpd的日志被收集，并且被解析

{

           "path" => "/var/log/httpd/access_log",

       "referrer" => "\"http://mini03/\"",

           "host" => "mini03",

       "response" => "200",

        "message" => "10.0.0.1 - - [28/Aug/2018:22:35:31 +0800] \"GET /images/poweredby.png HTTP/1.1\" 200 3956 \"http://mini03/\" \"Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36\"",

           "auth" => "-",

      "timestamp" => "28/Aug/2018:22:35:31 +0800",

          "bytes" => "3956",

       "clientip" => "10.0.0.1",

          "agent" => "\"Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36\"",

       "@version" => "1",

     "@timestamp" => 2018-08-28T14:44:12.477Z,

    "httpversion" => "1.1",

           "type" => "httpd-access-log",

          "ident" => "-",

        "request" => "/images/poweredby.png",

           "verb" => "GET"

}

………………

7.4. httpd收集日志到ES

 [yun@mini03 config]$ pwd

 /app/logstash/config

 [yun@mini03 config]$ cat filter-grok_httpd.conf

 input{

   file{

     path => ["/var/log/httpd/access_log"]

     type => "httpd-access-log"

     start_position => "beginning"

   }

 }

 filter{

   grok {

     match => { "message" => "%{HTTPD_COMBINEDLOG}" }

   }

 }

 output{

   # es有3台，随便指定一台即可  也可以是多台如 ["127.0.0.1:9200","127.0.0.2:9200"]

   elasticsearch {

     hosts => ["mini01:9200", "mini02:9200", "mini03:9200"]

     index => "httpd-access-log-%{+YYYY.MM.dd}"

   }

 }

 ########## 使用root用户，涉及权限

 [root@mini03 ~]# /app/logstash/bin/logstash -f /app/logstash/config/filter-grok_httpd.conf

 ………………

7.5. 浏览器访问httpd

浏览器

 # 可以通过谷歌、火狐、IE访问

 http://mini03/

 http://mini03/indweg.html

Linux命令行访问

 [yun@mini02 ~]$ ab -n40 -c  http://mini03/

 [yun@mini02 ~]$ ab -n40 -c  http://mini03/wet/bdhw/

7.6. 信息查看

head访问

ELK-logstash-6.3.2-常用配置

kibana查看

ELK-logstash-6.3.2-常用配置

秒客网

ELK-logstash-6.3.2-常用配置

1. input-file收集日志信息

1.1. 浏览器访问

2. input-if判断【日志多点收集】

3. codec-multiline——多行合并收集【可用于java程序日志收集】

3.1. 命令行输入输出测试

3.2. 重新收集ES日志

3.2.1. 在ES上删除之前收集的mini03 ES日志

3.2.2. 删除logstash的标记

3.2.3. logstash配置并启动

3.2.4. 浏览器通过kibana查看

4. codec-json【收集Nginx访问日志】

4.1. nginx 部分日志配置

4.2. logstash配置并启动

4.3. 浏览器访问Nginx

4.4. 信息查看

5. input-rsyslog日志收集

5.1. rsyslog日志收集测试

5.2. rsyslog收集到ES配置

5.3. 浏览器查看

6. input-tcp收集

6.1. logstash配置

6.2. 在mini02测试

7. filter-Grok

7.1. 查看grok位置和文件

7.2. 命令行测试

7.3. httpd日志收集命令行测试

7.4. httpd收集日志到ES

7.5. 浏览器访问httpd

7.6. 信息查看

相关文章