提前安装好elk（elasticsearch、logstach、kibana）
一 启动logstash
$LOGSTASH_HOME默认位于/usr/share/logstash或/opt/logstash
1 nginx日志使用默认格式
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';
2 下载geo库
# cd /etc/logstash
# wget https://geolite.maxmind.com/download/geoip/database/GeoLite2-City.tar.gz
# tar xvf GeoLite2-City.tar.gz
3 增加logstash配置
# cat /etc/logstash/conf.d/nginx_access.conf
input {
file {
path => [ "/path/to/nginx/access.log" ]
start_position => "beginning"
ignore_older => 0
}
}
filter {
grok {
match => { "message" => "%{IPORHOST:client_ip} (%{USER:ident}|-) (%{USER:auth}|-) \[%{HTTPDATE:timestamp}\] \"(?:%{WORD:verb} (%{NOTSPACE:request}|-)(?: HTTP/%{NUMBER:http_version})?|-)\" (%{NUMBER:response}|-) (?:%{NUMBER:bytes}|-) %{QS:referrer} %{QS:agent}" }
}
geoip {
source => "client_ip"
target => "geoip"
database => "/etc/logstash/GeoLite2-City_20190122/GeoLite2-City.mmdb"
add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}" ]
add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}" ]
}
mutate {
convert => [ "[geoip][coordinates]", "float" ]
convert => [ "response","integer" ]
convert => [ "bytes","integer" ]
replace => { "type" => "nginx_access" }
remove_field => "message"
}
date {
match => [ "timestamp","dd/MMM/yyyy:HH:mm:ss Z"]
}
mutate {
remove_field => "timestamp"
}
}
output {
elasticsearch {
hosts => ["$es_server:9200"]
index => "logstash-nginx-access-%{+YYYY.MM.dd}"
}
stdout {codec => rubydebug}
}
修改其中的nginx日志路径以及es的host；
如果你修改过nginx log format，还需要修改grok格式，grok格式可以在这里调试：http://grokdebug.herokuapp.com/
如果格式有误，会报 _grokparsefailure
4 测试配置是否正常
$LOGSTASH_HOME/bin/logstash -t -f /etc/logstash/conf.d/nginx_access.conf
正常应该会打印Configuration OK
5 启动logstash
$LOGSTASH_HOME/bin/logstash -f /etc/logstash/conf.d/nginx_access.conf
二 查看elasticsearch
# curl http://$es_server:9200/_cat/indices?v
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
green open logstash-nginx-access-2019.01.26 -szaQCE3THyq-fXbU18riQ 5 1 7875 0 862.7kb 465.2kb
此时es中应该有了索引
三 配置kibana
浏览器打开：http://$kibana_server:5000
配置索引pattern为：logstash-nginx-access-*
配置Time-field为：@timestamp
然后可以配置各种visualization和dashboard