前期博客
基于CentOS6.5下Suricata（一款高性能的网络IDS、IPS和网络安全监控引擎）的搭建（图文详解）（博主推荐）
基于Ubuntu14.04下Suricata（一款高性能的网络IDS、IPS和网络安全监控引擎）的搭建（图文详解）（博主推荐）
参考官网
https://redmine.openinfosecfoundation.org/projects/suricata/wiki/_Logstash_Kibana_and_Suricata_JSON_output
注意，这官网，默认是适合Ubuntu系统的安装手法。
首先，
Make sure your Suricata is compiled/installed with libjansson support enabled:
[root@suricata ~]# suricata --build-info
This is Suricata version 3.1 RELEASE
Features: PCAP_SET_BUFF LIBPCAP_VERSION_MAJOR= AF_PACKET HAVE_PACKET_FANOUT LIBCAP_NG LIBNET1. HAVE_HTP_URI_NORMALIZE_HOOK HAVE_NSS HAVE_LIBJANSSON TLS
SIMD support: SSE_4_2 SSE_4_1 SSE_3
Atomic intrisics: byte(s)
-bits, Little-endian architecture
GCC version 4.4. (Red Hat 4.4.-), C version
compiled with _FORTIFY_SOURCE=
L1 cache line size (CLS)=
thread local storage method: __thread
compiled with LibHTP v0.5.20, linked against LibHTP v0.5.20 Suricata Configuration:
AF_PACKET support: yes
PF_RING support: no
NFQueue support: no
NFLOG support: no
IPFW support: no
Netmap support: no
DAG enabled: no
Napatech enabled: no Unix socket enabled: yes
Detection enabled: yes libnss support: yes
libnspr support: yes
libjansson support: yes
hiredis support: no
Prelude support: no
PCRE jit: no
LUA support: no
libluajit: no
libgeoip: no
Non-bundled htp: no
Old barnyard2 support: no
CUDA enabled: no
Hyperscan support: no
Libnet support: yes Suricatasc install: yes Profiling enabled: no
Profiling locks enabled: no Development settings:
Coccinelle / spatch: no
Unit tests enabled: no
Debug output enabled: no
Debug validation enabled: no Generic build parameters:
Installation prefix: /usr/local
Configuration directory: /usr/local/etc/suricata/
Log directory: /usr/local/var/log/suricata/ --prefix /usr/local
--sysconfdir /usr/local/etc
--localstatedir /usr/local/var Host: x86_64-pc-linux-gnu
Compiler: gcc (exec name) / gcc (real)
GCC Protect enabled: no
GCC march native enabled: yes
GCC Profile enabled: no
Position Independent Executable enabled: no
CFLAGS -g -O2 -march=native
PCAP_CFLAGS
SECCFLAGS
[root@suricata ~]#
然后，需要配置
Configure suricata
[root@suricata suricata-3.1]# pwd
/root/suricata-3.1
[root@suricata suricata-3.1]# ls
aclocal.m4 compile config.h.in config.status configure.ac depcomp libhtp ltmain.sh Makefile.am qa scripts suricata.yaml
ChangeLog config.guess config.log config.sub contrib doc libtool m4 Makefile.in reference.config src suricata.yaml.in
classification.config config.h config.rpath configure COPYING install-sh LICENSE Makefile missing rules stamp-h1 threshold.config
[root@suricata suricata-3.1]# sudo vim suricata.yaml
# Extensible Event Format (nicknamed EVE) event log in JSON format
- eve-log:
enabled: yes
filetype: regular #regular|syslog|unix_dgram|unix_stream|redis
filename: eve.json
#prefix: "@cee: " # prefix to prepend to each log entry
# the following are valid when type: syslog above
#identity: "suricata"
#facility: local5
#level: Info ## possible levels: Emergency, Alert, Critical,
## Error, Warning, Notice, Info, Debug
#redis:
# server: 127.0.0.1
# port:
# mode: list ## possible values: list (default), channel
# key: suricata ## key or channel to use (default to suricata)
# Redis pipelining set up. This will enable to only do a query every
# 'batch-size' events. This should lower the latency induced by network
# connection at the cost of some memory. There is no flushing implemented
# so this setting as to be reserved to high traffic suricata.
# pipelining:
# enabled: yes ## set enable to yes to enable query pipelining
# batch-size: ## number of entry to keep in buffer
types:
- alert:
# payload: yes # enable dumping payload in Base64
# payload-buffer-size: 4kb # max size of payload buffer to output in eve-log
# payload-printable: yes # enable dumping payload in printable (lossy) format
# packet: yes # enable dumping of packet (without stream segments)
http: yes # enable dumping of http fields
tls: yes # enable dumping of tls fields
ssh: yes # enable dumping of ssh fields
smtp: yes # enable dumping of smtp fields # HTTP X-Forwarded-For support by adding an extra field or overwriting
# the source or destination IP address (depending on flow direction)
# with the one reported in the X-Forwarded-For HTTP header. This is
# helpful when reviewing alerts for traffic that is being reverse
# or forward proxied.
xff:
enabled: no
# Two operation modes are available, "extra-data" and "overwrite".
mode: extra-data
# Two proxy deployments are supported, "reverse" and "forward". In
# a "reverse" deployment the IP address used is the last one, in a
# "forward" deployment the first IP address is used.
deployment: reverse
# Header name where the actual IP address will be reported, if more
# than one IP address is present, the last IP address will be the
# one taken into consideration.
header: X-Forwarded-For
- http:
extended: yes # enable this for extended logging information
# custom allows additional http fields to be included in eve-log
# the example below adds three additional fields when uncommented
#custom: [Accept-Encoding, Accept-Language, Authorization]
- dns
- tls:
extended: yes # enable this for extended logging information
- files:
force-magic: no # force logging magic on all logged files
force-md5: no # force logging of md5 checksums
#- drop:
# alerts: no # log alerts that caused drops
- smtp:
#extended: yes # enable this for extended logging information
# this includes: bcc, message-id, subject, x_mailer, user-agent
# custom fields logging from the list:
# reply-to, bcc, message-id, subject, x-mailer, user-agent, received,
# x-originating-ip, in-reply-to, references, importance, priority,
# sensitivity, organization, content-md5, date
#custom: [received, x-mailer, x-originating-ip, relays, reply-to, bcc]
# output md5 of fields: body, subject
# for the body you need to set app-layer.protocols.smtp.mime.body-md5
# to yes
#md5: [body, subject] - ssh
- stats:
totals: yes # stats for all threads merged together
threads: no # per thread stats
deltas: no # include delta values
# bi-directional flows
- flow
# uni-directional flows
#- netflow
改为
# "United" event log in JSON format
- eve-log:
enabled: yes
type: file #file|syslog|unix_dgram|unix_stream
filename: eve.json
# the following are valid when type: syslog above
#identity: "suricata"
#facility: local5
#level: Info ## possible levels: Emergency, Alert, Critical,
## Error, Warning, Notice, Info, Debug
types:
- alert
- http:
extended: yes # enable this for extended logging information
- dns
- tls:
extended: yes # enable this for extended logging information
- files:
force-magic: yes # force logging magic on all logged files
force-md5: yes # force logging of md5 checksums
#- drop
- ssh
- smtp
然后，接下是安装
Install ELK (elasticsearch, logstash, kibana)
先卸载自带的openjdk
rpm -qa|grep java
rpm -e --nodeps tzdata-java-2013g-.el6.noarch
rpm -e --nodeps java-1.7.-openjdk-1.7.0.45-2.4.3.3.el6.x86_64
rpm -e --nodeps java-1.6.-openjdk-1.6.0.0-1.66.1.13..el6.x86_64
注意：这里一定要是jdk1.8，不然会报如下错误
[root@suricata ~]# sudo /etc/init.d/elasticsearch start
Starting elasticsearch: Elasticsearch requires at least Java but your Java version from /usr/local/jdk/jdk1..0_79/bin/java does not meet this requirement
[FAILED]
[root@suricata ~]#
官网提示了，建议安装orcal 的 jdk。
我这里安装jdk18。
[root@suricata suricata-3.1]# cd /usr/local/
[root@suricata local]# ls
bin etc games include lib lib64 libexec sbin share src var
[root@suricata local]# mkdir jdk
[root@suricata local]# cd jdk/
[root@suricata jdk]# pwd
/usr/local/jdk
[root@suricata jdk]# ll
total
[root@suricata jdk]# rz
[root@suricata jdk]# pwd
/usr/local/jdk
[root@suricata jdk]# ll
total
-rw-r--r--. root root Sep jdk-8u60-linux-x64.tar.gz
[root@suricata jdk]# tar -zxvf jdk-8u60-linux-x64.tar.gz
然后，再vim /etc/profile
#java
export JAVA_HOME=/usr/local/jdk/jdk1..0_60
export CLASSPATH=.:$JAVA_HOME/jre/lib/rt.jar:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
export PATH=$PATH:$JAVA_HOME/bin
文件生效
source /etc/profile
然后，接下来就是安装ELk和配置JDK。（注意以下只是个例子）
wget https://download.elasticsearch.org/kibana/kibana/kibana-3.0.0.tar.gz
wget https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-1.6.1.deb
wget https://download.elastic.co/logstash/logstash/packages/debian/logstash_1.5.3-1_all.deb tar -C /var/www/ -xzf kibana-3.0..tar.gz
dpkg -i elasticsearch-1.6..deb
dpkg -i logstash_1.5.3-1_all.deb
我这里啊，Elasticsearch选取5.5.0
所以，总的是
[root@suricata ~]# wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-5.5.0.deb
[root@suricata ~]# wget https://artifacts.elastic.co/downloads/logstash/logstash-5.5.0.deb
[root@suricata ~]# wget https://artifacts.elastic.co/downloads/kibana/kibana-5.5.0-linux-x86_64.tar.gz
最后下载，得到
[root@suricata ~]# pwd
/root
[root@suricata ~]# ls
anaconda-ks.cfg Desktop Downloads install.log kibana-5.5.-linux-x86_64.tar.gz Music Public suricata-3.1.tar.gz Videos
data Documents elasticsearch-5.5..deb install.log.syslog logstash-5.5..deb Pictures suricata-3.1 Templates
[root@suricata ~]#
然后，进行对kibana的安装
[root@suricata ~]# pwd
/root
[root@suricata ~]# ls
anaconda-ks.cfg Desktop Downloads install.log kibana-5.5.-linux-x86_64.tar.gz Music Public suricata-3.1.tar.gz Videos
data Documents elasticsearch-5.5..deb install.log.syslog logstash-5.5..deb Pictures suricata-3.1 Templates
[root@suricata ~]# tar -C /var/www/ -xzf kibana-5.5.-linux-x86_64.tar.gz
得到
[root@suricata www]# pwd
/var/www
[root@suricata www]# ls
cgi-bin error html icons kibana-5.5.-linux-x86_64
[root@suricata www]#
若大家没安装过这个命令，则安装下即可。
[root@suricata ~]# yum -y install dpkg
然后，进行对elasticsearch的安装
[root@suricata ~]# pwd
/root
[root@suricata ~]# ls
anaconda-ks.cfg Desktop Downloads install.log kibana-5.5.-linux-x86_64.tar.gz Music Public suricata-3.1.tar.gz Videos
data Documents elasticsearch-5.5..deb install.log.syslog logstash-5.5..deb Pictures suricata-3.1 Templates
[root@suricata ~]# dpkg -i elasticsearch-5.5..deb
这里啊，我建议大家还是去安装tar.gz包，这种方式比较好。
所以，后面，我还是下载了，rpm包，以这种方式来进行安装。
[root@suricata ~]# wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-5.5.0.rpm
[root@suricata ~]# wget https://artifacts.elastic.co/downloads/logstash/logstash-5.5.0.rpm
进行elasticsearch-5.5.0的安装
[root@suricata ~]# pwd
/root
[root@suricata ~]# ls
anaconda-ks.cfg Desktop Downloads elasticsearch-5.5..rpm install.log.syslog logstash-5.5..deb Music Public suricata-3.1.tar.gz Videos
data Documents elasticsearch-5.5..deb install.log kibana-5.5.-linux-x86_64.tar.gz logstash-5.5..rpm Pictures suricata-3.1 Templates
[root@suricata ~]# rpm -ivh elasticsearch-5.5.0.rpm
warning: elasticsearch-5.5.0.rpm: Header V4 RSA/SHA512 Signature, key ID d88e42b4: NOKEY
Preparing... ########################################### [100%]
Creating elasticsearch group... OK
Creating elasticsearch user... OK
1:elasticsearch ########################################### [100%]
### NOT starting on installation, please execute the following statements to configure elasticsearch service to start automatically using chkconfig
sudo chkconfig --add elasticsearch
### You can start elasticsearch service by executing
sudo service elasticsearch start
安装完，默认即有
然后，进行logstash的安装
[root@suricata ~]# pwd
/root
[root@suricata ~]# ls
anaconda-ks.cfg Desktop Downloads elasticsearch-5.5..rpm install.log.syslog logstash-5.5..deb Music Public suricata-3.1.tar.gz Videos
data Documents elasticsearch-5.5..deb install.log kibana-5.5.-linux-x86_64.tar.gz logstash-5.5..rpm Pictures suricata-3.1 Templates
[root@suricata ~]# rpm -ivh logstash-5.5..rpm
warning: logstash-5.5..rpm: Header V4 RSA/SHA512 Signature, key ID d88e42b4: NOKEY
Preparing... ########################################### [%]
:logstash ########################################### [%]
Using provided startup.options file: /etc/logstash/startup.options
/usr/share/logstash/vendor/jruby/bin/jruby: line : /usr/bin/java: No such file or directory
/usr/share/logstash/vendor/jruby/bin/jruby: line : exec: /usr/bin/java: cannot execute: No such file or directory
Unable to install system startup script for Logstash.
[root@suricata ~]#
安装完，默认即有
然后，下面是
Logstash configuration
[root@suricata conf.d]# pwd
/etc/logstash/conf.d
[root@suricata conf.d]# ll
total
[root@suricata conf.d]# touch /etc/logstash/conf.d/logstash.conf
插入如下内容
input {
file {
path => ["/var/log/suricata/eve.json"]
sincedb_path => ["/var/lib/logstash/"]
codec => json
type => "SuricataIDPS"
}
}
filter {
if [type] == "SuricataIDPS" {
date {
match => [ "timestamp", "ISO8601" ]
}
ruby {
code => "if event['event_type'] == 'fileinfo'; event['fileinfo']['type']=event['fileinfo']['magic'].to_s.split(',')[0]; end;"
}
}
if [src_ip] {
geoip {
source => "src_ip"
target => "geoip"
#database => "/opt/logstash/vendor/geoip/GeoLiteCity.dat"
add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}" ]
add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}" ]
}
mutate {
convert => [ "[geoip][coordinates]", "float" ]
}
if ![geoip.ip] {
if [dest_ip] {
geoip {
source => "dest_ip"
target => "geoip"
#database => "/opt/logstash/vendor/geoip/GeoLiteCity.dat"
add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}" ]
add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}" ]
}
mutate {
convert => [ "[geoip][coordinates]", "float" ]
}
}
}
}
}
output {
elasticsearch {
host => localhost
#protocol => http
}
}
然后，再
Configure the start-up services
update-rc.d elasticsearch defaults 10 （这两个是ubuntu系统里的做法，我这里是centos系统，不需要）
update-rc.d logstash defaults
service httpd start 启动apache
service elasticsearch start
service logstash start
或者 启动服务sudo /etc/init.d/elasticsearch start
停止服务sudo /etc/init.d/elasticsearch stop
启动服务sudo /etc/init.d/logstash/start
停止服务sudo /etc/init.d/logstashstop
如果大家在启动过程中，出现如下问题。
[root@suricata ~]# sudo /etc/init.d/elasticsearch start
which: no java in (/sbin:/usr/sbin:/bin:/usr/bin)
Could not find any executable java binary. Please install java in your PATH or set JAVA_HOME
[root@suricata ~]#
则见
ElasticSearch里JAVA环境丢失的问题
-rwxr-x---. 1 root root 5113 Jul 1 07:20 elasticsearch ，经过
[root@suricata init.d]# chmod elasticsearch
[root@suricata init.d]# pwd
/etc/init.d
[root@suricata init.d]#
改为
-rwxr-xr-x. 1 root root 5113 Jul 1 07:20 elasticsearch
大家，自行按照自己的jdk位置。
export JAVA_HOME=/usr/local/jdk/jdk1..0_60
然后，就可以成功了。
[root@suricata ~]# sudo /etc/init.d/elasticsearch start
Starting elasticsearch: [ OK ]
[root@suricata ~]#
然后，大家也许在启动，会遇到这样的问题：
Logstash默认的配置文件位置为./config和/etc/logstash/，后者已经存在，但直接运行依然会报错：
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash.
见
Logstash使用grok解析IIS日志
简单起见，直接在Logstash根目录创建软链接即可：
cd /usr/share/logstash
[root@suricata logstash]# pwd
/usr/share/logstash
[root@suricata logstash]# ln -s /etc/logstash ./config
[root@suricata logstash]#
设置配置文件实时生效，从而不用频繁地启停Logstash。
修改/etc/logstash/logstash.yml
config.reload.automatic: true
Logstash的运行
与Elasticsearch、Kibana不同，Logstash默认并不作为系统服务安装，我们也不建议作为服务启动。主要原因为：大多数情况下，Elasticsearch、Kibana在每台服务器上只有一个实例，可以作为服务启动；而一个Logstash实例一般只服务于一个数据流，因此一台服务器上经常运行多个实例 。编写一个配置文件helloworld.conf进行测试。注意，不要把该文件放在/etc/logstash/conf.d下，以避免不同实例间产生混淆，因为该文件夹是Logstash默认读取的文件夹。简单起见，可直接放在Logstash根目录下。
input {
stdin {}
}
output {
stdout {}
}
这是一个命令行输入、命令行输出的Logstash实例，运行成功则说明已配置正确：
bin/logstash -f helloworld.conf
虽然如此，很多时候我们还是希望各个Logstash实例能随系统而启动并在后台运行。将其加入计划任务即可：
nohup /usr/share/logstash/bin/logstash -f /usr/share/logstash/helloworld.conf > /dev/null &
然后，
Now make sure Suricata is running and you have logs written in your JSON log files
http://localhost/kibana-5.5.0
https://redmine.openinfosecfoundation.org/projects/suricata/wiki/_Logstash_Kibana_and_Suricata_JSON_output
继续