ubuntu 20.04 Docker 快速部署 单机 elasticsearch+kibana+filebeat 8.5.0

硬件信息：8核心 32G内存 250G硬盘

下载地址

如果想要二进制安装 可以去下面地址去下载 docker 不需要

全部下载： https://www.elastic.co/cn/downloads

ES: https://www.elastic.co/cn/downloads/elasticsearch

Kibana: https://www.elastic.co/cn/downloads/kibana

Logstash: https://www.elastic.co/cn/downloads/logstash

Filebeat: https://www.elastic.co/cn/downloads/beats/filebeat

基础配置

cat > /etc/apt/sources.list <<'EOF'
deb http://mirrors.aliyun.com/ubuntu/ focal main restricted   #优先检索focal的main和restricted组件。
deb http://mirrors.aliyun.com/ubuntu/ focal-updates main restricted
deb http://mirrors.aliyun.com/ubuntu/ focal universe
deb http://mirrors.aliyun.com/ubuntu/ focal-updates universe
deb http://mirrors.aliyun.com/ubuntu/ focal multiverse
deb http://mirrors.aliyun.com/ubuntu/ focal-updates multiverse
deb http://mirrors.aliyun.com/ubuntu/ focal-backports main restricted universe multiverse
deb http://mirrors.aliyun.com/ubuntu focal-security main restricted
deb http://mirrors.aliyun.com/ubuntu focal-security universe
deb http://mirrors.aliyun.com/ubuntu focal-security multiverse
EOF


更新源
apt-get update

1.安装docker

1.先卸载docker 防止已经安装过docker
sudo apt-get remove docker docker-engine docker.io containerd runc

2.安装依赖
sudo apt-get install \
    apt-transport-https \
    ca-certificates \
    curl \
    gnupg-agent \
    software-properties-common

5.导入阿里云证书
curl -fsSL https://mirrors.aliyun.com/docker-ce/linux/ubuntu/gpg | sudo apt-key add -

6.设置阿里云稳定仓库
sudo add-apt-repository "deb [arch=amd64] https://mirrors.aliyun.com/docker-ce/linux/ubuntu $(lsb_release -cs) stable"


清华源证书（与阿里云二选一）
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
清华源稳定仓库（与阿里云二选一）
sudo add-apt-repository "deb [arch=amd64] https://mirrors.tuna.tsinghua.edu.cn/docker-ce/linux/ubuntu $(lsb_release -cs) stable"


7.更新仓库 （其实就是在 /etc/apt/sources.list 加了docker源）
apt update

8.查看docker都有那些版本
apt-cache madison docker-ce

9.安装指定版本docker 19.03.15版本 比较稳定 用的人多
apt install docker-ce=5:19.03.15~3-0~ubuntu-focal docker-ce-cli=5:19.03.15~3-0~ubuntu-focal

检查是否安装成功
 docker --version
 显示如下
Docker version 19.03.15, build 99e3ed8919

#############以下操作为可选##############
为了以后管理docker方便 安装一个portainer 用于管理docker容器

1.下载portainer 镜像
docker pull portainer/portainer

2.运行portainer 下载镜像并启动容器 开机自启动
docker run -d -p 9000:9000 --restart=always -v /var/run/docker.sock:/var/run/docker.sock --name prtainer-test portainer/portainer

3.验证安装 
浏览器访问服务器9000端口
http://10.10.201.161:9000/
并设置admin密-码
（portainer的local的Container里 可以看到容器的启动情况）

2.docker 安装elasticsearch

# 创建目录 (如果想配置中文分词器直接去github下载拷贝到plugins目录再重启即可)
mkdir -p /opt/elasticsearch/{config,plugins,data}  

#配置elasticsearch.yml文件

cat <<EOF> /opt/elasticsearch/config/elasticsearch.yml
xpack.security.enabled: true
xpack.license.self_generated.type: basic
xpack.security.transport.ssl.enabled: false  
xpack.security.enrollment.enabled: true
http.host: 0.0.0.0
# xpack.security.enabled: false
EOF

#建立elasticsearch使用的docker网络 网络名为：elastic
docker network create -d bridge elastic

#赋予目录权限
chmod -R 777 /opt/elasticsearch


#运行elasticsearch 期间会自动下载镜像 并启动

docker run --name elasticsearch -p 9200:9200 -p 9300:9300 \
--net elastic \
-e "discovery.type=single-node" \
-e ES_JAVA_OPTS="-Xms1024m -Xmx1024m" \
-v /opt/elasticsearch/config/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml \
-v /opt/elasticsearch/data:/usr/share/elasticsearch/data \
-v /opt/elasticsearch/plugins:/usr/share/elasticsearch/plugins \
-d elasticsearch:8.5.0


# 注意：不要设置自启，怕死机 --restart=always

#查看elasticsearch 日志
docker logs -f elasticsearch

elasticsearch必要的一些配置（确认elasticsearch已启动 ） 显示下面的"@timestamp" 基本就要已经下载好镜像 并启动了

（elasticsearch 不要关闭  不要按 CTRL+C ）

另外再开启一个SSH访问客户端

确认elasticsearch已经启动后
另外开启一个新的ssh 使用下面的命令重置下面三个密码

1.重设所有密码 我是本机 所以后面是http://localhost:9200
 docker exec -it elasticsearch elasticsearch-setup-passwords interactive -u  http://localhost:9200 
 
显示如下： 设置你的所有的新的密码
Please confirm that you would like to continue [y/N]y


Enter password for [elastic]: 
Reenter password for [elastic]: 
Enter password for [apm_system]: 
Reenter password for [apm_system]: 
Enter password for [kibana_system]: 
Reenter password for [kibana_system]: 
Enter password for [logstash_system]: 
Reenter password for [logstash_system]: 
Enter password for [beats_system]: 
Reenter password for [beats_system]: 
Enter password for [remote_monitoring_user]: 
Reenter password for [remote_monitoring_user]: 
Changed password for user [apm_system]
Changed password for user [kibana_system]
Changed password for user [kibana]
Changed password for user [logstash_system]
Changed password for user [beats_system]
Changed password for user [remote_monitoring_user]
Changed password for user [elastic]


2.重置 elasticsearch密码  -i 表示自定义密码 给java客户端用的
docker exec -it elasticsearch bin/elasticsearch-reset-password -u elastic  -i  
显示如下
Please confirm that you would like to continue [y/N]y


Enter password for [elastic]: 
Re-enter password for [elastic]: 
Password for the [elastic] user successfully reset.


3. 重置 elasticsearch密码 给 kibana 用的
docker exec -it elasticsearch bin/elasticsearch-reset-password -u kibana_system -i  
显示如下
Please confirm that you would like to continue [y/N]y


Enter password for [kibana_system]: 
Re-enter password for [kibana_system]: 
Password for the [kibana_system] user successfully reset.

3..docker安装kibana

1.建立目录
mkdir -p /opt/kibana/{config,data}

2.下载镜像并运行kibana
docker run --name kibana -p 5601:5601 -d kibana:8.5.0

3. 拷贝配置文件
docker cp kibana:/usr/share/kibana/config/kibana.yml /opt/kibana/config/kibana.yml

4.配置kibana 配置文件 按你自己的设置hosts  password

cat <<EOF> /opt/kibana/config/kibana.yml
server.host: "0.0.0.0"  
server.shutdownTimeout: "5s"
elasticsearch.hosts: [ "http://10.10.201.161:9200" ]
elasticsearch.username: "kibana_system"  
elasticsearch.password: "123456"
i18n.locale: "zh-CN"
EOF


5.停止kibana 并删除kibana容器
docker stop kibana && docker rm kibana

6.按新的配置文件 启动一个新的kibana

sudo docker run --name kibana \
--net elastic \
-v /opt/kibana/config/kibana.yml:/usr/share/kibana/config/kibana.yml \
-p 5601:5601 -d kibana:8.5.0


7.查看日志（可以开启一个新的ssh 运行）
docker logs -f kibana

###es 不允许使用elastic用户登录kibana 所以这里需要创建一个自定义用户####

#进入es容器
docker exec -it elasticsearch bash

#添加一个用户administrator 可以换名字 并设置密码 要记住 之后访问kibana会用
bin/elasticsearch-users useradd  administrator

#给新的用户角色授权

bin/elasticsearch-users roles -a superuser  administrator
bin/elasticsearch-users roles -a kibana_system administrator

#退出elasticsearch容器
exit

完成

4.浏览器登陆kibana测试

用户名：administrator

密码：你之前设置的密码

portainer里的容器效果

一些docker 常用命令

查看已经启动的容器
docker ps 

启动容器 elasticsearch 
docker start elasticsearch 
启动容器kibana
docker start kibana


关闭容器 elasticsearch 
docker start elasticsearch
关闭容器kibana
docker start kibana

从启动容器
docker restart elasticsearch
docker restart kibana

4.二进制安装filebeat 8.5.0

因为客户端服务器不一定都是安装了docker 或者不让随便安装 所以使用二进制安装

下载

https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-8.5.0-linux-x86_64.tar.gz

开始安装

登陆客户端服务器后

1.建立存放文件的目录
mkdir /home/filebeat
2.上传filebeat-8.5.0-linux-x86_64.tar.gz 文件到 /home/filebeat 中

3.解压缩
tar zxvf /home/filebeat/filebeat-8.5.0-linux-x86_64.tar.gz

编辑filebeat配置文件

1.编辑文件filebeat.yml
cd /home/filebeat/filebeat-8.5.0-linux-x86_64

vim   /home/filebeat/filebeat-8.5.0-linux-x86_64/filebeat.yml

######需要更改的内容为#####
1.开启抓取日志的功能 enabled: true  
2.设置抓取文件的路径（我抓取nginx） /var/log/nginx/access.log

输出到Elasticsearch
在Elasticsearch Output中更改为如下
3.取消前面的#注释
output.elasticsearch:

4.  Elasticsearch的服务器地址
hosts: ["10.10.201.161:9200"]
5.设置elastic的用户名和密码 上面你自己设置过的
  username: "elastic"
  password: "123456"

如下 其他都不需要更改
# ============================== Filebeat inputs ===============================

filebeat.inputs:

# Each - is an input. Most options can be set at the input level, so
# you can use different inputs for various configurations.
# Below are the input specific configurations.

# filestream is an input for collecting log messages from files.
- type: filestream

  # Unique ID among all inputs, an ID is required.
  id: my-filestream-id

  # Change to true to enable this input configuration.
  enabled: true

  # Paths that should be crawled and fetched. Glob based paths.
  paths:
    - /var/log/nginx/access.log
    #- c:\programdata\elasticsearch\logs\*

    
# ---------------------------- Elasticsearch Output ----------------------------
output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["10.10.201.161:9200"]

  # Protocol - either `http` (default) or `https`.
  #protocol: "https"

  # Authentication credentials - either API key or username/password.
  #api_key: "id:api_key"
  username: "elastic"
  password: "123456"

启动filebeat

1.进入目录
cd /home/filebeat/filebeat-8.5.0-linux-x86_64

2.检查配置文件是否配置正确
./filebeat test config
显示 Config OK

3.检查配置文件 连接Elasticsearch 是否正确
./filebeat test output
显示 elasticsearch: http://10.10.201.161:9200...
  parse url... OK
  connection...
    parse host... OK
    dns lookup... OK
    addresses: 10.10.201.161
    dial up... OK
  TLS... WARN secure connection disabled
  talk to server... OK
  version: 8.5.0


4.执行启动服务
./filebeat -e -c filebeat.yml

如果想后台启动
nohup ./filebeat -e -c filebeat.yml >/dev/null 2>&1 &

filebeat开机自启动设置

1. 进入自启动目录：
cd /lib/systemd/system

2.建立文件
vim filebeat.service

3.配置文件
如下

[Unit]
Description=filebeat
Wants=network-online.target
After=network-online.target

[Service]
User=root
ExecStart=/home/filebeat/filebeat-8.5.0-linux-x86_64/filebeat -e -c /home/filebeat/filebeat-8.5.0-linux-x86_64/filebeat.yml

#上面ExecStart后面的和-e -c后面的/usr/local/filebeat/filebeat-7.6.2-linux-x86_64/替换你的目录

Restart=always
#设置为掉线自动重启，进程强制杀掉后会自动重新启动  

[Install]
WantedBy=multi-user.target

保存退出
4.加载配置
systemctl daemon-reload  

5. 设置开机自启动
systemctl enable filebeat             
6.其他的一些命令

systemctl start filebeat              #启动filebeat服务
systemctl enable filebeat             #设置开机自启动
systemctl disable filebeat            #停止开机自启动
systemctl status filebeat             #查看服务当前状态
systemctl restart filebeat　          #重新启动服务
systemctl list-units --type=service  #查看所有已启动的服务

5.Docker 安装 Logstash（还未完成 待后期更新）

1.下载镜像
docker pull logstash:8.5.0
2.建立目录
mkdir /opt/logstash
3.给文件夹授权
chmod -R 777 /opt/logstash

4.第一次执行docker启动脚本
docker run -it \
	-d \
	--name logstash \
	-p 9600:9600 \
	-p 5044:5044 \
	--net elastic \
	logstash:8.5.0
 
 5.将容器内的文件复制到主机上
docker cp logstash:/usr/share/logstash/config /opt/logstash/

docker cp logstash:/usr/share/logstash/pipeline /opt/logstash/

6.修改配置logstash.yml
vim /opt/logstash/config
如下

http.host: "0.0.0.0"
xpack.monitoring.enabled: true
xpack.monitoring.elasticsearch.hosts: [ "http://10.10.201.161:9200" ]
xpack.monitoring.elasticsearch.username: "elastic"
xpack.monitoring.elasticsearch.password: "nihao123456!"
xpack.monitoring.elasticsearch.ssl.certificate_authority: "/usr/share/logstash/config/certs/http_ca.crt"
xpack.monitoring.elasticsearch.ssl.ca_trusted_fingerprint: "第一次启动elasticsearch是保存的信息中查找"


vim /opt/logstash/config

input {
  tcp {
    mode => "server"
    host => "0.0.0.0"
    port => 5044
    codec => json_lines
  }
}
output {
  elasticsearch {
    hosts => "10.10.201.161:9200"
    index => "%{[spring.application.name]}-%{+YYYY.MM.dd}"
    user  => "logstash_system"
    password => "nihao123456!"
 }
}
~  

vim /opt/logstash/logstash.yml

http.host: "0.0.0.0"
xpack.monitoring.elasticsearch.hosts: [ "http://10.10.201.161:9200" ]
xpack.monitoring.enabled: false








docker run -d --name logstash -m 1000M --restart=always -p 5044:5044 -p 9600:9600 --privileged=true -e ES_JAVA_OPTS="-Duser.timezone=Asia/Shanghai" -v /opt/logstash/logstash.conf:/usr/share/logstash/pipeline/logstash.conf -v /opt/logstash/logstash.yml:/usr/share/logstash/config/logstash.yml logstash:8.5.0