Elasticsearch 7.1.1 集群 + 配置身份验证

一、安装Elasticsearch

1.1 环境说明

Centos7.6

Elasticsearch7.1.1

#挂载数据盘

fdisk /dev/vdb

n,p,,回车，回车，wq

fdisk -l

mkfs.ext4 /dev/vdb1

echo '/dev/vdb1 /opt ext4 defaults 0 0' >>/etc/fstab

mount -a

df -h

#时间同步

yum install -y ntp

systemctl enable ntpd && systemctl start ntpd

timedatectl set-timezone Asia/Shanghai

timedatectl set-ntp yes

ntpq -p

1.2 操作系统调优

cat >> /etc/sysctl.conf <<EOF

fs.file-max=

vm.max_map_count =

EOF

sysctl -p

vim /etc/security/limits.conf

* soft nproc

* hard nproc

* soft nofile

* hard nofile

* soft memlock unlimited

* hard memlock unlimited

vim /etc/security/limits.d/20-nproc.conf

* soft nproc

1.3 安装JDK

yum install -y java-1.8.0-openjdk*

vim /etc/profile

# set java environment

export JAVA_HOME=/usr/lib/jvm/java-1.8.-openjdk-1.8.0.212.b04-.el7_6.x86_64

export PATH=$PATH:$JAVA_HOME/bin

export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar

source /etc/profile

echo "source /etc/profile" >> /etc/bashrc

1.4 安装es

1）新建用户

groupadd elsearch
useradd elsearch -g elsearch -p elasticsearch

2）下载
cd /opt
wget https://img.yiyao.cc/elasticsearch-7.1.1-linux-x86_64.tar.gz
tar -zxvf elasticsearch-7.1.1-linux-x86_64.tar.gz
mv elasticsearch-7.1.1 elasticsearch
chown -R elsearch.elsearch ./elasticsearch

3）JVM调优
物理内存一半
vim /opt/elasticsearch/config/jvm.options

-Xms8g

-Xmx8g

4）配置es，三个节点同时作为 master 和 data

vim /opt/elasticsearch/config/elasticsearch.yml

#节点1

cluster.name: wmqees

node.name: es-node1

node.master: true

node.data: true

path.data: /opt/elasticsearch/data

path.logs: /opt/elasticsearch/logs

bootstrap.memory_lock: true

network.host: 172.16.2.141

http.port:

discovery.zen.minimum_master_nodes:

discovery.zen.ping.unicast.hosts: ["172.16.2.141:9300","172.16.2.142:9300","172.16.2.143:9300"]
cluster.initial_master_nodes: ["es-node1", "es-node2", "es-node3"]
http.cors.enabled: true
http.cors.allow-origin: "*"

cluster.initial_master_nodes参数说明：es7 引用了 Bootstrapping a cluster 后，首次启动Elasticsearch集群需要在集群中的一个或多个符合主节点的节点上显式定义初始的符合主节点的节点集。这称为群集自举，这仅在群集首次启动时才需要。

#节点2

cluster.name: wmqees

node.name: es-node2

node.master: true

node.data: true

path.data: /opt/elasticsearch/data

path.logs: /opt/elasticsearch/logs

bootstrap.memory_lock: true

network.host: 172.16.2.142

http.port:

discovery.zen.minimum_master_nodes:

discovery.zen.ping.unicast.hosts: ["172.16.2.141:9300","172.16.2.142:9300","172.16.2.143:9300"]
cluster.initial_master_nodes: ["es-node1", "es-node2", "es-node3"]
http.cors.enabled: true
http.cors.allow-origin: "*"

#节点3

cluster.name: wmqees

node.name: es-node3

node.master: true

node.data: true

path.data: /opt/elasticsearch/data

path.logs: /opt/elasticsearch/logs

bootstrap.memory_lock: true

network.host: 172.16.2.143

http.port:

discovery.zen.minimum_master_nodes:

discovery.zen.ping.unicast.hosts: ["172.16.2.141:9300","172.16.2.142:9300","172.16.2.143:9300"]
cluster.initial_master_nodes: ["es-node1", "es-node2", "es-node3"]
http.cors.enabled: true
http.cors.allow-origin: "*"

5）启动

su - elsearch -c "/opt/elasticsearch/bin/elasticsearch -d"

6）验证
curl "172.16.2.143:9200/_xpack"

{"build":{"hash":"7a013de","date":"2019-05-23T14:05:50.009976Z"},"license":{"uid":"344f983f-9d20-4476-851a-4172fd669f12","type":"basic","mode":"basic","status":"active"},"features":{"ccr":{"description":"Cross Cluster Replication","available":false,"enabled":true},"graph":{"description":"Graph Data Exploration for the Elastic Stack","available":false,"enabled":true},"ilm":{"description":"Index lifecycle management for the Elastic Stack","available":true,"enabled":true},"logstash":{"description":"Logstash management component for X-Pack","available":false,"enabled":true},"ml":{"description":"Machine Learning for the Elastic Stack","available":false,"enabled":true,"native_code_info":{"version":"7.1.1","build_hash":"fd619a36eb77df"}},"monitoring":{"description":"Monitoring for the Elastic Stack","available":true,"enabled":true},"rollup":{"description":"Time series pre-aggregation and rollup","available":true,"enabled":true},"security":{"description":"Security for the Elastic Stack","available":true,"enabled":false},"sql":{"description":"SQL access to Elasticsearch","available":true,"enabled":true},"watcher":{"description":"Alerting, Notification and Automation for the Elastic Stack","available":false,"enabled":true}},"tagline":"You know, for X"}

说明：显示 license 不为空则安装成功。es7版本默认已经包含xpack认证，无需注册。

1.5 开机自启

有 systemd 和 service 两种方式进行设置开机自启，推荐 systemd 方式可以设置 es 异常挂起后能够重启。

1.5.1 Systemd 方式（推荐）

1）新建环境配置文件，指定Java路径

vim /etc/sysconfig/elasticsearch

################################

# Elasticsearch

################################

# Elasticsearch home directory

#ES_HOME=/usr/share/elasticsearch

# Elasticsearch Java path

JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.212.b10-0.el7_6.x86_64

CLASSPATH=.:/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.212.b10-0.el7_6.x86_64/lib:/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.212.b10-0.el7_6.x86_64/jre/lib

# Elasticsearch configuration directory

#ES_PATH_CONF=${path.conf}

# Elasticsearch PID directory

#PID_DIR=/var/run/elasticsearch

# Additional Java OPTS

#ES_JAVA_OPTS=

# Configure restart on package upgrade (true, every other setting will lead to not restarting)

#RESTART_ON_UPGRADE=true

################################

# Elasticsearch service

################################

# SysV init.d

#

# The number of seconds to wait before checking if Elasticsearch started successfully as a daemon process

ES_STARTUP_SLEEP_TIME=

################################

# System properties

################################

# Specifies the maximum file descriptor number that can be opened by this process

# When using Systemd, this setting is ignored and the LimitNOFILE defined in

# /usr/lib/systemd/system/elasticsearch.service takes precedence

#MAX_OPEN_FILES=

# The maximum number of bytes of memory that may be locked into RAM

# Set to "unlimited" if you use the 'bootstrap.memory_lock: true' option

# in elasticsearch.yml.

# When using systemd, LimitMEMLOCK must be set in a unit file such as

# /etc/systemd/system/elasticsearch.service.d/override.conf.

#MAX_LOCKED_MEMORY=unlimited

# Maximum number of VMA (Virtual Memory Areas) a process can own

# When using Systemd, this setting is ignored and the 'vm.max_map_count'

# property is set at boot time in /usr/lib/sysctl.d/elasticsearch.conf

#MAX_MAP_COUNT=

官网样例：https://github.com/elastic/elasticsearch/blob/master/distribution/packages/src/common/env/elasticsearch

2）创建服务文件

vim /usr/lib/systemd/system/elasticsearch.service

[Unit]

Description=Elasticsearch

Documentation=http://www.elastic.co

Wants=network-online.target

After=network-online.target

[Service]

Restart=always

Type=simple

PrivateTmp=true

Environment=ES_HOME=/opt/elasticsearch

Environment=ES_PATH_CONF=/opt/elasticsearch/config

Environment=PID_DIR=/opt/elasticsearch

Environment=ES_SD_NOTIFY=true

EnvironmentFile=/etc/sysconfig/elasticsearch

WorkingDirectory=/opt/elasticsearch

User=elsearch

Group=elsearch

ExecStart=/opt/elasticsearch/bin/elasticsearch -p ${PID_DIR}/elasticsearch.pid --quiet

# StandardOutput is configured to redirect to journalctl since

# some error messages may be logged in standard output before

# elasticsearch logging system is initialized. Elasticsearch

# stores its logs in /var/log/elasticsearch and does not use

# journalctl by default. If you also want to enable journalctl

# logging, you can simply remove the "quiet" option from ExecStart.

StandardOutput=journal

StandardError=inherit

# Specifies the maximum file descriptor number that can be opened by this process

LimitNOFILE=

# Specifies the maximum number of processes

LimitNPROC=20480

LimitMEMLOCK=infinity

# Specifies the maximum size of virtual memory

LimitAS=infinity

# Specifies the maximum file size

LimitFSIZE=infinity

# Disable timeout logic and wait until process is stopped

TimeoutStopSec=

# SIGTERM signal is used to stop the Java process

KillSignal=SIGTERM

# Send the signal only to the JVM rather than its control group

KillMode=process

# Java process is never killed

SendSIGKILL=no

# When a JVM receives a SIGTERM signal it exits with code

SuccessExitStatus=

[Install]

WantedBy=multi-user.target

# Built for ${project.name}-${project.version} (${project.name})

官网样例：https://github.com/elastic/elasticsearch/blob/master/distribution/packages/src/common/systemd/elasticsearch.service

3）启动

systemctl daemon-reload

systemctl enable elasticsearch.service

systemctl start elasticsearch.service

可以 kill 掉 es 进程，es 会再次启动。

1.5.2 service 方式（不推荐）

1）创建启动脚本

vim /etc/init.d/elasticsearch

#!/bin/sh

#chkconfig:

#description: elasticsearch

#processname: elasticsearch-7.1.

export JAVA_HOME=/usr/lib/jvm/java-1.8.-openjdk-1.8.0.212.b04-.el7_6.x86_64

export JAVA_BIN=/usr/lib/jvm/java-1.8.-openjdk-1.8.0.212.b04-.el7_6.x86_64/bin

export PATH=$PATH:$JAVA_HOME/bin

export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar

export ES_HOME=/opt/elasticsearch

case $ in

    start)

        su elsearch<<!

        cd $ES_HOME

        ./bin/elasticsearch -d -p pid

        exit

!

        echo "elasticsearch is started"

        ;;

    stop)

        pid=`cat $ES_HOME/pid`

        kill - $pid

        echo "elasticsearch is stopped"

        ;;

    restart)

        pid=`cat $ES_HOME/pid`

        kill - $pid

        echo "elasticsearch is stopped"

        sleep

        su elsearch<<!

        cd $ES_HOME

        ./bin/elasticsearch -d -p pid

        exit

!

        echo "elasticsearch is started"

        ;;

    *)

        echo "start|stop|restart"

        ;;

esac

exit

说明：需指定JDK环境，要不然会默认使用es自带的JDK，自带的版本太新，去除了GC。

2）启动

# 添加到开机启动任务

chmod +x /etc/init.d/elasticsearch

chkconfig --add elasticsearch

# 启动

service elasticsearch start

二、配置 TLS 和身份验证

2.1 创建证书文件

在一个master上执行即可

cd /opt/elasticsearch

./bin/elasticsearch-certutil ca

两次回车

./bin/elasticsearch-certutil cert --ca elastic-stack-ca.p12

三次回车

赋予权限

mkdir config/certs
mv elastic-*.p12 config/certs/
chown -R elsearch:elsearch config/certs/

再把证书文件 elastic-certificates.p12 复制到其他master节点并赋予权限。

2.2 修改配置

#所有主机配置文件添加ssl

cat >> config/elasticsearch.yml <<EOF

xpack.security.enabled: true

xpack.security.transport.ssl.enabled: true

xpack.security.transport.ssl.verification_mode: certificate

xpack.security.transport.ssl.keystore.path: certs/elastic-certificates.p12

xpack.security.transport.ssl.truststore.path: certs/elastic-certificates.p12

EOF

重启 elasticsearch

service elasticsearch restart

2.3 生成客户端证书

cd /opt/elasticsearch

bin/elasticsearch-certutil cert --ca \

config/certs/elastic-stack-ca.p12 \

-name "CN=esuser,OU=dev,DC=weqhealth,DC=com"

回车

client.p12

回车

拆分证书

mv client.p12 config/certs/

cd config/certs/

openssl pkcs12 -in client.p12 -nocerts -nodes > client-key.pem

openssl pkcs12 -in client.p12 -clcerts -nokeys  > client.crt

openssl pkcs12 -in client.p12 -cacerts -nokeys -chain > client-ca.crt

chown elsearch:elsearch client*

2.4 设置默认密码

bin/elasticsearch-setup-passwords interactive

y，分别设置 elastic、apm_system、kibana、logstash_system、beats_system、remote_monitoring_user账号的密码。

2.5 配置kibana

修改 kibana.yml 文件

elasticsearch.username: "kibana"

elasticsearch.password: "elasticxxxxxxx"

然后用超级管理员账号 elastic 登入到 kibana。在kibana中设置角色和账号，也可以修改账号密码。

2.6 验证集群状态

因为开启了xpack验证，需要指定账号密码

curl --user elastic:elasticxxxxxx -XGET '172.16.2.143:9200/_cat/health?v&pretty'

epoch      timestamp cluster   status node.total node.data shards pri relo init unassign pending_tasks max_task_wait_time active_shards_percent

 ::  wmqees green                                                                           -                100.0%

参考：https://www.elastic.co/cn/blog/getting-started-with-elasticsearch-security

三、基本设置

3.1 设置分片数

es7默认主分片数和主分片副本数都为1，通过 default_template 指定分片数

PUT _template/default_template

{

  "index_patterns" : ["*"],

  "settings": {

    "number_of_shards": ,

    "number_of_replicas" :

  }

}

number_of_shards：每个索引的主分片数，默认值是 1 不再是5。这个配置在索引创建后不能修改。

number_of_replicas：每个主分片的副本数，默认值是 1 。对于活动的索引库，这个配置可以随时修改。

3.2 集群最大分片数

1）说明

基于集群中节点的数量，集群中的分片数量有一个软限制（数据节点数 * 1000），如果群集中没有数据节点，则不会执行该限制。

1、创建新索引，还原索引快照或打开关闭的索引会增加分片；关闭或删除某些索引会减少分片。

2、副本数计入此限制，但封闭索引不计入。具有5个主要分片和2个副本的索引将计为15个分片。无论封闭索引包含多少个分片和副本，任何封闭索引都将计为0。

2）设置每个数据节点在集群中允许的分片数量

PUT _cluster/settings?pretty

{

  "persistent": {

    "cluster.max_shards_per_node":

  }

}

3）查询设置结果

GET _cluster/settings?pretty

# 结果如下

{

  "persistent" : {

    "cluster" : {

      "max_shards_per_node" : ""

    }

  },

  "transient" : { }

}

如果集群有 3 个数据节点，这样集群总分片数为 9000 个。

官网解释集群分配数限制

3.2 安全重启es

1）禁用分片分配

关闭节点时，分配过程将等待 index.unassigned.node_left.delayed_timeout 1分钟（默认情况下为1分钟），然后开始将该节点上的分片复制到集群中的其他节点，这可能涉及大量I/O。由于该节点不久将要重新启动，因此该I/O是不必要的，通过在关闭节点之前禁用副本分配。

PUT _cluster/settings

{

  "persistent": {

    "cluster.routing.allocation.enable": "primaries"

  }

}

设置成primaries，索引的主分片会均分到集群的各个node，副本分片处于unassigined状态。

2）重启es

依次挨个重启一个节点，启动好了后再重启另外节点。

service elasticsearch restart

3）开启分片分配

PUT _cluster/settings

{

  "persistent": {

    "cluster.routing.allocation.enable": null

  }

}

官网参考：分配种类，重启操作，分配规则

秒客网

Elasticsearch 7.1.1 集群 + 配置身份验证

一、安装Elasticsearch

1.1 环境说明

1.2 操作系统调优

1.3 安装JDK

1.4 安装es

1.5 开机自启

1.5.1 Systemd 方式（推荐）

1.5.2 service 方式（不推荐）

二、配置 TLS 和身份验证

2.1 创建证书文件

2.2 修改配置

2.3 生成客户端证书

2.4 设置默认密码

2.5 配置kibana

2.6 验证集群状态

三、基本设置

3.1 设置分片数

3.2 集群最大分片数

1）说明

2）设置每个数据节点在集群中允许的分片数量

3）查询设置结果

3.2 安全重启es

1）禁用分片分配

2）重启es

3）开启分片分配

相关文章

Elasticsearch 7.1.1 集群 + 配置身份验证

一、安装Elasticsearch

1.1 环境说明

1.2 操作系统调优

1.3 安装JDK

1.4 安装es

1.5 开机自启

1.5.1 Systemd 方式（推荐）

1.5.2 service 方式（不推荐）

二、配置 TLS 和身份验证

2.1 创建证书文件

2.2 修改配置

2.3 生成客户端证书

2.4 设置默认密码

2.5 配置kibana

2.6 验证集群状态

三、基本设置

3.1 设置分片数

3.2 集群最大分片数

1） 说明

2）设置每个数据节点在集群中允许的分片数量

3）查询设置结果

3.2 安全重启es

1）禁用分片分配

2）重启es

3）开启分片分配

相关文章

1）说明