es集群无缝迁移扩容

新节点加入老集群

cluster.name: soc-cluster
node.name: node-2test
path.data: /home/secure/elastic/data
path.logs: /home/secure/elastic/logs
network.host: 0.0.0.0
cluster.initial_master_nodes: ["node-2"]
discovery.seed_hosts: ["10.32.3.2:9300","10.32.3.18:9300"]
http.port: 9201
transport.tcp.port: 9301
xpack.security.enabled: true
xpack.security.http.ssl.enabled: true
xpack.security.transport.ssl.enabled: true

xpack.license.self_generated.type: basic
xpack.security.http.ssl.key: certs/instance/instance.key
xpack.security.http.ssl.certificate: certs/instance/instance.crt
xpack.security.http.ssl.certificate_authorities: certs/ca/ca.crt

xpack.security.transport.ssl.verification_mode: none
xpack.security.transport.ssl.key: certs/instance/instance.key
xpack.security.transport.ssl.certificate: certs/instance/instance.crt
xpack.security.transport.ssl.certificate_authorities: certs/ca/ca.crt

xpack.security.http.ssl.supported_protocols: TLSv1.2
xpack.security.transport.ssl.supported_protocols: TLSv1.2

新节点配置

cluster.name: soc-cluster
node.name: node-2
cluster.initial_master_nodes: ["node-2"]
path.data: /data/soc/elastic/data
path.logs: /data/soc/elastic/logs
network.host: 0.0.0.0
discovery.seed_hosts: ["10.32.3.2", "10.32.3.18", "10.32.3.20"]
xpack.security.enabled: true
xpack.security.http.ssl.enabled: true
xpack.security.transport.ssl.enabled: true

xpack.license.self_generated.type: basic
xpack.security.http.ssl.key: certs/instance/instance.key
xpack.security.http.ssl.certificate: certs/instance/instance.crt
xpack.security.http.ssl.certificate_authorities: certs/ca/ca.crt

xpack.security.transport.ssl.verification_mode: none
xpack.security.transport.ssl.key: certs/instance/instance.key
xpack.security.transport.ssl.certificate: certs/instance/instance.crt
xpack.security.transport.ssl.certificate_authorities: certs/ca/ca.crt

xpack.security.http.ssl.supported_protocols: TLSv1.2
xpack.security.transport.ssl.supported_protocols: TLSv1.2

老节点配置

es集群无缝迁移扩容_elastic

es集群无缝迁移扩容_elastic_02

数据迁移

1.旧集群添加新节点后自动会触发分片均衡策略

es集群无缝迁移扩容_json_03

es集群无缝迁移扩容_json_04

es集群无缝迁移扩容_elastic_05

2.锁定分片只能分配到新节点上

集群中每个节点都能尽量分配到相等的分片数量

es集群无缝迁移扩容_elastic_06

手动干预集群的分片策略

es集群无缝迁移扩容_elastic_07

3.手动驱逐旧节点上的分片到新节点上

es集群无缝迁移扩容_重启_08

es集群无缝迁移扩容_json_09

curl -u elastic:11111 --insecure https://10.32.3.2:9200/_cat/nodes

transient   //暂时配置 重启后失效
persistent  //永久配置 重启后仍然有效

1.源集群操作
  触发集群分片均衡策略,把集群老节点上的分片全部移动到新的集群节点上
 curl -u elastic:1111 --insecure -H "Content-Type: application/json" -XPUT https://10.32.3.2:9200/_cluster/settings -d '{
  "transient" : {
     "cluster.routing.allocation.include._name" : null, //设置集群分片可以包含的节点名称
     "cluster.routing.allocation.exclude._name" : "node-2,node-18,node-20,node-4"  //设置集群分片排除的节点名称
  }
}'


在分片迁移的中间状态由于分片迁移会消耗一定的性能。因此为了稳定性考虑,可以通过如下API将迁移的并发度和迁移速度调小一点。
curl -u elastic:11111 --insecure -H "Content-Type: application/json" -XPUT https://10.32.3.2:9200/_cluster/settings -d '{ "persistent": {
            "cluster.routing.allocation.node_concurrent_recoveries": 2,
            "indices.recovery.max_bytes_per_sec": "100mb"  //设置节点的分片传输速率
 }
}'


2.集群分离
  停止集群中老节点的es服务进程
  老节点下线需要滚动进行,且每个节点的下线间隔需要大于10s,否则会有失主的风险
  原来的主节点需要最后下线
  

4.还原es集群的配置
 curl -u elastic:11111 --insecure -H "Content-Type: application/json" -XPUT https://10.32.3.2:9200/_cluster/settings -d '{
  "transient" : {
     "cluster.routing.allocation.include._name" : null,  //还原集群的默认配置
     "cluster.routing.allocation.exclude._name" : null   //还原集群的默认配置
  }
}'

View Code

1.把老节点上的分片迁移到新节点上

es集群无缝迁移扩容_elastic_10