Elasticsearch 8.x ELK 搭建并配置 SSL

ELK安装: 版本: 8.9.0 准备: 更新apt: 命令行 apt-get update 命令行 apt-get upgrade 命令行 apt-get install unzip JDK安装: - 1查看JDK版本: 命令行 apt search openjdk - 2安装JDK: 命令行 apt install openjdk-21-jdk -y （找一个最高的来安装） 创建ELK目录: 命令行 mkdir -p /iscsi/elk 命令行 mkdir -p /iscsi/elk/elasticsearch 命令行 mkdir -p /iscsi/elk/elasticsearch/data 命令行 mkdir -p /iscsi/elk/elasticsearch/logs 命令行 mkdir -p /iscsi/elk/elasticsearch/plugins 命令行 mkdir -p /iscsi/elk/kibana 命令行 mkdir -p /iscsi/elk/kibana/config 命令行 mkdir -p /iscsi/elk/kibana/data 命令行 mkdir -p /iscsi/elk/kibana/logs 命令行 mkdir -p /iscsi/elk/logstash 命令行 mkdir -p /iscsi/elk/logstash/config 命令行 mkdir -p /iscsi/elk/logstash/logs 命令行 mkdir -p /iscsi/elk/logstash/pipeline 文件夹提权: 命令行 chmod 777 -R /iscsi/elk/* 创建用户和组: 1查看用户: 命令行 cat /etc/passwd 2查看组: 命令行 cat /etc/group 3修改原来uid为1000的用户: 命令行 usermod -u 1001 tonywoo 4新建elasticsearch 用户: 命令行 useradd -u 1000 -g root elasticsearch ??useradd -u 1000 -g tonywoo elasticsearch 6新建组: 命令行 groupadd elasticsearch 5修改elasticsearch 用户密码: 命令行 passwd elasticsearch 设置目录权限: 1设置目录拥有者: 命令行 chown -R elasticsearch:root /iscsi/elk/elasticsearch/ 2设置目录拥有组: 命令行 chgrp -R 0 /iscsi/elk/elasticsearch 3备注: /iscsi/elk/elasticsearch 下的子目录的拥有者都要设置为 elasticsearch 这个用户 内存设置: - 1查看用户内存权限: 命令行 sysctl -a|grep vm.max_map_count - 2设置用户内存权限: 命令行 vim /etc/sysctl.conf - 3禁止内存与硬盘交换: vm.swappiness=1 - 4配置最大映射数量: vm.max_map_count=262144 - 5使配置生效: 退出vim，命令行 sysctl -p 修改打开文件数: - 1进入文件: 命令行 vim /etc/security/limits.conf - 2追加内容: ` # elasticsearch是用户，也可以使用*代替所有用户） elasticsearch soft nofile 65536 elasticsearch hard nofile 65536 #内存锁定交换 soft memlock unlimited hard memlock unlimited ` 查看docker网络: 命令行 docker network list 开始: 新建docker-compose文件: 命令行 touch /iscsi/elk/docker-compose-elk.yml 修改docker-compose文件如下: ` version: '3.7' services: elasticsearch: image: elasticsearch:8.9.0 container_name: elasticsearch hostname: elasticsearch restart: "no" volumes: - /etc/localtime:/etc/localtime #- /iscsi/elk/elasticsearch/data:/usr/share/elasticsearch/data:rw #- /iscsi/elk/elasticsearch/config:/usr/share/elasticsearch/config:rw #- /iscsi/elk/elasticsearch/logs:/usr/share/elasticsearch/logs:rw #- /iscsi/elk/elasticsearch/plugins:/usr/share/elasticsearch/plugins:rw environment: - TZ="Asia/Shanghai" - ES_JAVA_OPTS=-Xms512m -Xmx512m - discovery.type=single-node ports: - "9200:9200" - "9300:9300" networks: elastic: ipv4_address: 172.99.0.2 aliases: - elasticsearch kibana: image: kibana:8.9.0 container_name: kibana hostname: kibana restart: "no" volumes: - /etc/localtime:/etc/localtime #- /iscsi/elk/kibana/data:/usr/share/kibana/data:rw #- /iscsi/elk/kibana/config:/usr/share/kibana/config:rw #- /iscsi/elk/kibana/logs:/usr/share/kibana/logs:rw ports: - 5601:5601 depends_on: - elasticsearch networks: elastic: ipv4_address: 172.99.0.3 aliases: - kibana logstash: image: logstash:8.9.0 container_name: logstash hostname: logstash restart: "no" volumes: - /etc/localtime:/etc/localtime #- /iscsi/elk/logstash/config:/usr/share/logstash/config:rw #- /iscsi/elk/logstash/logs:/usr/share/logstash/logs:rw #- /iscsi/elk/logstash/pipeline:/usr/share/logstash/pipeline:rw ports: - 5044:5044 - 9066:9066 - 21068:21068 - "5000:5000/tcp" - "5000:5000/udp" depends_on: - elasticsearch networks: elastic: ipv4_address: 172.99.0.4 aliases: - logstash ##自定义网络 networks: elastic: ipam: driver: default config: - subnet: 172.99.0.0/16 ` 上面带有#号 的后面要解除 首次启动: /iscsi/elk下执行 命令行 docker-compose -f docker-compose-elk.yml up -d 复制容器内目录到宿主机: 命令行 docker cp elasticsearch:/usr/share/elasticsearch/config /iscsi/elk/elasticsearch/ 命令行 docker cp kibana:/usr/share/kibana/config /iscsi/elk/kibana/ 命令行 docker cp logstash:/usr/share/logstash/config /iscsi/elk/logstash/ 命令行 docker cp logstash:/usr/share/logstash/pipeline /iscsi/elk/logstash/ 命令行 chmod 777 -R /iscsi/elk/kibana/* 命令行 chmod 777 -R /iscsi/elk/logstash/* 命令行 chown -R elasticsearch:root /iscsi/elk/elasticsearch/config/ 命令行 chown -R elasticsearch:root /iscsi/elk/elasticsearch/config/certs/ 修改elasticsearch的jvm文件: 打开 /iscsi/elk/elasticsearch/config/jvm.options添加下面两项 命令行 vim /iscsi/elk/elasticsearch/config/jvm.options ` -Xms512m -Xmx512m ` 修改logstash的jvm文件: 打开 /iscsi/elk/logstash/config/jvm.options添加下面两项 命令行 vim /iscsi/elk/logstash/config/jvm.options 修改如下 #-Xms1g 改为 -Xms512m #-Xmx1g 改为 -Xmx512m 放开注释: 放开docker-compose-elk.yml文件内挂载数据卷的注释 命令行 cd /iscsi/elk 命令行 docker-compose -f docker-compose-elk.yml stop 命令行 docker-compose -f docker-compose-elk.yml rm 命令行 docker-compose -f docker-compose-elk.yml up -d 配置SSL: 设置目录拥有者: 命令行 chown -R elasticsearch:root /iscsi/elk/elasticsearch/config/* 命令行 chown -R elasticsearch:root /iscsi/elk/elasticsearch/config/certs/* 进入elasticsearch容器: 命令行 docker exec -it elasticsearch /bin/bash 生成elastic-stack-ca.p12文件: 命令行 ./bin/elasticsearch-certutil ca 需要在 `Enter password for elastic-stack-ca.p12:` 哪里设置密码 生成elastic-certificates.p12: 命令行 ./bin/elasticsearch-certutil cert --ca elastic-stack-ca.p12 `Enter password for CA(elastic-stack-ca.p12):`后输入 elastic-stack-ca.p12设置的密码 复制文件到config文件夹: 命令行 mv elastic-certificates.p12 config/certs/ 命令行 mv elastic-stack-ca.p12 ./config/ 备注 开放了docker-compose-elk.yml的注释复制文件到config宿主机文件夹会同时改变；elastic-stack-ca.p12文件后续也需要用到 设置文件拥有权: 退出容器 命令行 chmod 777 /iscsi/elk/elasticsearch/* 命令行 chmod 777 /iscsi/elk/elasticsearch/config/* 命令行 chmod 777 /iscsi/elk/elasticsearch/config/certs/* 命令行 chown -R elasticsearch:root /iscsi/elk/elasticsearch/* 设置elasticsearch.yml配置文件: 命令行 vim /iscsi/elk/elasticsearch/config/elasticsearch.yml 修改为如下 ` # Enable encryption and mutual authentication between cluster nodes xpack.security.transport.ssl: enabled: true verification_mode: certificate keystore.path: certs/elastic-certificates.p12 truststore.path: certs/elastic-certificates.p12 ` 修改密码: 如果certificate设置了密码，需要执行一下两个命令 退回到容器根目录 命令行 ./bin/elasticsearch-keystore add xpack.security.transport.ssl.keystore.secure_password 命令行 ./bin/elasticsearch-keystore add xpack.security.transport.ssl.truststore.secure_password 备注 这个密码就是 elastic-certificates.p12 文件设置的密码 重启elasticsearch容器: docker restart elasticsearch 配置elasticsearch和kibana开启https访问: 进入elasticsearch容器: 命令行 docker exec -it elasticsearch /bin/bash 生成elasticsearch-ssl-http.zip: 命令行 ./bin/elasticsearch-certutil http 操作如下 ## Elasticsearch HTTP Certificate Utility The 'http' command guides you through the process of generating certificates for use on the HTTP (Rest) interface for Elasticsearch. This tool will ask you a number of questions in order to generate the right set of files for your needs. ## Do you wish to generate a Certificate Signing Request (CSR)? A CSR is used when you want your certificate to be created by an existing Certificate Authority (CA) that you do not control (that is, you don't have access to the keys for that CA). If you are in a corporate environment with a central security team, then you may have an existing Corporate CA that can generate your certificate for you. Infrastructure within your organisation may already be configured to trust this CA, so it may be easier for clients to connect to Elasticsearch if you use a CSR and send that request to the team that controls your CA. If you choose not to generate a CSR, this tool will generate a new certificate for you. That certificate will be signed by a CA under your control. This is a quick and easy way to secure your cluster with TLS, but you will need to configure all your clients to trust that custom CA. ## 生成CSR 输入n Generate a CSR? [y/N]n ## Do you have an existing Certificate Authority (CA) key-pair that you wish to use to sign your certificate? If you have an existing CA certificate and key, then you can use that CA to sign your new http certificate. This allows you to use the same CA across multiple Elasticsearch clusters which can make it easier to configure clients, and may be easier for you to manage. If you do not have an existing CA, one will be generated for you. ## 是否使用存在的ca 输入y（在基础配置时生成了） Use an existing CA? [y/N]y ## What is the path to your CA? Please enter the full pathname to the Certificate Authority that you wish to use for signing your new http certificate. This can be in PKCS#12 (.p12), JKS (.jks) or PEM (.crt, .key, .pem) format. ## 输入ca文件的地址 CA Path: /usr/share/elasticsearch/config/elastic-stack-ca.p12 Reading a PKCS12 keystore requires a password. It is possible for the keystore's password to be blank, in which case you can simply press <ENTER> at the prompt ## 输入文件设置的密码 Password for elastic-stack-ca.p12: ## How long should your certificates be valid? Every certificate has an expiry date. When the expiry date is reached clients will stop trusting your certificate and TLS connections will fail. Best practice suggests that you should either: (a) set this to a short duration (90 - 120 days) and have automatic processes to generate a new certificate before the old one expires, or (b) set it to a longer duration (3 - 5 years) and then perform a manual update a few months before it expires. You may enter the validity period in years (e.g. 3Y), months (e.g. 18M), or days (e.g. 90D) ## 设置过期时间 For how long should your certificate be valid? [5y] 10y ## Do you wish to generate one certificate per node? If you have multiple nodes in your cluster, then you may choose to generate a separate certificate for each of these nodes. Each certificate will have its own private key, and will be issued for a specific hostname or IP address. Alternatively, you may wish to generate a single certificate that is valid across all the hostnames or addresses in your cluster. If all of your nodes will be accessed through a single domain (e.g. node01.es.example.com, node02.es.example.com, etc) then you may find it simpler to generate one certificate with a wildcard hostname (*.es.example.com) and use that across all of your nodes. However, if you do not have a common domain name, and you expect to add additional nodes to your cluster in the future, then you should generate a certificate per node so that you can more easily generate new certificates when you provision new nodes. ## 是否为每一个节点生成证书 输入n Generate a certificate per node? [y/N]n ## Which hostnames will be used to connect to your nodes? These hostnames will be added as "DNS" names in the "Subject Alternative Name" (SAN) field in your certificate. You should list every hostname and variant that people will use to connect to your cluster over http. Do not list IP addresses here, you will be asked to enter them later. If you wish to use a wildcard certificate (for example *.es.example.com) you can enter that here. ## 节点的hostname,设置为elasticsearch，敲两次回车 Enter all the hostnames that you need, one per line. When you are done, press <ENTER> once more to move on to the next step. elasticsearch You entered the following hostnames. - elasticsearch ## 配置是否正确 Is this correct [Y/n]y ## Which IP addresses will be used to connect to your nodes? If your clients will ever connect to your nodes by numeric IP address, then you can list these as valid IP "Subject Alternative Name" (SAN) fields in your certificate. If you do not have fixed IP addresses, or not wish to support direct IP access to your cluster then you can just press <ENTER> to skip this step. ## 节点的ip(可以在宿主机通过命令docker inspect elasticsearch查看),设置为172.99.0.2，敲两次回车 Enter all the IP addresses that you need, one per line. When you are done, press <ENTER> once more to move on to the next step. 172.99.0.2 You entered the following IP addresses. - 172.99.0.2 ## 配置是否正确 Is this correct [Y/n]y ## Other certificate options The generated certificate will have the following additional configuration values. These values have been selected based on a combination of the information you have provided above and secure defaults. You should not need to change these values unless you have specific requirements. Key Name: elasticsearch Subject DN: CN=elasticsearch Key Size: 2048 ## 是否更改任意项 Do you wish to change any of these options? [y/N]n ## What password do you want for your private key(s)? Your private key(s) will be stored in a PKCS#12 keystore file named "http.p12". This type of keystore is always password protected, but it is possible to use a blank password. If you wish to use a blank password, simply press <enter> at the prompt below. ## 输入生成文件的密码（可不设置，设置需要在后面进行配置） Provide a password for the "http.p12" file: [<ENTER> for none] ## 再次输入生成文件的密码 Repeat password to confirm: ## Where should we save the generated files? A number of files will be generated including your private key(s), public certificate(s), and sample configuration options for Elastic Stack products. These files will be included in a single zip archive. ## 生成压缩文件的地址和名称，直接敲回车即可 What filename should be used for the output zip file? [/usr/share/elasticsearch/elasticsearch-ssl-http.zip] 移动elasticsearch-ssl-http.zip压缩包: 命令行 mv elasticsearch-ssl-http.zip ./config/ 解压文件: 退出容器 命令行 unzip /iscsi/elk/elasticsearch/config/elasticsearch-ssl-http.zip 解压后会在原目录下新增两个目录分别是 elasticsearch 和 kibana 命令行 mv /iscsi/elk/elasticsearch/config/elasticsearch/http.p12 /iscsi/elk/elasticsearch/config/certs/ 复制elasticsearch-ca.pem到kibana的config文件夹内: 命令行 cp /iscsi/elk/elasticsearch/config/kibana/elasticsearch-ca.pem /iscsi/elk/kibana/config/ 删除文件夹: 命令行 rm -rf /iscsi/elk/elasticsearch/certs/kibana 文件提权: 命令行 chmod 777 /iscsi/elk/elasticsearch/config/certs/http.p12 设置http密码: 命令行 docker exec -it elasticsearch /bin/bash 命令行 ./bin/elasticsearch-keystore add xpack.security.http.ssl.keystore.secure_password 退出容器 重启elasticsearch容器: 命令行 docker restart elasticsearch 设置elastic用户的密码: 进入容器: 命令行 docker exec -it elasticsearch /bin/bash 设置密码: ./bin/elasticsearch-reset-password -u elastic -i 设置kibana_system密码: 命令行 ./bin/elasticsearch-reset-password -u kibana_system -i 生成kibana用https访问的公钥和私钥: 命令行 ./bin/elasticsearch-certutil csr -name kibana-server 备注 生成csr-bundle.zip文件夹 复制csr-bundle.zip到kibana: 退出容器 命令行 docker cp elasticsearch:/usr/share/elasticsearch/csr-bundle.zip /iscsi/elk/kibana/ 解压csr-bundle.zip: 命令行 cd /iscsi/elk/kibana/ 命令行 unzip /iscsi/elk/kibana/csr-bundle.zip 备注 解压后会生成 kibana-server 文件夹 移动文件到kibana的配置目录: 命令行 mv /iscsi/elk/kibana/kibana-server/* /iscsi/elk/kibana/config/ 命令行 rm -rf /iscsi/elk/kibana/kibana-server 生成kibana-server.crt文件: 命令行 cd /iscsi/elk/kibana/config 命令行 openssl x509 -req -days 3650 -in kibana-server.csr -signkey kibana-server.key -out kibana-server.crt 文件提权: 命令行 chmod 777 elasticsearch-ca.pem kibana-server.csr kibana-server.key kibana-server.crt 修改kibana.yml文件: 命令行 vim /iscsi/elk/kibana/config/kibana.yml 最终文件为 ` # # ** THIS IS AN AUTO-GENERATED FILE ** # # Default Kibana configuration for docker target server.host: "0.0.0.0" server.shutdownTimeout: "5s" elasticsearch.hosts: [ "https:/172.99.0.2:9200" ] monitoring.ui.container.elasticsearch.enabled: true elasticsearch.ssl.certificateAuthorities: ["/usr/share/kibana/config/elasticsearch-ca.pem"] elasticsearch.username: "kibana_system" elasticsearch.password: "[e-mail]" server.ssl.certificate: "/usr/share/kibana/config/kibana-server.crt" server.ssl.key: "/usr/share/kibana/config/kibana-server.key" server.ssl.enabled: true # 设置中文访问 i18n.locale: "zh-CN" ` 重启kiban: 命令行 docker restart kibana 配置logstash: 设置logstash_system密码: 命令行 docker exec -it elasticsearch /bin/bash 命令行 ./bin/elasticsearch-reset-password -u logstash_system -i 生成logstash.pem文件: 退出容器 命令行 openssl pkcs12 -in elasticsearch/config/certs/elastic-certificates.p12 -cacerts -nokeys -chain -out logstash.pem 移动logstash.pem文件到logstash配置文件目录下: 命令行 mv /iscsi/elk/elasticsearch/config/certs/logstash.pem /iscsi/elk/logstash/config/ 提权logstash.pem: 命令行 chmod 777 /iscsi/elk/logstash/config/logstash.pem 配置logstash.yml文件: 命令行 vim /iscsi/elk/logstash/config/logstash.yml 最终文件 ` http.host: "0.0.0.0" xpack.monitoring.elasticsearch.hosts: [ "https://172.99.0.2:9200" ] #你的ca.pem 的所在路径 xpack.monitoring.elasticsearch.ssl.certificate_authority: "/usr/share/logstash/config/logstash.pem" xpack.monitoring.elasticsearch.ssl.verification_mode: certificate # 探嗅 es节点，设置为 false xpack.monitoring.elasticsearch.sniffing: false xpack.monitoring.elasticsearch.username: "logstash_system" xpack.monitoring.elasticsearch.password: "[e-mail]" ` 配置logstash.conf文件: 命令行 vim /iscsi/elk/logstash/pipeline/logstash.conf 最终文件 ` input { tcp { port => 21068 codec => json_lines } } output { elasticsearch { hosts => ["https://172.99.0.2:9200"] index => "tonywoo-%{+YYYY.MM.dd}" user => "elastic" password => "[e-mail]" ssl_enabled => true ssl_certificate_authorities => ["/usr/share/logstash/config/logstash.pem"] } } ` 重启logstash: docker restart logstash 设置自动启动: ` cat > /etc/systemd/system/docker-compose-elk.service << EOF [Unit] Description=Docker Compose Application Service Requires=docker.service After=docker.service [Service] Type=oneshot RemainAfterExit=yes WorkingDirectory=/iscsi/elk/ ExecStart=/iscsi/elk/docker-compose -f docker-compose-elk.yml up -d ExecStop=/iscsi/elk/docker-compose -f docker-compose-elk.yml up down TimeoutStartSec=0 [Install] WantedBy=multi-user.target ` 回车 命令行 ctrl+d