es定期删除数据
1、定期删除索引
使用sentinl报警后，会产生大量如下索引，虽然不占空间，但时间久了也不好，故写个脚本定期删除
脚本如下：
1 #!/bin/bash
2 #只保留5天内的日志索引
3 LAST_DATA=`date -d "-5 days" "+%Y.%m.%d"`
4 #删除上个月份所有的索引
5 curl -XDELETE 'http://10.139.xx.xx:9200/*-'${LAST_DATA}''
再在设置一个定时策略即可
1 0 1 * * * /data1/elk/scripts/clear-index.sh
2、定期删除索引
Curator 是elasticsearch 官方的一个索引管理工具，可以删除、创建、关闭、段合并等等功能
安装
参考官网：https://www.elastic.co/guide/en/elasticsearch/client/curator/current/installation.html
pip install elasticsearch-curator
安装完如果curator 和curator_cli说明安装成功
curator核心在于俩个配置文件，配置文件名称随意无要求：
配置文件config.yml：配置要连接的ES地址、日志配置、日志级别等；
执行文件action.yml: 配置要执行的操作(可批量）、配置索引的格式（前缀匹配、正则匹配方式等）
config.yml样例
具体参数解析见官网：https://www.elastic.co/guide/en/elasticsearch/client/curator/4.2/configfile.html
client:
hosts:
- 127.0.0.1
port: 9200
url_prefix:
use_ssl: False
certificate:
client_cert:
client_key:
ssl_no_validate: False
http_auth:
timeout: 30
master_only: False logging:
loglevel: INFO
logfile: /var/log/elasticsearch-curator.log
logformat: default
blacklist: []
action.yml样例(删除3天前的数据)：
参数具体意思参见官网：https://www.elastic.co/guide/en/elasticsearch/client/curator/4.2/actionfile.html
actions:
1:
action: delete_indices
description: >-
Delete metric indices older than 3 days (based on index name), for
zou_data-2018-05-01
prefixed indices. Ignore the error if the filter does not result in an
actionable list of indices (ignore_empty_list) and exit cleanly.
options :
ignore_empty_list: True
disable_action: True
filters:
- filtertype: pattern
kind: regex
value: '^(zou_data-).*$'
- filtertype: age
source: name
direction: older
timestring: '%Y-%m-%d'
unit: days
unit_count: 3
运行curator
单次运行
curator --config config.yml action.yml
定时任务运行
0 0 */1 * * curator --config /opt/elasticsearch-curator/config.yml /opt/elasticsearch-curator/action.yml
3、定期删除索引内的数据
#!/bin/bash
indexs=` curl -X GET 'http://10.10.10.10:9200/_cat/indices?v' | awk '{print $3}' | grep -vE '(kibana|index|watcher|monitoring)'`
for index in $indexs
do
curl -X POST "10.139.34.129:9200/$index/_delete_by_query?pretty" -H 'Content-Type:application/json' -d '
{
"query": {
"bool": {
"must": [
{
"range": {
"@timestamp": {
"gte": "now-7d",
"lte": "now",
"format": "epoch_millis"
}
}
}
],
"must_not": []
}
}
}'
echo "已清除$index 索引内七天前数据~"
done
0 1 * * * /data1/elk/scripts/clear-data.sh
ES的删除操作，不会立即生效，跟更新操作类似。只是会被标记为已删除状态，ES后期会自动删除。
es启动脚本
#!/bin/bash
#set -x
cd `dirname $0`
data_dir=/data1/elk/elasticsearch if [ ! -d $data_dir/data ]; then
mkdir $data_dir/data && chown -R dev.dev $data_dir
fi bin_dir=$data_dir/bin PID=`ps -ef | grep elasticsearch | grep -v grep | grep root | grep -v bash |awk '{print $2}'` if [ -n "$PID" ]
then kill -9 $PID
echo "before: $PID"
cd $bin_dir && nohup su - dev -c "$bin_dir/elasticsearch" >> /dev/null 2>&1 &
sleep 3
P=`ps -ef | grep elasticsearch | grep -v grep | grep root | grep -v bash |awk '{print $2}'`
echo "now : $P"
else
echo "starting"
cd $bin_dir && nohup su - dev -c "$bin_dir/elasticsearch" >> /dev/null 2>&1 &
P=`ps -ef | grep elasticsearch | grep -v grep | grep root | grep -v bash |awk '{print $2}'`
echo "now : $P"
fi
kinaba启动脚本
#!/bin/bash
#set -x
cd `dirname $0`
data_dir=/data1/elk/kibana if [ ! -d $data_dir/data ]; then
mkdir $data_dir/data && chown -R dev.dev $data_dir
fi bin_dir=$data_dir/bin PID=`netstat -nlpt | grep 5601 | awk '{print $7}' | cut -d / -f1` if [ -n "$PID" ]
then kill -9 $PID
echo "before: $PID"
cd $bin_dir && nohup su - dev -c "$bin_dir/kibana" >> $data_dir/logs/kibana.log 2>&1 &
sleep 3
P=`netstat -nlpt | grep 5601 | awk '{print $7}' | cut -d / -f1`
echo "now : $P"
else
echo "starting"
cd $bin_dir && nohup su - dev -c "$bin_dir/kibana" >> $data_dir/logs/kibana.log 2>&1 &
P=`netstat -nlpt | grep 5601 | awk '{print $7}' | cut -d / -f1`
echo "now : $P"
fi