elasticsearch filebeat (status=403)/blocked by: [FORBIDDEN/12/index read-only / allow delete ()

1、现象，elk 有部分服务的日志无法查询。查看落盘日志是有新日志更新的，无法查询到日志

2、查看filebeat 日志内容

kubectl logs -f filebeat-xxxx -n log

2022-10-20T03:06:02.403Z  WARN  elasticsearch/client.go:511 Cannot index event publisher.Event{Content:beat.Event{Timestamp:
time.Time{, Meta:common.MapStr(nil), Fields:common.MapStr{"agent"
:common.MapStr{"hostname":"filebeat-dxxxv", "id":"xxxxxc-xxxx-xxxx-xxx-xxxxx", "type":"filebeat", "version":"7.5.1"}, 
"ecs":common.MapStr{}, "env":"xxxx", "host":common.MapStr{"name":"filebeat-dcbzv"}, "input":common.MapStr{"type":"log"}, 
"k8s":common.MapStr{"node":"x.x.x.x"}, "level":"INFO", "log":common.MapStr{"file":common.MapStr{"path":"/opt/xxx/xxx/xx/xx.log"}, 
"offset":1141177}, "logger_name":"xxxxxxx", "message":"==== create native sql query ====", "pod":common.MapStr{"ip":"172.16.0.252"}, 
"service":"xxx", "su":"service_unit_key_IS_UNDEFINED", "thread_name":"quartzScheduler_Worker-7"}, Private:file.State{Id:"", Finished:false, 
Fileinfo:(*os.fileStat)(0xc000927040), Source:"/opt/xxx/xxx/xxx/xxx.log", Offset:1141466, Timestamp:time.Time{wall:0xc0cc4c4dc92f6b06, 
ext:2431695925687, loc:(*time.Location)(0x4e5b700)}, TTL:-1, Type:"log", Meta:map[string]string(nil), 
FileStateOS:file.StateOS{Inode:0x120518, Device:0xfd01}}, TimeSeries:false}, Flags:0x1, Cache:publisher.EventCache{m:common.MapStr(nil)}} 
(status=403): {"type":"cluster_block_exception",
"reason":"index [log-gmtest-2022.10.20] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"}

明显的提示为403 与read-only，filebeat端口已经禁止写入

(status=403): {"type":"cluster_block_exception",
"reason":"index [log-gmtest-2022.10.20] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"}

2、官方文档描述 当磁盘容量到达95%会将index设置为Read-only模式，防止磁盘打暴。

es端的日志

[2022-10-20T00:59:20,021][WARN ][o.e.c.r.a.DiskThresholdMonitor] [node-1] flood stage disk watermark [95%] exceeded on [F9JFqZzCT1WT2EwFSJ5wBw][node-1][/var/lib/elasticsearch/nodes/0] free: 4.7gb[4.8%], all indices on this node will be marked read-only

​​https://www.elastic.co/guide/en/elasticsearch/reference/6.8/disk-allocator.html​​

3、将索引删除，清理出空间即可

curl -XDELETE x.x.x.x:9200/*
 #支持通配符
  curl -XDELETE x.x.x.x:9200/abc*