Can't see the messages that elasticsearch has

hi people

i can’t see messages that are older than 4 days when i do a query like message:ssh
every time i’m getting this error

now for example if i execute a search in elasticsearch using curl i get something but not everything…

curl -XGET http://10.161.90.41:9200/_all/_search?q=message:ssh
{"took":86,"timed_out":false,"num_reduce_phases":2,"_shards":{"total":908,"successful":908,"skipped":0,"failed":0},"hits":{"total":16701418,"max_score":18.795755,"hits":[{"_index":"zscaler_1997","_type":"message","_id":"17098e83-002a-11ea-b1e0-00505686b4ab","_score":18.795755,"_source":{"reason":"Allowed","file_Type":"None reqSize=1995 responseSize=844 totalSize=2839 sTime=35 cTime=36","referer":"www.google.com/","app_Name":"Google Search","gl2_remote_ip":"10.161.90.60","risk_Score":"0","gl2_remote_port":38187,"resp_Version":"1.1","source":"10.161.90.60","gl2_source_input":"5c1299f2f27cf219513df994","web_hostname":"www.google.com","dst_ip":"216.58.193.132","src_ip":"192.168.10.227","app_Class":"WebSearch","protocol":"HTTPS","action":"Allowed","gl2_source_node":"6dfaf83b-9635-43e0-98fc-a243e3ae61d2","url_Super_Cat":"Information Technology","dlp_Dict":

a curious thing is that when i do these queries for a short period of time elasticsearch doesn’t index anything or graylog doesn’t send anything to elasticsearch, the cool thing is that no messages are lost.

anyone has any idea how can i fix these types of queries in graylog or in elasticsearch?

please note that elasticsearch is up and running. and the load on the elasticsearch data nodes is not going to high, it’s arround 11 or max 15…

thanks,
Marius.

you should check the elasticsearch log file. Graylog gets a 500 in return and that is something elasticsearch will log.

That should give you guidance to find the reason.

just ran multiple tests and no info in the elasticsearch log appear on any of the cluster nodes…