1. Describe your incident:
My graylog has stopped writing logs, i think it is related to graylog_deflector.
2. Describe your environment:
- OS Information:
3. What steps have you already taken to try and solve the problem?
It started with that my graylog_deflector complaid :
Graylog_deflector exists as an indexer and is not an alias
I stopped graylog servers and removed that index using elastic api.
but i think something broke as well.
I have tried to recalculate and rotate using gui.
My elastic is green and this is my aliases.
curl -X GET “http://:9200$IP/_cat/aliases?v”
alias index filter routing.index routing.search
gl-events_deflector gl-events_249 - - -
gl-system-events_deflector gl-system-events_252 - - -
atlassian_deflector atlassian_112 - - -
“cluster_name” : “graylog2”,
“status” : “green”,
“timed_out” : false,
“number_of_nodes” : 3,
“number_of_data_nodes” : 2,
“active_primary_shards” : 894,
“active_shards” : 1788,
“relocating_shards” : 0,
“initializing_shards” : 0,
“unassigned_shards” : 0,
“delayed_unassigned_shards” : 0,
“number_of_pending_tasks” : 0,
“number_of_in_flight_fetch” : 0,
“task_max_waiting_in_queue_millis” : 0,
Logs from elastic do not complain on anything, but one of my graylog that is master complains about the following
2023-05-15T15:23:05.313+02:00 WARN [IndexFieldTypePollerPeriodical] Active write index for index set “Default index set” (5991a880b0537403a942df26) doesn’t exist yet
2023-05-15T15:23:05.772+02:00 WARN [IndexFieldTypePollerPeriodical] Active write index for index set “winbeats_logs” (5ba0fcd3c62d021b94668d29) doesn’t exist yet
If i locate my graylog_deflector in gui .
system → Indices & Index Sets → Default index set → graylog2_deflector
Time range of index is unknown, because index range is not available. Please recalculate index ranges manually. (176.3GiB / 442,719,136 messages
i can see it hold some TB of data. but when i rotate nothing get writen or anything.
If i expand the graylog_deflector and choose recalculate i get warnings.
Error starting index ranges recalculation for graylog2_deflector
Could not create a job to start index ranges recalculation for graylog2_deflector, reason: Error: cannot POST http://172.16.231.22:9000/api/system/indices/ranges/graylog2_deflector/rebuild (400)
Helpful Posting Tips: Tips for Posting Questions that Get Answers [Hold down CTRL and link on link to open tips documents in a separate tab]