Graylog deflector problem

Graylog Central (peer support)
1

1. Describe your incident:
My graylog has stopped writing logs, i think it is related to graylog_deflector.

2. Describe your environment:

  • OS Information:
    redhat 7.
    Graylog 3.3.5

3. What steps have you already taken to try and solve the problem?
It started with that my graylog_deflector complaid :
Graylog_deflector exists as an indexer and is not an alias
I stopped graylog servers and removed that index using elastic api.
but i think something broke as well.
I have tried to recalculate and rotate using gui.
My elastic is green and this is my aliases.

curl -X GET “http://:9200$IP/_cat/aliases?v”
alias index filter routing.index routing.search
gl-events_deflector gl-events_249 - - -
gl-system-events_deflector gl-system-events_252 - - -
atlassian_deflector atlassian_112 - - -

#elastic status
{
“cluster_name” : “graylog2”,
“status” : “green”,
“timed_out” : false,
“number_of_nodes” : 3,
“number_of_data_nodes” : 2,
“active_primary_shards” : 894,
“active_shards” : 1788,
“relocating_shards” : 0,
“initializing_shards” : 0,
“unassigned_shards” : 0,
“delayed_unassigned_shards” : 0,
“number_of_pending_tasks” : 0,
“number_of_in_flight_fetch” : 0,
“task_max_waiting_in_queue_millis” : 0,

Logs from elastic do not complain on anything, but one of my graylog that is master complains about the following
2023-05-15T15:23:05.313+02:00 WARN [IndexFieldTypePollerPeriodical] Active write index for index set “Default index set” (5991a880b0537403a942df26) doesn’t exist yet
2023-05-15T15:23:05.772+02:00 WARN [IndexFieldTypePollerPeriodical] Active write index for index set “winbeats_logs” (5ba0fcd3c62d021b94668d29) doesn’t exist yet

If i locate my graylog_deflector in gui .
system → Indices & Index Sets → Default index set → graylog2_deflector

Time range of index is unknown, because index range is not available. Please recalculate index ranges manually. (176.3GiB / 442,719,136 messages

i can see it hold some TB of data. but when i rotate nothing get writen or anything.
If i expand the graylog_deflector and choose recalculate i get warnings.
Error starting index ranges recalculation for graylog2_deflector
Could not create a job to start index ranges recalculation for graylog2_deflector, reason: Error: cannot POST http://172.16.231.22:9000/api/system/indices/ranges/graylog2_deflector/rebuild (400)

Helpful Posting Tips: Tips for Posting Questions that Get Answers [Hold down CTRL and link on link to open tips documents in a separate tab]

2

My graylog has stopped writing logs

Are you able to expand on this? Your cluster was working with issue and then suddenly stopped working? Its really difficult to say whats wrong without knowing what lead up to the issue.

The only helpful thing i can offer is to rebuild graylog using a more recent version (3.3 is fairly outdated).

Are you able to share your server.log file?

3

Hey @landychev

what version of Elasticsearch and mongoDb do you have?

4

Mongo is 4.4.10 and elastic is 6.8.8

5

It´s hard to analysize what went wrong.
All i know is that one of elastic node went down.
i removed the bad node from the cluster and then i saw the error with alias.

How to rebuild the graylog ?

6

After doing some more digging i can see that my stream All messages that point to a 2tb index is broken.
How to repair it ?
Or do i need to create a new stream ?

graylog-stream
graylog-stream1621×109 13.1 KB

7

You can do a clean install via Installing Graylog.

The above doc is useful, but if you want a set of install commands that has been simplified as much as possible: se-poc-docs/src/On Prem POC at main · Graylog2/se-poc-docs · GitHub.

Hope that helps.

8

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.