Graylog stopped working randomly

Graylog randomly stopped working and I can’t find any hints to why in the log files. My current Graylog Version is “3.3” and it is running on Centos 7 on Linux. If you need any additional Information please tell me and I will provide the necessary data.

Best regards,
beepjeep

PS: All the Processes are “Running” and “Active”.

What exactly do you mean by “it stops working”?

Everything is apparently “running” but I don’t receive any messages anymore. I’ve looked into all the log files and even deleted some to have a fresh log file to look at and I don’t see anything that’s wrong. @Ponet

If you run tcpdump on your graylog server, can you see packets being received on your input ports?

Have you checked the message buffers under System > Nodes ?

Is elasticsearch healthy? Are the indexes writable?

Could you tell me how I can do that? I’m really fresh to Linux and Centos. Elasticsearch is “Healthy” and the indexes are writable. Would you like to have a look at the log files maybe? I’ve seen something along the lines of "Winlogbeat expected [winlogbeat_user_name] but got [keyword] instead."
@Ponet (sorry for the ping, wasn’t sure if you would see this)

To check the message buffers, go to the node page on in the web interface.
You’ll find it under System > Nodes in the navbar.

For using tcpdump, see the below link:

From the error message, it sounds like there is a datatype mapping issue which could also be causing the problem.

This is what is on “Nodes”. Packets are being received. What exactly do you mean by “datatype missing issue” ?

Click on the node to view the details.

As for the ES mapping, that isn’t really my strong suit but, when ES receives messages it analyses the fields and assigns a datatype in the field mappings. From that error, it looks as though the datatype ES is picking up for winlogbeat_user_name field value is ‘keyword’ but, that is not what it was expecting.
I’d recommend looking at the documentation:
https://docs.graylog.org/en/3.3/pages/configuration/elasticsearch.html#custom-index-mappings

I’ll take a look into that but I’ll post this now.


I’ll keep you updated.

Sadly I can’t seem to get it working… I don’t know what to do… Is there anyone on this community that you would recommend? @Ponet

Hi,

do you run a cluster or a standalone?

  1. Go on the Overview of the nodes https://yourgraylog.example/system/nodes
  2. click on “More actions” on your stopped node:
    grafik

Have a look at the thread dump and the process-buffer dump. Is there anything suspicous? We had this once with a runaway grok pattern.

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.