[2.4.0, 2.4.1] Graylog hangs at startup

Graylog Central (peer support)
1

Hello!

I’ve been experiencing this issue before, and I “resolved” returning to a previous state of the data (mongo/elasticsearch/graylog).

This is happening using the official Docker image. Analysing the Graylog’s log, it hangs without any warning after “INFO : org.graylog2.bindings.providers.RulesEngineProvider - No static rules file loaded.”… No inputs are initialised, no web api, so I don’t really know where to look now.

May you please help me?
This is the full log: https://pastebin.com/PTLjFW4j

2

Please provide the complete configuration of Graylog (also docker-compose.yml if you’re using Docker Compose).

3

Hello jochen,

here is the docker-compose configuration:

https://pastebin.com/GxHX5hEZ

As per the configuration file, is the one which comes with the official docker image (image id: f93945a7d67c).

Thank you very much :slight_smile:

P.S.
I can enter and interact with the mongo database, and the elasticsearch’s health API reports the status green.

4

What are the hardware specs of the machine running these Docker containers?

5

It’s an Amazon c5.2xlarge instance (8 vCPU, 16 GB of RAM, running with a 300GB SSD). Running a new instance of the graylog stack via docker-compose it works, but obviously I’d loose the previous data and, worst thing, it’d have the same problem again in the future.

6

Try starting Graylog in debug mode by setting the environment variable GRAYLOG_SERVER_OPTS to --debug.

7

It keeps repeating the following log (again and again):

graylog_1        | 2018-01-23 14:40:18,581 DEBUG: org.mongodb.driver.protocol.command - Sending command {find : BsonString{value='roles'}} to database graylog on connection [connectionId{localValue:2, serverValue:4}] to server mongo:27017
graylog_1        | 2018-01-23 14:40:18,582 DEBUG: org.mongodb.driver.protocol.command - Command execution completed
8

Please post the complete log and not just some arbitrary lines.

9

Here it is. After those lines it will continue spamming the two lines above.

https://pastebin.com/zPmAQrjB

Sorry, it was capped to ~6000 lines
http://termbin.com/vcnj

10

As an additional information, I can confirm that the broken docker-composed environment has the very same (mongo) graylog.roles collection as a newly created one… if of any interest in debugging…

11

That’s a red herring. The “roles” collection is read each second by default.

12

Do you see anything particular in the given log?

13

Playing around with the mongo collections, I wiped the pipeline ones and graylog started working again… strange, also because I don’t really know why it wasn’t logged anywhere…

14

Do you happen to have a backup of the pipeline-related collections (or better: the complete MongoDB database) so we could try to reproduce the problem?

15

I could try recovering it. But I think I can recover the pipeline now:

rule "Drop unsupported browsers errors"
when
    has_field("user_agent_browser_name_and_version")
    AND lookup_value("unsupported-browsers", $message.user_agent_browser_name_and_version, "0") == "1"
then
    drop_message();
end

Where “unsupported-browsers” is a cached CSV lookup table

16

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.