Graylog not reading messages from journal


(Peter Schoenegger) #1

We have installed a Graylog server and send in log data from multiple systems. We get an average of 3000 messages per second. For some reason Graylog is not efficiently reading messages from the journal. Any help would be highly appreciated!!! What could be the problem here?

48,881,103 unprocessed messages are currently in the journal, in 314 segments.
1,524 messages have been appended in the last second, 1,060 messages have been read in the last second.

The journal contains 48,857,121 unprocessed messages in 314 segments. 2,943 messages appended, 0 messages read in the last second.

That’s our graylog-settings.json file:
root@xyz:/home/ubuntu# vim /etc/graylog/graylog-settings.json
{
“timezone”: “Europe/Vienna”,
“smtp_server”: “mailhub.xyz.at”,
“smtp_port”: 2500,
“smtp_user”: “”,
“smtp_password”: “”,
“smtp_from_email”: “graylog@xyz.at”,
“smtp_web_url”: “http://graylog-beta”,
“smtp_no_tls”: true,
“smtp_no_ssl”: true,
“master_node”: “127.0.0.1”,
“local_connect”: false,
“current_address”: “172.20.45.29”,
“last_address”: “172.20.45.29”,
“enforce_ssl”: false,
“journal_size”: 100,
“node_id”: false,
“internal_logging”: false,
“web_listen_uri”: false,
“web_endpoint_uri”: false,
“rest_listen_uri”: false,
“rest_transport_uri”: false,
“external_rest_uri”: false,
“custom_attributes”: {
“graylog-server”: {
“memory”: “10240m”
},
“elasticsearch”: {
“memory”: “10240m”
}
}
}

image


(Jochen) #2

Is your Elasticsearch cluster healthy and able to cope with the message throughput?


(Peter Schoenegger) #3

Hi, Thank you so much for your help which I really appreciate!!! Elasticsearch settings have not been configured or changed from my side. We have the following amount of data currently in Graylog: 33 indices, 3,398,194,851 documents, 3.2TB (Index rotation strategy: Index Time, Rotation period: P1D (1d, a day), Index retention strategy: Delete, Max number of indices: 40)

This is what I get back when running (found on google: http://syslog.xyz.at:9200/_cluster/health?pretty):

{
“cluster_name” : “graylog”,
“status” : “yellow”,
“timed_out” : false,
“number_of_nodes” : 1,
“number_of_data_nodes” : 1,
“active_primary_shards” : 132,
“active_shards” : 132,
“relocating_shards” : 0,
“initializing_shards” : 0,
“unassigned_shards” : 132,
“delayed_unassigned_shards” : 0,
“number_of_pending_tasks” : 0,
“number_of_in_flight_fetch” : 0,
“task_max_waiting_in_queue_millis” : 0,
“active_shards_percent_as_number” : 50.0
}

Please find the files /opt/graylog/embedded/cookbooks/graylog/attributes/default.rb and vim /opt/graylog/conf/graylog.conf attached…


(Peter Schoenegger) #4

Oh sorry, I can’t upload the files…


(Peter Schoenegger) #5

How can I find out if the cluster is able to cope with the message throughput? Thanks a lot!!!


#6

It does seem your ES cluster is too slow. If it were fast enough, the output buffer would not be 100%. Check if you have enough RAM in the ES node and consider making an ES cluster.


(Peter Schoenegger) #7

Thank you for your help! For some reason the operation went back to a perfectly normal state after doing a reboot yesterday evening! Appreciate your help!!!


(system) #8

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.