"Nothing found in stream All Messages" and Java is at 493% CPU

Graylog Central (peer support)
1

Hi,
My graylog install has been working well for months. Today I noticed the last message in the ‘all messages’ stream is from 5am today. It’s around 12:30pm right now.

I have syslog:514 listening on this box and graylog Local inputs is fed from this port. The System > Inputs status shows 31Msgs/s for last 1 minute average.

I rebooted the Graylog box within the past hour and a Java process seems to be running quite high. I’m not a Graylog expert by any means. Any thoughts where to look for the problem?

top - 12:28:54 up  1:26,  1 user,  load average: 9.51, 9.69, 10.16
Tasks: 217 total,   1 running, 216 sleeping,   0 stopped,   0 zombie
%Cpu(s): 33.2 us,  1.1 sy, 55.1 ni, 10.5 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem:  13192254+total,  7198040 used, 12472451+free,   162696 buffers
KiB Swap: 33554428 total,        0 used, 33554428 free.  3425244 cached Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                                                                                                                                        
 2442 root      22   2 7016916 1.380g  23940 S 491.8  1.1 372:34.40 java                                                                                                                                           
23380 graylog   20   0 4535476 694068  23684 S 232.9  0.5   0:22.38 java                                                                                                                                           
16658 root      20   0 1333848 395812  10332 S   6.5  0.3   0:36.56 Suricata-Main

The last 10 messages from graylog-server log are below. Nothing much to go on there:

tail -10 /var/log/graylog-server/server.log  
2017-06-14T12:26:02.418-05:00 INFO  [AggregatesMaintenance] removed 0 history items
2017-06-14T12:27:02.418-05:00 INFO  [AggregatesMaintenance] removed 0 history items
2017-06-14T12:28:02.418-05:00 INFO  [AggregatesMaintenance] removed 0 history items
2017-06-14T12:29:02.418-05:00 INFO  [AggregatesMaintenance] removed 0 history items
2017-06-14T12:30:02.421-05:00 INFO  [AggregatesMaintenance] removed 0 history items
2017-06-14T12:31:02.418-05:00 INFO  [AggregatesMaintenance] removed 0 history items
2017-06-14T12:32:02.418-05:00 INFO  [AggregatesMaintenance] removed 0 history items
2017-06-14T12:33:02.418-05:00 INFO  [AggregatesMaintenance] removed 0 history items
2017-06-14T12:34:02.418-05:00 INFO  [AggregatesMaintenance] removed 0 history items
2017-06-14T12:35:02.418-05:00 INFO  [AggregatesMaintenance] removed 0 history items
2

In case anyone else has the same problem, it appears to be botched permssions on the /var/lib/graylog-server directory. I started graylog in debug mode (GRAYLOG_SERVER_ARGS="-d" in /etc/default/graylog-server) and immediately found errors about “Failed to acquire lock” on things under that directory. as well as ‘permission denied’.

Seems to have fixed things with chown graylog:graylog /var/lib/graylog-server -R. No idea how it got that way. shrug

3

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.