Unprocessed messages are currently in the journal

[kevin_bolton]

I am using graylog-2.4.0-1.ova.

Below is a message in System / Nodes / Disk Journal / Utilization:
1,066,447 unprocessed messages are currently in the journal, in 2 segments.

I believe that is an error message and will cause incomplete search result.

I am a really newer.
Is any SOP that can help me to find out the problem or ask graylog to process the unprocessed messages?

[Magneton (Jake Smith)]

Hi Kevin,

When I have seen this it usually indicated a performance problem in that the graylog server has experienced a high peak of events and is waiting to write events to the journal whilst trying to process incoming events.

Can you give your vm extra resources? It would also be helpful if you can provide some information on your setup such as

How many log sources and types of logs eg 10 windows and one linux server
Current VM resources assigned

Though in your screen shot your journal utilisation looks very low - 0.18%

Are you seeing a historical message.

Try and give some more information when posting as it may enable someone here to help you easily. Obviously, don’t include sensitive config info.

Jake

[kevin_bolton]

Hi Magneton,

I re-create a new graylog server as below:
Graylog VM(5GB RAM) that running in Windows 10(i5-4310U, 8GB RAM)

I have 515 log files in a folder(c:\temp) that total size is 21GB.

I move 5~ 6 files every time from temp folder to filebeat log folder(c:\BELogs) and wait graylog become 0 unprocessed messages
When BELogs contain 200 files that total size is 8.44 GB, Unprocessed number is slowly increasing.
I do not see any error message.
What can I do?

[kevin_bolton]

[kevin_bolton]

[jochen]

What are the JVM settings for Elasticsearch and Graylog on that machine?
http://docs.graylog.org/en/2.4/pages/configuration/file_location.html

[kevin_bolton]

JVM settings for Elasticsearch and Graylog are the same file.

BTW, I have two NICs, eth0:10.0.2.9 and eth1:192.168.56.3
I am using http://192.168.56.3 as web GUI address and http://192.168.56.3:9000/api as server_url
I am not sure current_address and last_address as 10.0.2.9 are right or wrong.

[Magneton (Jake Smith)]

Kevin,

First start by using one ip address to make things less complicated.

Set elasticsearch correctly in /elasticsearch.yml.

cluster.name: graylog

Set the following(Graylog) server.conf - /etc/graylog/erver/server.conf

is_master = true
rest_listen_uri = http://10.0.1.177:9000/api/
web_listen_uri = http://10.0.1.177:9000/
elasticsearch_hosts = http://10.0.1.177:9200

You will now have everything on the one interface.

I would suggest that your log files are probably too big for the size of your VM.

Try sending logs on from a windows or linux host to start. I like to use NxLog rather than filebeat, just a personnel preference.

See that your logs are received and processed correctly, then try your large file.

This is from the graylog sizing guie which suggests that it will not work due to large file size without increasing VM performance.

Regards

Jake

[jan (Jan Doberstein)]

@Magneton if @kevin_bolton would not use the OVA - the virtual appliance - your suggestions would be correct.

@kevin_bolton did you run graylog-ctl reconfigure after you had changed anything? Some details about the script and how you work with the OVA can be found here in the documentation

http://docs.graylog.org/en/stable/pages/configuration/graylog_ctl.html

[Magneton (Jake Smith)]

Jan and Kevin,

I am using a VM built on top of Ubuntu built with bash script.

Jake

[jtkarvo]

Any errors in the Elasticsearch or Graylog log files?

[kevin_bolton]

I ran “graylog-ctl reconfigure” after
both of listen_uri IP from 0.0.0.0 to 192.168.56.3
and
elasticsearch_hosts IP from 10.0.2.9 to 192.168.56.3

then I check /opt/graylog/conf/graylog.conf again, it back to before (0.0.0.0 and 10.0.2.9)
I don’t know why, but,

1. unprocessed messages become 0
2. Graylog back to work
3. In overview page,
   3.1 Elasticsearch cluster
   Elasticsearch cluster is yellow. Shards: 8 active, 0 initializing, 0 relocating, 8 unassigned,
   3.2 Indexer failures
   There were 702 failed indexing attempts in the last 24 hours.
   
   

[kevin_bolton]

[jtkarvo]

Do you have Elasticsearch replicas=1 ?

If you have just one node, you need to have 0 replicas. After editing index settings, you can use curl to change that setting to the existing ones. See https://www.elastic.co/guide/en/elasticsearch/guide/current/replica-shards.html

[kevin_bolton]

Yes,
/opt/graylog/conf/graylog.conf > Elasticsearch replicas=1

I done
sudo vi /opt/graylog/conf/graylog.conf > Elasticsearch replicas=0
sudo graylog-ctl reconfigure
/opt/graylog/conf/graylog.conf > Elasticsearch replicas back to 1

why?

[jan (Jan Doberstein)]

you would like to re-read http://docs.graylog.org/en/stable/pages/configuration/graylog_ctl.html

You use the OVA - with onyl limited abilities to customize.

[jtkarvo]

Setting the number of replicas happens in the UI (System/Indices). It is in the index set settings.

After setting that, you can use curl to modify the existing indices.

[jtkarvo]

And another thing; check your elasticsearch logs. I suspect you ran out of ES disk space. See disk watermark settings of ES.