Journal & JVM HeapSize Tweeks

[Paulpretorius (Paul)]

Hi All,

Is there any way to Increase the Journal Size? its 1G at the moment. I loaded some hosts onto the system (16 via a script) they all started sending their messages to graylog and the Journal filled. I then had a message saying that messages had been discarded and not logged due to over utilization.

Also, How do i edit ES_HEAP_SIZE environment variable as mentioned : http://docs.graylog.org/en/2.2/pages/configuration/elasticsearch.html?highlight=ES_HEAP_SIZE

Thanks
Paul.

[jochen]

Hi Paul,

@Paulpretorius

Is there any way to Increase the Journal Size?

Yes, of course.

Graylog2/graylog2-server/blob/2.2.1/misc/graylog.conf#L405-L409

@Paulpretorius

Also, How do i edit ES_HEAP_SIZE environment variable

This depends on how you’ve installed Elasticsearch. See http://docs.graylog.org/en/2.2/pages/configuration/file_location.html for some hints.

[Paulpretorius (Paul)]

@jochen

message_journal_max_size

Hi jochen

Thanks for getting back to me.

I managed to change the journal size, but found that after I run a graylog-ctl reconfigure, the changes are wiped out so in this case, restart keeps the settings :
ubuntu@graylog:/$ sudo vi /opt/graylog/conf/graylog.conf
ubuntu@graylog:/$ sudo graylog-ctl restart
ok: run: elasticsearch: (pid 31331) 0s
ok: run: etcd: (pid 31358) 1s
ok: run: graylog-server: (pid 31408) 0s
ok: run: mongodb: (pid 31431) 1s
ok: run: nginx: (pid 31449) 0s
ubuntu@graylog:/$ sudo cat /opt/graylog/conf/graylog.conf | grep “message_journal_max_size”
message_journal_max_size = 4gb

The same seems to be true for “indices.store.throttle.max_bytes_per_sec” that you can put into /opt/graylog/elasticsearch/config/elasticsearch.yml. After a reconfigure, it wipes the changes.
indices.store.throttle.max_bytes_per_sec: 150mb

The heapsize, I have set … but still only using 1.4G even though I have set it to 4.
Reconfigure and restart don’t seem to wake it up … Also tried sticking it into /etc/environment but that didn’t do it either.

ubuntu@graylog:/$ sudo cat /etc/default/elasticsearch | grep “ES_HEAP_SIZE”
.# Set ES_HEAP_SIZE to 50% of available RAM, but no more than 31g
ES_HEAP_SIZE=4g
.# in elasticsearch.yml (ES_HEAP_SIZE must also be set).

ubuntu@graylog:/$ sudo cat /etc/elasticsearch/elasticsearch.yml | grep “ES_HEAP_SIZE”
.# Make sure that the ES_HEAP_SIZE environment variable is set to about half the memory
ES_HEAP_SIZE=4g

ubuntu@graylog:/$ sudo cat /opt/graylog/elasticsearch/config/elasticsearch.yml | grep “ES_HEAP_SIZE”
ES_HEAP_SIZE=4g

ubuntu@graylog:/$ cat /etc/environment | grep “ES_”
ES_HEAP_SIZE=4G

Almost there!!

[jochen]

If you’re using the Graylog virtual machine appliance, you have to use the graylog-ctl command to configure it. Otherwise your changes will be overwritten: http://docs.graylog.org/en/2.2/pages/configuration/graylog_ctl.html

[Paulpretorius (Paul)]

Thanks jochen.

I’m not seeing anything for Journal Size and heap size in those commands ? Am I missing something ?

[jan (Jan Doberstein)]

hej @Paulpretorius

you need to scroll down until you see the advanced setting

But just to have once written:

• the journal is just a buffer, if this get filled up and the data is not written to elasticsearch, you need more elasticsearch power to be able to ingest at the given message rate
• you should not use the omnibus package/the appliance in production
• just because you are not able to change all settings that you might want to tune
• read the docs, what you can set http://docs.graylog.org/en/2.2/pages/configuration/graylog_ctl.html#
• journal size is not editable

[Paulpretorius (Paul)]

Thanks Jan!

I have edited the .json and reconfigured / restart … But still only using 1.4G

“enforce_ssl”: false,
“journal_size”: 8,
“node_id”: false,
“internal_logging”: true,
“web_listen_uri”: false,
“web_endpoint_uri”: false,
“rest_listen_uri”: false,
“rest_transport_uri”: false,
“external_rest_uri”: false,
“custom_attributes”: {
“elasticsearch”: {
“memory”: “4400m”
}
}
}

The journal grew past 5G and over 20000000 unprocessed messages. I was hoping to increase the RAM on the JVM to help with processing lol

[jan (Jan Doberstein)]

@Paulpretorius

as already written - you are bound to the defaults https://github.com/Graylog2/graylog2-server/blob/2.2.1/misc/graylog.conf#L405-L409

The Omnibus Package does not give you the option to modify that.

if you have that amount of messages in your journal the elasticsearch you like to ingest is way underpowered for what you put in.

[Paulpretorius (Paul)]

Haha I realise its touch and go yes

We have about 150 Windows Clients and 20 Linux Client sending into it … Somes clients are quite old, so I’m hovering historical events.

[jan (Jan Doberstein)]

if you had left the defaults of the virtual appliance and you have some noisy clients the appliance is way underpowered for that.

You can create a cluster with some more appliances but I would recommend to build your own environment for this.

[harryw]

To change the Java heap size for the graylog-server do the following ;

sudo vi /opt/graylog/service/graylog-server/run

and change the Xms and Xmx values.
I use -Xms=2000m and -Xmx2500m on three graylog-server appliances with two elasticsearch appliances.

I also increased the journal size to ‘4g’ in the /opt/graylog/conf/graylog.conf file

I can process around 30000 messages per minute without causing a journal file backlog.

Ensure that you configure your graylog-server and elasticsearch servers with the correct CPU/memory configurations as per the documentation.

[jan (Jan Doberstein)]

When you are running OVA please do not edit configuration “by hand” on the next Update or run of graylog-ctl this will be overwritten.

We have covered raising heap in our documentation.

[Paulpretorius (Paul)]

I shall give this a go! Thanks Harry