How do i change the gc.log and JVM temp files location from my datanode

Graylog Central (peer support)
1

Hi there,

Today i received some alarms that my data-node servers have 100% usage of the tmp directory.
Looking inside it i found that the datanode process had created some files (gc logs and tmp opensearch-xxx directories).

But i have a separated disk this stuff, i even changed the /etc/graylog/datanode/jvm.options but it did not work.

Looking around i found that i can change some parameters it in the /etc/graylog/datanode/datanode.conf (Data Node Configuration)

But nothing says about changing the JVM tmp directory and GC files.
I’m using Graylog and Datanode in version 6.1.7 with RHEL9.
Any help to change is appreciated because i cannot run this stuff in the /tmp/.

2

Hi @tadeu.alves
Those opensearch-xxx directories hold configuration of the currently running opensearch process. They should be automatically deleted in the moment the process terminates. Generally, there are only several text/configuration files with negligible size. If you see suspiciously many opensearch-xxx directories, then there is some problem with the cleanup and I’d be interested to debug it further.

You can relocate these opensearch-xxx to a different partition by configuring opensearch_config_location in your datanode.conf.

You can also relocate all the logs by configuring opensearch_logs_location path.

You can also define OPENSEARCH_TMPDIR env property, telling the datanode-managed opensearch where to put temp files.

Finally, you can adapt the jvm.options file. Look for the JDK 9+ GC logging section and adapt the file path and/or verbosity.

3

Hi @Tdvorak thanks for the help!

About what you explained.

1 - opensearch_logs_location - This configuration is fine i have a dedicated partition to the /var/log so no problem here.

2 - Where do i change the OPENSEARCH_TMPDIR because in the Documentation of the conf file doesn’t show where i can change it.

3 - I changed the JDK 9+ GC logging in the /etc/graylog/datanode/jvm.options but the configuration was ignored.. even tried different file locations like /etc/graylog/datanode/opensearch-conf/opensearch/jvm.options or /usr/share/graylog-datanode/dist/opensearch-2.15.0-linux-x64/config/jvm.options but in all of them the configuration was ignored.

4 - This is my ps -ef | grep java in the datanode machine:

/usr/share/graylog-datanode/dist/opensearch-2.15.0-linux-x64/jdk/bin/java -Xshare:auto -Dopensearch.networkaddress.cache.ttl=60 -Dopensearch.networkaddress.cache.negative.ttl=10 -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -XX:-OmitStackTraceInFastThrow -XX:+ShowCodeDetailsInExceptionMessages -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dio.netty.allocator.numDirectArenas=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Djava.security.manager=allow -Djava.locale.providers=SPI,COMPAT -Xms1g -Xmx1g -XX:+UseG1GC -XX:G1ReservePercent=25 -XX:InitiatingHeapOccupancyPercent=30 -Djava.io.tmpdir=/tmp/opensearch-6944561981453929329 -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=data -XX:ErrorFile=/tmp/hs_err_pid%p.log -Xlog:gc*,gc+age=trace,safepoint:file=/tmp/gc.log:utctime,pid,tags:filecount=32,filesize=64m -Djava.security.manager=allow -Xms8g -Xmx8g -Dopensearch.transport.cname_in_publish_address=true -Djavax.net.ssl.trustStore=/etc/graylog/datanode/opensearch-conf/opensearch/datanode-truststore.p12 -Djavax.net.ssl.trustStorePassword=STVXpJfkpBtEyFmCZnZLkZGHRAxQqDQRJoqpVSjPYMnbNtrSagEjvSdJINunxezqJmlSItVwjPOrFvfBXhEuFAIjZsutRLLRHFfxauPvFJzPiTmtgJXliSrZaVHNDREffyPcniRxGJtodzbWvfpPliYExztewEuoAnTfRnQHqJtyUxgVhEEbzQhcMJHSjYxCcReJCvkNPuihMzBkdkthjZcepmWGyiQfqmfyhlBXBkZZLJXFlMLsamEoUCCbZljP -Djavax.net.ssl.trustStoreType=pkcs12 -XX:MaxDirectMemorySize=4294967296 -Dopensearch.path.home=/usr/share/graylog-datanode/dist/opensearch-2.15.0-linux-x64 -Dopensearch.path.conf=/etc/graylog/datanode/opensearch-conf/opensearch -Dopensearch.distribution.type=tar -Dopensearch.bundled_jdk=true -cp /usr/share/graylog-datanode/dist/opensearch-2.15.0-linux-x64/lib/* org.opensearch.bootstrap.OpenSearch

If there’s any information needed just tell me what to get :smiley:

Anyways thanks again @Tdvorak for the help.