How do i change the gc.log and JVM temp files location from my datanode

[tadeu.alves]

Hi there,

Today i received some alarms that my data-node servers have 100% usage of the tmp directory.
Looking inside it i found that the datanode process had created some files (gc logs and tmp opensearch-xxx directories).

But i have a separated disk this stuff, i even changed the /etc/graylog/datanode/jvm.options but it did not work.

Looking around i found that i can change some parameters it in the /etc/graylog/datanode/datanode.conf (Data Node Configuration)

But nothing says about changing the JVM tmp directory and GC files.
I’m using Graylog and Datanode in version 6.1.7 with RHEL9.
Any help to change is appreciated because i cannot run this stuff in the /tmp/.

[Tdvorak (Tomas Dvorak)]

Hi @tadeu.alves
Those opensearch-xxx directories hold configuration of the currently running opensearch process. They should be automatically deleted in the moment the process terminates. Generally, there are only several text/configuration files with negligible size. If you see suspiciously many opensearch-xxx directories, then there is some problem with the cleanup and I’d be interested to debug it further.

You can relocate these opensearch-xxx to a different partition by configuring opensearch_config_location in your datanode.conf.

You can also relocate all the logs by configuring opensearch_logs_location path.

You can also define OPENSEARCH_TMPDIR env property, telling the datanode-managed opensearch where to put temp files.

Finally, you can adapt the jvm.options file. Look for the JDK 9+ GC logging section and adapt the file path and/or verbosity.

[tadeu.alves]

Hi @Tdvorak thanks for the help!

About what you explained.

1 - opensearch_logs_location - This configuration is fine i have a dedicated partition to the /var/log so no problem here.

2 - Where do i change the OPENSEARCH_TMPDIR because in the Documentation of the conf file doesn’t show where i can change it.

3 - I changed the JDK 9+ GC logging in the /etc/graylog/datanode/jvm.options but the configuration was ignored.. even tried different file locations like /etc/graylog/datanode/opensearch-conf/opensearch/jvm.options or /usr/share/graylog-datanode/dist/opensearch-2.15.0-linux-x64/config/jvm.options but in all of them the configuration was ignored.

4 - This is my ps -ef | grep java in the datanode machine:

/usr/share/graylog-datanode/dist/opensearch-2.15.0-linux-x64/jdk/bin/java -Xshare:auto -Dopensearch.networkaddress.cache.ttl=60 -Dopensearch.networkaddress.cache.negative.ttl=10 -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -XX:-OmitStackTraceInFastThrow -XX:+ShowCodeDetailsInExceptionMessages -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dio.netty.allocator.numDirectArenas=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Djava.security.manager=allow -Djava.locale.providers=SPI,COMPAT -Xms1g -Xmx1g -XX:+UseG1GC -XX:G1ReservePercent=25 -XX:InitiatingHeapOccupancyPercent=30 -Djava.io.tmpdir=/tmp/opensearch-6944561981453929329 -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=data -XX:ErrorFile=/tmp/hs_err_pid%p.log -Xlog:gc*,gc+age=trace,safepoint:file=/tmp/gc.log:utctime,pid,tags:filecount=32,filesize=64m -Djava.security.manager=allow -Xms8g -Xmx8g -Dopensearch.transport.cname_in_publish_address=true -Djavax.net.ssl.trustStore=/etc/graylog/datanode/opensearch-conf/opensearch/datanode-truststore.p12 -Djavax.net.ssl.trustStorePassword=STVXpJfkpBtEyFmCZnZLkZGHRAxQqDQRJoqpVSjPYMnbNtrSagEjvSdJINunxezqJmlSItVwjPOrFvfBXhEuFAIjZsutRLLRHFfxauPvFJzPiTmtgJXliSrZaVHNDREffyPcniRxGJtodzbWvfpPliYExztewEuoAnTfRnQHqJtyUxgVhEEbzQhcMJHSjYxCcReJCvkNPuihMzBkdkthjZcepmWGyiQfqmfyhlBXBkZZLJXFlMLsamEoUCCbZljP -Djavax.net.ssl.trustStoreType=pkcs12 -XX:MaxDirectMemorySize=4294967296 -Dopensearch.path.home=/usr/share/graylog-datanode/dist/opensearch-2.15.0-linux-x64 -Dopensearch.path.conf=/etc/graylog/datanode/opensearch-conf/opensearch -Dopensearch.distribution.type=tar -Dopensearch.bundled_jdk=true -cp /usr/share/graylog-datanode/dist/opensearch-2.15.0-linux-x64/lib/* org.opensearch.bootstrap.OpenSearch

If there’s any information needed just tell me what to get

Anyways thanks again @Tdvorak for the help.

[Tdvorak (Tomas Dvorak)]

Thanks for your tests! You are right, there is no simple and straightforward way how to adapt the jvm.options. The version that opensearch is starting with is distributed in datanode jar and can’t be easily modified.

I have created a feature request for that: Allow modifying datanode jvm.options · Issue #22652 · Graylog2/graylog2-server · GitHub and linked your topic in it.

You can, however, configure at least the opensearch temp dir. Depending how you start your datanode, you should be able to configure env properties there. This is not datanode configuration option, just generic env property OPENSEARCH_TMPDIR.

Running datanode manually from a downloaded tarball, I verified that this command is working:

OPENSEARCH_TMPDIR=/my/tmp/dir bin/graylog-datanode datanode -f datanode.conf

In your system, you probably have some descriptor file or configuration for running system services. There you can configure env for your datanode. Datanode will then forward OPENSEARCH_TMPDIR to the opensearch. This will solve the /tmp/opensearch-xxx directories you see.

For the gc.log I have no easy option but I’ll try to ask colleagues if they have some tips.

[tadeu.alves]

I’ll tried changing the systemd service file to force the gc.log location.. but that did not work…

The only way to make it work is to run the opensearch as a standalone and not as a data-node

[Tdvorak (Tomas Dvorak)]

yeah, the gc.log location is hardcoded and there is currently no way to configure it. Please follow/subscribe the github issue if you want to be informed as soon as there is any progress in this.