Graylog-datanode and java tmp dir not at /tmp

Before you post: Your responses to these questions will help the community help you. Please complete this template if you’re asking a support question.
Don’t forget to select tags to help index your topic!

1. Describe your incident:
I have a requirement to not run noexec on /tmp. Per hardening guidelines.
I have tried to configure datanode to run at /run/graylog-datanode, but a portion of the opensearch service stlil tries to hit /tmp/opensearch-/libzstd-jni-1.5.5-.so

in /etc/graylog/datanode/jvm.options
I configured

Custom temporary directory for OpenSearch (avoid /tmp)

-Djava.io.tmpdir=/run/graylog-datanode/tmp

Custom location for heap dumps and GC logs (avoid /tmp)

-XX:HeapDumpPath=/run/graylog-datanode/logs
-XX:ErrorFile=/run/graylog-datanode/logs/hs_err_pid%p.log
-Xlog:gc*,gc+age=trace,safepoint:file=/run/graylog-datanode/logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m

When I look at ps aux
It shows two calls for -Djava.io.tmpdir.
First one listed is mine in jvm.options, the second one is called from something else -Djava.io.tmpdir=/tmp/opensearch-10572032389771119066

2. Describe your environment:

  • OS Information:
    RHEL9

  • Package Version:
    6.1.8 (or latest)

  • Service logs, configurations, and environment variables:

graylog+ 22400 84.2 3.7 5816188 606396 ? Ssl 15:27 0:19 /usr/share/graylog-datanode/jvm/bin/java -Dlog4j.configurationFile=file:///etc/graylog/datanode/log4j2.xml -Xms2g -Xmx2g -XX:+UseG1GC -XX:-OmitStackTraceInFastThrow -XX:+UnlockExperimentalVMOptions -Djdk.tls.acknowledgeCloseNotify=true -Djava.io.tmpdir=/run/graylog-datanode/tmp -jar /usr/share/graylog-datanode/graylog-datanode.jar datanode -f /etc/graylog/datanode/datanode.conf -np
graylog+ 22829 232 54.8 12006260 8832404 ? Sl 15:27 0:27 /usr/share/graylog-datanode/dist/opensearch-2.15.0-linux-x64/jdk/bin/java -Xshare:auto -Dopensearch.networkaddress.cache.ttl=60 -Dopensearch.networkaddress.cache.negative.ttl=10 -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -XX:-OmitStackTraceInFastThrow -XX:+ShowCodeDetailsInExceptionMessages -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dio.netty.allocator.numDirectArenas=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Djava.security.manager=allow -Djava.locale.providers=SPI,COMPAT -Xms1g -Xmx1g -XX:+UseG1GC -XX:G1ReservePercent=25 -XX:InitiatingHeapOccupancyPercent=30 -Djava.io.tmpdir=/tmp/opensearch-10572032389771119066 -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=data -XX:ErrorFile=/tmp/hs_err_pid%p.log -Xlog:gc*,gc+age=trace,safepoint:file=/tmp/gc.log:utctime,pid,tags:filecount=32,filesize=64m -Djava.security.manager=allow -Xms8g -Xmx8g -Dopensearch.transport.cname_in_publish_address=true -Djavax.net.ssl.trustStore=/var/lib/graylog-datanode/opensearch/config/opensearch/datanode-truststore.p12 -Djavax.net.ssl.trustStorePassword=GSRqyznMmRqoOSrPCMvSvBGJyHPosRkbRCAdiHsJUeQPeWUQotoRFOUgNZyKLUoMXMTYzJKrLmgXJtGwqtCahLCEMjdOSVyMSyapXQHVtNXJaUvGGvNlBKhrFFUqUcafICyMBJjCxwrkMpNQKKIYofnJnmzWQCHkmwvyNvlbNaapZfxvqXOwMzdbBgapOlUnMNqstQirsZQCUDaILTenQEtyuxcONjpVJJgluzMoJumDGxzTTwXNVfGrXUcfpStj -Djavax.net.ssl.trustStoreType=pkcs12 -XX:MaxDirectMemorySize=4294967296 -Dopensearch.path.home=/usr/share/graylog-datanode/dist/opensearch-2.15.0-linux-x64 -Dopensearch.path.conf=/var/lib/graylog-datanode/opensearch/config/opensearch -Dopensearch.distribution.type=tar -Dopensearch.bundled_jdk=true -cp /usr/share/graylog-datanode/dist/opensearch-2.15.0-linux-x64/lib/* org.opensearch.bootstrap.OpenSearch

3. What steps have you already taken to try and solve the problem?
Editing systemd file, editing datanode/jvm.options

4. How can the community help?
getting datanode to fully run not in /tmp/

Helpful Posting Tips: Tips for Posting Questions that Get Answers [Hold down CTRL and link on link to open tips documents in a separate tab]

It definitely feels like the purpose of the datanode service to manage opensearch is not being fulfilled?

/var/lib/graylog-datanode/opensearch/config/opensearch/jvm.options has settings that are hard coded. specifically the xms/xmx settings. which appear to not being honored from whats set in the datanode/jvm.options. same with the java tmp_dir.

I think I have a work around but im not sure if its a best approach.
Edit the service file directly, and put in
Environment=“OPENSEARCH_TMPDIR=/run/graylog-datanode/tmp”
above the execstart. Kind of leads me to believe that datanode isnt passing all the potential variables seting it datanode/jvm.options, down to the opensearch wrapper service. I already see that the xms/xmx values are not getting set to the wrapper service. just the default 1g xms/xmx.

Anyone else notice this?

I’m not sure about the tmp directory stuff, but in the case of the jvm.options the design is that you use jvm.options to set the heap of the datanode service itself, but then the heap of opensearch is managed by datanode by way of settings in datanode.conf rather than setting them manually for opensearch.

I found that it doesn’t

Also found that xms/xmx aren’t being set when the open search executable spins up.

I can send you more details.

Hey Joel,

Good Morning,

I wanted to share some findings regarding an issue with OpenSearch’s temporary directory configuration under Graylog Data Node. Below are some key data points:

[root@portal graylog-datanode]# ps aux | grep graylog-datanode | grep tmpdir

graylog+ 27568 45.0 4.1 5817580 671772 ? Ssl 15:59 0:21 /usr/share/graylog-datanode/jvm/bin/java -Dlog4j.configurationFile=file:///etc/graylog/datanode/log4j2.xml -Xms2g -Xmx2g -XX:+UseG1GC -XX:-OmitStackTraceInFastThrow -XX:+UnlockExperimentalVMOptions -Djdk.tls.acknowledgeCloseNotify=true -Djava.io.tmpdir=/run/graylog-datanode/tmp -jar /usr/share/graylog-datanode/graylog-datanode.jar datanode -f /etc/graylog/datanode/datanode.conf -np

graylog+ 27997 107 55.1 12400232 8892728 ? Sl 15:59 0:38 /usr/share/graylog-datanode/dist/opensearch-2.15.0-linux-x64/jdk/bin/java -Xshare:auto -Dopensearch.networkaddress.cache.ttl=60 -Dopensearch.networkaddress.cache.negative.ttl=10 -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -XX:-OmitStackTraceInFastThrow -XX:+ShowCodeDetailsInExceptionMessages -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dio.netty.allocator.numDirectArenas=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Djava.security.manager=allow -Djava.locale.providers=SPI,COMPAT -Xms1g -Xmx1g -XX:+UseG1GC -XX:G1ReservePercent=25 -XX:InitiatingHeapOccupancyPercent=30 -Djava.io.tmpdir=/run/graylog-datanode/tmp -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=data -XX:ErrorFile=/tmp/hs_err_pid%p.log -Xlog:gc*,gc+age=trace,safepoint:file=/tmp/gc.log:utctime,pid,tags:filecount=32,filesize=64m -Djava.security.manager=allow -Xms8g -Xmx8g -Dopensearch.transport.cname_in_publish_address=true -Djavax.net.ssl.trustStore=/var/lib/graylog-datanode/opensearch/config/opensearch/datanode-truststore.p12 -Djavax.net.ssl.trustStorePassword=aLMOYOKMTvGxHBTCVDtUggynXDdkKyAKAaoQkavFpHeOdfGrJXIjXuPMcDuAoMZlbdJqEEInLjiBzJlLrqddMCxMWUJTpkNWXmiBYITDZLiTnnOHGUosxSSzYOLSDqPTCcVQyxpwiHBjRHQDHmbfXiWYjjwhIryqNQsPqFHRLEzQaLPoyUeyULWEyriJsCLyeJcKQAdqKmDxQYjNzsrGRudXtCQRioXnjTyRFnJtjwdpEFUzMhdfaAoPTmjQEwuk -Djavax.net.ssl.trustStoreType=pkcs12 -XX:MaxDirectMemorySize=4294967296 -Dopensearch.path.home=/usr/share/graylog-datanode/dist/opensearch-2.15.0-linux-x64 -Dopensearch.path.conf=/var/lib/graylog-datanode/opensearch/config/opensearch -Dopensearch.distribution.type=tar -Dopensearch.bundled_jdk=true -cp /usr/share/graylog-datanode/dist/opensearch-2.15.0-linux-x64/lib/* org.opensearch.bootstrap.OpenSearch

Observations & Issue:- I have explicitly set Xms/Xmx to 2g in datanode.conf, but OpenSearch (spawned by Graylog Data Node) is using 1g instead.

  • Setting -Djava.io.tmpdir=/run/graylog-datanode/tmp in jvm.options was not applied globally to all processes under the graylog-datanode service.
  • The OpenSearch process expects the OPENSEARCH_TMPDIR environment variable to be set.
    • If it’s not set, OpenSearch defaults to /tmp as the temporary directory.