I have a VM with RHEL8 where I prepare a graylog-appliance for a customer.
Everything worked fine 2 months ago, the project was put on pause then. Now as I return to it, graylog and ES seem not to work anymore.
At first I found an issue around the ES temp dir and mounting it with “noexec” flag.
I edited /etc/sysconfig/elasticsearch to poiunt it to “ES_TMPDIR=/var/log/elasticsearch”, that made ES start again.
GL does not work though. It starts and then after a while logs as mentioned below.
2. Describe your environment:
OS Information:
NAME=“Red Hat Enterprise Linux”
VERSION=“8.5 (Ootpa)”
Package Version:
Name : graylog-server
Version : 4.1.12
Name : elasticsearch-oss
Version : 7.10.2
Service logs, configurations, and environment variables:
Feb 14 09:22:12 graylog-server[95335]: Exception in thread "main" com.google.inject.CreationException: Unable to create injector, see the following errors:
Feb 14 09:22:12 graylog-server[95335]: 1) [Guice/ErrorInjectingConstructor]: NoClassDefFoundError: Could not initialize class Native
Feb 14 09:22:12 graylog-server[95335]: at OshiFsProbe.<init>(OshiFsProbe.java:49)
Feb 14 09:22:12 graylog-server[95335]: while locating OshiFsProbe
Feb 14 09:22:12 graylog-server[95335]: at SystemStatsModule.configure(SystemStatsModule.java:55)
Feb 14 09:22:12 graylog-server[95335]: while locating FsProbe
Feb 14 09:22:12 graylog-server[95335]: Learn more:
Feb 14 09:22:12 graylog-server[95335]: https://github.com/google/guice/wiki/ERROR_INJECTING_CONSTRUCTOR
Feb 14 09:22:12 graylog-server[95335]: 1 error
Feb 14 09:22:12 graylog-server[95335]: ======================
Feb 14 09:22:12 graylog-server[95335]: Full classname legend:
Feb 14 09:22:12 graylog-server[95335]: ======================
Feb 14 09:22:12 graylog-server[95335]: FsProbe: "org.graylog2.shared.system.stats.fs.FsProbe"
Feb 14 09:22:12 graylog-server[95335]: Native: "com.sun.jna.Native"
Feb 14 09:22:12 graylog-server[95335]: OshiFsProbe: "org.graylog2.shared.system.stats.fs.OshiFsProbe"
Feb 14 09:22:12 graylog-server[95335]: SystemStatsModule: "org.graylog2.shared.system.stats.SystemStatsModule"
Feb 14 09:22:12 graylog-server[95335]: ========================
Feb 14 09:22:12 graylog-server[95335]: End of classname legend:
Feb 14 09:22:12 graylog-server[95335]: ========================
Feb 14 09:22:12 graylog-server[95335]: at com.google.inject.internal.Errors.throwCreationExceptionIfErrorsExist(Errors.java:568)
Feb 14 09:22:12 graylog-server[95335]: at com.google.inject.internal.InternalInjectorCreator.injectDynamically(InternalInjectorCreator.java:190)
Feb 14 09:22:12 graylog-server[95335]: at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:113)
Feb 14 09:22:12 graylog-server[95335]: at com.google.inject.Guice.createInjector(Guice.java:87)
Feb 14 09:22:12 graylog-server[95335]: at org.graylog2.shared.bindings.GuiceInjectorHolder.createInjector(GuiceInjectorHolder.java:34)
Feb 14 09:22:12 graylog-server[95335]: at org.graylog2.bootstrap.CmdLineTool.setupInjector(CmdLineTool.java:460)
Feb 14 09:22:12 graylog-server[95335]: at org.graylog2.bootstrap.CmdLineTool.run(CmdLineTool.java:264)
Feb 14 09:22:12 graylog-server[95335]: at org.graylog2.bootstrap.Main.main(Main.java:45)
Feb 14 09:22:12 graylog-server[95335]: Caused by: java.lang.NoClassDefFoundError: Could not initialize class com.sun.jna.Native
Feb 14 09:22:12 graylog-server[95335]: at com.sun.jna.platform.linux.Udev.<clinit>(Udev.java:37)
Feb 14 09:22:12 graylog-server[95335]: at oshi.hardware.platform.linux.LinuxHWDiskStore.getDisks(LinuxHWDiskStore.java:160)
Feb 14 09:22:12 graylog-server[95335]: at oshi.hardware.platform.linux.LinuxHWDiskStore.getDisks(LinuxHWDiskStore.java:151)
Feb 14 09:22:12 graylog-server[95335]: at oshi.hardware.platform.linux.LinuxHardwareAbstractionLayer.getDiskStores(LinuxHardwareAbstractionLayer.java:75)
Feb 14 09:22:12 graylog-server[95335]: at org.graylog2.shared.system.stats.fs.OshiFsProbe.lambda$init$8(OshiFsProbe.java:78)
Feb 14 09:22:12 graylog-server[95335]: at java.util.Optional.map(Optional.java:215)
Feb 14 09:22:12 graylog-server[95335]: at org.graylog2.shared.system.stats.fs.OshiFsProbe.init(OshiFsProbe.java:76)
Feb 14 09:22:12 graylog-server[95335]: at org.graylog2.shared.system.stats.fs.OshiFsProbe.<init>(OshiFsProbe.java:59)
Feb 14 09:22:12 graylog-server[95335]: at org.graylog2.shared.system.stats.fs.OshiFsProbe$$FastClassByGuice$$100900467.GUICE$TRAMPOLINE(<generated>)
Feb 14 09:22:12 graylog-server[95335]: at org.graylog2.shared.system.stats.fs.OshiFsProbe$$FastClassByGuice$$100900467.apply(<generated>)
Feb 14 09:22:12 graylog-server[95335]: at com.google.inject.internal.DefaultConstructionProxyFactory$FastClassProxy.newInstance(DefaultConstructionProxyFactory.java:82)
Feb 14 09:22:12 graylog-server[95335]: at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:114)
Feb 14 09:22:12 graylog-server[95335]: at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:91)
Feb 14 09:22:12 graylog-server[95335]: at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:296)
Feb 14 09:22:12 graylog-server[95335]: at com.google.inject.internal.FactoryProxy.get(FactoryProxy.java:60)
Feb 14 09:22:12 graylog-server[95335]: at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:41)
Feb 14 09:22:12 graylog-server[95335]: at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:169)
3. What steps have you already taken to try and solve the problem?
edit /etc/sysconfig/elasticsearch to point it to “ES_TMPDIR=/var/log/elasticsearch”
edit /etc/fstab to remove “noexec” flag from /tmp mount
“yum update” to look for possible updates that fix things
check “df -h” for full filesystems, rotated logs etc : /var was full from logs,
Still working on fixing that.
ES runs now although there might be an issue around that mount-option “noexec” (I have to harden the system against DISA STIG).
graylog-server tells me:
2022-02-14T14:32:17.252+01:00 ERROR [LinuxFileSystem] Failed to get file counts from statvfs. /var/lib/graylog-server/.cache/JNA/temp/jna4212077980307545931.tmp: /var/lib/graylog-server/.cache/JNA/temp/jna4212077980307545931.tmp: cannot open shared object file: Operation not permitted
2022-02-14T14:32:17.254+01:00 ERROR [LinuxFileSystem] Failed to get file counts from statvfs. Could not initialize class com.sun.jna.Native
It runs with user/group graylog/graylog, and the dir looks like:
total 0
drwxr-xr-x. 2 graylog graylog 40 Feb 14 14:32 temp
# ls -l /var/lib/graylog-server/.cache/JNA/temp/
total 112
-rw-r--r--. 1 graylog graylog 112848 Feb 14 14:32 jna4212077980307545931.tmp
so it should be permitted to access that file, right?
If I remove it and restart the service it gets re-created and fails again.
I also found this information here
How was elasticsearch installed?
EDIT: took a look at the link you provided, by chance have you tried to set Elasticsearch service in
systemd to something like this? Or PWD to something similar?
selinux is enabled, yes. As mentioned I have to make the box compliant to the “DISA STIG for RHEL8”, maybe that hardens things too much for using with graylog. I can’t exactly tell if GL stopped working before my hardening playbooks or not.
Yes, I set that environment variable in the unit file as well. ES seems to run without issues.
Ok now I understand why you have a tmp file that directory. So using Ansible it manually installed the package.
You could find more by executing these commands
sealert -a /var/log/audit/audit.log
root # ls -aZ <— Shows the labels to files
But since your in permissive mode I believe its not a problem, But don’t disable Selinux or you will lose all your permissions installed that is if you don’t want them.
Depend on what you can do, not sure if this is prod env or dev, but try reinstall elasticsearch NOTE: make sure you don’t go over 7.10 , to prevent this pin your elasticsearch package another suggestion don’t use “Y” flag when running updates.
Oh boy… That will do it
Selinux configuration is pretty easy, Set SELinux in permissive mode.
sudo yum -y install curl vim policycoreutils python3-policycoreutils
sudo setsebool -P httpd_can_network_connect 1
sudo semanage port -a -t http_port_t -p tcp 9000
sudo semanage port -a -t http_port_t -p tcp 9200
sudo semanage port -a -t mongod_port_t -p tcp 27017
Execute reboot and run this command
sealert -a /var/log/audit/audit.log
If there are any problems the output will show you how to take care of it and give you the commands to do so. When you have no errors Set SElinux back to Enforce mode and reboot.