Graylog-Installation does not work anymore

1. Describe your incident:

I have a VM with RHEL8 where I prepare a graylog-appliance for a customer.
Everything worked fine 2 months ago, the project was put on pause then. Now as I return to it, graylog and ES seem not to work anymore.

At first I found an issue around the ES temp dir and mounting it with “noexec” flag.
I edited /etc/sysconfig/elasticsearch to poiunt it to “ES_TMPDIR=/var/log/elasticsearch”, that made ES start again.

GL does not work though. It starts and then after a while logs as mentioned below.

2. Describe your environment:

  • OS Information:
    NAME=“Red Hat Enterprise Linux”
    VERSION=“8.5 (Ootpa)”

  • Package Version:
    Name : graylog-server
    Version : 4.1.12
    Name : elasticsearch-oss
    Version : 7.10.2

  • Service logs, configurations, and environment variables:

Feb 14 09:22:12  graylog-server[95335]: Exception in thread "main" com.google.inject.CreationException: Unable to create injector, see the following errors:
Feb 14 09:22:12  graylog-server[95335]: 1) [Guice/ErrorInjectingConstructor]: NoClassDefFoundError: Could not initialize class Native
Feb 14 09:22:12  graylog-server[95335]:   at OshiFsProbe.<init>(OshiFsProbe.java:49)
Feb 14 09:22:12  graylog-server[95335]:   while locating OshiFsProbe
Feb 14 09:22:12  graylog-server[95335]:   at SystemStatsModule.configure(SystemStatsModule.java:55)
Feb 14 09:22:12  graylog-server[95335]:   while locating FsProbe
Feb 14 09:22:12  graylog-server[95335]: Learn more:
Feb 14 09:22:12  graylog-server[95335]:   https://github.com/google/guice/wiki/ERROR_INJECTING_CONSTRUCTOR
Feb 14 09:22:12  graylog-server[95335]: 1 error
Feb 14 09:22:12  graylog-server[95335]: ======================
Feb 14 09:22:12  graylog-server[95335]: Full classname legend:
Feb 14 09:22:12  graylog-server[95335]: ======================
Feb 14 09:22:12  graylog-server[95335]: FsProbe:              "org.graylog2.shared.system.stats.fs.FsProbe"
Feb 14 09:22:12  graylog-server[95335]: Native:               "com.sun.jna.Native"
Feb 14 09:22:12  graylog-server[95335]: OshiFsProbe:          "org.graylog2.shared.system.stats.fs.OshiFsProbe"
Feb 14 09:22:12  graylog-server[95335]: SystemStatsModule:    "org.graylog2.shared.system.stats.SystemStatsModule"
Feb 14 09:22:12  graylog-server[95335]: ========================
Feb 14 09:22:12  graylog-server[95335]: End of classname legend:
Feb 14 09:22:12  graylog-server[95335]: ========================
Feb 14 09:22:12  graylog-server[95335]:         at com.google.inject.internal.Errors.throwCreationExceptionIfErrorsExist(Errors.java:568)
Feb 14 09:22:12  graylog-server[95335]:         at com.google.inject.internal.InternalInjectorCreator.injectDynamically(InternalInjectorCreator.java:190)
Feb 14 09:22:12  graylog-server[95335]:         at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:113)
Feb 14 09:22:12  graylog-server[95335]:         at com.google.inject.Guice.createInjector(Guice.java:87)
Feb 14 09:22:12  graylog-server[95335]:         at org.graylog2.shared.bindings.GuiceInjectorHolder.createInjector(GuiceInjectorHolder.java:34)
Feb 14 09:22:12  graylog-server[95335]:         at org.graylog2.bootstrap.CmdLineTool.setupInjector(CmdLineTool.java:460)
Feb 14 09:22:12  graylog-server[95335]:         at org.graylog2.bootstrap.CmdLineTool.run(CmdLineTool.java:264)
Feb 14 09:22:12  graylog-server[95335]:         at org.graylog2.bootstrap.Main.main(Main.java:45)
Feb 14 09:22:12  graylog-server[95335]: Caused by: java.lang.NoClassDefFoundError: Could not initialize class com.sun.jna.Native
Feb 14 09:22:12  graylog-server[95335]:         at com.sun.jna.platform.linux.Udev.<clinit>(Udev.java:37)
Feb 14 09:22:12  graylog-server[95335]:         at oshi.hardware.platform.linux.LinuxHWDiskStore.getDisks(LinuxHWDiskStore.java:160)
Feb 14 09:22:12  graylog-server[95335]:         at oshi.hardware.platform.linux.LinuxHWDiskStore.getDisks(LinuxHWDiskStore.java:151)
Feb 14 09:22:12  graylog-server[95335]:         at oshi.hardware.platform.linux.LinuxHardwareAbstractionLayer.getDiskStores(LinuxHardwareAbstractionLayer.java:75)
Feb 14 09:22:12  graylog-server[95335]:         at org.graylog2.shared.system.stats.fs.OshiFsProbe.lambda$init$8(OshiFsProbe.java:78)
Feb 14 09:22:12  graylog-server[95335]:         at java.util.Optional.map(Optional.java:215)
Feb 14 09:22:12  graylog-server[95335]:         at org.graylog2.shared.system.stats.fs.OshiFsProbe.init(OshiFsProbe.java:76)
Feb 14 09:22:12  graylog-server[95335]:         at org.graylog2.shared.system.stats.fs.OshiFsProbe.<init>(OshiFsProbe.java:59)
Feb 14 09:22:12  graylog-server[95335]:         at org.graylog2.shared.system.stats.fs.OshiFsProbe$$FastClassByGuice$$100900467.GUICE$TRAMPOLINE(<generated>)
Feb 14 09:22:12  graylog-server[95335]:         at org.graylog2.shared.system.stats.fs.OshiFsProbe$$FastClassByGuice$$100900467.apply(<generated>)
Feb 14 09:22:12  graylog-server[95335]:         at com.google.inject.internal.DefaultConstructionProxyFactory$FastClassProxy.newInstance(DefaultConstructionProxyFactory.java:82)
Feb 14 09:22:12  graylog-server[95335]:         at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:114)
Feb 14 09:22:12  graylog-server[95335]:         at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:91)
Feb 14 09:22:12  graylog-server[95335]:         at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:296)
Feb 14 09:22:12  graylog-server[95335]:         at com.google.inject.internal.FactoryProxy.get(FactoryProxy.java:60)
Feb 14 09:22:12  graylog-server[95335]:         at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:41)
Feb 14 09:22:12  graylog-server[95335]:         at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:169)

3. What steps have you already taken to try and solve the problem?

  • edit /etc/sysconfig/elasticsearch to point it to “ES_TMPDIR=/var/log/elasticsearch”
  • edit /etc/fstab to remove “noexec” flag from /tmp mount
  • “yum update” to look for possible updates that fix things
  • check “df -h” for full filesystems, rotated logs etc : /var was full from logs,

4. How can the community help?

  • interpret what the system is telling me
  • point me at some solution

thanks!

Helpful Posting Tips: Tips for Posting Questions that Get Answers [Hold down CTRL and link on link to open tips documents in a separate tab]

Still working on fixing that.
ES runs now although there might be an issue around that mount-option “noexec” (I have to harden the system against DISA STIG).

graylog-server tells me:

2022-02-14T14:32:17.252+01:00 ERROR [LinuxFileSystem] Failed to get file counts from statvfs. /var/lib/graylog-server/.cache/JNA/temp/jna4212077980307545931.tmp: /var/lib/graylog-server/.cache/JNA/temp/jna4212077980307545931.tmp: cannot open shared object file: Operation not permitted
2022-02-14T14:32:17.254+01:00 ERROR [LinuxFileSystem] Failed to get file counts from statvfs. Could not initialize class com.sun.jna.Native

It runs with user/group graylog/graylog, and the dir looks like:

total 0
drwxr-xr-x. 2 graylog graylog 40 Feb 14 14:32 temp

# ls -l /var/lib/graylog-server/.cache/JNA/temp/
total 112
-rw-r--r--. 1 graylog graylog 112848 Feb 14 14:32 jna4212077980307545931.tmp

so it should be permitted to access that file, right?

If I remove it and restart the service it gets re-created and fails again.

Hello,
I might be able to help

/var/lib/graylog-server/.cache/JNA/temp/jna4212077980307545931.tmp: cannot open shared object file: Operation not permitted

what is you permission on ls -al /var/lib/graylog-server/ ?

Here is mine if this helps.

[root@graylog graylog_user]# ls -al /var/lib/graylog-server
total 16K
drwxr-xr-x.  6 graylog graylog   70 Aug 16 21:40 .
drwxr-xr-x. 58 root    root    4.0K Dec  7 21:11 ..
drwxr-xr-x.  3 graylog graylog   16 Aug 16 21:40 .cache
drwxr-xr-x.  3 graylog graylog 4.0K Feb 14 18:56 journal
drwxr-xr-x.  3 root    root    4.0K Apr  7  2021 journal_backup
drwxr-xr-x.  2 graylog graylog 4.0K Feb  7 16:09 libnative
[root@graylog graylog_user]#

AND

[root@graylog graylog_user]# ls -l /var/lib/graylog-server/.cache/JNA/
total 0
drwxr-xr-x. 3 graylog graylog 17 Aug 16 21:40 .
drwxr-xr-x. 3 graylog graylog 16 Aug 16 21:40 ..
drwxr-xr-x. 2 graylog graylog  6 Feb  7 16:09 temp
[root@graylog graylog_user]#

The permissions on my system are the same as on yours. The problem is
not solved yet.

The ES-issue is something around this: Elasticsearch no longer works under systemd (7.4.0 on CentOS 7.7.1908)? - Elasticsearch - Discuss the Elastic Stack

But still the startup issues with GL, even when I remove that tempfile, “chmod 777” it etc

It looks like this:

-rw-r--r--. 1 graylog graylog 112848 Feb 15 13:18 jna2528244358987533199.tmp

so I would assume that the graylog-server which runs with

User=graylog
Group=graylog

should be allowed to fully access it. As far as I see it creates it anyway.

I’d appreciate any help here, I should get this box up and running asap.

Maybe I have to upgrade to some other releases of ElasticSearch or Graylog here?

Hello @sgw

Do you have Selinux enabled?

I also found this information here
How was elasticsearch installed?

EDIT: took a look at the link you provided, by chance have you tried to set Elasticsearch service in
systemd to something like this? Or PWD to something similar?

Environment=ES_TMPDIR=/usr/share/elasticsearch/tmp

It looks like you have full upgrade ES and GL already. I wouldn’t go past ES 7.10

Graylog was installed using the ansible role “graylog2.graylog” (should be GitHub - Graylog2/graylog-ansible-role: Ansible role which installs and configures Graylog). I will look into that role to check for selinux-specific tasks or so.

selinux is enabled, yes. As mentioned I have to make the box compliant to the “DISA STIG for RHEL8”, maybe that hardens things too much for using with graylog. I can’t exactly tell if GL stopped working before my hardening playbooks or not.

Yes, I set that environment variable in the unit file as well. ES seems to run without issues.

thanks so far

1 Like

looked at the role, nothing selinux-specific IMO.
Turned selinux to permissive mode, same failure.

The system runs “auditd”, and I grep that tempfile in “/var/log/audit/audit.log”:

node=node1.loc type=PATH msg=audit(1645001959.323:3448375): item=0 name="/var/lib/graylog-server/.cache/JNA/temp/jna2624504139031619181.tmp" inode=4266041 dev=fd:03 mode=0100644 ouid=991 ogid=989 rdev=00:00 obj=system_u:object_r:var_lib_t:s0 nametype=NORMAL cap_fp=0 cap_fi=0 cap_fe=0 cap_fver=0 cap_frootid=0OUID="graylog" OGID="graylog"
node=node1.loc type=PATH msg=audit(1645001959.405:3448376): item=0 name="/var/lib/graylog-server/.cache/JNA/temp/jna2624504139031619181.tmp" inode=4266041 dev=fd:03 mode=0100644 ouid=991 ogid=989 rdev=00:00 obj=system_u:object_r:var_lib_t:s0 nametype=NORMAL cap_fp=0 cap_fi=0 cap_fe=0 cap_fver=0 cap_frootid=0OUID="graylog" OGID="graylog"
node=node1.loc type=PATH msg=audit(1645001959.407:3448377): item=0 name="/var/lib/graylog-server/.cache/JNA/temp/jna2624504139031619181.tmp" inode=4266041 dev=fd:03 mode=0100644 ouid=991 ogid=989 rdev=00:00 obj=system_u:object_r:var_lib_t:s0 nametype=NORMAL cap_fp=0 cap_fi=0 cap_fe=0 cap_fver=0 cap_frootid=0OUID="graylog" OGID="graylog"

I assume it’ s only logging the access but not blocking it?

Hello,

Ok now I understand why you have a tmp file that directory. So using Ansible it manually installed the package.

You could find more by executing these commands

sealert -a /var/log/audit/audit.log

root # ls -aZ <— Shows the labels to files

But since your in permissive mode I believe its not a problem, But don’t disable Selinux or you will lose all your permissions installed that is if you don’t want them.

Depend on what you can do, not sure if this is prod env or dev, but try reinstall elasticsearch NOTE: make sure you don’t go over 7.10 , to prevent this pin your elasticsearch package another suggestion don’t use “Y” flag when running updates.

root # yum reinstall elasticsearch-oss

Example:

thanks in general!
I rolled back to an older snapshot as all my hardening attempts did in fact screw up the system.

Now I harden against CIS Level 2 and graylog + ES are running, although there is a GL-update pending.

Currently I see the 2 processes java and graylog-server as unconfined daemons in terms of SElinux. I will research how to configure that.

The missing update: I will do it only after my hardening is done … and for sure after another snapshot :wink:

Maybe the issue comes up again then, we will see. thanks!

1 Like

Hello,

Oh boy… That will do it :laughing:
Selinux configuration is pretty easy, Set SELinux in permissive mode.

sudo yum -y install curl vim policycoreutils python3-policycoreutils
sudo setsebool -P httpd_can_network_connect 1
sudo semanage port -a -t http_port_t -p tcp 9000
sudo semanage port -a -t http_port_t -p tcp 9200
sudo semanage port -a -t mongod_port_t -p tcp 27017

Execute reboot and run this command

sealert -a /var/log/audit/audit.log

If there are any problems the output will show you how to take care of it and give you the commands to do so. When you have no errors Set SElinux back to Enforce mode and reboot.

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.