Graylog Open 6.1.1 using uninstalled library?

Graylog Central (peer support)
1

This one is interesting, if anybody can help.

Yesterday I upgraded from 6.0 to 6.1. This problem might have been present then, but my monitors only run this particular check once a day. This morning I upgraded from 6.1 to 6.1.1.

6.1.1+9bd27f8, codename Noir

I monitor all services on my systems with needsrestart. Today after the updates I get this:

[main] #3495 uses deleted /var/lib/graylog-server/libnative/libzstd-jni-1.5.6-613249988640211524658.so
[main] #3495 is a child of #3490
[main] #3490 exe => /usr/bin/bash
[main] #3490 is graylog-server.service

I have restarted the service, and rebooted the node, and still get this error.

I have confirmed that file does not exist on the system.

Any ideas?

2

It appears that graylog itself removes this file. I created it and on graylog-server restart the file is removed.

Can reference to this library be removed in source?

3

Are you having an issue with graylog or does graylog not start or is the web interface inaccessible?

What linux distro/version are you using and are you installing graylog via a package manager (e.g. apt/yum)?

Also for context, this is what the contents of my libnative dir (defaults to /var/lib/graylog-server/libnative) looks like:

<user>@<hostname>:/var/lib/graylog-server/libnative$ ls -l
total 13748
-rw------- 1 graylog graylog 2684104 Aug 18 00:49 libnetty_tcnative_linux_x86_6417396452431933687271.so
-rw------- 1 graylog graylog 2684104 Aug 18 17:37 libnetty_tcnative_linux_x86_644210672352286664324.so
-rw------- 1 graylog graylog 2684104 Jul  3 13:16 libnetty_tcnative_linux_x86_64479320830875938353.so
-rw------- 1 graylog graylog 2688360 Oct 25 16:59 libnetty_tcnative_linux_x86_645483110867199748997.so
-rw------- 1 graylog graylog 2810664 May  5 19:16 libnetty_tcnative_linux_x86_647516904040934455876.so
-rw------- 1 graylog graylog   99422 Jul  3 13:16 libnetty_transport_native_epoll_x86_641388341869190226814.so
-rw------- 1 graylog graylog   99422 Aug 18 00:49 libnetty_transport_native_epoll_x86_6414645005496518944720.so
-rw------- 1 graylog graylog   99422 Aug 18 17:37 libnetty_transport_native_epoll_x86_643638996723415474830.so
-rw------- 1 graylog graylog   99563 Oct 25 16:59 libnetty_transport_native_epoll_x86_647379059230842972927.so
-rw------- 1 graylog graylog   99422 May  5 19:16 libnetty_transport_native_epoll_x86_64956328721841476754.so

I don’t see any libzstd-jni* files and its possible that its removed because its not part of graylog?

4

Everything runs fine, but my monitors seem to think that graylog is trying to use a file that gets removed when the process starts. This is on Alma 9 using packages.

My directory looks like this:

root@graylog:~ 
ll /var/lib/graylog-server/libnative/
total 100
-rw------- 1 graylog graylog 99563 Oct 25 09:24 libnetty_transport_native_epoll_x86_6412433652931218598535.so

needsrestart output:

root@graylog:~ 
# needrestart -ma -v -rl
[main] eval /etc/needrestart/needrestart.conf
[main] eval /etc/needrestart/conf.d/schweb.conf
[main] needrestart v3.6
[main] running in root mode
[Core] Using UI 'NeedRestart::UI::stdio'...
[main] systemd detected
[Core] #633 is a NeedRestart::Interp::Java
[Core] #807 is a NeedRestart::Interp::Python
[Python] #807: source=/usr/sbin/tuned
[main] #80521 uses deleted /var/lib/graylog-server/libnative/libzstd-jni-1.5.6-616812283262751137345.so
[main] #80521 is a child of #80520
[main] #80520 exe => /usr/bin/bash
[main] #80520 is graylog-server.service
[Kernel] Linux: kernel release 5.14.0-427.40.1.el9_4.x86_64, kernel version #1 SMP PREEMPT_DYNAMIC Wed Oct 16 07:08:17 EDT 2024
[Kernel/Linux] /boot/vmlinuz-5.14.0-427.40.1.el9_4.x86_64 => 5.14.0-427.40.1.el9_4.x86_64 (mockbuild@x64-builder01.almalinux.org) #1 SMP PREEMPT_DYNAMIC Wed Oct 16 07:08:17 EDT 2024 [5.14.0-427.40.1.el9_4.x86_64]*
[Kernel/Linux] /boot/vmlinuz-5.14.0-427.37.1.el9_4.x86_64 => 5.14.0-427.37.1.el9_4.x86_64 (mockbuild@x64-builder02.almalinux.org) #1 SMP PREEMPT_DYNAMIC Tue Sep 24 17:44:03 EDT 2024 [5.14.0-427.37.1.el9_4.x86_64]
[Kernel/Linux] /boot/vmlinuz-5.14.0-427.35.1.el9_4.x86_64 => 5.14.0-427.35.1.el9_4.x86_64 (mockbuild@x64-builder01.almalinux.org) #1 SMP PREEMPT_DYNAMIC Thu Sep 12 11:21:43 EDT 2024 [5.14.0-427.35.1.el9_4.x86_64]
[Kernel/Linux] using RPM version sorting
[Kernel/Linux] Expected linux version: 5.14.0-427.40.1.el9_4.x86_64

Running kernel seems to be up-to-date.

Services to be restarted:
 systemctl restart graylog-server.service

No containers need to be restarted.

No user sessions are running outdated binaries.

No VM guests are running outdated hypervisor (qemu) binaries on this host.



root@graylog:~ 
# rpm -qa | grep graylog
graylog-6.1-repository-1-1.noarch
graylog-server-6.1.1-1.x86_64
5

Hi,

to clarify: needrestart looks for processes using deleted (aka updated) libraries. If for instance libssl gets updated by a new package version, it is crucial that every service which uses libssl (apache, ssh, nginx…) gets restarted to make sure it uses the updated version of the library.

In this case here the detection seems to be a false positive as the deleted library does not belong to any package but gets created and deleted by graylog itself.

There are two ways to remove this false positive:

  • configure needrestart to ignore the deleted library
  • patch graylog not to produce this detectable behaviour

The first solution can be implemented by every user (who understands needrestarts config), the second one only by graylog’s developers (who understand what the hell graylog is doing there).

As I see exactly the same issue here on Debian, I will try the first solution.

6

Here is the first solution:

Create a file /etc/needrestart/conf.d/graylog.conf containing

push @{$nrconf{blacklist_mappings}},
	qr(^/tmp/),
	qr(^/var/lib/graylog-server/.cache/JNA/temp/),
	qr(^/var/lib/graylog-server/libnative/),
	qr(/temp/jna-);

This will instruct needrestart to ignore deleted libraries in the mentioned paths.

Of course I would prefer graylog not deleting the file in the first place.

7

Thanks for digging into this!

I did end up configuring needrestart to ignore this. Great explanation about what is going on.

Yeah, hopefully the devs fix whatever this is.

8

Appreciate the further explanation. I mis-read the earlier posts.

Do you have any steps that can be taken to reproduce this behavior? Does it only occur when upgrading graylog between versions (e.g. 6.0->6.1) ?

9

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.

10

FYI tracking further discussion on this via Debian needrestart always want graylog-server restart in 6.1 · Issue #20949 · Graylog2/graylog2-server · GitHub