I am running a vulnerability scan using Tenable Nessus, which has identified several Log4j files related to the datanode:
Path : /usr/share/graylog-datanode/dist/opensearch-2.19.5-linux-x64/lib/log4j-core-2.25.3.jarInstalled version : 2.25.3
Fixed version : 2.25.4
I have already manually installed a newer version of Graylog, but it still shows these vulnerabilities.
Is there a solution to this issue?
Hello,
The solution is we need to wait until the maintainer pull the latest release of opensearch which contain the bug fix and backport.
I patched to latest version of graylog 7.1.3 and verify only log4j of opensearch which is not update. for datanode’s logj4 is up to date.
Regards,
Hey @baadaasz ,
For compatibility reasons and possible migration issues that could hit users when we switch directly to 3.x, Graylog Datanode is still distributing and running 2.19.5, the latest 2.x opensearch release.
Opensearch seems to have an open PR with backport of the log4j update for the 2.x branch: [2.19] update log4j2 to 2.25.4 by rursprung · Pull Request #21445 · opensearch-project/OpenSearch · GitHub, but there is no new release yet.
The datanode and graylog server itself are already depending on 2.25.4 in the 7.1 releases.
Just thought I’d pile on here 
– the locked version dependency for opensearch is a real headache for keeping vulnerabilities patched.
@Tdvorak said
The datanode and graylog server itself are already depending on 2.25.4 in the 7.1 releases.
The docs show 2.19.5 – is this incorrect? Graylog Docs > Install Graylog - Compatibility Matrix [my mistake - 2.25.4 is log4j version]
The problem gets worse if depending on backports, as with opensearch 2.* – it looks like there is still active maintenance, but the 3.* release GA was over a year ago, so this will inevitably slow down.
I am unclear what’s the roadmap for 3.* support – did not find anything on github, but I’m not that familiar with how the project is managed so I probably missed it.
For some background – like the original poster, I’m working in an environment where we do vulnerability scanning. That includes pressure from management to patch quickly, regardless of the actual risk involved (often little to none, if you look at the CVEs…).
2.19.5 is the current distributed opensearch version in datanode, 2.25.4 is the log4j version which is used by the datanode and graylog server itself.
There are several reasons why it takes time to release datanode with new opensearch versions. Opensearch releases often contain bugs or breaking changes that we are discovering during our internal testing and we simply can’t use such opensearch release, because it actually breaks things.
With the 2.x → 3.x update, there are additional problems. For example you can’t start the 3.x opensearch server with any index created in opensearch 1.x or elasticsearch 7.0. You have to reindex these, manually. Opensearch doesn’t solve this, it leaves all the work to the users or external tools.
To make things even more complicated, opensearch created a completely new java client between 2.x and 3.x releases, backwards incompatible, with several bugs and parts of implementation missing.
We are working really hard on the opensearch 3.x support. But all of this takes some time to develop, integrate, fix, and properly test, if you don’t want to release a fragile and half-baked software.
Sorry I confused opensearch and log4j versions – I added a clarification on my comment.
Implementing opensearch 3.x sounds like the main challenge is providing a clean upgrade path. I understand that’s probably a hard requirement for many users. However a release with 3.x support and zero backward compatibility might be good for some users. I would consider it – either run two systems in parallel for a while, or roll my own export-import tools.
For updates on 3.x support, are there any issues or pull requests on the graylog2 project I can follow? I could not find any.
I see Tomas opened a new PR for datanode to update log4j versions – I believe this is simply replacing those bundled in the opensearch distro with the current patched versions.
I did this myself yesterday and have seen no side effects. (I tested first of course!)
I have a self-managed opensearch cluster, not datanode – you can replace the log4j jars in /usr/share/opensearch/lib and restart the opensearch processes. That path is for redhat-flavor hosts with yum/dnf packages, other packages may vary. And I’m not sure how the package manager will handle this going forward, it may be necessary to remove these when upgrading opensearch next time around.
Yes, we are trying to mitigate some problems and the lack of recent releases of the 2.x branch of opensearch. Replacing outdated log4j libraries seems like a way to go.
The datanode aims to simplify configuration, maintenance and manage the opensearch clusters for users, so it will provide backwards compatibility and migration tools for old indices, even those created in os1/es7.
