After upgrading from 3.3 to 4, im getting Server currently unavailable

Graylog
#1

I am using the OVA. graylog was working fine… when i was logged in , it said i wasnt running the latest version…
so i downloaded the graylog-4.0-repository_latest.deb and ran the update.
wget https://packages.graylog2.org/repo/packages/graylog-4.0-repository_latest.deb
dpkg -i graylog-4.0-repository_latest.deb
apt-get update
apt-get install graylog-server

it finished without error.
Initially when i tried to refresh the page, I got " Graylog is currently not reachable…"
when i researched that, i found that the plugins were not compatible so i removed all non 4.0.1
jar files… now after i restarted GL, i get the front page but it says
“We are experiencing problems connecting to the Graylog server running on http://192.168.1.10:9000/api/ . Please verify that the server is healthy and working correctly.”

9001 is listening… and /etc/init.d/graylog-server status says its running.

here are ERROR lines in the server.log file

2020-12-17T04:23:15.736Z ERROR [VersionProbe] Unable to retrieve version from Elasticsearch node:
2020-12-17T04:23:19.375Z ERROR [CmdLineTool]
ERROR: Unable to probe any host for Elasticsearch version!
2020-12-17T04:23:43.679Z ERROR [AuditLogger] Unable to write audit log entry because there is no valid license.
2020-12-17T04:23:44.179Z ERROR [MongoAuditLogPeriodical] Not running cleanup for auditlog entries in MongoDB because there is no valid license.
2020-12-17T04:24:22.122Z ERROR [AuditLogger] Unable to write audit log entry because there is no valid license.
2020-12-17T04:33:22.486Z ERROR [AuditLogger] Unable to write audit log entry because there is no valid license.
2020-12-17T04:33:26.567Z ERROR [AuditLogger] Unable to write audit log entry because there is no valid license.
2020-12-17T04:33:40.062Z ERROR [AuditLogger] Unable to write audit log entry because there is no valid license.
2020-12-17T04:33:40.592Z ERROR [MongoAuditLogPeriodical] Not running cleanup for auditlog entries in MongoDB because there is no valid license.
2020-12-17T04:34:17.026Z ERROR [AuditLogger] Unable to write audit log entry because there is no valid license.
2020-12-17T04:39:58.158Z ERROR [AuditLogger] Unable to write audit log entry because there is no valid license.
2020-12-17T04:40:02.373Z ERROR [AuditLogger] Unable to write audit log entry because there is no valid license.
2020-12-16T23:40:16.239-05:00 ERROR [AuditLogger] Unable to write audit log entry because there is no valid license.
2020-12-16T23:40:17.001-05:00 ERROR [MongoAuditLogPeriodical] Not running cleanup for auditlog entries in MongoDB because there is no valid license.

#2

Hello @jronpaul,

Are elasticsearch and mongod running? What are their versions?

#3

@jronpaul I was going to mention what @ttsandrew had asked. Ensuring that all of Graylog’s dependencies are up and running is a good first place to start.

#4

thanks guys… it appears they are both up… however, it seems elasticsearch runs under java?

root@graylog:~# ps -ef|grep mongo
mongodb 407 1 1 Dec16 ? 00:16:07 /usr/bin/mongod --config /etc/mongod.conf

root@graylog:~# systemctl status elasticsearch
● elasticsearch.service - Elasticsearch
Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: enabled)
Active: active (running) since Thu 2020-12-17 21:51:52 EST; 5s ago
Docs: http://www.elastic.co
Main PID: 6676 (java)
Tasks: 16 (limit: 4677)
CGroup: /system.slice/elasticsearch.service
└─6676 /usr/bin/java -Xms1g -Xmx1g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitia

Dec 17 21:51:52 graylog systemd[1]: Started Elasticsearch.
Dec 17 21:51:52 graylog elasticsearch[6676]: warning: Falling back to java on path. This behavior is deprecated. Specify JAVA
root@graylog:~#

#5

Yep, Elasticsearch is a Java application. What version are you on? This looks like it’s from 7.X, which IIRC requires you to set the JAVA_HOME environment variable in /etc/default/elasticsearch. You can find that by running:

update-alternatives --config java
There is only one alternative in link group java (providing /usr/bin/java): /usr/lib/jvm/java-11-openjdk-amd64/bin/java

So in the example above, you’d use /usr/lib/jvm/java-11-openjdk-amd64 as the path. Try setting that and restarting Elasticsearch and Graylog. Also, as a note, just because systemd indicates is running doesn’t mean that the service is listening on the port or is actively responding. Running a netstat or even curling the elastic API would serve as a more useful troubleshooting method than getting the status through systemd.

#6

ok… thats odd because i havent changed paths or anything in my env.
ubuntu@graylog:~ update-alternatives --config java There is only one alternative in link group java (providing /usr/bin/java): /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java Nothing to configure. ubuntu@graylog:~ sudo su -
root@graylog:~# update-alternatives --config java
There is only one alternative in link group java (providing /usr/bin/java): /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java
Nothing to configure.
root@graylog:~#

interesting that JAVA_HOME is commented out in the elasticsearch config file…
i didnt change it… maybe an update did?

root@graylog:~# grep -v “#” /etc/default/elasticsearch|grep -v ^$
ES_PATH_CONF=/etc/elasticsearch
ES_STARTUP_SLEEP_TIME=5
root@graylog:~#

so i set JAVA_HOME to /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java and restarted the service…
still get the same error though.

thanks for staying with me on this.

#7

if i put the IP:9000 in my browser… i get the login… but creds are not working.

#8

here us curl
root@graylog:~# curl http://127.0.0.1:9000/api
{“cluster_id”:“445l89c2-48r8-4914-r8f6-9r95lg6lf6d6”,“node_id”:“4424r455-6g8c-47c7-9r4r-84d381r731g2”,“version”:“4.0.1+6a0cc0b”,“tagline”:“Manage your logs in the dark and have lasers going and make it look like you’re from space!”}root@graylog:~#

#9

Can’t remember the exact setting but look for CORS in the config. Enable that and see if helps.

#10

http_enable_cors = true is what you’re looking for.

1 Like
#11

ok tried that.
still got
We are experiencing problems connecting to the Graylog server running on http://192.168.1.241:9000/api/ . Please verify that the server is healthy and working correctly.

graylog log says
[2020-12-23T22:11:30,029][INFO ][o.e.c.r.a.AllocationService] [OMj8A95] Cluster health status changed from [RED] to [GREEN] (reason: [shards started [[graylog_0][3]] …]).

im seeing this in the gc.log.0.current.
2020-12-23T22:37:50.052-0500: 1608.503: Total time for which application threads were stopped: 0.0004017 seconds, Stopping threads took: 0.0001811 seconds
2020-12-23T22:37:50.058-0500: 1608.509: Total time for which application threads were stopped: 0.0003766 seconds, Stopping threads took: 0.0001249 seconds
2020-12-23T22:37:50.059-0500: 1608.510: Total time for which application threads were stopped: 0.0003183 seconds, Stopping threads took: 0.0001646 seconds
2020-12-23T22:37:50.072-0500: 1608.523: [GC (Allocation Failure) 2020-12-23T22:37:50.072-0500: 1608.523: [ParNew
Desired survivor size 8716288 bytes, new threshold 6 (max 6)

#12

I’m also having very similar issues on a debian system, having just upgraded today from 3.3. Mongo + ElasticSearch both look good.

Oddly enough, after enable CORS, I can now get to the login page like I’m used to, but my credentials don’t work? I only have one Graylog server and I was able to login before the upgrade.

#13

@sparis per the upgrade documentation LDAP config is inactivated on upgrade to 4 until you log in with local admin account to review and reactivate it. This is because the feature has changed significantly. Probably this is your issue.

#14

@ttsandrew I took your advice and was able to reset the local admin password to get back into the system. Thankfully all the data is still here (streams, settings, user accounts, etc).

I’ve looked through the updated documentation on the changes to LDAP / Active Directory and i’m confused. I’m using, what I believe to be, local authentication directly against our GrayLog server. We never bothered to integrate LDAP/AD because we have such a small install (i.e 3 people that login). Does the authentication still require that I setup LDAP even though the users exist on the graylog server directly?

I went so far as to create a brand new user, set a password, and still can’t login as that new user. Looking at the log i see this:

ERROR [AESTools] Could not encrypt value.
java.security.InvalidKeyException: Illegal key size

#15

I was able to upgrade to java 11 from 8, and this seems to have taken care of the issue.

Thanks @ttsandrew for the help!

1 Like
#16

im already at java 11… and im only using local auth

#17

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.