Upgraded server to 5.2.8.1 now it will not launch

Graylog Central (peer support)
1

I upgraded our server to 5.2.8.1 and it will not launch

The server log shows
2024-06-18T16:36:00.934-07:00 ERROR [Messages] Caught exception during bulk indexing: ElasticsearchException{message=ElasticsearchException[An error occurred: ]; nested: ConnectException[Connection refused]; nested: ConnectException[Connection refused];, errorDetails=}, retrying (attempt #15).

  • OS Information: Red Hat Enterprise Linux release 8.10 (Ootpa)

  • Package Version:
    elasticsearch-oss.x86_64 7.10.2-1 @elasticsearch-7.x
    graylog-5.2-repository.noarch 1-2 @System
    graylog-server.x86_64 5.2.8-1 @graylog
    graylog-sidecar.x86_64 1.1.0-1 @graylog-sidecar
    graylog-sidecar-repository.noarch 1-2 @System
    mongodb-org.x86_64 4.2.25-1.el8 @mongodb-org-4.2
    mongodb-org-mongos.x86_64 4.2.25-1.el8 @mongodb-org-4.2
    mongodb-org-server.x86_64 4.2.25-1.el8 @mongodb-org-4.2
    mongodb-org-shell.x86_64 4.2.25-1.el8 @mongodb-org-4.2
    mongodb-org-tools.x86_64 4.2.25-1.el8 @mongodb-org-4.2

elasticsearch log

  • Service logs, configurations, and environment variables:
    [2024-06-18T16:36:26,834][INFO ][o.e.h.AbstractHttpServerTransport] [gandalf.bookingcenter.com] publish_address {127.0.0.1:9200}, bound_addresses {[::1]:9200}, {127.0.0.1:9200}
    [2024-06-18T16:36:26,835][INFO ][o.e.n.Node ] [xxx.bookingcenter.com] started
    [2024-06-18T16:36:27,019][INFO ][o.e.g.GatewayService ] [xx.bookingcenter.com] recovered [24] indices into cluster_state
    [2024-06-18T16:36:30,847][INFO ][o.e.c.r.a.AllocationService] [xxx.bookingcenter.com] Cluster health status changed from [RED] to [GREEN] (reason: [shards started [[gl-failures_58][0]]]).

checked config files and queried elasticsearch

curl -XGET http://localhost:9200/_cluster/health?pretty=true
{
“cluster_name” : “graylog”,
“status” : “green”,
“timed_out” : false,
“number_of_nodes” : 1,
“number_of_data_nodes” : 1,
“active_primary_shards” : 61,
“active_shards” : 61,
“relocating_shards” : 0,
“initializing_shards” : 0,
“unassigned_shards” : 0,
“delayed_unassigned_shards” : 0,
“number_of_pending_tasks” : 0,
“number_of_in_flight_fetch” : 0,
“task_max_waiting_in_queue_millis” : 0,
“active_shards_percent_as_number” : 100.0
}
curl -XGET http://localhost:9200
{
“name” : “xxxxx.bookingcenter.com”,
“cluster_name” : “graylog”,
“cluster_uuid” : “OF9VAHXESDa7GPpdR75v0A”,
“version” : {
“number” : “7.10.2”,
“build_flavor” : “oss”,
“build_type” : “rpm”,
“build_hash” : “747e1cc71def077253878a59143c1f785afa92b9”,
“build_date” : “2021-01-13T00:42:12.435326Z”,
“build_snapshot” : false,
“lucene_version” : “8.7.0”,
“minimum_wire_compatibility_version” : “6.8.0”,
“minimum_index_compatibility_version” : “6.0.0-beta1”
},
“tagline” : “You Know, for Search”
}

config files look fine, so I am a bit stuck

2

Hey @the_tree

You mean Graylog service will not start?

3

Yes it won’t start

elasticsearch status is green and I turned up the elasticsearch logger level to DEBUG but no errors i can see.

I found something relating to HEAP size on the web, so i will try that
thx
steve

4

If its your service heap might not help, check you journal

journalctl -xeu graylog-server

5

Thanks for that, i see a java error
Exception in thread “main” java.lang.UnsupportedClassVersionError: org/graylog2/bootstrap/Main

Found a thread on here with a possible solution to use bundled JVM instead of the OS version, I will let you how it goes

6

OK i commented out JAVA=/usr/bin/java in
/etc/sysconfig/graylog-server

Now i am getting a mongodb version issue, so progress has been made :wink:

Exception in thread “main” org.graylog2.bootstrap.preflight.PreflightCheckException: You’re running MongoDB 4.

7

Awesome @the_tree :+1:

8

well i managed to upgrade mongodb and get graylog server running but now it seems all my graylog configurations are gone :frowning:

9

Rule #1 when updating anything, create backups :cowboy_hat_face:

10

Yeah , we usually take machine snapshots but not for this machine

I should have backed up mongodb, damnit

“the only backup you will ever need is the one you didn’t make or the one you can’t read from” !

Thanks for all your help @gsmith !

1 Like
11

Np anytime @the_tree , have gn man

12

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.