Rolling upgrade from 2.3 to 2.4?

hi,

is it possible to do a rolling upgrade from Graylog 2.3 to 2.4? (i.e. one server of the running cluster at a time).

If it is, are there any special concerns, such as master first/last etc.?

If you’re upgrading the Graylog master node first, a rolling upgrade should work.

But be aware that we didn’t test this, so your feedback is most welcome. :wink:

Rolling upgrade of a 3 node CentOs/RHEL cluster from 2.3 to 2.4 worked fine for me. Stopped the Graylog service on the master node and upgraded it first. Brought the master back online and upgraded the rest.

1 Like

I have a 4 nodes(gray01-04) cluster using Centos 7, with each node running cluster of graylog-server, mongodb and elasticsearch. gray01 is the graylog master. I did rollowing upgrade on 04-03-02 one at time without issue. Watching nodes number from 4 to 3 is quite a joy.

But my problem is the last node gray01 which has is_master set to true in server.conf.

after the upgrade gray01 can’t join back. I had to set gray02 as master. So I am down to a 3 nodes graylog cluster.

  • following is the error message, hope someone give me a pointer where to debug.
<snipped> /var/log/graylog-server/server.log
2018-01-04T15:43:20.233-05:00 ERROR [CmdLineTool] Guice error (more detail on log level debug): Error injecting constructor, java.lang.IllegalAccessError: tried to access method com.google.common.util.concurrent.SimpleTimeLimiter.<init>(Ljava/util/concurrent/ExecutorService;)V from class org.graylog.plugin.filter.dns.DnsResolverFilter
[root@gray01 graylog-server]# 

  • mongodb and elasticsearch logs looks ok
/var/log/elasticsearch/graylog.log
/var/log/mongodb/mongod.log 
  • my rolling upgrade procedure on all 4 nodes
rpm -e graylog-2.3-repository-1-5.noarch
rpm -Uvh https://packages.graylog2.org/repo/packages/graylog-2.4-repository_latest.rpm
yum upgrade graylog-server 
systemctl restart graylog-server

It looks like you’re using a plugin which is incompatible with Graylog 2.4.x (e. g. the DNS Resolver plugin).

2 Likes

@jochen, right to the point. I removed /usr/share/graylog-server/plugingr/aylog-plugin-dnsresolver-1.1.2.jar and gray01 was able to startup without error.

Thanks for the constant free support you provided to the forum.

Had the same problem with DNS resolver plugin - Google couldn’t help, so I will post my errors here to help others.
Indications:

  1. can’t connect to Web service, except for 9200 port

  2. systemctl status graylog-server shows green status: Active: active (running), but each time some seconds after restart the following errors appear (status still green):
    Jan 05 17:56:52 syslog-srv01-prod-srv-vm-lv01 graylog-server[7844]: at org.graylog2.bootstrap.Main.main(Main.java:44)
    Jan 05 17:56:52 syslog-srv01-prod-srv-vm-lv01 graylog-server[7844]: 12 errors
    Jan 05 17:56:52 syslog-srv01-prod-srv-vm-lv01 graylog-server[7844]: at com.google.inject.internal.Errors.throwCreationExceptionIfErrorsExist(Errors.java:470)
    Jan 05 17:56:52 syslog-srv01-prod-srv-vm-lv01 graylog-server[7844]: at com.google.inject.internal.InternalInjectorCreator.injectDynamically(InternalInjectorCreator.java:184)
    Jan 05 17:56:52 syslog-srv01-prod-srv-vm-lv01 graylog-server[7844]: at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:110)
    Jan 05 17:56:52 syslog-srv01-prod-srv-vm-lv01 graylog-server[7844]: at com.google.inject.Guice.createInjector(Guice.java:99)
    Jan 05 17:56:52 syslog-srv01-prod-srv-vm-lv01 graylog-server[7844]: at org.graylog2.shared.bindings.GuiceInjectorHolder.createInjector(GuiceInjectorHolder.java:34)
    Jan 05 17:56:52 syslog-srv01-prod-srv-vm-lv01 graylog-server[7844]: at org.graylog2.bootstrap.CmdLineTool.setupInjector(CmdLineTool.java:379)
    Jan 05 17:56:52 syslog-srv01-prod-srv-vm-lv01 graylog-server[7844]: at org.graylog2.bootstrap.CmdLineTool.run(CmdLineTool.java:193)
    Jan 05 17:56:52 syslog-srv01-prod-srv-vm-lv01 graylog-server[7844]: at org.graylog2.bootstrap.Main.main(Main.java:44)

  3. var/log/graylog-server shows the following errors:
    2018-01-05T17:56:52.079+02:00 ERROR [CmdLineTool] Guice error (more detail on log level debug): Error injecting constructor, java.lang.IllegalAccessError: tried to access method com.google.common.util.concurrent.SimpleTimeLimiter.(Ljava/util/concurrent/ExecutorService;)V from class org.graylog.plugin.filter.dns.DnsResolverFilter

  4. disabling of dns_resolver_enabled in /etc/graylog/server/server.conf (I put # in front of the line) with restart doesn’t help, you have to (re)move jar file as tjyang did.

+1 goes to jochen, thanks !

2 Likes

@arvo, Thanks for taking time to make this thread easier to be googled for others.

Seemed to work fine; I updated a cluster in CentOS, too.

1 Like

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.