Error after updating Graylog from 2.2.3 to 2.3.1

ashwath · September 13, 2017, 4:23pm

Hi,

Today i update my Graylog appliance to 2.3.1 using graylog_latest.deb file. Post update i initiated the reconfigure command, then finally it throws me an error message.

Then i tried to restarted the graylog services and now the web application is taking me to the below shown image.

I wonder where i made a mistake while performing update. I have also attached the error message below.

Requesting your input on this to solve this issue, i am totally new to Graylog and its components.

Error while re-configuring:
 ============================================================================                                                                             ====
    Error executing action `run` on resource 'ruby_block[add node to server list                                                                             ]'
    ============================================================================                                                                             ====

    Errno::ECONNREFUSED
    -------------------
    Connection refused - connect(2) for "127.0.0.1" port 4001

    Cookbook Trace:
    ---------------
    /opt/graylog/embedded/cookbooks/graylog/libraries/registry.rb:17:in `set_mas                                                                             ter'
    /opt/graylog/embedded/cookbooks/graylog/recipes/graylog-server.rb:86:in `blo                                                                             ck (2 levels) in from_file'

    Resource Declaration:
    ---------------------
    # In /opt/graylog/embedded/cookbooks/graylog/recipes/graylog-server.rb

     84: ruby_block "add node to server list" do
     85:   block do
     86:     $registry.set_master
     87:     $registry.add_gl_server(node['ipaddress'])
     88:     $registry.add_es_node(node['ipaddress'])
     89:   end
     90:   retries 15
     91: end

    Compiled Resource:
    ------------------
    # Declared in /opt/graylog/embedded/cookbooks/graylog/recipes/graylog-server                                                                             .rb:84:in `from_file'

    ruby_block("add node to server list") do
      action [:run]
      retries 15
      retry_delay 2
      default_guard_interpreter :default
      block_name "add node to server list"
      declared_type :ruby_block
      cookbook_name :graylog
      recipe_name "graylog-server"
      block #<Proc:0x00000002bbb938@/opt/graylog/embedded/cookbooks/graylog/reci                                                                             pes/graylog-server.rb:85>
    end

Recipe: graylog::elasticsearch
  * service[elasticsearch] action restart
    - restart service service[elasticsearch]
Recipe: graylog::graylog-server
  * service[graylog-server] action restart
    - restart service service[graylog-server]

Running handlers:
[2017-09-13T20:27:09+05:30] ERROR: Running exception handlers
Running handlers complete
[2017-09-13T20:27:09+05:30] ERROR: Exception handlers complete
Chef Client failed. 16 resources updated in 03 minutes 02 seconds
[2017-09-13T20:27:09+05:30] FATAL: Stacktrace dumped to /opt/graylog/embedded/co                                                                             okbooks/cache/chef-stacktrace.out
[2017-09-13T20:27:09+05:30] FATAL: Please provide the contents of the stacktrace                                                                             .out file if you file a bug report
[2017-09-13T20:27:09+05:30] ERROR: ruby_block[add node to server list] (graylog:                                                                             :graylog-server line 84) had an error: Errno::ECONNREFUSED: Connection refused -                                                                              connect(2) for "127.0.0.1" port 4001
[2017-09-13T20:27:09+05:30] FATAL: Chef::Exceptions::ChildConvergeError: Chef ru                                                                             n process exited unsuccessfully (exit code 1)
 

Web page:
![image|576x255](upload://2u6fO3wdD468EA9QRVhP5Cp85L2.png)

**Stack tracer dump:**

 Generated at 2017-09-13 20:27:09 +0530
Errno::ECONNREFUSED: ruby_block[add node to server list] (graylog::graylog-server line 84) had an error: Errno::ECONNREFUSED: Connection refused - connect(2) for "127.0.0.1" port 4001
/opt/graylog/embedded/lib/ruby/2.1.0/net/http.rb:879:in `initialize'
/opt/graylog/embedded/lib/ruby/2.1.0/net/http.rb:879:in `open'
/opt/graylog/embedded/lib/ruby/2.1.0/net/http.rb:879:in `block in connect'
/opt/graylog/embedded/lib/ruby/2.1.0/timeout.rb:75:in `timeout'
/opt/graylog/embedded/lib/ruby/2.1.0/net/http.rb:878:in `connect'
/opt/graylog/embedded/lib/ruby/2.1.0/net/http.rb:863:in `do_start'
/opt/graylog/embedded/lib/ruby/2.1.0/net/http.rb:852:in `start'
/opt/graylog/embedded/lib/ruby/2.1.0/net/http.rb:1375:in `request'
/opt/graylog/embedded/lib/ruby/gems/2.1.0/gems/etcd-0.3.0/lib/etcd/client.rb:111:in `api_execute'
/opt/graylog/embedded/lib/ruby/gems/2.1.0/gems/etcd-0.3.0/lib/etcd/keys.rb:39:in `set'
/opt/graylog/embedded/cookbooks/graylog/libraries/registry.rb:17:in `set_master'
/opt/graylog/embedded/cookbooks/graylog/recipes/graylog-server.rb:86:in `block (2 levels) in from_file'
/opt/graylog/embedded/lib/ruby/gems/2.1.0/gems/chef-12.6.0/lib/chef/provider/ruby_block.rb:35:in `call'
/opt/graylog/embedded/lib/ruby/gems/2.1.0/gems/chef-12.6.0/lib/chef/provider/ruby_block.rb:35:in `block in action_run'
/opt/graylog/embedded/lib/ruby/gems/2.1.0/gems/chef-12.6.0/lib/chef/mixin/why_run.rb:52:in `call'
/opt/graylog/embedded/lib/ruby/gems/2.1.0/gems/chef-12.6.0/lib/chef/mixin/why_run.rb:52:in `add_action'
/opt/graylog/embedded/lib/ruby/gems/2.1.0/gems/chef-12.6.0/lib/chef/provider.rb:175:in `converge_by'
/opt/graylog/embedded/lib/ruby/gems/2.1.0/gems/chef-12.6.0/lib/chef/provider/ruby_block.rb:34:in `action_run'
/opt/graylog/embedded/lib/ruby/gems/2.1.0/gems/chef-12.6.0/lib/chef/provider.rb:144:in `run_action'
/opt/graylog/embedded/lib/ruby/gems/2.1.0/gems/chef-12.6.0/lib/chef/resource.rb:596:in `run_action'
/opt/graylog/embedded/lib/ruby/gems/2.1.0/gems/chef-12.6.0/lib/chef/runner.rb:74:in `run_action'
/opt/graylog/embedded/lib/ruby/gems/2.1.0/gems/chef-12.6.0/lib/chef/runner.rb:106:in `block (2 levels) in converge'
/opt/graylog/embedded/lib/ruby/gems/2.1.0/gems/chef-12.6.0/lib/chef/runner.rb:106:in `each'
/opt/graylog/embedded/lib/ruby/gems/2.1.0/gems/chef-12.6.0/lib/chef/runner.rb:106:in `block in converge'
/opt/graylog/embedded/lib/ruby/gems/2.1.0/gems/chef-12.6.0/lib/chef/resource_collection/resource_list.rb:83:in `block in execute_each_resource'
/opt/graylog/embedded/lib/ruby/gems/2.1.0/gems/chef-12.6.0/lib/chef/resource_collection/stepable_iterator.rb:116:in `call'
/opt/graylog/embedded/lib/ruby/gems/2.1.0/gems/chef-12.6.0/lib/chef/resource_collection/stepable_iterator.rb:116:in `call_iterator_block'
/opt/graylog/embedded/lib/ruby/gems/2.1.0/gems/chef-12.6.0/lib/chef/resource_collection/stepable_iterator.rb:85:in `step'
/opt/graylog/embedded/lib/ruby/gems/2.1.0/gems/chef-12.6.0/lib/chef/resource_collection/stepable_iterator.rb:104:in `iterate'
/opt/graylog/embedded/lib/ruby/gems/2.1.0/gems/chef-12.6.0/lib/chef/resource_collection/stepable_iterator.rb:55:in `each_with_index'
/opt/graylog/embedded/lib/ruby/gems/2.1.0/gems/chef-12.6.0/lib/chef/resource_collection/resource_list.rb:81:in `execute_each_resource'
/opt/graylog/embedded/lib/ruby/gems/2.1.0/gems/chef-12.6.0/lib/chef/runner.rb:105:in `converge'
/opt/graylog/embedded/lib/ruby/gems/2.1.0/gems/chef-12.6.0/lib/chef/client.rb:658:in `block in converge'
/opt/graylog/embedded/lib/ruby/gems/2.1.0/gems/chef-12.6.0/lib/chef/client.rb:653:in `catch'
/opt/graylog/embedded/lib/ruby/gems/2.1.0/gems/chef-12.6.0/lib/chef/client.rb:653:in `converge'
/opt/graylog/embedded/lib/ruby/gems/2.1.0/gems/chef-12.6.0/lib/chef/client.rb:692:in `converge_and_save'
/opt/graylog/embedded/lib/ruby/gems/2.1.0/gems/chef-12.6.0/lib/chef/client.rb:271:in `run'
/opt/graylog/embedded/lib/ruby/gems/2.1.0/gems/chef-12.6.0/lib/chef/application.rb:261:in `block in fork_chef_client'
/opt/graylog/embedded/lib/ruby/gems/2.1.0/gems/chef-12.6.0/lib/chef/application.rb:249:in `fork'
/opt/graylog/embedded/lib/ruby/gems/2.1.0/gems/chef-12.6.0/lib/chef/application.rb:249:in `fork_chef_client'
/opt/graylog/embedded/lib/ruby/gems/2.1.0/gems/chef-12.6.0/lib/chef/application.rb:215:in `block in run_chef_client'
/opt/graylog/embedded/lib/ruby/gems/2.1.0/gems/chef-12.6.0/lib/chef/local_mode.rb:44:in `with_server_connectivity'
/opt/graylog/embedded/lib/ruby/gems/2.1.0/gems/chef-12.6.0/lib/chef/application.rb:203:in `run_chef_client'
/opt/graylog/embedded/lib/ruby/gems/2.1.0/gems/chef-12.6.0/lib/chef/application/solo.rb:286:in `block in interval_run_chef_client'
/opt/graylog/embedded/lib/ruby/gems/2.1.0/gems/chef-12.6.0/lib/chef/application/solo.rb:275:in `loop'
/opt/graylog/embedded/lib/ruby/gems/2.1.0/gems/chef-12.6.0/lib/chef/application/solo.rb:275:in `interval_run_chef_client'
/opt/graylog/embedded/lib/ruby/gems/2.1.0/gems/chef-12.6.0/lib/chef/application/solo.rb:253:in `run_application'
/opt/graylog/embedded/lib/ruby/gems/2.1.0/gems/chef-12.6.0/lib/chef/application.rb:58:in `run'
/opt/graylog/embedded/lib/ruby/gems/2.1.0/gems/chef-12.6.0/bin/chef-solo:25:in `<top (required)>'
/opt/graylog/embedded/bin/chef-solo:23:in `load'
/opt/graylog/embedded/bin/chef-solo:23:in `<main>'

Thank you
Ashwath Kumar

bubba198 · September 13, 2017, 10:49pm

I did the same some time ago – 2.1.0+62db7e0, codename Smuttynose to the latest 2.3.1 and got different but equally critical errors so I had to roll back. Has anyone successfully upgraded an OVA appliance around here? Is it supposed to work or we have to wait for a new appliance OVA to be released? Of course new OVA means no data is preserved which would not be ideal…

jochen · September 14, 2017, 7:42am

It looks like etcd isn’t running.

Please read the following issue on GitHub for a related problem:

jochen · September 14, 2017, 7:43am

Yes.

It’s supposed to work as described at http://docs.graylog.org/en/2.3/pages/configuration/graylog_ctl.html#upgrade-graylog

ashwath · September 15, 2017, 4:41pm

Hi,

Thanks for the tip, as mentioned in that thread deleting etcd content and re-configuring graylog worked.

Regards
Ashwath Kumar R

ashwath · September 15, 2017, 4:47pm

Hi,

Download the latest version of .deb file from omnibus package, stop graylog services and then delete the contents inside etcd folder(sudo rm -r /var/opt/graylog/data/etcd/*).

Then install graylog (sudo dpkg -G -i graylog_latest.deb)

Finally reconfigure graylog and reboot the machine(sudo graylog-ctl reconfigure)

This will work if you are running a single host instance.

Thank you
Ashwath Kumar R

dannlori · September 15, 2017, 8:03pm

Thanks. Worked perfectly.

bubba198 · September 28, 2017, 2:51am

Thanks @ashwath I did delete etcd as you suggested and the install does function but now I am getting a huge amount of meaningless java errors logged to the host with source of graylog-server; none seen prior to the upgrade. Had to revert back to the snapshot yet again.

Here’s an example text of the errors:

at com.codahale.metrics.InstrumentedThreadFactory$InstrumentedRunnable.run(InstrumentedThreadFactory.java:66) [graylog.jar:?]
at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:42) [graylog.jar:?]

jochen · September 28, 2017, 6:46am

Please post the complete logs or at least the complete error message (including the stack trace).

What you’ve posted is just a tiny part of the stack trace.

bubba198 · September 28, 2017, 5:46pm

Thanks @jochen I actually did another clean test; loaded a brand new 2.1.0+62db7e0 OVA and then upgraded it using the procedure discussed here. Then exposed it to a live incoming stream and here’s my /var/log/graylog/server/current (link to the full gist is at the bottom of the post)

2017-09-28_16:58:13.40268 It looks like you are trying to access MongoDB over HTTP on the native driver port.
2017-09-28_16:58:23.72851 INFO  [CmdLineTool] Loaded plugin: Splunk Output 0.4.0 [com.graylog.splunk.output.SplunkOutputPlugin]
2017-09-28_16:58:23.72906 INFO  [CmdLineTool] Loaded plugin: Elastic Beats Input 2.3.1 [org.graylog.plugins.beats.BeatsInputPlugin]
2017-09-28_16:58:23.73971 INFO  [CmdLineTool] Loaded plugin: Collector 2.3.1 [org.graylog.plugins.collector.CollectorPlugin]
2017-09-28_16:58:23.74167 INFO  [CmdLineTool] Loaded plugin: Enterprise Integration Plugin 2.3.1 [org.graylog.plugins.enterprise_integration.EnterpriseIntegrationPlugin]
2017-09-28_16:58:23.74621 INFO  [CmdLineTool] Loaded plugin: MapWidgetPlugin 2.3.1 [org.graylog.plugins.map.MapWidgetPlugin]
2017-09-28_16:58:23.78201 INFO  [CmdLineTool] Loaded plugin: Pipeline Processor Plugin 2.3.1 [org.graylog.plugins.pipelineprocessor.ProcessorPlugin]
2017-09-28_16:58:23.79649 INFO  [CmdLineTool] Loaded plugin: Anonymous Usage Statistics 2.3.1 [org.graylog.plugins.usagestatistics.UsageStatsPlugin]
2017-09-28_16:58:23.79829 INFO  [CmdLineTool] Loaded plugin: SnmpPlugin 0.3.0 [org.graylog.snmp.SnmpPlugin]
2017-09-28_16:58:26.04790 INFO  [CmdLineTool] Running with JVM arguments: -Xms1g -Xmx1500m -XX:NewRatio=1 -XX:+ResizeTLAB -XX:+UseConcMarkSweepGC -XX:+CMSConcurrentMTEnabled -XX:+CMSClassUnloadingEnabled -XX:+UseParNewGC -XX:-OmitStackTraceInFastThrow -Dlog4j.configurationFile=file:///opt/graylog/conf/log4j2.xml -Djava.library.path=/opt/graylog/server/lib/sigar/ -

https://gist.github.com/bubba198/420d2b950f4f887a60a4696637b84c20

JulioQc · September 28, 2017, 6:17pm

@bubba198 I had a bunch of the errors you experience. Apparently linked to the new ES 5.5 mapping formats. Here is how I solved it:

delete all custom mappings you had, making a copy in some file (i.e notepad)
rotate the deflector index
update your custom mapping to the new ES 5.5 format
post your new custom mappings
reboot

Can’t say it will solve all the issues but a bunch should be fixed that way. Hope it helps

system · October 12, 2017, 6:17pm

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.