Error after updating Graylog from 2.2.3 to 2.3.1


(Ashwath Kumar) #1

Hi,

Today i update my Graylog appliance to 2.3.1 using graylog_latest.deb file. Post update i initiated the reconfigure command, then finally it throws me an error message.

Then i tried to restarted the graylog services and now the web application is taking me to the below shown image.

I wonder where i made a mistake while performing update. I have also attached the error message below.

Requesting your input on this to solve this issue, i am totally new to Graylog and its components.

Error while re-configuring:
 ============================================================================                                                                             ====
    Error executing action `run` on resource 'ruby_block[add node to server list                                                                             ]'
    ============================================================================                                                                             ====

    Errno::ECONNREFUSED
    -------------------
    Connection refused - connect(2) for "127.0.0.1" port 4001

    Cookbook Trace:
    ---------------
    /opt/graylog/embedded/cookbooks/graylog/libraries/registry.rb:17:in `set_mas                                                                             ter'
    /opt/graylog/embedded/cookbooks/graylog/recipes/graylog-server.rb:86:in `blo                                                                             ck (2 levels) in from_file'

    Resource Declaration:
    ---------------------
    # In /opt/graylog/embedded/cookbooks/graylog/recipes/graylog-server.rb

     84: ruby_block "add node to server list" do
     85:   block do
     86:     $registry.set_master
     87:     $registry.add_gl_server(node['ipaddress'])
     88:     $registry.add_es_node(node['ipaddress'])
     89:   end
     90:   retries 15
     91: end

    Compiled Resource:
    ------------------
    # Declared in /opt/graylog/embedded/cookbooks/graylog/recipes/graylog-server                                                                             .rb:84:in `from_file'

    ruby_block("add node to server list") do
      action [:run]
      retries 15
      retry_delay 2
      default_guard_interpreter :default
      block_name "add node to server list"
      declared_type :ruby_block
      cookbook_name :graylog
      recipe_name "graylog-server"
      block #<Proc:0x00000002bbb938@/opt/graylog/embedded/cookbooks/graylog/reci                                                                             pes/graylog-server.rb:85>
    end

Recipe: graylog::elasticsearch
  * service[elasticsearch] action restart
    - restart service service[elasticsearch]
Recipe: graylog::graylog-server
  * service[graylog-server] action restart
    - restart service service[graylog-server]

Running handlers:
[2017-09-13T20:27:09+05:30] ERROR: Running exception handlers
Running handlers complete
[2017-09-13T20:27:09+05:30] ERROR: Exception handlers complete
Chef Client failed. 16 resources updated in 03 minutes 02 seconds
[2017-09-13T20:27:09+05:30] FATAL: Stacktrace dumped to /opt/graylog/embedded/co                                                                             okbooks/cache/chef-stacktrace.out
[2017-09-13T20:27:09+05:30] FATAL: Please provide the contents of the stacktrace                                                                             .out file if you file a bug report
[2017-09-13T20:27:09+05:30] ERROR: ruby_block[add node to server list] (graylog:                                                                             :graylog-server line 84) had an error: Errno::ECONNREFUSED: Connection refused -                                                                              connect(2) for "127.0.0.1" port 4001
[2017-09-13T20:27:09+05:30] FATAL: Chef::Exceptions::ChildConvergeError: Chef ru                                                                             n process exited unsuccessfully (exit code 1)
 

Web page:
![image|576x255](upload://2u6fO3wdD468EA9QRVhP5Cp85L2.png)

**Stack tracer dump:**

 Generated at 2017-09-13 20:27:09 +0530
Errno::ECONNREFUSED: ruby_block[add node to server list] (graylog::graylog-server line 84) had an error: Errno::ECONNREFUSED: Connection refused - connect(2) for "127.0.0.1" port 4001
/opt/graylog/embedded/lib/ruby/2.1.0/net/http.rb:879:in `initialize'
/opt/graylog/embedded/lib/ruby/2.1.0/net/http.rb:879:in `open'
/opt/graylog/embedded/lib/ruby/2.1.0/net/http.rb:879:in `block in connect'
/opt/graylog/embedded/lib/ruby/2.1.0/timeout.rb:75:in `timeout'
/opt/graylog/embedded/lib/ruby/2.1.0/net/http.rb:878:in `connect'
/opt/graylog/embedded/lib/ruby/2.1.0/net/http.rb:863:in `do_start'
/opt/graylog/embedded/lib/ruby/2.1.0/net/http.rb:852:in `start'
/opt/graylog/embedded/lib/ruby/2.1.0/net/http.rb:1375:in `request'
/opt/graylog/embedded/lib/ruby/gems/2.1.0/gems/etcd-0.3.0/lib/etcd/client.rb:111:in `api_execute'
/opt/graylog/embedded/lib/ruby/gems/2.1.0/gems/etcd-0.3.0/lib/etcd/keys.rb:39:in `set'
/opt/graylog/embedded/cookbooks/graylog/libraries/registry.rb:17:in `set_master'
/opt/graylog/embedded/cookbooks/graylog/recipes/graylog-server.rb:86:in `block (2 levels) in from_file'
/opt/graylog/embedded/lib/ruby/gems/2.1.0/gems/chef-12.6.0/lib/chef/provider/ruby_block.rb:35:in `call'
/opt/graylog/embedded/lib/ruby/gems/2.1.0/gems/chef-12.6.0/lib/chef/provider/ruby_block.rb:35:in `block in action_run'
/opt/graylog/embedded/lib/ruby/gems/2.1.0/gems/chef-12.6.0/lib/chef/mixin/why_run.rb:52:in `call'
/opt/graylog/embedded/lib/ruby/gems/2.1.0/gems/chef-12.6.0/lib/chef/mixin/why_run.rb:52:in `add_action'
/opt/graylog/embedded/lib/ruby/gems/2.1.0/gems/chef-12.6.0/lib/chef/provider.rb:175:in `converge_by'
/opt/graylog/embedded/lib/ruby/gems/2.1.0/gems/chef-12.6.0/lib/chef/provider/ruby_block.rb:34:in `action_run'
/opt/graylog/embedded/lib/ruby/gems/2.1.0/gems/chef-12.6.0/lib/chef/provider.rb:144:in `run_action'
/opt/graylog/embedded/lib/ruby/gems/2.1.0/gems/chef-12.6.0/lib/chef/resource.rb:596:in `run_action'
/opt/graylog/embedded/lib/ruby/gems/2.1.0/gems/chef-12.6.0/lib/chef/runner.rb:74:in `run_action'
/opt/graylog/embedded/lib/ruby/gems/2.1.0/gems/chef-12.6.0/lib/chef/runner.rb:106:in `block (2 levels) in converge'
/opt/graylog/embedded/lib/ruby/gems/2.1.0/gems/chef-12.6.0/lib/chef/runner.rb:106:in `each'
/opt/graylog/embedded/lib/ruby/gems/2.1.0/gems/chef-12.6.0/lib/chef/runner.rb:106:in `block in converge'
/opt/graylog/embedded/lib/ruby/gems/2.1.0/gems/chef-12.6.0/lib/chef/resource_collection/resource_list.rb:83:in `block in execute_each_resource'
/opt/graylog/embedded/lib/ruby/gems/2.1.0/gems/chef-12.6.0/lib/chef/resource_collection/stepable_iterator.rb:116:in `call'
/opt/graylog/embedded/lib/ruby/gems/2.1.0/gems/chef-12.6.0/lib/chef/resource_collection/stepable_iterator.rb:116:in `call_iterator_block'
/opt/graylog/embedded/lib/ruby/gems/2.1.0/gems/chef-12.6.0/lib/chef/resource_collection/stepable_iterator.rb:85:in `step'
/opt/graylog/embedded/lib/ruby/gems/2.1.0/gems/chef-12.6.0/lib/chef/resource_collection/stepable_iterator.rb:104:in `iterate'
/opt/graylog/embedded/lib/ruby/gems/2.1.0/gems/chef-12.6.0/lib/chef/resource_collection/stepable_iterator.rb:55:in `each_with_index'
/opt/graylog/embedded/lib/ruby/gems/2.1.0/gems/chef-12.6.0/lib/chef/resource_collection/resource_list.rb:81:in `execute_each_resource'
/opt/graylog/embedded/lib/ruby/gems/2.1.0/gems/chef-12.6.0/lib/chef/runner.rb:105:in `converge'
/opt/graylog/embedded/lib/ruby/gems/2.1.0/gems/chef-12.6.0/lib/chef/client.rb:658:in `block in converge'
/opt/graylog/embedded/lib/ruby/gems/2.1.0/gems/chef-12.6.0/lib/chef/client.rb:653:in `catch'
/opt/graylog/embedded/lib/ruby/gems/2.1.0/gems/chef-12.6.0/lib/chef/client.rb:653:in `converge'
/opt/graylog/embedded/lib/ruby/gems/2.1.0/gems/chef-12.6.0/lib/chef/client.rb:692:in `converge_and_save'
/opt/graylog/embedded/lib/ruby/gems/2.1.0/gems/chef-12.6.0/lib/chef/client.rb:271:in `run'
/opt/graylog/embedded/lib/ruby/gems/2.1.0/gems/chef-12.6.0/lib/chef/application.rb:261:in `block in fork_chef_client'
/opt/graylog/embedded/lib/ruby/gems/2.1.0/gems/chef-12.6.0/lib/chef/application.rb:249:in `fork'
/opt/graylog/embedded/lib/ruby/gems/2.1.0/gems/chef-12.6.0/lib/chef/application.rb:249:in `fork_chef_client'
/opt/graylog/embedded/lib/ruby/gems/2.1.0/gems/chef-12.6.0/lib/chef/application.rb:215:in `block in run_chef_client'
/opt/graylog/embedded/lib/ruby/gems/2.1.0/gems/chef-12.6.0/lib/chef/local_mode.rb:44:in `with_server_connectivity'
/opt/graylog/embedded/lib/ruby/gems/2.1.0/gems/chef-12.6.0/lib/chef/application.rb:203:in `run_chef_client'
/opt/graylog/embedded/lib/ruby/gems/2.1.0/gems/chef-12.6.0/lib/chef/application/solo.rb:286:in `block in interval_run_chef_client'
/opt/graylog/embedded/lib/ruby/gems/2.1.0/gems/chef-12.6.0/lib/chef/application/solo.rb:275:in `loop'
/opt/graylog/embedded/lib/ruby/gems/2.1.0/gems/chef-12.6.0/lib/chef/application/solo.rb:275:in `interval_run_chef_client'
/opt/graylog/embedded/lib/ruby/gems/2.1.0/gems/chef-12.6.0/lib/chef/application/solo.rb:253:in `run_application'
/opt/graylog/embedded/lib/ruby/gems/2.1.0/gems/chef-12.6.0/lib/chef/application.rb:58:in `run'
/opt/graylog/embedded/lib/ruby/gems/2.1.0/gems/chef-12.6.0/bin/chef-solo:25:in `<top (required)>'
/opt/graylog/embedded/bin/chef-solo:23:in `load'
/opt/graylog/embedded/bin/chef-solo:23:in `<main>'

Thank you
Ashwath Kumar


(bubba198) #2

I did the same some time ago – 2.1.0+62db7e0, codename Smuttynose to the latest 2.3.1 and got different but equally critical errors so I had to roll back. Has anyone successfully upgraded an OVA appliance around here? Is it supposed to work or we have to wait for a new appliance OVA to be released? Of course new OVA means no data is preserved which would not be ideal…


(Jochen) #3

It looks like etcd isn’t running.

Please read the following issue on GitHub for a related problem:


(Jochen) #4

Yes.

It’s supposed to work as described at http://docs.graylog.org/en/2.3/pages/configuration/graylog_ctl.html#upgrade-graylog


(Ashwath Kumar) #5

Hi,

Thanks for the tip, as mentioned in that thread deleting etcd content and re-configuring graylog worked.

Regards
Ashwath Kumar R


(Ashwath Kumar) #6

Hi,

Download the latest version of .deb file from omnibus package, stop graylog services and then delete the contents inside etcd folder(sudo rm -r /var/opt/graylog/data/etcd/*).

Then install graylog (sudo dpkg -G -i graylog_latest.deb)

Finally reconfigure graylog and reboot the machine(sudo graylog-ctl reconfigure)

This will work if you are running a single host instance.

Thank you
Ashwath Kumar R


(Dan Nelson) #7

Thanks. Worked perfectly.


(bubba198) #8

Thanks @ashwath I did delete etcd as you suggested and the install does function but now I am getting a huge amount of meaningless java errors logged to the host with source of graylog-server; none seen prior to the upgrade. Had to revert back to the snapshot yet again.

Here’s an example text of the errors:

at com.codahale.metrics.InstrumentedThreadFactory$InstrumentedRunnable.run(InstrumentedThreadFactory.java:66) [graylog.jar:?]
at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:42) [graylog.jar:?]

(Jochen) #9

Please post the complete logs or at least the complete error message (including the stack trace).

What you’ve posted is just a tiny part of the stack trace.


(bubba198) #10

Thanks @jochen I actually did another clean test; loaded a brand new 2.1.0+62db7e0 OVA and then upgraded it using the procedure discussed here. Then exposed it to a live incoming stream and here’s my /var/log/graylog/server/current (link to the full gist is at the bottom of the post)

2017-09-28_16:58:13.40268 It looks like you are trying to access MongoDB over HTTP on the native driver port.
2017-09-28_16:58:23.72851 INFO  [CmdLineTool] Loaded plugin: Splunk Output 0.4.0 [com.graylog.splunk.output.SplunkOutputPlugin]
2017-09-28_16:58:23.72906 INFO  [CmdLineTool] Loaded plugin: Elastic Beats Input 2.3.1 [org.graylog.plugins.beats.BeatsInputPlugin]
2017-09-28_16:58:23.73971 INFO  [CmdLineTool] Loaded plugin: Collector 2.3.1 [org.graylog.plugins.collector.CollectorPlugin]
2017-09-28_16:58:23.74167 INFO  [CmdLineTool] Loaded plugin: Enterprise Integration Plugin 2.3.1 [org.graylog.plugins.enterprise_integration.EnterpriseIntegrationPlugin]
2017-09-28_16:58:23.74621 INFO  [CmdLineTool] Loaded plugin: MapWidgetPlugin 2.3.1 [org.graylog.plugins.map.MapWidgetPlugin]
2017-09-28_16:58:23.78201 INFO  [CmdLineTool] Loaded plugin: Pipeline Processor Plugin 2.3.1 [org.graylog.plugins.pipelineprocessor.ProcessorPlugin]
2017-09-28_16:58:23.79649 INFO  [CmdLineTool] Loaded plugin: Anonymous Usage Statistics 2.3.1 [org.graylog.plugins.usagestatistics.UsageStatsPlugin]
2017-09-28_16:58:23.79829 INFO  [CmdLineTool] Loaded plugin: SnmpPlugin 0.3.0 [org.graylog.snmp.SnmpPlugin]
2017-09-28_16:58:26.04790 INFO  [CmdLineTool] Running with JVM arguments: -Xms1g -Xmx1500m -XX:NewRatio=1 -XX:+ResizeTLAB -XX:+UseConcMarkSweepGC -XX:+CMSConcurrentMTEnabled -XX:+CMSClassUnloadingEnabled -XX:+UseParNewGC -XX:-OmitStackTraceInFastThrow -Dlog4j.configurationFile=file:///opt/graylog/conf/log4j2.xml -Djava.library.path=/opt/graylog/server/lib/sigar/ -

https://gist.github.com/bubba198/420d2b950f4f887a60a4696637b84c20


(Julien) #11

@bubba198 I had a bunch of the errors you experience. Apparently linked to the new ES 5.5 mapping formats. Here is how I solved it:

  • delete all custom mappings you had, making a copy in some file (i.e notepad)
  • rotate the deflector index
  • update your custom mapping to the new ES 5.5 format
  • post your new custom mappings
  • reboot

Can’t say it will solve all the issues but a bunch should be fixed that way. Hope it helps :slight_smile:


(system) #12

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.