Ubuntu 14.04, graylog after upgrade

Graylog Central (peer support)
1

Hi

Running ubuntu 14.04, upgraded

  • graylog to 2.4.3
  • elasticsearch to 5.6.8
  • mongodb-org to 3.6.3

Had to set path.data in elasticsearch.yml. The graylog-server does not respond. In the log it seems as if it is looping, end with “INFO [Version] HV000001: Hibernate Validator 5.1.3.Final”

From the elasticsearch-log:

[2018-03-22T14:28:24,180][WARN ][o.e.t.n.Netty4Transport  ] [37y1JQC] exception caught on transport layer [[id: 0x63f6d071, L:/0:0:0:0:0:0:0:1:9300 ! R:/0:0:0:0:0:0:0:1:60338]], closing connection
io.netty.handler.codec.DecoderException: java.io.StreamCorruptedException: invalid internal transport message format, got (d,a,d,a)

mongodb is running (when I try mongo --host 127.0.0.1:27017).

# curl -XGET 'localhost:9200/_cluster/health?pretty'
{
  "cluster_name" : "graylog",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 1,
  "number_of_data_nodes" : 1,
  "active_primary_shards" : 30,
  "active_shards" : 30,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 100.0
}

Any ideas?

2

Are you sure you’re running Graylog 2.4.3?

Because Hibernate Validator 5.1.3.Final was used by Graylog 1.2.x.

3

Ok. Where do change that?

4

You have to upgrade Graylog to a version which supports Elasticsearch 5.x (which is Graylog 2.3.0 and later) or downgrade to an Elasticsearch version supported by the Graylog version you’re running.

In general, please refer to the documentation for installation and upgrade instructions:
http://docs.graylog.org/en/2.4/pages/installation.html

5

I am running version 2.4.3

~# dpkg -l graylog-server
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name                            Version              Architecture         Description
+++-===============================-====================-====================-===================================================================
ii  graylog-server                  2.4.3-1              all                  Graylog server
6

Not according to the log snippet you’ve posted.

Maybe you want to provide some more details?

7

This is what is in the log (looping):

2018-03-23T10:59:40.962+01:00 INFO  [Version] HV000001: Hibernate Validator 5.1.3.Final
2018-03-23T10:59:45.426+01:00 INFO  [CmdLineTool] Loaded plugin: AWS plugins 2.4.3 [org.graylog.aws.plugin.AWSPlugin]
2018-03-23T10:59:45.428+01:00 INFO  [CmdLineTool] Loaded plugin: Elastic Beats Input 2.4.3 [org.graylog.plugins.beats.BeatsInputPlugin]
2018-03-23T10:59:45.439+01:00 INFO  [CmdLineTool] Loaded plugin: CEF Input 2.4.3 [org.graylog.plugins.cef.CEFInputPlugin]
2018-03-23T10:59:45.440+01:00 INFO  [CmdLineTool] Loaded plugin: Collector 2.4.3 [org.graylog.plugins.collector.CollectorPlugin]
2018-03-23T10:59:45.441+01:00 INFO  [CmdLineTool] Loaded plugin: Enterprise Integration Plugin 2.4.3 [org.graylog.plugins.enterprise_integration.EnterpriseIntegrationPlugin]
2018-03-23T10:59:45.442+01:00 INFO  [CmdLineTool] Loaded plugin: MapWidgetPlugin 2.4.3 [org.graylog.plugins.map.MapWidgetPlugin]
2018-03-23T10:59:45.443+01:00 INFO  [CmdLineTool] Loaded plugin: NetFlow Plugin 2.4.3 [org.graylog.plugins.netflow.NetFlowPlugin]
2018-03-23T10:59:45.452+01:00 INFO  [CmdLineTool] Loaded plugin: Pipeline Processor Plugin 2.4.3 [org.graylog.plugins.pipelineprocessor.ProcessorPlugin]
2018-03-23T10:59:45.453+01:00 INFO  [CmdLineTool] Loaded plugin: Threat Intelligence Plugin 2.4.3 [org.graylog.plugins.threatintel.ThreatIntelPlugin]
2018-03-23T10:59:45.833+01:00 INFO  [CmdLineTool] Running with JVM arguments: -Xms2g -Xmx2g -XX:NewRatio=1 -XX:+ResizeTLAB -XX:+UseConcMarkSweepGC -XX:+CMSConcurrentMTEnabled -XX:+CMSClassUnloadingEnabled -XX:+UseParNewGC -XX:-OmitStackTraceInFastThrow -Dlog4j.configurationFile=file:///etc/graylog/server/log4j2.xml -Djava.library.path=/usr/share/graylog-server/lib/sigar -Dgraylog2.installation_source=deb
2018-03-23T10:59:46.066+01:00 INFO  [Version] HV000001: Hibernate Validator 5.1.3.Final
8

server.conf:

is_master = true
node_id_file = /etc/graylog/server/node-id
password_secret = 'x'
root_password_sha2 = y
root_timezone = Europe/Oslo
plugin_dir = /usr/share/graylog-server/plugin
rest_listen_uri = http://192.168.95.27:12900/
rest_transport_uri = http://192.168.95.27:12900/
web_enable = true
web_listen_uri = http://192.168.95.27:9000/
rotation_strategy = count
elasticsearch_max_docs_per_index = 20000000
rotation_strategy = count
elasticsearch_max_docs_per_index = 20000000
elasticsearch_max_number_of_indices = 20
retention_strategy = delete
elasticsearch_max_number_of_indices = 20
retention_strategy = delete
elasticsearch_shards = 1
elasticsearch_replicas = 0
elasticsearch_index_prefix = graylog
allow_leading_wildcard_searches = false
allow_highlighting = false
elasticsearch_cluster_name = graylog
elasticsearch_discovery_zen_ping_unicast_hosts = 127.0.0.1:9300
elasticsearch_transport_tcp_port = 9300
elasticsearch_discovery_zen_ping_multicast_enabled = false
elasticsearch_analyzer = standard
output_batch_size = 500
output_flush_interval = 1
output_fault_count_threshold = 5
output_fault_penalty_seconds = 30
processbuffer_processors = 5
outputbuffer_processors = 3
processor_wait_strategy = blocking
ring_size = 65536
inputbuffer_ring_size = 65536
inputbuffer_processors = 2
inputbuffer_wait_strategy = blocking
message_journal_enabled = true
message_journal_dir = /var/lib/graylog-server/journal
message_journal_max_age = 24h
message_journal_max_size = 10gb
lb_recognition_period_seconds = 3
mongodb_uri = mongodb://localhost/graylog
mongodb_max_connections = 1000
mongodb_threads_allowed_to_block_multiplier = 5
rules_file = /etc/graylog/server/rules.drl
transport_email_enabled = true
transport_email_hostname = zzz.zzzzzz.zz
transport_email_port = 25
transport_email_use_tls = false
transport_email_use_ssl = false
transport_email_subject_prefix = [graylog]
transport_email_from_email = graylog@spacetec.no
transport_email_web_interface_url = https://xxx.yyy.zz:9000
content_packs_dir = /usr/share/graylog-server/contentpacks
content_packs_auto_load = grok-patterns.json

elasticsearch.yml:

cluster.name: graylog
path.data: /var/lib/elasticsearch/graylog
9

Try running Graylog in debug mode (adding --debug or -d to the start script).

Also, what’s in the logs of your Elasticsearch node?

And last but not least, please read the upgrade notes for Graylog 2.3.x and 2.4.x, since you seem to be missing some configuration settings:

10

I tried to run with debug. What should I be loooking for?

elasticsearch log (some of it):

[2018-03-23T08:20:02,126][INFO ][o.e.n.Node ] [37y1JQC] started
[2018-03-23T08:20:04,260][INFO ][o.e.c.s.ClusterSettings ] [37y1JQC] updating [cluster.routing.allocation.enable] from [ALL] to [all]
[2018-03-23T08:20:05,419][INFO ][o.e.g.GatewayService ] [37y1JQC] recovered [30] indices into cluster_state
[2018-03-23T08:20:12,179][INFO ][o.e.c.r.a.AllocationService] [37y1JQC] Cluster health status changed from [RED] to [GREEN] (reason: [shards started [[graylog_179][0], [graylog_178][0], [graylog_177][0]] …]).
[2018-03-23T08:20:33,860][INFO ][o.e.n.Node ] [37y1JQC] stopping …
[2018-03-23T08:20:34,045][INFO ][o.e.n.Node ] [37y1JQC] stopped
[2018-03-23T08:20:34,046][INFO ][o.e.n.Node ] [37y1JQC] closing …
[2018-03-23T08:20:34,056][INFO ][o.e.n.Node ] [37y1JQC] closed
[2018-03-23T08:20:37,184][INFO ][o.e.n.Node ] [] initializing …
[2018-03-23T08:20:37,350][INFO ][o.e.e.NodeEnvironment ] [37y1JQC] using [1] data paths, mounts [[/disk1 (/dev/sda1)]], net usable_space [533.6gb], net total_space [1tb], spins? [possibly], types [ext4]
[2018-03-23T08:20:37,350][INFO ][o.e.e.NodeEnvironment ] [37y1JQC] heap size [2gb], compressed ordinary object pointers [true]
[2018-03-23T08:20:37,426][INFO ][o.e.n.Node ] node name [37y1JQC] derived from node ID [37y1JQCvQsueEvMYhTXAwg]; set [node.name] to override
[2018-03-23T08:20:37,426][INFO ][o.e.n.Node ] version[5.6.8], pid[6452], build[688ecce/2018-02-16T16:46:30.010Z], OS[Linux/3.13.0-143-generic/amd64], JVM[Oracle Corporation/Java HotSpot™ 64-Bit Server VM/9.0.4/9.0.4+11]
[2018-03-23T08:20:37,426][INFO ][o.e.n.Node ] JVM arguments [-Xms2g, -Xmx2g, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -Djdk.io.permissionsUseCanonicalPath=true, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j.skipJansi=true, -XX:+HeapDumpOnOutOfMemoryError, -Des.path.home=/usr/share/elasticsearch]
[2018-03-23T08:20:38,759][INFO ][o.e.p.PluginsService ] [37y1JQC] loaded module [aggs-matrix-stats]
[2018-03-23T08:20:38,759][INFO ][o.e.p.PluginsService ] [37y1JQC] loaded module [ingest-common]
[2018-03-23T08:20:38,759][INFO ][o.e.p.PluginsService ] [37y1JQC] loaded module [lang-expression]
[2018-03-23T08:20:38,759][INFO ][o.e.p.PluginsService ] [37y1JQC] loaded module [lang-groovy]
[2018-03-23T08:20:38,759][INFO ][o.e.p.PluginsService ] [37y1JQC] loaded module [lang-mustache]
[2018-03-23T08:20:38,760][INFO ][o.e.p.PluginsService ] [37y1JQC] loaded module [lang-painless]
[2018-03-23T08:20:38,760][INFO ][o.e.p.PluginsService ] [37y1JQC] loaded module [parent-join]
[2018-03-23T08:20:38,760][INFO ][o.e.p.PluginsService ] [37y1JQC] loaded module [percolator]
[2018-03-23T08:20:38,760][INFO ][o.e.p.PluginsService ] [37y1JQC] loaded module [reindex]
[2018-03-23T08:20:38,760][INFO ][o.e.p.PluginsService ] [37y1JQC] loaded module [transport-netty3]
[2018-03-23T08:20:38,760][INFO ][o.e.p.PluginsService ] [37y1JQC] loaded module [transport-netty4]
[2018-03-23T08:20:38,760][INFO ][o.e.p.PluginsService ] [37y1JQC] no plugins loaded
[2018-03-23T08:20:40,892][INFO ][o.e.d.DiscoveryModule ] [37y1JQC] using discovery type [zen]
[2018-03-23T08:20:41,947][INFO ][o.e.n.Node ] initialized
[2018-03-23T08:20:41,947][INFO ][o.e.n.Node ] [37y1JQC] starting …
[2018-03-23T08:20:42,187][INFO ][o.e.t.TransportService ] [37y1JQC] publish_address {127.0.0.1:9300}, bound_addresses {[::1]:9300}, {127.0.0.1:9300}
[2018-03-23T08:20:45,289][INFO ][o.e.c.s.ClusterService ] [37y1JQC] new_master {37y1JQC}{37y1JQCvQsueEvMYhTXAwg}{XLEJWooyTzS9livsyTKPZQ}{127.0.0.1}{127.0.0.1:9300}, reason: zen-disco-elected-as-master ([0] nodes joined)
[2018-03-23T08:20:45,335][INFO ][o.e.h.n.Netty4HttpServerTransport] [37y1JQC] publish_address {127.0.0.1:9200}, bound_addresses {[::1]:9200}, {127.0.0.1:9200}
[2018-03-23T08:20:45,335][INFO ][o.e.n.Node ] [37y1JQC] started
[2018-03-23T08:20:47,687][INFO ][o.e.c.s.ClusterSettings ] [37y1JQC] updating [cluster.routing.allocation.enable] from [ALL] to [all]
[2018-03-23T08:20:48,890][INFO ][o.e.g.GatewayService ] [37y1JQC] recovered [30] indices into cluster_state
[2018-03-23T08:20:56,414][INFO ][o.e.c.r.a.AllocationService] [37y1JQC] Cluster health status changed from [RED] to [GREEN] (reason: [shards started [[graylog_177][0]] …]).

I saw some parameters that removed in 2.3, and removed it. No difference.

This is the plugin directory:

ll /usr/share/graylog-server/plugin/

total 37960
drwxr-xr-x 2 root root 4096 mars 23 09:05 ./
drwxr-xr-x 6 root root 4096 mars 23 09:05 …/
-rw-r–r-- 1 root root 13997198 jan. 24 23:30 graylog-plugin-aws-2.4.3.jar
-rw-r–r-- 1 root root 27034 jan. 24 23:30 graylog-plugin-beats-2.4.3.jar
-rw-r–r-- 1 root root 60153 jan. 24 23:30 graylog-plugin-cef-2.4.3.jar
-rw-r–r-- 1 root root 2971112 jan. 24 23:30 graylog-plugin-collector-2.4.3.jar
-rw-r–r-- 1 root root 4296855 jan. 24 23:30 graylog-plugin-enterprise-integration-2.4.3.jar
-rw-r–r-- 1 root root 6617711 jan. 24 23:30 graylog-plugin-map-widget-2.4.3.jar
-rw-r–r-- 1 root root 705987 jan. 24 23:30 graylog-plugin-netflow-2.4.3.jar
-rw-r–r-- 1 root root 5594731 jan. 24 23:30 graylog-plugin-pipeline-processor-2.4.3.jar
-rw-r–r-- 1 root root 4574666 jan. 24 23:30 graylog-plugin-threatintel-2.4.3.jar

11

Any error messages in the logs of your Graylog node (they might also come on DEBUG level).

Are Graylog and Elasticsearch running on the same machine?

12

A lot of info, this is one thing, perhaps:

2018-03-23T14:36:32.386+01:00 DEBUG [PluginProperties] No value found for attribute in JAR manifest of file </usr/share/graylog-server/plugin/graylog-plugin-netflow-2.4.3.jar>

Everything on the same server.

13

Please don’t just post arbitrary excerpts from the logs.

14

Should I post the whole log here?

15

I was not able to post what was logged.

16

@jochen

Could it be the java version?

java -version

java version “9.0.4”
Java™ SE Runtime Environment (build 9.0.4+11)
Java HotSpot™ 64-Bit Server VM (build 9.0.4+11, mixed mode)

I have tried a lot of different things now, still the same problem

17

Please use a pastebin service such as https://gist.github.com/ or https://0bin.net/.

Yes. Graylog 2.x doesn’t work with Java 9.

18

Thanks! That was the problem.

19

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.