Upgrade to v2.3 no out messages

Graylog Central (peer support)
1

After upgrade to graylog 2.3 there are no out messages. I also upgraded to elastic 5.

Logs show messages:
2017-08-07T14:14:24.579Z INFO [RebuildIndexRangesJob] Done calculating index ranges for 37 indices. Took 559590ms.
2017-08-07T14:14:24.579Z INFO [SystemJobManager] SystemJob <69531000-7b79-11e7-b7c4-000d3a00be98> [org.graylog2.indexer.ranges.RebuildIndexRangesJob] finished in 559937ms.
2017-08-07T14:15:29.995Z ERROR [Messages] Caught exception during bulk indexing: java.net.SocketTimeoutException: Read timed out, retrying (attempt #1).
2017-08-07T14:15:32.010Z ERROR [Messages] Caught exception during bulk indexing: java.net.SocketTimeoutException: Read timed out, retrying (attempt #1).
2017-08-07T14:15:48.210Z INFO [Messages] Bulk indexing finally successful (attempt #2).
2017-08-07T14:16:00.280Z INFO [Messages] Bulk indexing finally successful (attempt #2).
2017-08-07T14:17:03.626Z ERROR [Messages] Caught exception during bulk indexing: java.net.SocketTimeoutException: Read timed out, retrying (attempt #1).
2017-08-07T14:17:04.910Z ERROR [Messages] Caught exception during bulk indexing: java.net.SocketTimeoutException: Read timed out, retrying (attempt #1).
2017-08-07T14:17:50.078Z INFO [Messages] Bulk indexing finally successful (attempt #2).
2017-08-07T14:17:53.882Z INFO [Messages] Bulk indexing finally successful (attempt #2).
2017-08-07T14:18:03.341Z WARN [NodePingThread] Did not find meta info of this node. Re-registering.
2017-08-07T14:18:53.906Z ERROR [Messages] Caught exception during bulk indexing: java.net.SocketTimeoutException: Read timed out, retrying (attempt #1).
2017-08-07T14:18:57.331Z ERROR [Messages] Caught exception during bulk indexing: java.net.SocketTimeoutException: Read timed out, retrying (attempt #1).
2017-08-07T14:19:30.009Z INFO [Messages] Bulk indexing finally successful (attempt #2).
2017-08-07T14:19:30.823Z INFO [Messages] Bulk indexing finally successful (attempt #2).
2017-08-07T14:23:51.422Z ERROR [Messages] Caught exception during bulk indexing: java.net.SocketTimeoutException: Read timed out, retrying (attempt #1).

Elastic just seems to have join node:
[2017-08-07T14:03:06,592][INFO ][o.e.c.s.ClusterService ] [ESEARCH01] added {{ESEARCH10}IP{IP:9300},}, reason: zen-disco-receive(from master [master {ESEARCH02}{IP}{IP:9300} committed version [406]])

Config of graylog:
is_master = False
node_id_file = /etc/graylog/server/node-id
root_timezone = UTC
plugin_dir = /usr/share/graylog-server/plugin
rest_listen_uri = h_t_t_pgraylogIP:9000/api/
rest_enable_cors = True
web_listen_uri = h_t_t_pgraylogIP:9000
web_endpoint_uri = h_t_t_pgraylogIP:9000/api/
web_enable_cors = True
web_enable_gzip = True
elasticsearch_hosts = h_t_t_pelastic1:9200,h_t_t_pelastic2:9200,h_t_t_pelastic3:9200,h_t_t_pelastic4:9200
elasticsearch_max_total_connections = 50
elasticsearch_max_total_connections_per_route = 10
elasticsearch_discovery_enabled = true
retention_strategy = delete
allow_leading_wildcard_searches = false
allow_highlighting = false
output_batch_size = 40000
output_flush_interval = 1
output_fault_count_threshold = 5
output_fault_penalty_seconds = 30
processbuffer_processors = 5
outputbuffer_processors = 4
processor_wait_strategy = blocking
ring_size = 65536
inputbuffer_ring_size = 65536
inputbuffer_processors = 2
inputbuffer_wait_strategy = blocking
message_journal_enabled = true
message_journal_dir = /var/lib/graylog-server/journal
message_journal_max_age = 12h
message_journal_max_size = 10gb
message_journal_flush_age = 1m
message_journal_flush_interval = 1000000
message_journal_segment_age = 1h
message_journal_segment_size = 100mb
lb_recognition_period_seconds = 3
mongodb_uri = mongodb://graylog:CONECTION_STRING?replicaSet="graylog"
mongodb_max_connections = 100
mongodb_threads_allowed_to_block_multiplier = 5
content_packs_dir = /usr/share/graylog-server/contentpacks
content_packs_auto_load = grok-patterns.json
proxied_requests_thread_pool_size = 32

Please note h_t_t_p is only because of forum limitations for new users.

2

Triggered manual recalculation of indexes:
2017-08-07T16:48:05+02:00 551f9f49 / Done calculating index ranges for 59 indices. Took 1834585ms.

3

Elastic config:
bootstrap.memory_lock: true
cluster.name: "Elastic-cluster"
discovery.zen.ping.unicast.hosts: [“GRAYLOG01”,“GRAYLOG02”,“GRAYLOG03”,“ESEARCH01”,“ESEARCH02”,“ESEARCH10”]
http.port: 9200
node.master: true
node.data: true
transport.tcp.port: 9300
network.host: [“ESEARCH01”,“IP”,“127.0.0.1”]
network.publish_host: [“ESEARCH01”,“IP”]
node.name: {HOSTNAME} path.conf: /etc/elasticsearch path.data: /mnt/data/elasticsearch/{HOSTNAME}
path.logs: /var/log/elasticsearch/${HOSTNAME}

4

You can prevent this by properly formatting your text snippets.

Example:

```
TEXT
```

Also see Markdown Reference

5

These logs aren’t complete and it’s certainly only from one of the three Graylog nodes. Additionally, you’re missing the logs of the Elasticsearch nodes.

Graylog 2.3.0 and later doesn’t use an embedded Elasticsearch node to join the Elasticsearch cluster. Please refer to http://docs.graylog.org/en/2.3/pages/upgrade/graylog-2.3.html#graylog-switches-to-elasticsearch-http-client for details.

This looks wrong. Please refer to Network Settings | Elasticsearch Reference [5.5] | Elastic for details.

6

Dumped all config to the default in v2.3.
Did not modify the elastic search and rebooted all the graylog nodes, now I see output of the nodes.

Thank you for your support

7

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.