Upgrade from 2.2x to 2.3 broke searches

Graylog Central (peer support)
1

This is what I get when I try to perform a search

Could not execute search
There was an error executing your search. Please check your Graylog server logs for more information.

Error Message:
Unable to perform search query. Index not found for query: 1_0. Try recalculating your index ranges.
Details:
Index not found for query: 1_0. Try recalculating your index ranges.
Search status code:
500
Search response:
cannot GET http://192.168.8.50:12900/search/universal/relative?query=*&range=300&limit=150&sort=timestamp%3Adesc (500)

I tried to recalculate the index range and this is what I get

grayloguser@cot-graylog02:/var/log/graylog-server$ sudo curl -XPOST http://127.0.0.1:9000/api/system/indices/ranges/rebuild
curl: (7) Failed to connect to 127.0.0.1 port 9000: Connection refused

2

What’s in the logs of your Elasticsearch and Graylog nodes?
http://docs.graylog.org/en/2.3/pages/configuration/file_location.html

3

There isn’t anything interesting in either log

grayloguser@cot-graylog02:/var/log/elasticsearch$ tail graylog.log
[2017-08-21 09:11:50,400][INFO ][node                     ] [Cerise] starting ...
[2017-08-21 09:11:50,523][INFO ][transport                ] [Cerise] publish_address {127.0.0.1:9300}, bound_addresses {[::1]:9300}, {127.0.0.1:9300}
[2017-08-21 09:11:50,538][INFO ][discovery                ] [Cerise] graylog/joet24PRTf-Q6DhgBea34A
[2017-08-21 09:11:53,665][INFO ][cluster.service          ] [Cerise] new_master {Cerise}{joet24PRTf-Q6DhgBea34A}{127.0.0.1}{127.0.0.1:9300}, reason: zen-disco-join(elected_as_master, [0] joins received)
[2017-08-21 09:11:53,698][INFO ][http                     ] [Cerise] publish_address {127.0.0.1:9200}, bound_addresses {[::1]:9200}, {127.0.0.1:9200}
[2017-08-21 09:11:53,699][INFO ][node                     ] [Cerise] started
[2017-08-21 09:11:54,027][INFO ][gateway                  ] [Cerise] recovered [16] indices into cluster_state
[2017-08-21 09:12:10,702][INFO ][cluster.routing.allocation] [Cerise] Cluster health status changed from [RED] to [GREEN] (reason: [shards started [[graylog_1][0], [graylog_1][0]] ...]).
[2017-08-21 09:46:56,780][INFO ][cluster.metadata         ] [Cerise] [cot_0] creating index, cause [api], templates [cot-template], shards [4]/[0], mappings [message]
[2017-08-21 09:46:56,889][INFO ][cluster.routing.allocation] [Cerise] Cluster health status changed from [RED] to [GREEN] (reason: [shards started [[cot_0][0], [cot_0][0]] ...]).

grayloguser@cot-graylog02:/var/log/graylog-server$ tail server.log
2017-08-21T09:57:22.429-07:00 INFO  [RebuildIndexRangesJob] Created ranges for index graylog_12: MongoIndexRange{id=null, indexName=graylog_12, begin=2017-05-12T23:17:01.033Z, end=2017-07-20T07:03:06.000Z, calculatedAt=2017-08-21T16:57:18.305Z, calculationDuration=4122, streamIds=[000000000000000000000001, 57c85a713628170a8a05f06f, 57d96769362817760f375d82]}
2017-08-21T09:57:26.336-07:00 INFO  [MongoIndexRangeService] Calculated range of [graylog_3] in [3904ms].
2017-08-21T09:57:26.338-07:00 INFO  [RebuildIndexRangesJob] Created ranges for index graylog_3: MongoIndexRange{id=null, indexName=graylog_3, begin=2016-10-04T20:17:01.325Z, end=2017-04-22T21:58:53.000Z, calculatedAt=2017-08-21T16:57:22.432Z, calculationDuration=3904, streamIds=[000000000000000000000001, 57c85a713628170a8a05f06f, 57d96769362817760f375d82]}
2017-08-21T09:57:30.320-07:00 INFO  [MongoIndexRangeService] Calculated range of [graylog_11] in [3980ms].
2017-08-21T09:57:30.324-07:00 INFO  [RebuildIndexRangesJob] Created ranges for index graylog_11: MongoIndexRange{id=null, indexName=graylog_11, begin=2017-01-11T16:28:31.000Z, end=2017-07-10T07:07:14.000Z, calculatedAt=2017-08-21T16:57:26.340Z, calculationDuration=3980, streamIds=[000000000000000000000001, 57c85a713628170a8a05f06f, 57d96769362817760f375d82, 5876631ad9c77d0a9f89e942]}
2017-08-21T09:57:34.168-07:00 INFO  [MongoIndexRangeService] Calculated range of [graylog_2] in [3842ms].
2017-08-21T09:57:34.175-07:00 INFO  [RebuildIndexRangesJob] Created ranges for index graylog_2: MongoIndexRange{id=null, indexName=graylog_2, begin=2016-10-01T16:39:01.120Z, end=2017-04-14T01:04:57.000Z, calculatedAt=2017-08-21T16:57:30.325Z, calculationDuration=3842, streamIds=[000000000000000000000001, 57c85a713628170a8a05f06f, 57d96769362817760f375d82]}
2017-08-21T09:57:34.175-07:00 INFO  [RebuildIndexRangesJob] Done calculating index ranges for 16 indices. Took 68148ms.
2017-08-21T09:57:34.178-07:00 INFO  [SystemJobManager] SystemJob <ab49a640-8691-11e7-870d-56d26345c40a> [org.graylog2.indexer.ranges.RebuildIndexRangesJob] finished in 68154ms.
2017-08-21T10:20:11.654-07:00 INFO  [IndexRangesCleanupPeriodical] Removing index range information for unavailable indices: [1_0]
4

I can drop a bigger chunk of the log file if you need me to. Just let me know what you want to see but I don’t see anything interesting

5

/var/log/graylog-server/server.log

2017-08-21T09:20:14.933-07:00 INFO  [ServerBootstrap] Services started, startup times in ms: {OutputSetupService [RUNNING]=42, BufferSynchronizerService [RUNNING]=66, KafkaJournal [RUNNING]=84, InputSetupService [RUNNING]=129, StreamCacheService [RUNNING]=222, JournalReader [RUNNING]=223, LookupTableService [RUNNING]=230, ConfigurationEtagService [RUNNING]=250, PeriodicalsService [RUNNING]=268, JerseyService [RUNNING]=18383}
2017-08-21T09:20:14.933-07:00 INFO  [ServiceManagerListener] Services are healthy
2017-08-21T09:20:14.934-07:00 INFO  [InputSetupService] Triggering launching persisted inputs, node transitioned from Uninitialized [LB:DEAD] to Running [LB:ALIVE]
2017-08-21T09:20:14.955-07:00 INFO  [ServerBootstrap] Graylog server up and running.
2017-08-21T09:20:15.025-07:00 INFO  [InputStateListener] Input [GELF UDP/57a5152036281721679b6834] is now STARTING
2017-08-21T09:20:15.069-07:00 INFO  [InputStateListener] Input [Raw/Plaintext UDP/58f13800cbf5701d856426f6] is now STARTING
2017-08-21T09:20:15.071-07:00 INFO  [InputStateListener] Input [Syslog UDP/57a5187c36281721679b6bdc] is now STARTING
2017-08-21T09:20:15.244-07:00 WARN  [NettyTransport] receiveBufferSize (SO_RCVBUF) for input SyslogUDPInput{title=Linux, type=org.graylog2.inputs.syslog.udp.SyslogUDPInput, nodeId=null} should be 262144 but is 212992.
2017-08-21T09:20:15.244-07:00 WARN  [NettyTransport] receiveBufferSize (SO_RCVBUF) for input RawUDPInput{title=Orwell, type=org.graylog2.inputs.raw.udp.RawUDPInput, nodeId=null} should be 262144 but is 212992.
2017-08-21T09:20:15.250-07:00 INFO  [InputStateListener] Input [Syslog UDP/57a5187c36281721679b6bdc] is now RUNNING
2017-08-21T09:20:15.253-07:00 INFO  [InputStateListener] Input [Raw/Plaintext UDP/58f13800cbf5701d856426f6] is now RUNNING
2017-08-21T09:20:15.253-07:00 WARN  [NettyTransport] receiveBufferSize (SO_RCVBUF) for input GELFUDPInput{title=Windows, type=org.graylog2.inputs.gelf.udp.GELFUDPInput, nodeId=99fb7c0e-bf36-46ef-a1d6-2527b2e3113d} should be 262144 but is 212992.
2017-08-21T09:20:15.312-07:00 INFO  [InputStateListener] Input [GELF UDP/57a5152036281721679b6834] is now RUNNING
2017-08-21T09:46:56.645-07:00 INFO  [MongoIndexSet] Did not find a deflector alias. Setting one up now.
2017-08-21T09:46:56.649-07:00 INFO  [MongoIndexSet] There is no index target to point to. Creating one now.
2017-08-21T09:46:56.657-07:00 INFO  [MongoIndexSet] Cycling from <none> to <cot_0>.
2017-08-21T09:46:56.657-07:00 INFO  [MongoIndexSet] Creating target index <cot_0>.
2017-08-21T09:46:56.734-07:00 INFO  [Indices] Successfully created index template cot-template
2017-08-21T09:46:56.850-07:00 INFO  [MongoIndexSet] Waiting for allocation of index <cot_0>.
2017-08-21T09:46:56.932-07:00 INFO  [MongoIndexSet] Index <cot_0> has been successfully allocated.
2017-08-21T09:46:56.932-07:00 INFO  [MongoIndexSet] Pointing index alias <cot_deflector> to new index <cot_0>.
2017-08-21T09:46:56.951-07:00 INFO  [MongoIndexSet] Successfully pointed index alias <cot_deflector> to index <cot_0>.
2017-08-21T09:53:25.339-07:00 INFO  [InputStateListener] Input [Raw/Plaintext UDP/58f13800cbf5701d856426f6] is now STOPPING
2017-08-21T09:53:25.343-07:00 INFO  [InputStateListener] Input [Raw/Plaintext UDP/58f13800cbf5701d856426f6] is now STOPPED
2017-08-21T09:53:25.344-07:00 INFO  [InputStateListener] Input [Raw/Plaintext UDP/58f13800cbf5701d856426f6] is now TERMINATED
2017-08-21T09:54:35.372-07:00 INFO  [InputStateListener] Input [GELF UDP/599b104b318edf0c35a17669] is now STARTING
2017-08-21T09:54:35.402-07:00 WARN  [NettyTransport] receiveBufferSize (SO_RCVBUF) for input GELFUDPInput{title=WinLogs-gelf, type=org.graylog2.inputs.gelf.udp.GELFUDPInput, nodeId=null} should be 1048576 but is 212992.
2017-08-21T09:54:35.403-07:00 INFO  [InputStateListener] Input [GELF UDP/599b104b318edf0c35a17669] is now RUNNING
2017-08-21T09:54:35.700-07:00 INFO  [connection] Opened connection [connectionId{localValue:11, serverValue:24}] to localhost:27017
2017-08-21T09:54:35.700-07:00 INFO  [connection] Opened connection [connectionId{localValue:10, serverValue:25}] to localhost:27017
2017-08-21T09:54:35.703-07:00 INFO  [InputStateListener] Input [GELF UDP/599b104b318edf0c35a17669] is now STOPPING
2017-08-21T09:54:35.707-07:00 INFO  [InputStateListener] Input [GELF UDP/599b104b318edf0c35a17669] is now STOPPED
2017-08-21T09:54:35.708-07:00 INFO  [InputStateListener] Input [GELF UDP/599b104b318edf0c35a17669] is now TERMINATED
2017-08-21T09:54:35.716-07:00 INFO  [InputStateListener] Input [GELF UDP/599b104b318edf0c35a17669] is now STARTING
2017-08-21T09:54:35.721-07:00 WARN  [NettyTransport] receiveBufferSize (SO_RCVBUF) for input GELFUDPInput{title=WinLogs-gelf, type=org.graylog2.inputs.gelf.udp.GELFUDPInput, nodeId=null} should be 1048576 but is 212992.
2017-08-21T09:54:35.728-07:00 INFO  [InputStateListener] Input [GELF UDP/599b104b318edf0c35a17669] is now RUNNING
2017-08-21T09:54:39.590-07:00 INFO  [InputStateListener] Input [GELF UDP/599b104f318edf0c35a17690] is now STARTING
2017-08-21T09:54:39.594-07:00 ERROR [NettyTransport] Error in Input [GELF UDP/599b104f318edf0c35a17690] (channel [id: 0x242d2c0e])
java.net.BindException: Address already in use
        at sun.nio.ch.Net.bind0(Native Method) ~[?:1.8.0_131]
        at sun.nio.ch.Net.bind(Net.java:433) ~[?:1.8.0_131]
        at sun.nio.ch.DatagramChannelImpl.bind(DatagramChannelImpl.java:691) ~[?:1.8.0_131]
        at sun.nio.ch.DatagramSocketAdaptor.bind(DatagramSocketAdaptor.java:91) ~[?:1.8.0_131]
        at org.jboss.netty.channel.socket.nio.NioDatagramPipelineSink.bind(NioDatagramPipelineSink.java:129) [graylog.jar:?]
        at org.jboss.netty.channel.socket.nio.NioDatagramPipelineSink.eventSunk(NioDatagramPipelineSink.java:77) [graylog.jar:?]
        at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:779) [graylog.jar:?]
        at org.jboss.netty.channel.SimpleChannelHandler.bindRequested(SimpleChannelHandler.java:299) [graylog.jar:?]
        at org.jboss.netty.channel.SimpleChannelHandler.handleDownstream(SimpleChannelHandler.java:265) [graylog.jar:?]
        at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:591) [graylog.jar:?]
        at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:784) [graylog.jar:?]
        at org.jboss.netty.channel.SimpleChannelHandler.bindRequested(SimpleChannelHandler.java:299) [graylog.jar:?]
        at org.jboss.netty.channel.SimpleChannelHandler.handleDownstream(SimpleChannelHandler.java:265) [graylog.jar:?]
        at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:591) [graylog.jar:?]
        at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:784) [graylog.jar:?]
        at org.jboss.netty.channel.SimpleChannelHandler.bindRequested(SimpleChannelHandler.java:299) [graylog.jar:?]
        at org.jboss.netty.channel.SimpleChannelHandler.handleDownstream(SimpleChannelHandler.java:265) [graylog.jar:?]
        at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:591) [graylog.jar:?]
        at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:582) [graylog.jar:?]
        at org.jboss.netty.channel.Channels.bind(Channels.java:561) [graylog.jar:?]
        at org.jboss.netty.channel.AbstractChannel.bind(AbstractChannel.java:197) [graylog.jar:?]
        at org.jboss.netty.bootstrap.ConnectionlessBootstrap.bind(ConnectionlessBootstrap.java:198) [graylog.jar:?]
        at org.graylog2.plugin.inputs.transports.NettyTransport.launch(NettyTransport.java:136) [graylog.jar:?]
        at org.graylog2.plugin.inputs.MessageInput.launch(MessageInput.java:153) [graylog.jar:?]
        at org.graylog2.shared.inputs.InputLauncher$1.run(InputLauncher.java:84) [graylog.jar:?]
        at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176) [graylog.jar:?]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_131]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_131]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_131]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_131]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131]
2017-08-21T09:54:39.607-07:00 ERROR [InputLauncher] The [org.graylog2.inputs.gelf.udp.GELFUDPInput] input with ID <599b104f318edf0c35a17690> misfired. Reason: Address already in use.
org.graylog2.plugin.inputs.MisfireException: org.graylog2.plugin.inputs.MisfireException: org.jboss.netty.channel.ChannelException: Failed to bind to: /0.0.0.0:5414
        at org.graylog2.plugin.inputs.MessageInput.launch(MessageInput.java:156) ~[graylog.jar:?]
        at org.graylog2.shared.inputs.InputLauncher$1.run(InputLauncher.java:84) [graylog.jar:?]
        at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176) [graylog.jar:?]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_131]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_131]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_131]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_131]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131]
Caused by: org.graylog2.plugin.inputs.MisfireException: org.jboss.netty.channel.ChannelException: Failed to bind to: /0.0.0.0:5414
        at org.graylog2.plugin.inputs.transports.NettyTransport.launch(NettyTransport.java:155) ~[graylog.jar:?]
        at org.graylog2.plugin.inputs.MessageInput.launch(MessageInput.java:153) ~[graylog.jar:?]
        ... 7 more
Caused by: org.jboss.netty.channel.ChannelException: Failed to bind to: /0.0.0.0:5414
        at org.jboss.netty.bootstrap.ConnectionlessBootstrap.bind(ConnectionlessBootstrap.java:204) ~[graylog.jar:?]
        at org.graylog2.plugin.inputs.transports.NettyTransport.launch(NettyTransport.java:136) ~[graylog.jar:?]
        at org.graylog2.plugin.inputs.MessageInput.launch(MessageInput.java:153) ~[graylog.jar:?]
        ... 7 more
Caused by: java.net.BindException: Address already in use
        at sun.nio.ch.Net.bind0(Native Method) ~[?:1.8.0_131]
        at sun.nio.ch.Net.bind(Net.java:433) ~[?:1.8.0_131]
        at sun.nio.ch.DatagramChannelImpl.bind(DatagramChannelImpl.java:691) ~[?:1.8.0_131]
        at sun.nio.ch.DatagramSocketAdaptor.bind(DatagramSocketAdaptor.java:91) ~[?:1.8.0_131]
        at org.jboss.netty.channel.socket.nio.NioDatagramPipelineSink.bind(NioDatagramPipelineSink.java:129) ~[graylog.jar:?]
        at org.jboss.netty.channel.socket.nio.NioDatagramPipelineSink.eventSunk(NioDatagramPipelineSink.java:77) ~[graylog.jar:?]
        at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:779) ~[graylog.jar:?]
        at org.jboss.netty.channel.SimpleChannelHandler.bindRequested(SimpleChannelHandler.java:299) ~[graylog.jar:?]
        at org.jboss.netty.channel.SimpleChannelHandler.handleDownstream(SimpleChannelHandler.java:265) ~[graylog.jar:?]
        at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:591) ~[graylog.jar:?]
        at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:784) ~[graylog.jar:?]
        at org.jboss.netty.channel.SimpleChannelHandler.bindRequested(SimpleChannelHandler.java:299) ~[graylog.jar:?]
        at org.jboss.netty.channel.SimpleChannelHandler.handleDownstream(SimpleChannelHandler.java:265) ~[graylog.jar:?]
        at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:591) ~[graylog.jar:?]
        at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:784) ~[graylog.jar:?]
        at org.jboss.netty.channel.SimpleChannelHandler.bindRequested(SimpleChannelHandler.java:299) ~[graylog.jar:?]
        at org.jboss.netty.channel.SimpleChannelHandler.handleDownstream(SimpleChannelHandler.java:265) ~[graylog.jar:?]
        at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:591) ~[graylog.jar:?]
        at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:582) ~[graylog.jar:?]
        at org.jboss.netty.channel.Channels.bind(Channels.java:561) ~[graylog.jar:?]
        at org.jboss.netty.channel.AbstractChannel.bind(AbstractChannel.java:197) ~[graylog.jar:?]
        at org.jboss.netty.bootstrap.ConnectionlessBootstrap.bind(ConnectionlessBootstrap.java:198) ~[graylog.jar:?]
        at org.graylog2.plugin.inputs.transports.NettyTransport.launch(NettyTransport.java:136) ~[graylog.jar:?]
        at org.graylog2.plugin.inputs.MessageInput.launch(MessageInput.java:153) ~[graylog.jar:?]
        ... 7 more
2017-08-21T09:54:39.610-07:00 ERROR [graylog-eventbus] Exception thrown by subscriber method inputStateChanged(org.graylog2.plugin.events.inputs.IOStateChangedEvent) on subscriber org.graylog2.inputs.InputStateListener@35d4035f when dispatching event: IOStateChangedEvent{oldState=STARTING, newState=FAILED, changedState=InputState{stoppable=GELFUDPInput{title=WinLogs-gelf, type=org.graylog2.inputs.gelf.udp.GELFUDPInput, nodeId=null}, state=FAILED, startedAt=2017-08-21T16:54:39.588Z, detailedMessage='null'}}
java.lang.NullPointerException: null
        at java.util.Objects.requireNonNull(Objects.java:203) ~[?:1.8.0_131]
        at org.graylog2.shared.system.activities.Activity.<init>(Activity.java:34) ~[graylog.jar:?]
        at org.graylog2.inputs.InputStateListener.inputStateChanged(InputStateListener.java:57) ~[graylog.jar:?]
        at sun.reflect.GeneratedMethodAccessor252.invoke(Unknown Source) ~[?:?]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_131]
        at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_131]
        at com.google.common.eventbus.Subscriber.invokeSubscriberMethod(Subscriber.java:91) ~[graylog.jar:?]
        at com.google.common.eventbus.Subscriber$SynchronizedSubscriber.invokeSubscriberMethod(Subscriber.java:150) ~[graylog.jar:?]
        at com.google.common.eventbus.Subscriber$1.run(Subscriber.java:76) [graylog.jar:?]
        at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176) [graylog.jar:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_131]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_131]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131]
2017-08-21T09:54:39.661-07:00 INFO  [InputStateListener] Input [GELF UDP/599b104f318edf0c35a17690] is now STOPPING
2017-08-21T09:54:39.673-07:00 INFO  [InputStateListener] Input [GELF UDP/599b104f318edf0c35a17690] is now STOPPED
2017-08-21T09:54:39.674-07:00 INFO  [InputStateListener] Input [GELF UDP/599b104f318edf0c35a17690] is now TERMINATED
2017-08-21T09:54:39.676-07:00 INFO  [InputStateListener] Input [GELF UDP/599b104f318edf0c35a17690] is now STARTING
2017-08-21T09:54:39.724-07:00 ERROR [NettyTransport] Error in Input [GELF UDP/599b104f318edf0c35a17690] (channel [id: 0xbfb085f6])
java.net.BindException: Address already in use
        at sun.nio.ch.Net.bind0(Native Method) ~[?:1.8.0_131]
        at sun.nio.ch.Net.bind(Net.java:433) ~[?:1.8.0_131]
        at sun.nio.ch.DatagramChannelImpl.bind(DatagramChannelImpl.java:691) ~[?:1.8.0_131]
        at sun.nio.ch.DatagramSocketAdaptor.bind(DatagramSocketAdaptor.java:91) ~[?:1.8.0_131]
        at org.jboss.netty.channel.socket.nio.NioDatagramPipelineSink.bind(NioDatagramPipelineSink.java:129) [graylog.jar:?]
        at org.jboss.netty.channel.socket.nio.NioDatagramPipelineSink.eventSunk(NioDatagramPipelineSink.java:77) [graylog.jar:?]
        at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:779) [graylog.jar:?]
        at org.jboss.netty.channel.SimpleChannelHandler.bindRequested(SimpleChannelHandler.java:299) [graylog.jar:?]
        at org.jboss.netty.channel.SimpleChannelHandler.handleDownstream(SimpleChannelHandler.java:265) [graylog.jar:?]
        at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:591) [graylog.jar:?]
        at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:784) [graylog.jar:?]
        at org.jboss.netty.channel.SimpleChannelHandler.bindRequested(SimpleChannelHandler.java:299) [graylog.jar:?]
        at org.jboss.netty.channel.SimpleChannelHandler.handleDownstream(SimpleChannelHandler.java:265) [graylog.jar:?]
        at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:591) [graylog.jar:?]
        at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:784) [graylog.jar:?]
        at org.jboss.netty.channel.SimpleChannelHandler.bindRequested(SimpleChannelHandler.java:299) [graylog.jar:?]
        at org.jboss.netty.channel.SimpleChannelHandler.handleDownstream(SimpleChannelHandler.java:265) [graylog.jar:?]
        at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:591) [graylog.jar:?]
        at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:582) [graylog.jar:?]
        at org.jboss.netty.channel.Channels.bind(Channels.java:561) [graylog.jar:?]
        at org.jboss.netty.channel.AbstractChannel.bind(AbstractChannel.java:197) [graylog.jar:?]
        at org.jboss.netty.bootstrap.ConnectionlessBootstrap.bind(ConnectionlessBootstrap.java:198) [graylog.jar:?]
        at org.graylog2.plugin.inputs.transports.NettyTransport.launch(NettyTransport.java:136) [graylog.jar:?]
        at org.graylog2.plugin.inputs.MessageInput.launch(MessageInput.java:153) [graylog.jar:?]
        at org.graylog2.shared.inputs.InputLauncher$1.run(InputLauncher.java:84) [graylog.jar:?]
        at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176) [graylog.jar:?]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_131]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_131]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_131]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_131]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131]
2017-08-21T09:54:39.726-07:00 ERROR [InputLauncher] The [org.graylog2.inputs.gelf.udp.GELFUDPInput] input with ID <599b104f318edf0c35a17690> misfired. Reason: Address already in use.
org.graylog2.plugin.inputs.MisfireException: org.graylog2.plugin.inputs.MisfireException: org.jboss.netty.channel.ChannelException: Failed to bind to: /0.0.0.0:5414
        at org.graylog2.plugin.inputs.MessageInput.launch(MessageInput.java:156) ~[graylog.jar:?]
        at org.graylog2.shared.inputs.InputLauncher$1.run(InputLauncher.java:84) [graylog.jar:?]
        at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176) [graylog.jar:?]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_131]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_131]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_131]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_131]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131]
Caused by: org.graylog2.plugin.inputs.MisfireException: org.jboss.netty.channel.ChannelException: Failed to bind to: /0.0.0.0:5414
        at org.graylog2.plugin.inputs.transports.NettyTransport.launch(NettyTransport.java:155) ~[graylog.jar:?]
        at org.graylog2.plugin.inputs.MessageInput.launch(MessageInput.java:153) ~[graylog.jar:?]
        ... 7 more
Caused by: org.jboss.netty.channel.ChannelException: Failed to bind to: /0.0.0.0:5414
        at org.jboss.netty.bootstrap.ConnectionlessBootstrap.bind(ConnectionlessBootstrap.java:204) ~[graylog.jar:?]
        at org.graylog2.plugin.inputs.transports.NettyTransport.launch(NettyTransport.java:136) ~[graylog.jar:?]
        at org.graylog2.plugin.inputs.MessageInput.launch(MessageInput.java:153) ~[graylog.jar:?]
        ... 7 more
Caused by: java.net.BindException: Address already in use
        at sun.nio.ch.Net.bind0(Native Method) ~[?:1.8.0_131]
        at sun.nio.ch.Net.bind(Net.java:433) ~[?:1.8.0_131]
        at sun.nio.ch.DatagramChannelImpl.bind(DatagramChannelImpl.java:691) ~[?:1.8.0_131]
        at sun.nio.ch.DatagramSocketAdaptor.bind(DatagramSocketAdaptor.java:91) ~[?:1.8.0_131]
        at org.jboss.netty.channel.socket.nio.NioDatagramPipelineSink.bind(NioDatagramPipelineSink.java:129) ~[graylog.jar:?]
        at org.jboss.netty.channel.socket.nio.NioDatagramPipelineSink.eventSunk(NioDatagramPipelineSink.java:77) ~[graylog.jar:?]
        at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:779) ~[graylog.jar:?]
        at org.jboss.netty.channel.SimpleChannelHandler.bindRequested(SimpleChannelHandler.java:299) ~[graylog.jar:?]
        at org.jboss.netty.channel.SimpleChannelHandler.handleDownstream(SimpleChannelHandler.java:265) ~[graylog.jar:?]
        at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:591) ~[graylog.jar:?]
        at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:784) ~[graylog.jar:?]
        at org.jboss.netty.channel.SimpleChannelHandler.bindRequested(SimpleChannelHandler.java:299) ~[graylog.jar:?]
        at org.jboss.netty.channel.SimpleChannelHandler.handleDownstream(SimpleChannelHandler.java:265) ~[graylog.jar:?]
        at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:591) ~[graylog.jar:?]
        at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:784) ~[graylog.jar:?]
        at org.jboss.netty.channel.SimpleChannelHandler.bindRequested(SimpleChannelHandler.java:299) ~[graylog.jar:?]
        at org.jboss.netty.channel.SimpleChannelHandler.handleDownstream(SimpleChannelHandler.java:265) ~[graylog.jar:?]
        at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:591) ~[graylog.jar:?]
        at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:582) ~[graylog.jar:?]
        at org.jboss.netty.channel.Channels.bind(Channels.java:561) ~[graylog.jar:?]
        at org.jboss.netty.channel.AbstractChannel.bind(AbstractChannel.java:197) ~[graylog.jar:?]
        at org.jboss.netty.bootstrap.ConnectionlessBootstrap.bind(ConnectionlessBootstrap.java:198) ~[graylog.jar:?]
        at org.graylog2.plugin.inputs.transports.NettyTransport.launch(NettyTransport.java:136) ~[graylog.jar:?]
        at org.graylog2.plugin.inputs.MessageInput.launch(MessageInput.java:153) ~[graylog.jar:?]
        ... 7 more
2017-08-21T09:54:39.730-07:00 ERROR [graylog-eventbus] Exception thrown by subscriber method inputStateChanged(org.graylog2.plugin.events.inputs.IOStateChangedEvent) on subscriber org.graylog2.inputs.InputStateListener@35d4035f when dispatching event: IOStateChangedEvent{oldState=STARTING, newState=FAILED, changedState=InputState{stoppable=GELFUDPInput{title=WinLogs-gelf, type=org.graylog2.inputs.gelf.udp.GELFUDPInput, nodeId=null}, state=FAILED, startedAt=2017-08-21T16:54:39.674Z, detailedMessage='null'}}
java.lang.NullPointerException: null
        at java.util.Objects.requireNonNull(Objects.java:203) ~[?:1.8.0_131]
        at org.graylog2.shared.system.activities.Activity.<init>(Activity.java:34) ~[graylog.jar:?]
        at org.graylog2.inputs.InputStateListener.inputStateChanged(InputStateListener.java:57) ~[graylog.jar:?]
        at sun.reflect.GeneratedMethodAccessor252.invoke(Unknown Source) ~[?:?]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_131]
        at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_131]
        at com.google.common.eventbus.Subscriber.invokeSubscriberMethod(Subscriber.java:91) ~[graylog.jar:?]
        at com.google.common.eventbus.Subscriber$SynchronizedSubscriber.invokeSubscriberMethod(Subscriber.java:150) ~[graylog.jar:?]
        at com.google.common.eventbus.Subscriber$1.run(Subscriber.java:76) [graylog.jar:?]
        at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176) [graylog.jar:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_131]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_131]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131]
6

/var/log/elasticsearch/graylog.log

[2017-08-21 09:11:47,781][INFO ][env                      ] [Cerise] heap size [1007.3mb], compressed ordinary object pointers [true]
[2017-08-21 09:11:50,400][INFO ][node                     ] [Cerise] initialized
[2017-08-21 09:11:50,400][INFO ][node                     ] [Cerise] starting ...
[2017-08-21 09:11:50,523][INFO ][transport                ] [Cerise] publish_address {127.0.0.1:9300}, bound_addresses {[::1]:9300}, {127.0.0.1:9300}
[2017-08-21 09:11:50,538][INFO ][discovery                ] [Cerise] graylog/joet24PRTf-Q6DhgBea34A
[2017-08-21 09:11:53,665][INFO ][cluster.service          ] [Cerise] new_master {Cerise}{joet24PRTf-Q6DhgBea34A}{127.0.0.1}{127.0.0.1:9300}, reason: zen-disco-join(elected_as_master, [0] joins received)
[2017-08-21 09:11:53,698][INFO ][http                     ] [Cerise] publish_address {127.0.0.1:9200}, bound_addresses {[::1]:9200}, {127.0.0.1:9200}
[2017-08-21 09:11:53,699][INFO ][node                     ] [Cerise] started
[2017-08-21 09:11:54,027][INFO ][gateway                  ] [Cerise] recovered [16] indices into cluster_state
[2017-08-21 09:12:10,702][INFO ][cluster.routing.allocation] [Cerise] Cluster health status changed from [RED] to [GREEN] (reason: [shards started [[graylog_1][0], [graylog_1][0]] ...]).
[2017-08-21 09:46:56,780][INFO ][cluster.metadata         ] [Cerise] [cot_0] creating index, cause [api], templates [cot-template], shards [4]/[0], mappings [message]
[2017-08-21 09:46:56,889][INFO ][cluster.routing.allocation] [Cerise] Cluster health status changed from [RED] to [GREEN] (reason: [shards started [[cot_0][0], [cot_0][0]] ...]).
[2017-08-21 10:40:55,773][INFO ][cluster.metadata         ] [Cerise] [graylog_16] update_mapping [message]
[2017-08-21 10:41:00,640][INFO ][cluster.metadata         ] [Cerise] [graylog_16] update_mapping [message]
[2017-08-21 10:43:56,420][INFO ][node                     ] [Cerise] stopping ...
[2017-08-21 10:43:58,081][INFO ][node                     ] [Cerise] stopped
[2017-08-21 10:43:58,083][INFO ][node                     ] [Cerise] closing ...
[2017-08-21 10:43:58,127][INFO ][node                     ] [Cerise] closed
[2017-08-21 10:44:02,088][INFO ][node                     ] [Cybele] version[2.4.5], pid[4455], build[c849dd1/2017-04-24T16:18:17Z]
[2017-08-21 10:44:02,089][INFO ][node                     ] [Cybele] initializing ...
[2017-08-21 10:44:03,333][INFO ][plugins                  ] [Cybele] modules [reindex, lang-expression, lang-groovy], plugins [], sites []
[2017-08-21 10:44:03,526][INFO ][env                      ] [Cybele] using [1] data paths, mounts [[/ (/dev/mapper/cot--graylog03--vg-root)]], net usable_space [67.5gb], net total_space [93.7gb], spins? [no], types [ext4]
[2017-08-21 10:44:03,527][INFO ][env                      ] [Cybele] heap size [1007.3mb], compressed ordinary object pointers [true]
[2017-08-21 10:44:08,197][INFO ][node                     ] [Cybele] initialized
[2017-08-21 10:44:08,197][INFO ][node                     ] [Cybele] starting ...
[2017-08-21 10:44:08,438][INFO ][transport                ] [Cybele] publish_address {127.0.0.1:9300}, bound_addresses {[::1]:9300}, {127.0.0.1:9300}
[2017-08-21 10:44:08,448][INFO ][discovery                ] [Cybele] graylog/LzX60DGfTEylXMfZMjIznQ
[2017-08-21 10:44:11,518][INFO ][cluster.service          ] [Cybele] new_master {Cybele}{LzX60DGfTEylXMfZMjIznQ}{127.0.0.1}{127.0.0.1:9300}, reason: zen-disco-join(elected_as_master, [0] joins received)
[2017-08-21 10:44:11,536][INFO ][http                     ] [Cybele] publish_address {127.0.0.1:9200}, bound_addresses {[::1]:9200}, {127.0.0.1:9200}
[2017-08-21 10:44:11,536][INFO ][node                     ] [Cybele] started
[2017-08-21 10:44:12,181][INFO ][gateway                  ] [Cybele] recovered [17] indices into cluster_state
[2017-08-21 10:44:27,232][INFO ][cluster.routing.allocation] [Cybele] Cluster health status changed from [RED] to [GREEN] (reason: [shards started [[graylog_1][0]] ...]).
grayloguser@cot-graylog02:/var/log/elasticsearch$ clear
grayloguser@cot-graylog02:/var/log/elasticsearch$ tail -n 500 graylog
tail: cannot open 'graylog' for reading: No such file or directory
grayloguser@cot-graylog02:/var/log/elasticsearch$ tail -n 500 graylog.log
[2017-08-21 00:10:45,601][INFO ][cluster.metadata         ] [Jon Spectre] [graylog_16] update_mapping [message]
[2017-08-21 01:27:28,248][WARN ][transport                ] [Jon Spectre] Received response for a request that has timed out, sent [31228ms] ago, timed out [1221ms] ago, action [internal:discovery/zen/fd/ping], node [{graylog-99fb7c0e-bf36-46ef-a1d6-2527b2e3113d}{Eft_GQYXRc2MldfbW0dh1g}{127.0.0.1}{127.0.0.1:9350}{client=true, data=false, master=false}], id [5733024]
[2017-08-21 07:04:55,659][INFO ][cluster.metadata         ] [Jon Spectre] [graylog_16] update_mapping [message]
[2017-08-21 07:23:00,543][INFO ][cluster.metadata         ] [Jon Spectre] [graylog_16] update_mapping [message]
[2017-08-21 07:23:00,637][INFO ][cluster.metadata         ] [Jon Spectre] [graylog_16] update_mapping [message]
[2017-08-21 07:23:00,776][INFO ][cluster.metadata         ] [Jon Spectre] [graylog_16] update_mapping [message]
[2017-08-21 07:23:00,783][INFO ][cluster.metadata         ] [Jon Spectre] [graylog_16] update_mapping [message]
[2017-08-21 07:23:00,811][INFO ][cluster.metadata         ] [Jon Spectre] [graylog_16] update_mapping [message]
[2017-08-21 07:23:00,832][INFO ][cluster.metadata         ] [Jon Spectre] [graylog_16] update_mapping [message]
[2017-08-21 07:23:00,843][INFO ][cluster.metadata         ] [Jon Spectre] [graylog_16] update_mapping [message]
[2017-08-21 07:23:00,866][INFO ][cluster.metadata         ] [Jon Spectre] [graylog_16] update_mapping [message]
[2017-08-21 07:23:00,905][INFO ][cluster.metadata         ] [Jon Spectre] [graylog_16] update_mapping [message]
[2017-08-21 07:23:00,960][INFO ][cluster.metadata         ] [Jon Spectre] [graylog_16] update_mapping [message]
[2017-08-21 07:23:00,996][INFO ][cluster.metadata         ] [Jon Spectre] [graylog_16] update_mapping [message]
[2017-08-21 07:23:01,003][INFO ][cluster.metadata         ] [Jon Spectre] [graylog_16] update_mapping [message]
[2017-08-21 07:23:01,028][INFO ][cluster.metadata         ] [Jon Spectre] [graylog_16] update_mapping [message]
[2017-08-21 07:23:01,047][INFO ][cluster.metadata         ] [Jon Spectre] [graylog_16] update_mapping [message]
[2017-08-21 07:23:02,545][INFO ][cluster.metadata         ] [Jon Spectre] [graylog_16] update_mapping [message]
[2017-08-21 07:23:02,581][INFO ][cluster.metadata         ] [Jon Spectre] [graylog_16] update_mapping [message]
[2017-08-21 07:23:02,613][INFO ][cluster.metadata         ] [Jon Spectre] [graylog_16] update_mapping [message]
[2017-08-21 07:23:04,536][INFO ][cluster.metadata         ] [Jon Spectre] [graylog_16] update_mapping [message]
[2017-08-21 07:23:34,536][INFO ][cluster.metadata         ] [Jon Spectre] [graylog_16] update_mapping [message]
[2017-08-21 07:23:34,579][INFO ][cluster.metadata         ] [Jon Spectre] [graylog_16] update_mapping [message]
[2017-08-21 07:23:34,597][INFO ][cluster.metadata         ] [Jon Spectre] [graylog_16] update_mapping [message]
[2017-08-21 07:23:34,625][INFO ][cluster.metadata         ] [Jon Spectre] [graylog_16] update_mapping [message]
[2017-08-21 07:36:46,555][INFO ][cluster.metadata         ] [Jon Spectre] [graylog_16] update_mapping [message]
[2017-08-21 07:36:46,574][INFO ][cluster.metadata         ] [Jon Spectre] [graylog_16] update_mapping [message]
[2017-08-21 07:36:46,601][INFO ][cluster.metadata         ] [Jon Spectre] [graylog_16] update_mapping [message]
[2017-08-21 07:36:46,633][INFO ][cluster.metadata         ] [Jon Spectre] [graylog_16] update_mapping [message]
[2017-08-21 07:36:48,581][INFO ][cluster.metadata         ] [Jon Spectre] [graylog_16] update_mapping [message]
[2017-08-21 07:36:48,604][INFO ][cluster.metadata         ] [Jon Spectre] [graylog_16] update_mapping [message]
[2017-08-21 07:36:48,655][INFO ][cluster.metadata         ] [Jon Spectre] [graylog_16] update_mapping [message]
[2017-08-21 07:36:48,704][INFO ][cluster.metadata         ] [Jon Spectre] [graylog_16] update_mapping [message]
[2017-08-21 07:36:48,726][INFO ][cluster.metadata         ] [Jon Spectre] [graylog_16] update_mapping [message]
[2017-08-21 07:36:50,550][INFO ][cluster.metadata         ] [Jon Spectre] [graylog_16] update_mapping [message]
[2017-08-21 07:36:50,575][INFO ][cluster.metadata         ] [Jon Spectre] [graylog_16] update_mapping [message]
[2017-08-21 07:36:50,631][INFO ][cluster.metadata         ] [Jon Spectre] [graylog_16] update_mapping [message]
[2017-08-21 07:37:01,539][INFO ][cluster.metadata         ] [Jon Spectre] [graylog_16] update_mapping [message]
[2017-08-21 07:37:01,608][INFO ][cluster.metadata         ] [Jon Spectre] [graylog_16] update_mapping [message]
[2017-08-21 07:37:21,557][INFO ][cluster.metadata         ] [Jon Spectre] [graylog_16] update_mapping [message]
[2017-08-21 07:37:49,593][INFO ][cluster.metadata         ] [Jon Spectre] [graylog_16] update_mapping [message]
[2017-08-21 08:28:38,042][INFO ][node                     ] [Jon Spectre] stopping ...
[2017-08-21 08:28:42,469][INFO ][node                     ] [Jon Spectre] stopped
[2017-08-21 08:28:42,470][INFO ][node                     ] [Jon Spectre] closing ...
[2017-08-21 08:28:42,670][INFO ][node                     ] [Jon Spectre] closed
[2017-08-21 08:46:54,350][INFO ][node                     ] [Shellshock] version[2.4.5], pid[1905], build[c849dd1/2017-04-24T16:18:17Z]
[2017-08-21 08:46:54,354][INFO ][node                     ] [Shellshock] initializing ...
[2017-08-21 08:46:55,587][INFO ][plugins                  ] [Shellshock] modules [reindex, lang-expression, lang-groovy], plugins [], sites []
[2017-08-21 08:46:55,779][INFO ][env                      ] [Shellshock] using [1] data paths, mounts [[/ (/dev/mapper/cot--graylog03--vg-root)]], net usable_space [67.5gb], net total_space [93.7gb], spins? [no], types [ext4]
[2017-08-21 08:46:55,780][INFO ][env                      ] [Shellshock] heap size [1007.3mb], compressed ordinary object pointers [true]
[2017-08-21 08:47:00,948][INFO ][node                     ] [Shellshock] initialized
[2017-08-21 08:47:00,950][INFO ][node                     ] [Shellshock] starting ...
[2017-08-21 08:47:01,287][INFO ][transport                ] [Shellshock] publish_address {127.0.0.1:9300}, bound_addresses {[::1]:9300}, {127.0.0.1:9300}
[2017-08-21 08:47:01,296][INFO ][discovery                ] [Shellshock] graylog/go2AfF3NR-ii5sR-W0QjZQ
[2017-08-21 08:47:04,359][INFO ][cluster.service          ] [Shellshock] new_master {Shellshock}{go2AfF3NR-ii5sR-W0QjZQ}{127.0.0.1}{127.0.0.1:9300}, reason: zen-disco-join(elected_as_master, [0] joins received)
[2017-08-21 08:47:04,382][INFO ][http                     ] [Shellshock] publish_address {127.0.0.1:9200}, bound_addresses {[::1]:9200}, {127.0.0.1:9200}
[2017-08-21 08:47:04,383][INFO ][node                     ] [Shellshock] started
[2017-08-21 08:47:05,297][INFO ][gateway                  ] [Shellshock] recovered [16] indices into cluster_state
[2017-08-21 08:47:31,550][INFO ][cluster.routing.allocation] [Shellshock] Cluster health status changed from [RED] to [GREEN] (reason: [shards started [[graylog_1][1]] ...]).
[2017-08-21 09:11:45,238][INFO ][node                     ] [Shellshock] stopping ...
[2017-08-21 09:11:45,934][INFO ][node                     ] [Shellshock] stopped
[2017-08-21 09:11:45,934][INFO ][node                     ] [Shellshock] closing ...
[2017-08-21 09:11:45,942][INFO ][node                     ] [Shellshock] closed
[2017-08-21 09:11:47,027][INFO ][node                     ] [Cerise] version[2.4.5], pid[2635], build[c849dd1/2017-04-24T16:18:17Z]
[2017-08-21 09:11:47,028][INFO ][node                     ] [Cerise] initializing ...
[2017-08-21 09:11:47,690][INFO ][plugins                  ] [Cerise] modules [reindex, lang-expression, lang-groovy], plugins [], sites []
[2017-08-21 09:11:47,780][INFO ][env                      ] [Cerise] using [1] data paths, mounts [[/ (/dev/mapper/cot--graylog03--vg-root)]], net usable_space [67.5gb], net total_space [93.7gb], spins? [no], types [ext4]
[2017-08-21 09:11:47,781][INFO ][env                      ] [Cerise] heap size [1007.3mb], compressed ordinary object pointers [true]
[2017-08-21 09:11:50,400][INFO ][node                     ] [Cerise] initialized
[2017-08-21 09:11:50,400][INFO ][node                     ] [Cerise] starting ...
[2017-08-21 09:11:50,523][INFO ][transport                ] [Cerise] publish_address {127.0.0.1:9300}, bound_addresses {[::1]:9300}, {127.0.0.1:9300}
[2017-08-21 09:11:50,538][INFO ][discovery                ] [Cerise] graylog/joet24PRTf-Q6DhgBea34A
[2017-08-21 09:11:53,665][INFO ][cluster.service          ] [Cerise] new_master {Cerise}{joet24PRTf-Q6DhgBea34A}{127.0.0.1}{127.0.0.1:9300}, reason: zen-disco-join(elected_as_master, [0] joins received)
[2017-08-21 09:11:53,698][INFO ][http                     ] [Cerise] publish_address {127.0.0.1:9200}, bound_addresses {[::1]:9200}, {127.0.0.1:9200}
[2017-08-21 09:11:53,699][INFO ][node                     ] [Cerise] started
[2017-08-21 09:11:54,027][INFO ][gateway                  ] [Cerise] recovered [16] indices into cluster_state
[2017-08-21 09:12:10,702][INFO ][cluster.routing.allocation] [Cerise] Cluster health status changed from [RED] to [GREEN] (reason: [shards started [[graylog_1][0], [graylog_1][0]] ...]).
[2017-08-21 09:46:56,780][INFO ][cluster.metadata         ] [Cerise] [cot_0] creating index, cause [api], templates [cot-template], shards [4]/[0], mappings [message]
[2017-08-21 09:46:56,889][INFO ][cluster.routing.allocation] [Cerise] Cluster health status changed from [RED] to [GREEN] (reason: [shards started [[cot_0][0], [cot_0][0]] ...]).
[2017-08-21 10:40:55,773][INFO ][cluster.metadata         ] [Cerise] [graylog_16] update_mapping [message]
[2017-08-21 10:41:00,640][INFO ][cluster.metadata         ] [Cerise] [graylog_16] update_mapping [message]
[2017-08-21 10:43:56,420][INFO ][node                     ] [Cerise] stopping ...
[2017-08-21 10:43:58,081][INFO ][node                     ] [Cerise] stopped
[2017-08-21 10:43:58,083][INFO ][node                     ] [Cerise] closing ...
[2017-08-21 10:43:58,127][INFO ][node                     ] [Cerise] closed
[2017-08-21 10:44:02,088][INFO ][node                     ] [Cybele] version[2.4.5], pid[4455], build[c849dd1/2017-04-24T16:18:17Z]
[2017-08-21 10:44:02,089][INFO ][node                     ] [Cybele] initializing ...
[2017-08-21 10:44:03,333][INFO ][plugins                  ] [Cybele] modules [reindex, lang-expression, lang-groovy], plugins [], sites []
[2017-08-21 10:44:03,526][INFO ][env                      ] [Cybele] using [1] data paths, mounts [[/ (/dev/mapper/cot--graylog03--vg-root)]], net usable_space [67.5gb], net total_space [93.7gb], spins? [no], types [ext4]
[2017-08-21 10:44:03,527][INFO ][env                      ] [Cybele] heap size [1007.3mb], compressed ordinary object pointers [true]
[2017-08-21 10:44:08,197][INFO ][node                     ] [Cybele] initialized
[2017-08-21 10:44:08,197][INFO ][node                     ] [Cybele] starting ...
[2017-08-21 10:44:08,438][INFO ][transport                ] [Cybele] publish_address {127.0.0.1:9300}, bound_addresses {[::1]:9300}, {127.0.0.1:9300}
[2017-08-21 10:44:08,448][INFO ][discovery                ] [Cybele] graylog/LzX60DGfTEylXMfZMjIznQ
[2017-08-21 10:44:11,518][INFO ][cluster.service          ] [Cybele] new_master {Cybele}{LzX60DGfTEylXMfZMjIznQ}{127.0.0.1}{127.0.0.1:9300}, reason: zen-disco-join(elected_as_master, [0] joins received)
[2017-08-21 10:44:11,536][INFO ][http                     ] [Cybele] publish_address {127.0.0.1:9200}, bound_addresses {[::1]:9200}, {127.0.0.1:9200}
[2017-08-21 10:44:11,536][INFO ][node                     ] [Cybele] started
[2017-08-21 10:44:12,181][INFO ][gateway                  ] [Cybele] recovered [17] indices into cluster_state
[2017-08-21 10:44:27,232][INFO ][cluster.routing.allocation] [Cybele] Cluster health status changed from [RED] to [GREEN] (reason: [shards started [[graylog_1][0]] ...]).
7

Try deleting the index_ranges collection in MongoDB and then recalculate index ranges via the Graylog web interface (System / Indices / Index Set / Maintenance).

8

I did a successful rebuild via the web interface but still no luck so I just downgraded back to 2.2.3. Is there an update to elasticsearch that needs to be done in order to make this work?

9

The index name “1_0”, which is part of some of your error messages, looks very strange.

What index sets do exist in your Graylog setup and what’s their configuration?

10

I am using the default index set, max 20, 20000000 max documents, 0 index replicas, 4 index shards, 1 max number of segments, close index retention strategy.

11

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.