elastic version 5.5.2
graylog version 2.3.1
mongodb version 2.6.10
the problem
i’m trying to get messages from a php application on a remote server into graylog using gelf over udp. tcpdump shows traffic coming in when i send below test message. however the inputs in graylog show nothing incoming. i redacted some of the ip addresses.
what occurs to me already is the strange date on the logs of elastic. elastic does not seem to log anything new when restarting the server. however graylog reports that the elasticsearch cluster is green. (edit: the date problem has been fixed, the cluster log messages somehow weren’t writeable by the user the process runs under. i updated the logs with the correct version)
i was hoping if anyone could help me out troubleshoot this issue. thank you for reading.
send message command
echo -n '{ "version": "1.1", "host": "example.org", "short_message": "A short message", "level": 5, "_some_info": "foo" }' | nc -w0 -u bbb.bbb.bbb.bbb 12201
tcpdump output
sudo tcpdump -i ens3 udp port 12201
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on ens3, link-type EN10MB (Ethernet), capture size 262144 bytes
12:48:30.909255 IP aaa.aaa.aaa.aaa > bbb.bbb.bbb.bbb.12201: UDP, length 112
graylog input configuration
bind_address: bbb.bbb.bbb.bbb
decompress_size_limit: 8388608
override_source:
port: 12201
recv_buffer_size: 262144
elasticsearch graylog cluster log
[2017-09-05 13:34:26,084][INFO ][node ] [Vavavoom] version[2.4.4], pid[4517], build[fcbb46d/2017-01-03T11:33:16Z]
[2017-09-05 13:34:26,085][INFO ][node ] [Vavavoom] initializing ...
[2017-09-05 13:34:26,465][INFO ][plugins ] [Vavavoom] modules [reindex, lang-expression, lang-groovy], plugins [], sites []
[2017-09-05 13:34:26,481][INFO ][env ] [Vavavoom] using [1] data paths, mounts [[/ (/dev/vda1)]], net usable_space [107.3gb], net total_space [143.5gb], spins? [possibly], types [ext4]
[2017-09-05 13:34:26,481][INFO ][env ] [Vavavoom] heap size [495.3mb], compressed ordinary object pointers [true]
[2017-09-05 13:34:28,181][INFO ][node ] [Vavavoom] initialized
[2017-09-05 13:34:28,181][INFO ][node ] [Vavavoom] starting ...
[2017-09-05 13:34:28,236][INFO ][transport ] [Vavavoom] publish_address {127.0.0.1:9300}, bound_addresses {[::1]:9300}, {127.0.0.1:9300}
[2017-09-05 13:34:28,240][INFO ][discovery ] [Vavavoom] graylog/UuXnej4bQFmxq0yFml8qSQ
[2017-09-05 13:34:31,276][INFO ][cluster.service ] [Vavavoom] new_master {Vavavoom}{UuXnej4bQFmxq0yFml8qSQ}{127.0.0.1}{127.0.0.1:9300}, added {{graylog-dbcea0b8-d5cb-46a3-bdeb-b689bfbcd75c}{Evx8OsmkSqamRwLVDeTxMg}{127.0.0.1}{127.0.0.1:9350}{client=true, data=false, master=false},}, reason: zen-disco-join(elected_as_master, [0] joins received)
[2017-09-05 13:34:31,324][INFO ][http ] [Vavavoom] publish_address {127.0.0.1:9200}, bound_addresses {[::1]:9200}, {127.0.0.1:9200}
[2017-09-05 13:34:31,324][INFO ][node ] [Vavavoom] started
[2017-09-05 13:34:31,398][INFO ][gateway ] [Vavavoom] recovered [1] indices into cluster_state
[2017-09-05 13:34:31,824][INFO ][cluster.routing.allocation] [Vavavoom] Cluster health status changed from [RED] to [GREEN] (reason: [shards started [[graylog_0][0], [graylog_0][1], [graylog_0][1], [graylog_0][0]] ...]).
graylog logs
2017-09-05T12:55:25.880+02:00 INFO [CmdLineTool] Loaded plugin: Elastic Beats Input 2.2.3 [org.graylog.plugins.beats.BeatsInputPlugin]
2017-09-05T12:55:25.882+02:00 INFO [CmdLineTool] Loaded plugin: Collector 2.2.3 [org.graylog.plugins.collector.CollectorPlugin]
2017-09-05T12:55:25.882+02:00 INFO [CmdLineTool] Loaded plugin: Enterprise Integration Plugin 2.2.3 [org.graylog.plugins.enterprise_integration.EnterpriseIntegrationPlugin]
2017-09-05T12:55:25.883+02:00 INFO [CmdLineTool] Loaded plugin: MapWidgetPlugin 2.2.3 [org.graylog.plugins.map.MapWidgetPlugin]
2017-09-05T12:55:25.891+02:00 INFO [CmdLineTool] Loaded plugin: Pipeline Processor Plugin 2.2.3 [org.graylog.plugins.pipelineprocessor.ProcessorPlugin]
2017-09-05T12:55:25.891+02:00 INFO [CmdLineTool] Loaded plugin: Anonymous Usage Statistics 2.2.3 [org.graylog.plugins.usagestatistics.UsageStatsPlugin]
2017-09-05T12:55:26.192+02:00 INFO [CmdLineTool] Running with JVM arguments: -Xms256m -Xmx256m -XX:NewRatio=1 -XX:+ResizeTLAB -XX:+UseConcMarkSweepGC -XX:+CMSConcurrentMTEnabled -XX:+CMSClassUnloadingEnabled -XX:+UseParNewGC -XX:-OmitStackTraceInFastThrow -Djava.net.preferIPv4Stack=true -Dlog4j.configurationFile=file:///etc/graylog/server/log4j2.xml -Djava.library.path=/usr/share/graylog-server/lib/sigar -Dgraylog2.installation_source=deb
2017-09-05T12:55:26.425+02:00 INFO [Version] HV000001: Hibernate Validator null
2017-09-05T12:55:28.697+02:00 INFO [InputBufferImpl] Message journal is enabled.
2017-09-05T12:55:28.717+02:00 INFO [NodeId] Node ID: dbcea0b8-d5cb-46a3-bdeb-b689bfbcd75c
2017-09-05T12:55:28.896+02:00 INFO [LogManager] Loading logs.
2017-09-05T12:55:28.938+02:00 INFO [LogManager] Logs loading complete.
2017-09-05T12:55:28.938+02:00 INFO [KafkaJournal] Initialized Kafka based journal at /var/lib/graylog-server/journal
2017-09-05T12:55:28.961+02:00 INFO [InputBufferImpl] Initialized InputBufferImpl with ring size <65536> and wait strategy <BlockingWaitStrategy>, running 2 parallel message handlers.
2017-09-05T12:55:28.977+02:00 INFO [cluster] Cluster created with settings {hosts=[localhost:27017], mode=SINGLE, requiredClusterType=UNKNOWN, serverSelectionTimeout='30000 ms', maxWaitQueueSize=5000}
2017-09-05T12:55:29.022+02:00 INFO [cluster] No server chosen by ReadPreferenceServerSelector{readPreference=primary} from cluster description ClusterDescription{type=UNKNOWN, connectionMode=SINGLE, serverDescriptions=[ServerDescription{address=localhost:27017, type=UNKNOWN, state=CONNECTING}]}. Waiting for 30000 ms before timing out
2017-09-05T12:55:29.041+02:00 INFO [connection] Opened connection [connectionId{localValue:1, serverValue:11}] to localhost:27017
2017-09-05T12:55:29.043+02:00 INFO [cluster] Monitor thread successfully connected to server with description ServerDescription{address=localhost:27017, type=STANDALONE, state=CONNECTED, ok=true, version=ServerVersion{versionList=[2, 6, 10]}, minWireVersion=0, maxWireVersion=2, maxDocumentSize=16777216, roundTripTimeNanos=537531}
2017-09-05T12:55:29.048+02:00 INFO [connection] Opened connection [connectionId{localValue:2, serverValue:12}] to localhost:27017
2017-09-05T12:55:29.347+02:00 INFO [node] [graylog-dbcea0b8-d5cb-46a3-bdeb-b689bfbcd75c] version[2.4.4], pid[4208], build[fcbb46d/2017-01-03T11:33:16Z]
2017-09-05T12:55:29.347+02:00 INFO [node] [graylog-dbcea0b8-d5cb-46a3-bdeb-b689bfbcd75c] initializing ...
2017-09-05T12:55:29.352+02:00 INFO [plugins] [graylog-dbcea0b8-d5cb-46a3-bdeb-b689bfbcd75c] modules [], plugins [graylog-monitor], sites []
2017-09-05T12:55:30.753+02:00 INFO [node] [graylog-dbcea0b8-d5cb-46a3-bdeb-b689bfbcd75c] initialized
2017-09-05T12:55:30.849+02:00 INFO [ProcessBuffer] Initialized ProcessBuffer with ring size <65536> and wait strategy <BlockingWaitStrategy>.
2017-09-05T12:55:32.243+02:00 INFO [RulesEngineProvider] No static rules file loaded.
2017-09-05T12:55:32.509+02:00 WARN [GeoIpResolverEngine] GeoIP database file does not exist: /etc/graylog/server/GeoLite2-City.mmdb
2017-09-05T12:55:32.513+02:00 INFO [OutputBuffer] Initialized OutputBuffer with ring size <65536> and wait strategy <BlockingWaitStrategy>.
2017-09-05T12:55:32.567+02:00 WARN [GeoIpResolverEngine] GeoIP database file does not exist: /etc/graylog/server/GeoLite2-City.mmdb
2017-09-05T12:55:32.637+02:00 WARN [GeoIpResolverEngine] GeoIP database file does not exist: /etc/graylog/server/GeoLite2-City.mmdb
2017-09-05T12:55:32.700+02:00 WARN [GeoIpResolverEngine] GeoIP database file does not exist: /etc/graylog/server/GeoLite2-City.mmdb
2017-09-05T12:55:32.762+02:00 WARN [GeoIpResolverEngine] GeoIP database file does not exist: /etc/graylog/server/GeoLite2-City.mmdb
2017-09-05T12:55:33.311+02:00 INFO [ServerBootstrap] Graylog server 2.2.3+7adc951 starting up
2017-09-05T12:55:33.311+02:00 INFO [ServerBootstrap] JRE: Oracle Corporation 1.8.0_131 on Linux 4.4.0-78-generic
2017-09-05T12:55:33.311+02:00 INFO [ServerBootstrap] Deployment: deb
2017-09-05T12:55:33.311+02:00 INFO [ServerBootstrap] OS: Ubuntu 16.04.2 LTS (xenial)
2017-09-05T12:55:33.312+02:00 INFO [ServerBootstrap] Arch: amd64
2017-09-05T12:55:33.324+02:00 WARN [DeadEventLoggingListener] Received unhandled event of type <org.graylog2.plugin.lifecycles.Lifecycle> from event bus <AsyncEventBus{graylog-eventbus}>
2017-09-05T12:55:33.343+02:00 INFO [PeriodicalsService] Starting 26 periodicals ...
2017-09-05T12:55:33.346+02:00 INFO [Periodicals] Starting [org.graylog2.periodical.ThroughputCalculator] periodical in [0s], polling every [1s].
2017-09-05T12:55:33.345+02:00 INFO [node] [graylog-dbcea0b8-d5cb-46a3-bdeb-b689bfbcd75c] starting ...
2017-09-05T12:55:33.379+02:00 INFO [Periodicals] Starting [org.graylog2.periodical.AlertScannerThread] periodical in [10s], polling every [60s].
2017-09-05T12:55:33.381+02:00 INFO [Periodicals] Starting [org.graylog2.periodical.BatchedElasticSearchOutputFlushThread] periodical in [0s], polling every [1s].
2017-09-05T12:55:33.381+02:00 INFO [Periodicals] Starting [org.graylog2.periodical.ClusterHealthCheckThread] periodical in [120s], polling every [20s].
2017-09-05T12:55:33.392+02:00 INFO [Periodicals] Starting [org.graylog2.periodical.ContentPackLoaderPeriodical] periodical, running forever.
2017-09-05T12:55:33.400+02:00 INFO [Periodicals] Starting [org.graylog2.periodical.GarbageCollectionWarningThread] periodical, running forever.
2017-09-05T12:55:33.401+02:00 INFO [Periodicals] Starting [org.graylog2.periodical.IndexerClusterCheckerThread] periodical in [0s], polling every [30s].
2017-09-05T12:55:33.402+02:00 INFO [Periodicals] Starting [org.graylog2.periodical.IndexRetentionThread] periodical in [0s], polling every [300s].
2017-09-05T12:55:33.402+02:00 INFO [Periodicals] Starting [org.graylog2.periodical.IndexRotationThread] periodical in [0s], polling every [10s].
2017-09-05T12:55:33.404+02:00 INFO [IndexRetentionThread] Elasticsearch cluster not available, skipping index retention checks.
2017-09-05T12:55:33.408+02:00 INFO [Periodicals] Starting [org.graylog2.periodical.NodePingThread] periodical in [0s], polling every [1s].
2017-09-05T12:55:33.417+02:00 INFO [connection] Opened connection [connectionId{localValue:3, serverValue:13}] to localhost:27017
2017-09-05T12:55:33.422+02:00 INFO [Periodicals] Starting [org.graylog2.periodical.VersionCheckThread] periodical in [300s], polling every [1800s].
2017-09-05T12:55:33.423+02:00 INFO [Periodicals] Starting [org.graylog2.periodical.ThrottleStateUpdaterThread] periodical in [1s], polling every [1s].
2017-09-05T12:55:33.423+02:00 INFO [Periodicals] Starting [org.graylog2.events.ClusterEventPeriodical] periodical in [0s], polling every [1s].
2017-09-05T12:55:33.425+02:00 INFO [Periodicals] Starting [org.graylog2.events.ClusterEventCleanupPeriodical] periodical in [0s], polling every [86400s].
2017-09-05T12:55:33.425+02:00 INFO [Periodicals] Starting [org.graylog2.periodical.ClusterIdGeneratorPeriodical] periodical, running forever.
2017-09-05T12:55:33.426+02:00 INFO [Periodicals] Starting [org.graylog2.periodical.IndexRangesMigrationPeriodical] periodical, running forever.
2017-09-05T12:55:33.427+02:00 INFO [Periodicals] Starting [org.graylog2.periodical.IndexRangesCleanupPeriodical] periodical in [15s], polling every [3600s].
2017-09-05T12:55:33.433+02:00 INFO [connection] Opened connection [connectionId{localValue:5, serverValue:15}] to localhost:27017
2017-09-05T12:55:33.437+02:00 INFO [connection] Opened connection [connectionId{localValue:4, serverValue:14}] to localhost:27017
2017-09-05T12:55:33.440+02:00 INFO [connection] Opened connection [connectionId{localValue:7, serverValue:17}] to localhost:27017
2017-09-05T12:55:33.444+02:00 INFO [connection] Opened connection [connectionId{localValue:6, serverValue:16}] to localhost:27017
2017-09-05T12:55:33.444+02:00 INFO [connection] Opened connection [connectionId{localValue:8, serverValue:18}] to localhost:27017
2017-09-05T12:55:33.463+02:00 INFO [PeriodicalsService] Not starting [org.graylog2.periodical.UserPermissionMigrationPeriodical] periodical. Not configured to run on this node.
2017-09-05T12:55:33.464+02:00 INFO [Periodicals] Starting [org.graylog2.periodical.AlarmCallbacksMigrationPeriodical] periodical, running forever.
2017-09-05T12:55:33.466+02:00 INFO [Periodicals] Starting [org.graylog2.periodical.ConfigurationManagementPeriodical] periodical, running forever.
2017-09-05T12:55:33.473+02:00 INFO [Periodicals] Starting [org.graylog2.periodical.LdapGroupMappingMigration] periodical, running forever.
2017-09-05T12:55:33.477+02:00 INFO [Periodicals] Starting [org.graylog2.periodical.IndexFailuresPeriodical] periodical, running forever.
2017-09-05T12:55:33.483+02:00 INFO [Periodicals] Starting [org.graylog.plugins.usagestatistics.UsageStatsNodePeriodical] periodical in [300s], polling every [21600s].
2017-09-05T12:55:33.485+02:00 INFO [Periodicals] Starting [org.graylog.plugins.usagestatistics.UsageStatsClusterPeriodical] periodical in [300s], polling every [21600s].
2017-09-05T12:55:33.504+02:00 INFO [Periodicals] Starting [org.graylog.plugins.pipelineprocessor.periodical.LegacyDefaultStreamMigration] periodical, running forever.
2017-09-05T12:55:33.505+02:00 INFO [Periodicals] Starting [org.graylog.plugins.collector.periodical.PurgeExpiredCollectorsThread] periodical in [0s], polling every [3600s].
2017-09-05T12:55:33.517+02:00 INFO [IndexerClusterCheckerThread] Indexer not fully initialized yet. Skipping periodic cluster check.
2017-09-05T12:55:33.538+02:00 INFO [LegacyDefaultStreamMigration] Legacy default stream has no connections, no migration needed.
2017-09-05T12:55:33.624+02:00 INFO [V20161130141500_DefaultStreamRecalcIndexRanges] Cluster not connected yet, delaying migration until it is reachable.
2017-09-05T12:55:33.647+02:00 INFO [transport] [graylog-dbcea0b8-d5cb-46a3-bdeb-b689bfbcd75c] publish_address {127.0.0.1:9350}, bound_addresses {127.0.0.1:9350}
2017-09-05T12:55:33.652+02:00 INFO [discovery] [graylog-dbcea0b8-d5cb-46a3-bdeb-b689bfbcd75c] graylog/Evx8OsmkSqamRwLVDeTxMg
2017-09-05T12:55:33.817+02:00 INFO [JerseyService] Enabling CORS for HTTP endpoint
2017-09-05T12:55:36.654+02:00 WARN [discovery] [graylog-dbcea0b8-d5cb-46a3-bdeb-b689bfbcd75c] waited for 3s and no initial state was set by the discovery
2017-09-05T12:55:36.654+02:00 INFO [node] [graylog-dbcea0b8-d5cb-46a3-bdeb-b689bfbcd75c] started
2017-09-05T12:55:36.738+02:00 INFO [service] [graylog-dbcea0b8-d5cb-46a3-bdeb-b689bfbcd75c] detected_master {Galaxy Master}{CVOsZg3GR0qaHVvQ611W0Q}{127.0.0.1}{127.0.0.1:9300}, added {{Galaxy Master}{CVOsZg3GR0qaHVvQ611W0Q}{127.0.0.1}{127.0.0.1:9300},}, reason: zen-disco-receive(from master [{Galaxy Master}{CVOsZg3GR0qaHVvQ611W0Q}{127.0.0.1}{127.0.0.1:9300}])
2017-09-05T12:55:44.367+02:00 INFO [NetworkListener] Started listener bound to [xxx.xxx.xxx.xxx:9000]
2017-09-05T12:55:44.368+02:00 INFO [HttpServer] [HttpServer] Started.
2017-09-05T12:55:44.368+02:00 INFO [JerseyService] Started REST API at <http://xxx.xxx.xxx.xxx:9000/api/>
2017-09-05T12:55:44.368+02:00 INFO [JerseyService] Started Web Interface at <http://xxx.xxx.xxx.xxx:9000/>
2017-09-05T12:55:44.370+02:00 INFO [ServerBootstrap] Services started, startup times in ms: {OutputSetupService [RUNNING]=42, BufferSynchronizerService [RUNNING]=71, KafkaJournal [RUNNING]=71, InputSetupService [RUNNING]=123, JournalReader [RUNNING]=150, StreamCacheService [RUNNING]=167, PeriodicalsService [RUNNING]=194, ConfigurationEtagService [RUNNING]=195, IndexerSetupService [RUNNING]=3415, JerseyService [RUNNING]=11031}
2017-09-05T12:55:44.369+02:00 INFO [ServiceManagerListener] Services are healthy
2017-09-05T12:55:44.375+02:00 INFO [ServerBootstrap] Graylog server up and running.
2017-09-05T12:55:44.376+02:00 INFO [InputSetupService] Triggering launching persisted inputs, node transitioned from Uninitialized [LB:DEAD] to Running [LB:ALIVE]
2017-09-05T12:55:44.402+02:00 INFO [InputStateListener] Input [GELF UDP/59ae7741a956ef635cea7eb8] is now STARTING
2017-09-05T12:55:44.404+02:00 INFO [InputStateListener] Input [Raw/Plaintext UDP/59ae7c54a956ef635cea841e] is now STARTING
2017-09-05T12:55:44.437+02:00 WARN [NettyTransport] receiveBufferSize (SO_RCVBUF) for input RawUDPInput{title=test, type=org.graylog2.inputs.raw.udp.RawUDPInput, nodeId=null} should be 262144 but is 212992.
2017-09-05T12:55:44.437+02:00 WARN [NettyTransport] receiveBufferSize (SO_RCVBUF) for input GELFUDPInput{title=appliance-gelf-udp, type=org.graylog2.inputs.gelf.udp.GELFUDPInput, nodeId=null} should be 262144 but is 212992.
2017-09-05T12:55:44.439+02:00 INFO [InputStateListener] Input [Raw/Plaintext UDP/59ae7c54a956ef635cea841e] is now RUNNING
2017-09-05T12:55:44.440+02:00 INFO [InputStateListener] Input [GELF UDP/59ae7741a956ef635cea7eb8] is now RUNNING
2017-09-05T12:56:30.108+02:00 WARN [ProxiedResource] Unable to call http://xxx.xxx.xxx.xxx:9000/api/system/jobs on node <dbcea0b8-d5cb-46a3-bdeb-b689bfbcd75c>
java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method) ~[?:1.8.0_131]
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116) ~[?:1.8.0_131]
at java.net.SocketInputStream.read(SocketInputStream.java:171) ~[?:1.8.0_131]
at java.net.SocketInputStream.read(SocketInputStream.java:141) ~[?:1.8.0_131]
at okio.Okio$2.read(Okio.java:138) ~[graylog.jar:?]
at okio.AsyncTimeout$2.read(AsyncTimeout.java:236) ~[graylog.jar:?]
at okio.RealBufferedSource.indexOf(RealBufferedSource.java:325) ~[graylog.jar:?]
at okio.RealBufferedSource.indexOf(RealBufferedSource.java:314) ~[graylog.jar:?]
at okio.RealBufferedSource.readUtf8LineStrict(RealBufferedSource.java:210) ~[graylog.jar:?]
at okhttp3.internal.http1.Http1Codec.readResponse(Http1Codec.java:191) ~[graylog.jar:?]
at okhttp3.internal.http1.Http1Codec.readResponseHeaders(Http1Codec.java:132) ~[graylog.jar:?]
at okhttp3.internal.http.CallServerInterceptor.intercept(CallServerInterceptor.java:54) ~[graylog.jar:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[graylog.jar:?]
at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:45) ~[graylog.jar:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[graylog.jar:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67) ~[graylog.jar:?]
at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:93) ~[graylog.jar:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[graylog.jar:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67) ~[graylog.jar:?]
at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93) ~[graylog.jar:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[graylog.jar:?]
at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:120) ~[graylog.jar:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[graylog.jar:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67) ~[graylog.jar:?]
at org.graylog2.rest.RemoteInterfaceProvider.lambda$get$0(RemoteInterfaceProvider.java:59) ~[graylog.jar:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[graylog.jar:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67) ~[graylog.jar:?]
at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:179) ~[graylog.jar:?]
at okhttp3.RealCall.execute(RealCall.java:63) ~[graylog.jar:?]
at retrofit2.OkHttpCall.execute(OkHttpCall.java:174) ~[graylog.jar:?]
at org.graylog2.shared.rest.resources.ProxiedResource.lambda$null$0(ProxiedResource.java:76) ~[graylog.jar:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_131]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_131]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_131]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131]
2017-09-05T13:34:25.465+02:00 INFO [zen] [graylog-dbcea0b8-d5cb-46a3-bdeb-b689bfbcd75c] master_left [{Galaxy Master}{CVOsZg3GR0qaHVvQ611W0Q}{127.0.0.1}{127.0.0.1:9300}], reason [transport disconnected]
2017-09-05T13:34:25.465+02:00 WARN [zen] [graylog-dbcea0b8-d5cb-46a3-bdeb-b689bfbcd75c] master left (reason = transport disconnected), current nodes: {{graylog-dbcea0b8-d5cb-46a3-bdeb-b689bfbcd75c}{Evx8OsmkSqamRwLVDeTxMg}{127.0.0.1}{127.0.0.1:9350}{client=true, data=false, master=false},}
2017-09-05T13:34:25.468+02:00 INFO [service] [graylog-dbcea0b8-d5cb-46a3-bdeb-b689bfbcd75c] removed {{Galaxy Master}{CVOsZg3GR0qaHVvQ611W0Q}{127.0.0.1}{127.0.0.1:9300},}, reason: zen-disco-master_failed ({Galaxy Master}{CVOsZg3GR0qaHVvQ611W0Q}{127.0.0.1}{127.0.0.1:9300})
2017-09-05T13:34:25.469+02:00 WARN [ClusterStateMonitor] No Elasticsearch data nodes in cluster, cluster is completely offline.
2017-09-05T13:34:31.282+02:00 INFO [service] [graylog-dbcea0b8-d5cb-46a3-bdeb-b689bfbcd75c] detected_master {Vavavoom}{UuXnej4bQFmxq0yFml8qSQ}{127.0.0.1}{127.0.0.1:9300}, added {{Vavavoom}{UuXnej4bQFmxq0yFml8qSQ}{127.0.0.1}{127.0.0.1:9300},}, reason: zen-disco-receive(from master [{Vavavoom}{UuXnej4bQFmxq0yFml8qSQ}{127.0.0.1}{127.0.0.1:9300}])