I have just realised that no logs were received after 4.30 this morning.
All the services are up and running, I can access any old data no problem. Everything is working normally but no new log data?
We did have a disk space alert from Ubuntu which appears to have sorted itself out as we now have a few GB free. Has this affected something within Graylog?
All help more than welcome.
Hey @mr_m_cox,
is your disk usage still over the disk-level-watermark of elasticsearch? Default is 80% I think.
If yes, than your problem could be, that elasticsearch is not allowing a new index to be created because of too low disk space.
Take a look at system/overview in your Webinterface and look if your Elasticsearch is marked as green. Also, you could try curl -XGET localhost:9200/_cat/shards?h=index,shard,prirep,state,unassigned.reason| grep UNASSIGNED on your elasticsearch (change ip/port if needed) to see if any and which index is not assigned.
Greetings - Phil
This is my overview, the indexer issues have been there all along and I am yet to resolve but don’t appear to cause a direct problem to functionality like this has.
Sorry, I missed something. Try
curl -XGET "localhost:9200/_cat/shards?h=index,shard,prirep,state,unassigned.reason" --stderr - | grep "UNASSIGNED"
What does Graylog write in the Indexer Failure log?
Index Failure Log but as I said, its has been doing this all the time its been configured, not only since the 4.30 fail.
Well ok, what does the log file of your graylog-server say for the time since 4:30?
The button on the right saying “Pause Processing” was se to “Resume Processing”.
Now all works. would that have been a shutdown following disk space issues?
Tell a lie,I am getting messages in but not out. I have no sources or messages in the last few hours still, not even in the last 5 mins.
I have just had a google around and cant really find anything about why I would be getting input but no output.
I have cleared my journal as the journalreader service was stopped failed.
Any suggestions? Its clearly taking in messages but not outputting meaning all current time frame searches are empty and no sources.
UPDATE
Still broken. Got hundreds on ins and no outs.
Any advise or suggestions would be more than welcome. Everything looks set correctly in the web interface, just not happening
Have you tried restarting elasticsearch and graylog? This fixed it for me when I had the problem that the output wouldn’t restart after a full disk.
I have rebooted the box, restarted the graylog, elasticsearch and mongod services.
I think I have cleared the journal although a little unsure of the location. I dont know what else to do. It was working great yesterday, filled the disk overnight and now wont come back.
Arghh!
disk is now expanded, along with RAM to provide plenty of space now.
Can you provide your /var/log/graylog-server/server.log file or is this a security problem for your company?
<2017-03-24T14:32:58.431Z INFO [zen] [graylog-aff96a53-22dc-4b86-9c31-cd9adb49c34b] master_left [{Lancer}{RjWaKdXcTzyAEkzDRGZONA}{127.0.0.1}{127.0.0.1:9300}], reason [transport disconnected]
2017-03-24T14:32:58.433Z WARN [zen] [graylog-aff96a53-22dc-4b86-9c31-cd9adb49c34b] master left (reason = transport disconnected), current nodes: {{graylog-aff96a53-22dc-4b86-9c31-cd9adb49c34b}{gOgEooV6RAe28SwCCGUVPw}{10.10.0.79}{10.10.0.79:9350}{client=true, data=false, master=false},}
2017-03-24T14:32:58.433Z INFO [service] [graylog-aff96a53-22dc-4b86-9c31-cd9adb49c34b] removed {{Lancer}{RjWaKdXcTzyAEkzDRGZONA}{127.0.0.1}{127.0.0.1:9300},}, reason: zen-disco-master_failed ({Lancer}{RjWaKdXcTzyAEkzDRGZONA}{127.0.0.1}{127.0.0.1:9300})
2017-03-24T14:32:58.433Z WARN [ClusterStateMonitor] No Elasticsearch data nodes in cluster, cluster is completely offline.
2017-03-24T14:33:04.179Z INFO [service] [graylog-aff96a53-22dc-4b86-9c31-cd9adb49c34b] detected_master {Franklin Hall}{N12A7E_LQ_GJRgGD1f6Zfw}{127.0.0.1}{127.0.0.1:9300}, added {{Franklin Hall}{N12A7E_LQ_GJRgGD1f6Zfw}{127.0.0.1}{127.0.0.1:9300},}, reason: zen-disco-receive(from master [{Franklin Hall}{N12A7E_LQ_GJRgGD1f6Zfw}{127.0.0.1}{127.0.0.1:9300}])
2017-03-24T14:33:08.950Z INFO [Server] SIGNAL received. Shutting down.
2017-03-24T14:33:08.953Z INFO [GracefulShutdown] Graceful shutdown initiated.
2017-03-24T14:33:08.954Z INFO [GracefulShutdown] Node status: [Halting [LB:DEAD]]. Waiting <3sec> for possible load balancers to recognize state change.
2017-03-24T14:33:12.957Z INFO [InputSetupService] Attempting to close input <org.graylog2.inputs.syslog.udp.SyslogUDPInput.58d3add0380fe90f52bfe020> [Syslog UDP].
2017-03-24T14:33:12.960Z INFO [InputSetupService] Input <org.graylog2.inputs.syslog.udp.SyslogUDPInput.58d3add0380fe90f52bfe020> closed. Took [2ms]
2017-03-24T14:33:12.960Z INFO [InputSetupService] Attempting to close input <org.graylog2.inputs.gelf.udp.GELFUDPInput.58cbbfe9f7371a0d6e0971ba> [GELF UDP].
2017-03-24T14:33:12.961Z INFO [InputSetupService] Input <org.graylog2.inputs.gelf.udp.GELFUDPInput.58cbbfe9f7371a0d6e0971ba> closed. Took [1ms]
2017-03-24T14:33:12.968Z INFO [Buffers] Waiting until all buffers are empty.
2017-03-24T14:33:12.970Z INFO [Buffers] All buffers are empty. Continuing.
2017-03-24T14:33:12.973Z INFO [node] [graylog-aff96a53-22dc-4b86-9c31-cd9adb49c34b] stopping ...
2017-03-24T14:33:12.974Z INFO [OutputSetupService] Stopping output org.graylog2.outputs.BlockingBatchedESOutput
2017-03-24T14:33:12.974Z INFO [PeriodicalsService] Shutting down periodical [org.graylog2.periodical.AlertScannerThread].
2017-03-24T14:33:12.975Z INFO [PeriodicalsService] Shutdown of periodical [org.graylog2.periodical.AlertScannerThread] complete, took <0ms>.
2017-03-24T14:33:12.976Z INFO [PeriodicalsService] Shutting down periodical [org.graylog2.periodical.BatchedElasticSearchOutputFlushThread].
2017-03-24T14:33:12.976Z INFO [PeriodicalsService] Shutdown of periodical [org.graylog2.periodical.BatchedElasticSearchOutputFlushThread] complete, took <0ms>.
2017-03-24T14:33:12.976Z INFO [PeriodicalsService] Shutting down periodical [org.graylog2.periodical.ClusterHealthCheckThread].
2017-03-24T14:33:12.976Z INFO [PeriodicalsService] Shutdown of periodical [org.graylog2.periodical.ClusterHealthCheckThread] complete, took <0ms>.
2017-03-24T14:33:12.976Z INFO [PeriodicalsService] Shutting down periodical [org.graylog2.periodical.IndexerClusterCheckerThread].
2017-03-24T14:33:12.976Z INFO [PeriodicalsService] Shutdown of periodical [org.graylog2.periodical.IndexerClusterCheckerThread] complete, took <0ms>.
2017-03-24T14:33:12.976Z INFO [PeriodicalsService] Shutting down periodical [org.graylog2.periodical.IndexRetentionThread].
2017-03-24T14:33:12.977Z INFO [PeriodicalsService] Shutdown of periodical [org.graylog2.periodical.IndexRetentionThread] complete, took <0ms>.
2017-03-24T14:33:12.977Z INFO [PeriodicalsService] Shutting down periodical [org.graylog2.periodical.IndexRotationThread].
2017-03-24T14:33:12.977Z INFO [PeriodicalsService] Shutdown of periodical [org.graylog2.periodical.IndexRotationThread] complete, took <0ms>.
2017-03-24T14:33:12.977Z INFO [PeriodicalsService] Shutting down periodical [org.graylog2.periodical.VersionCheckThread].
2017-03-24T14:33:12.977Z INFO [PeriodicalsService] Shutdown of periodical [org.graylog2.periodical.VersionCheckThread] complete, took <0ms>.
2017-03-24T14:33:12.978Z INFO [PeriodicalsService] Shutting down periodical [org.graylog2.periodical.ThrottleStateUpdaterThread].
2017-03-24T14:33:12.978Z INFO [PeriodicalsService] Shutdown of periodical [org.graylog2.periodical.ThrottleStateUpdaterThread] complete, took <0ms>.
2017-03-24T14:33:12.978Z INFO [PeriodicalsService] Shutting down periodical [org.graylog2.events.ClusterEventPeriodical].
2017-03-24T14:33:12.978Z INFO [PeriodicalsService] Shutdown of periodical [org.graylog2.events.ClusterEventPeriodical] complete, took <0ms>.
2017-03-24T14:33:12.978Z INFO [PeriodicalsService] Shutting down periodical [org.graylog2.events.ClusterEventCleanupPeriodical].
2017-03-24T14:33:12.978Z INFO [PeriodicalsService] Shutdown of periodical [org.graylog2.events.ClusterEventCleanupPeriodical] complete, took <0ms>.
2017-03-24T14:33:12.978Z INFO [PeriodicalsService] Shutting down periodical [org.graylog2.periodical.IndexRangesCleanupPeriodical].
2017-03-24T14:33:12.978Z INFO [PeriodicalsService] Shutdown of periodical [org.graylog2.periodical.IndexRangesCleanupPeriodical] complete, took <0ms>.
2017-03-24T14:33:12.978Z INFO [PeriodicalsService] Shutting down periodical [org.graylog.plugins.usagestatistics.UsageStatsNodePeriodical].
2017-03-24T14:33:12.978Z INFO [PeriodicalsService] Shutdown of periodical [org.graylog.plugins.usagestatistics.UsageStatsNodePeriodical] complete, took <0ms>.
2017-03-24T14:33:12.978Z INFO [PeriodicalsService] Shutting down periodical [org.graylog.plugins.usagestatistics.UsageStatsClusterPeriodical].
2017-03-24T14:33:12.978Z INFO [PeriodicalsService] Shutdown of periodical [org.graylog.plugins.usagestatistics.UsageStatsClusterPeriodical] complete, took <0ms>.
2017-03-24T14:33:12.978Z INFO [PeriodicalsService] Shutting down periodical [org.graylog.plugins.collector.periodical.PurgeExpiredCollectorsThread].
2017-03-24T14:33:12.979Z INFO [PeriodicalsService] Shutdown of periodical [org.graylog.plugins.collector.periodical.PurgeExpiredCollectorsThread] complete, took <0ms>.
2017-03-24T14:33:12.979Z INFO [GracefulShutdown] Goodbye.
2017-03-24T14:33:12.979Z INFO [JerseyService] Shutting down HTTP listener at <http://graylog.mydomain.com:9000/api/>
2017-03-24T14:33:12.999Z INFO [LogManager] Shutting down.
2017-03-24T14:33:13.000Z INFO [node] [graylog-aff96a53-22dc-4b86-9c31-cd9adb49c34b] stopped
2017-03-24T14:33:13.000Z INFO [node] [graylog-aff96a53-22dc-4b86-9c31-cd9adb49c34b] closing ...
2017-03-24T14:33:13.007Z INFO [node] [graylog-aff96a53-22dc-4b86-9c31-cd9adb49c34b] closed
2017-03-24T14:33:13.023Z INFO [LogManager] Shutdown complete.
2017-03-24T14:33:13.045Z INFO [NetworkListener] Stopped listener bound to [graylog.mydomain.com:9000]
2017-03-24T14:33:13.046Z INFO [JerseyService] Shutting down HTTP listener at <http://graylog.mydomain.com:9001/>
2017-03-24T14:33:13.049Z INFO [NetworkListener] Stopped listener bound to [graylog.mydomain.com:9001]
2017-03-24T14:33:13.057Z INFO [JournalReader] Stopping.
2017-03-24T14:33:13.057Z INFO [ServiceManagerListener] Services are now stopped.
2017-03-24T14:33:15.952Z INFO [CmdLineTool] Loaded plugin: Elastic Beats Input 2.2.2 [org.graylog.plugins.beats.BeatsInputPlugin]
2017-03-24T14:33:15.954Z INFO [CmdLineTool] Loaded plugin: Collector 2.2.2 [org.graylog.plugins.collector.CollectorPlugin]
2017-03-24T14:33:15.955Z INFO [CmdLineTool] Loaded plugin: Enterprise Integration Plugin 2.2.2 [org.graylog.plugins.enterprise_integration.EnterpriseIntegrationPlugin]
2017-03-24T14:33:15.956Z INFO [CmdLineTool] Loaded plugin: MapWidgetPlugin 2.2.2 [org.graylog.plugins.map.MapWidgetPlugin]
2017-03-24T14:33:15.963Z INFO [CmdLineTool] Loaded plugin: Pipeline Processor Plugin 2.2.2 [org.graylog.plugins.pipelineprocessor.ProcessorPlugin]
2017-03-24T14:33:15.964Z INFO [CmdLineTool] Loaded plugin: Anonymous Usage Statistics 2.2.2 [org.graylog.plugins.usagestatistics.UsageStatsPlugin]
2017-03-24T14:33:16.153Z INFO [CmdLineTool] Running with JVM arguments: -Xms1g -Xmx1g -XX:NewRatio=1 -XX:+ResizeTLAB -XX:+UseConcMarkSweepGC -XX:+CMSConcurrentMTEnabled -XX:+CMSClassUnloadingEnabled -XX:+UseParNewGC -XX:-OmitStackTraceInFastThrow -Dlog4j.configurationFile=file:///etc/graylog/server/log4j2.xml -Djava.library.path=/usr/share/graylog-server/lib/sigar -Dgraylog2.installation_source=deb
2017-03-24T14:33:16.334Z INFO [Version] HV000001: Hibernate Validator null
2017-03-24T14:33:18.032Z INFO [InputBufferImpl] Message journal is enabled.
2017-03-24T14:33:18.051Z INFO [NodeId] Node ID: aff96a53-22dc-4b86-9c31-cd9adb49c34b
2017-03-24T14:33:18.232Z INFO [LogManager] Loading logs.
2017-03-24T14:33:18.290Z INFO [LogManager] Logs loading complete.
2017-03-24T14:33:18.290Z INFO [KafkaJournal] Initialized Kafka based journal at /var/lib/graylog-server/journal
2017-03-24T14:33:18.303Z INFO [InputBufferImpl] Initialized InputBufferImpl with ring size <65536> and wait strategy <BlockingWaitStrategy>, running 2 parallel message handlers.
2017-03-24T14:33:18.320Z INFO [cluster] Cluster created with settings {hosts=[localhost:27017], mode=SINGLE, requiredClusterType=UNKNOWN, serverSelectionTimeout='30000 ms', maxWaitQueueSize=5000}
2017-03-24T14:33:18.383Z INFO [cluster] No server chosen by ReadPreferenceServerSelector{readPreference=primary} from cluster description ClusterDescription{type=UNKNOWN, connectionMode=SINGLE, serverDescriptions=[ServerDescription{address=localhost:27017, type=UNKNOWN, state=CONNECTING}]}. Waiting for 30000 ms before timing out
2017-03-24T14:33:18.402Z INFO [connection] Opened connection [connectionId{localValue:1, serverValue:9}] to localhost:27017
2017-03-24T14:33:18.405Z INFO [cluster] Monitor thread successfully connected to server with description ServerDescription{address=localhost:27017, type=STANDALONE, state=CONNECTED, ok=true, version=ServerVersion{versionList=[3, 2, 12]}, minWireVersion=0, maxWireVersion=4, maxDocumentSize=16777216, roundTripTimeNanos=551753}
2017-03-24T14:33:18.411Z INFO [connection] Opened connection [connectionId{localValue:2, serverValue:10}] to localhost:27017
2017-03-24T14:33:18.652Z INFO [node] [graylog-aff96a53-22dc-4b86-9c31-cd9adb49c34b] version[2.4.4], pid[5942], build[fcbb46d/2017-01-03T11:33:16Z]
2017-03-24T14:33:18.653Z INFO [node] [graylog-aff96a53-22dc-4b86-9c31-cd9adb49c34b] initializing ...
2017-03-24T14:33:18.658Z INFO [plugins] [graylog-aff96a53-22dc-4b86-9c31-cd9adb49c34b] modules [], plugins [graylog-monitor], sites []
2017-03-24T14:33:20.107Z INFO [node] [graylog-aff96a53-22dc-4b86-9c31-cd9adb49c34b] initialized
2017-03-24T14:33:20.209Z INFO [ProcessBuffer] Initialized ProcessBuffer with ring size <65536> and wait strategy <BlockingWaitStrategy>.
2017-03-24T14:33:21.753Z INFO [RulesEngineProvider] No static rules file loaded.
2017-03-24T14:33:22.065Z WARN [GeoIpResolverEngine] GeoIP database file does not exist: /etc/graylog/server/GeoLite2-City.mmdb
2017-03-24T14:33:22.072Z INFO [OutputBuffer] Initialized OutputBuffer with ring size <65536> and wait strategy <BlockingWaitStrategy>.
2017-03-24T14:33:22.140Z WARN [GeoIpResolverEngine] GeoIP database file does not exist: /etc/graylog/server/GeoLite2-City.mmdb
2017-03-24T14:33:22.254Z WARN [GeoIpResolverEngine] GeoIP database file does not exist: /etc/graylog/server/GeoLite2-City.mmdb
2017-03-24T14:33:22.299Z WARN [GeoIpResolverEngine] GeoIP database file does not exist: /etc/graylog/server/GeoLite2-City.mmdb
2017-03-24T14:33:22.365Z WARN [GeoIpResolverEngine] GeoIP database file does not exist: /etc/graylog/server/GeoLite2-City.mmdb
2017-03-24T14:33:22.891Z INFO [ServerBootstrap] Graylog server 2.2.2+691b4b7 starting up
2017-03-24T14:33:22.891Z INFO [ServerBootstrap] JRE: Oracle Corporation 1.8.0_121 on Linux 4.8.0-41-generic
2017-03-24T14:33:22.892Z INFO [ServerBootstrap] Deployment: deb
2017-03-24T14:33:22.892Z INFO [ServerBootstrap] OS: Ubuntu 16.04.2 LTS (xenial)
2017-03-24T14:33:22.892Z INFO [ServerBootstrap] Arch: amd64
2017-03-24T14:33:22.898Z WARN [DeadEventLoggingListener] Received unhandled event of type <org.graylog2.plugin.lifecycles.Lifecycle> from event bus <AsyncEventBus{graylog-eventbus}>
2017-03-24T14:33:22.931Z INFO [PeriodicalsService] Starting 26 periodicals ...
2017-03-24T14:33:22.932Z INFO [Periodicals] Starting [org.graylog2.periodical.ThroughputCalculator] periodical in [0s], polling every [1s].
2017-03-24T14:33:22.932Z INFO [Periodicals] Starting [org.graylog2.periodical.AlertScannerThread] periodical in [10s], polling every [60s].
2017-03-24T14:33:22.932Z INFO [Periodicals] Starting [org.graylog2.periodical.BatchedElasticSearchOutputFlushThread] periodical in [0s], polling every [1s].
2017-03-24T14:33:22.933Z INFO [Periodicals] Starting [org.graylog2.periodical.ClusterHealthCheckThread] periodical in [120s], polling every [20s].
2017-03-24T14:33:22.934Z INFO [Periodicals] Starting [org.graylog2.periodical.ContentPackLoaderPeriodical] periodical, running forever.
2017-03-24T14:33:22.936Z INFO [Periodicals] Starting [org.graylog2.periodical.GarbageCollectionWarningThread] periodical, running forever.
2017-03-24T14:33:22.940Z INFO [Periodicals] Starting [org.graylog2.periodical.IndexerClusterCheckerThread] periodical in [0s], polling every [30s].
2017-03-24T14:33:22.947Z INFO [Periodicals] Starting [org.graylog2.periodical.IndexRetentionThread] periodical in [0s], polling every [300s].
2017-03-24T14:33:22.947Z INFO [Periodicals] Starting [org.graylog2.periodical.IndexRotationThread] periodical in [0s], polling every [10s].
2017-03-24T14:33:22.947Z INFO [IndexRetentionThread] Elasticsearch cluster not available, skipping index retention checks.
2017-03-24T14:33:22.949Z INFO [node] [graylog-aff96a53-22dc-4b86-9c31-cd9adb49c34b] starting ...
2017-03-24T14:33:22.959Z INFO [Periodicals] Starting [org.graylog2.periodical.NodePingThread] periodical in [0s], polling every [1s].
2017-03-24T14:33:22.959Z INFO [Periodicals] Starting [org.graylog2.periodical.VersionCheckThread] periodical in [300s], polling every [1800s].
2017-03-24T14:33:22.961Z INFO [connection] Opened connection [connectionId{localValue:3, serverValue:11}] to localhost:27017
2017-03-24T14:33:22.962Z INFO [Periodicals] Starting [org.graylog2.periodical.ThrottleStateUpdaterThread] periodical in [1s], polling every [1s].
2017-03-24T14:33:22.962Z INFO [connection] Opened connection [connectionId{localValue:4, serverValue:12}] to localhost:27017
2017-03-24T14:33:22.964Z INFO [Periodicals] Starting [org.graylog2.events.ClusterEventPeriodical] periodical in [0s], polling every [1s].
2017-03-24T14:33:22.971Z INFO [Periodicals] Starting [org.graylog2.events.ClusterEventCleanupPeriodical] periodical in [0s], polling every [86400s].
2017-03-24T14:33:22.971Z INFO [Periodicals] Starting [org.graylog2.periodical.ClusterIdGeneratorPeriodical] periodical, running forever.
2017-03-24T14:33:22.972Z INFO [Periodicals] Starting [org.graylog2.periodical.IndexRangesMigrationPeriodical] periodical, running forever.
2017-03-24T14:33:22.973Z INFO [Periodicals] Starting [org.graylog2.periodical.IndexRangesCleanupPeriodical] periodical in [15s], polling every [3600s].
2017-03-24T14:33:22.976Z INFO [connection] Opened connection [connectionId{localValue:6, serverValue:14}] to localhost:27017
2017-03-24T14:33:22.976Z INFO [connection] Opened connection [connectionId{localValue:5, serverValue:13}] to localhost:27017
2017-03-24T14:33:22.989Z INFO [connection] Opened connection [connectionId{localValue:7, serverValue:15}] to localhost:27017
2017-03-24T14:33:22.991Z INFO [connection] Opened connection [connectionId{localValue:8, serverValue:16}] to localhost:27017
2017-03-24T14:33:23.001Z INFO [PeriodicalsService] Not starting [org.graylog2.periodical.UserPermissionMigrationPeriodical] periodical. Not configured to run on this node.
2017-03-24T14:33:23.014Z INFO [Periodicals] Starting [org.graylog2.periodical.AlarmCallbacksMigrationPeriodical] periodical, running forever.
2017-03-24T14:33:23.018Z INFO [Periodicals] Starting [org.graylog2.periodical.ConfigurationManagementPeriodical] periodical, running forever.
2017-03-24T14:33:23.020Z INFO [Periodicals] Starting [org.graylog2.periodical.LdapGroupMappingMigration] periodical, running forever.
2017-03-24T14:33:23.022Z INFO [Periodicals] Starting [org.graylog2.periodical.IndexFailuresPeriodical] periodical, running forever.
2017-03-24T14:33:23.036Z INFO [Periodicals] Starting [org.graylog.plugins.usagestatistics.UsageStatsNodePeriodical] periodical in [300s], polling every [21600s].
2017-03-24T14:33:23.040Z INFO [Periodicals] Starting [org.graylog.plugins.usagestatistics.UsageStatsClusterPeriodical] periodical in [300s], polling every [21600s].
2017-03-24T14:33:23.052Z INFO [Periodicals] Starting [org.graylog.plugins.pipelineprocessor.periodical.LegacyDefaultStreamMigration] periodical, running forever.
2017-03-24T14:33:23.052Z INFO [Periodicals] Starting [org.graylog.plugins.collector.periodical.PurgeExpiredCollectorsThread] periodical in [0s], polling every [3600s].
2017-03-24T14:33:23.060Z INFO [IndexerClusterCheckerThread] Indexer not fully initialized yet. Skipping periodic cluster check.
2017-03-24T14:33:23.076Z INFO [LegacyDefaultStreamMigration] Legacy default stream has no connections, no migration needed.
2017-03-24T14:33:23.165Z INFO [V20161130141500_DefaultStreamRecalcIndexRanges] Cluster not connected yet, delaying migration until it is reachable.
2017-03-24T14:33:23.179Z INFO [transport] [graylog-aff96a53-22dc-4b86-9c31-cd9adb49c34b] publish_address {10.10.0.79:9350}, bound_addresses {10.10.0.79:9350}
2017-03-24T14:33:23.183Z INFO [discovery] [graylog-aff96a53-22dc-4b86-9c31-cd9adb49c34b] graylog/wZkTp1vwTx2C1T2b__ez6g
2017-03-24T14:33:23.553Z INFO [JerseyService] Enabling CORS for HTTP endpoint
2017-03-24T14:33:26.186Z WARN [discovery] [graylog-aff96a53-22dc-4b86-9c31-cd9adb49c34b] waited for 3s and no initial state was set by the discovery
2017-03-24T14:33:26.186Z INFO [node] [graylog-aff96a53-22dc-4b86-9c31-cd9adb49c34b] started
2017-03-24T14:33:26.259Z INFO [service] [graylog-aff96a53-22dc-4b86-9c31-cd9adb49c34b] detected_master {Franklin Hall}{N12A7E_LQ_GJRgGD1f6Zfw}{127.0.0.1}{127.0.0.1:9300}, added {{Franklin Hall}{N12A7E_LQ_GJRgGD1f6Zfw}{127.0.0.1}{127.0.0.1:9300},}, reason: zen-disco-receive(from master [{Franklin Hall}{N12A7E_LQ_GJRgGD1f6Zfw}{127.0.0.1}{127.0.0.1:9300}])
2017-03-24T14:33:31.521Z INFO [NetworkListener] Started listener bound to [graylog.mydomain.com:9000]
2017-03-24T14:33:31.523Z INFO [HttpServer] [HttpServer] Started.
2017-03-24T14:33:31.523Z INFO [JerseyService] Started REST API at <http://graylog.mydomain.com:9000/api/>
2017-03-24T14:33:34.040Z INFO [NetworkListener] Started listener bound to [graylog.mydomain.com:9001]
2017-03-24T14:33:34.041Z INFO [HttpServer] [HttpServer-1] Started.
2017-03-24T14:33:34.041Z INFO [JerseyService] Started Web Interface at <http://graylog.mydomain.com:9001/>
2017-03-24T14:33:34.043Z INFO [ServiceManagerListener] Services are healthy
2017-03-24T14:33:34.043Z INFO [ServerBootstrap] Services started, startup times in ms: {InputSetupService [RUNNING]=10, OutputSetupService [RUNNING]=12, BufferSynchronizerService [RUNNING]=13, KafkaJournal [RUNNING]=26, JournalReader [RUNNING]=88, ConfigurationEtagService [RUNNING]=96, StreamCacheService [RUNNING]=128, PeriodicalsService [RUNNING]=149, IndexerSetupService [RUNNING]=3341, JerseyService [RUNNING]=11130}
2017-03-24T14:33:34.047Z INFO [ServerBootstrap] Graylog server up and running.
2017-03-24T14:33:34.054Z INFO [InputSetupService] Triggering launching persisted inputs, node transitioned from Uninitialized [LB:DEAD] to Running [LB:ALIVE]
2017-03-24T14:33:34.087Z INFO [InputStateListener] Input [GELF UDP/58cbbfe9f7371a0d6e0971ba] is now STARTING
2017-03-24T14:33:34.091Z INFO [InputStateListener] Input [Syslog UDP/58d3add0380fe90f52bfe020] is now STARTING
2017-03-24T14:33:34.128Z WARN [NettyTransport] receiveBufferSize (SO_RCVBUF) for input GELFUDPInput{title=Windows Event Logs, type=org.graylog2.inputs.gelf.udp.GELFUDPInput, nodeId=null} should be 262144 but is 212992.
2017-03-24T14:33:34.129Z WARN [NettyTransport] receiveBufferSize (SO_RCVBUF) for input SyslogUDPInput{title=Linux_Input, type=org.graylog2.inputs.syslog.udp.SyslogUDPInput, nodeId=null} should be 262144 but is 212992.
2017-03-24T14:33:34.131Z INFO [InputStateListener] Input [GELF UDP/58cbbfe9f7371a0d6e0971ba] is now RUNNING
2017-03-24T14:33:34.132Z INFO [InputStateListener] Input [Syslog UDP/58d3add0380fe90f52bfe020] is now RUNNING>This should, if i cut the right part, show the graylog and es services restarting and then the system coming up again.
Could you please edit your post and put the log in
```
< Log here >
```
Tags for better readability?
Well, there seems nothing wrong. How does your elasticsearch log look like? Any errors in there?