Hello,
I have the following docker setup:
rsyslog -> redis -> logstash -> graylog -> elasticsearch (and mongodb in replica set)
- 2 graylog nodes (v2.2.1) 1 master - 1 slave
- 12 elasticsearch data nodes (v2.4.0 - lucenne version is 5.5.2)
Symtoms : Graylog journal size never stop to increase
graylog.conf:
elasticsearch_cluster_name = graylog
elasticsearch_discovery_zen_ping_unicast_hosts = elasticsearch:9300
elasticsearch_network_host = 0.0.0.0
elasticsearch.yaml:
cluster.name: graylog
discovery.zen.ping.multicast.enabled: false
network.host: 0.0.0.0
discovery.zen.fd.ping_timeout: 30s
discovery.zen.ping_timeout: 30s
discovery.zen.minimum_master_nodes: 1
Graylog is able to connect to elastic cluster and he is healthy:
/usr/share/graylog# curl elasticsearch:9200/_cluster/health?pretty
{
"cluster_name" : "graylog",
"status" : "green",
"timed_out" : false,
"number_of_nodes" : 12,
"number_of_data_nodes" : 8,
"active_primary_shards" : 243,
"active_shards" : 726,
"relocating_shards" : 2,
"initializing_shards" : 0,
"unassigned_shards" : 0,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0,
"task_max_waiting_in_queue_millis" : 0,
"active_shards_percent_as_number" : 100.0
}
Graylog logs from graylog master :
graylog_1 | 2017-08-08 12:36:43,757 INFO : org.graylog2.bootstrap.CmdLineTool - Loaded plugin: Elastic Beats Input 2.2.1 [org.graylog.plugins.beats.BeatsInputPlugin]
graylog_1 | 2017-08-08 12:36:43,759 INFO : org.graylog2.bootstrap.CmdLineTool - Loaded plugin: CEF Input 1.0.0 [org.graylog.plugins.cef.CEFInputPlugin]
graylog_1 | 2017-08-08 12:36:43,760 INFO : org.graylog2.bootstrap.CmdLineTool - Loaded plugin: Collector 2.2.1 [org.graylog.plugins.collector.CollectorPlugin]
graylog_1 | 2017-08-08 12:36:43,761 INFO : org.graylog2.bootstrap.CmdLineTool - Loaded plugin: Enterprise Integration Plugin 2.2.1 [org.graylog.plugins.enterprise_integration.EnterpriseIntegrationPlugin]
graylog_1 | 2017-08-08 12:36:43,762 INFO : org.graylog2.bootstrap.CmdLineTool - Loaded plugin: MapWidgetPlugin 2.2.1 [org.graylog.plugins.map.MapWidgetPlugin]
graylog_1 | 2017-08-08 12:36:43,763 INFO : org.graylog2.bootstrap.CmdLineTool - Loaded plugin: Pipeline Processor Plugin 2.2.1 [org.graylog.plugins.pipelineprocessor.ProcessorPlugin]
graylog_1 | 2017-08-08 12:36:43,764 INFO : org.graylog2.bootstrap.CmdLineTool - Loaded plugin: Anonymous Usage Statistics 2.2.1 [org.graylog.plugins.usagestatistics.UsageStatsPlugin]
graylog_1 | 2017-08-08 12:36:44,023 INFO : org.graylog2.bootstrap.CmdLineTool - Running with JVM arguments: -Xms1g -Xmx2g -XX:NewRatio=1 -XX:MaxMetaspaceSize=256m -XX:+ResizeTLAB -XX:+UseConcMarkSweepGC -XX:+CMSConcurrentMTEnabled -XX:+CMSClassUnloadingEnabled -XX:+UseParNewGC -XX:-OmitStackTraceInFastThrow -Dlog4j.configurationFile=/usr/share/graylog/data/config/log4j2.xml -Djava.library.path=/usr/share/graylog/lib/sigar/ -Dgraylog2.installation_source=docker
graylog_1 | 2017-08-08 12:36:44,278 INFO : org.hibernate.validator.internal.util.Version - HV000001: Hibernate Validator null
graylog_1 | 2017-08-08 12:36:46,820 INFO : org.graylog2.shared.buffers.InputBufferImpl - Message journal is enabled.
graylog_1 | 2017-08-08 12:36:46,845 INFO : org.graylog2.plugin.system.NodeId - No node ID file found. Generated: 9497e0b0-254a-4e9f-acd5-555d10a7c348
graylog_1 | 2017-08-08 12:36:47,069 INFO : kafka.log.LogManager - Loading logs.
graylog_1 | 2017-08-08 12:36:47,075 INFO : kafka.log.LogManager - Logs loading complete.
graylog_1 | 2017-08-08 12:36:47,122 INFO : kafka.log.LogManager - Created log for partition [messagejournal,0] in /usr/share/graylog/data/journal with properties {file.delete.delay.ms -> 60000, compact -> false, max.message.bytes -> 104857600, min.insync.replicas -> 1, segment.jitter.ms -> 0, index.interval.bytes -> 4096, min.cleanable.dirty.ratio -> 0.5, unclean.leader.election.enable -> true, retention.bytes -> 5368709120, delete.retention.ms -> 86400000, flush.ms -> 60000, segment.bytes -> 104857600, segment.ms -> 3600000, retention.ms -> 43200000, flush.messages -> 1000000, segment.index.bytes -> 1048576}.
graylog_1 | 2017-08-08 12:36:47,122 INFO : org.graylog2.shared.journal.KafkaJournal - Initialized Kafka based journal at /usr/share/graylog/data/journal
graylog_1 | 2017-08-08 12:36:47,138 INFO : org.graylog2.shared.buffers.InputBufferImpl - Initialized InputBufferImpl with ring size <65536> and wait strategy <BlockingWaitStrategy>, running 2 parallel message handlers.
graylog_1 | 2017-08-08 12:36:47,160 INFO : org.mongodb.driver.cluster - Cluster created with settings {hosts=[10.249.130.112:27017, 10.249.130.112:27018, 10.249.130.113:27017], mode=MULTIPLE, requiredClusterType=UNKNOWN, serverSelectionTimeout='30000 ms', maxWaitQueueSize=500}
graylog_1 | 2017-08-08 12:36:47,161 INFO : org.mongodb.driver.cluster - Adding discovered server 10.249.130.112:27017 to client view of cluster
graylog_1 | 2017-08-08 12:36:47,196 INFO : org.mongodb.driver.cluster - Adding discovered server 10.249.130.112:27018 to client view of cluster
graylog_1 | 2017-08-08 12:36:47,198 INFO : org.mongodb.driver.cluster - Adding discovered server 10.249.130.113:27017 to client view of cluster
graylog_1 | 2017-08-08 12:36:47,219 INFO : org.mongodb.driver.cluster - No server chosen by ReadPreferenceServerSelector{readPreference=primary} from cluster description ClusterDescription{type=UNKNOWN, connectionMode=MULTIPLE, serverDescriptions=[ServerDescription{address=10.249.130.113:27017, type=UNKNOWN, state=CONNECTING}, ServerDescription{address=10.249.130.112:27017, type=UNKNOWN, state=CONNECTING}, ServerDescription{address=10.249.130.112:27018, type=UNKNOWN, state=CONNECTING}]}. Waiting for 30000 ms before timing out
graylog_1 | 2017-08-08 12:36:47,247 INFO : org.mongodb.driver.connection - Opened connection [connectionId{localValue:1, serverValue:18}] to 10.249.130.112:27018
graylog_1 | 2017-08-08 12:36:47,247 INFO : org.mongodb.driver.connection - Opened connection [connectionId{localValue:3, serverValue:8}] to 10.249.130.113:27017
graylog_1 | 2017-08-08 12:36:47,247 INFO : org.mongodb.driver.connection - Opened connection [connectionId{localValue:2, serverValue:46}] to 10.249.130.112:27017
graylog_1 | 2017-08-08 12:36:47,250 INFO : org.mongodb.driver.cluster - Monitor thread successfully connected to server with description ServerDescription{address=10.249.130.112:27018, type=REPLICA_SET_ARBITER, state=CONNECTED, ok=true, version=ServerVersion{versionList=[3, 2, 10]}, minWireVersion=0, maxWireVersion=4, maxDocumentSize=16777216, roundTripTimeNanos=983759, setName='graylog', canonicalAddress=10.249.130.112:27018, hosts=[10.249.130.113:27017, 10.249.130.112:27017], passives=[], arbiters=[10.249.130.112:27018], primary='10.249.130.112:27017', tagSet=TagSet{[]}, electionId=null, setVersion=4, lastWriteDate=null, lastUpdateTimeNanos=9695511288117981}
graylog_1 | 2017-08-08 12:36:47,250 INFO : org.mongodb.driver.cluster - Monitor thread successfully connected to server with description ServerDescription{address=10.249.130.113:27017, type=REPLICA_SET_SECONDARY, state=CONNECTED, ok=true, version=ServerVersion{versionList=[3, 2, 10]}, minWireVersion=0, maxWireVersion=4, maxDocumentSize=16777216, roundTripTimeNanos=1657460, setName='graylog', canonicalAddress=10.249.130.113:27017, hosts=[10.249.130.113:27017, 10.249.130.112:27017], passives=[], arbiters=[10.249.130.112:27018], primary='10.249.130.112:27017', tagSet=TagSet{[]}, electionId=null, setVersion=4, lastWriteDate=null, lastUpdateTimeNanos=9695511288295940}
graylog_1 | 2017-08-08 12:36:47,250 INFO : org.mongodb.driver.cluster - Monitor thread successfully connected to server with description ServerDescription{address=10.249.130.112:27017, type=REPLICA_SET_PRIMARY, state=CONNECTED, ok=true, version=ServerVersion{versionList=[3, 2, 10]}, minWireVersion=0, maxWireVersion=4, maxDocumentSize=16777216, roundTripTimeNanos=929454, setName='graylog', canonicalAddress=10.249.130.112:27017, hosts=[10.249.130.113:27017, 10.249.130.112:27017], passives=[], arbiters=[10.249.130.112:27018], primary='10.249.130.112:27017', tagSet=TagSet{[]}, electionId=7fffffff0000000000000204, setVersion=4, lastWriteDate=null, lastUpdateTimeNanos=9695511288112846}
graylog_1 | 2017-08-08 12:36:47,253 INFO : org.mongodb.driver.cluster - Discovered cluster type of REPLICA_SET
graylog_1 | 2017-08-08 12:36:47,254 INFO : org.mongodb.driver.cluster - Setting max election id to 7fffffff0000000000000204 from replica set primary 10.249.130.112:27017
graylog_1 | 2017-08-08 12:36:47,254 INFO : org.mongodb.driver.cluster - Setting max set version to 4 from replica set primary 10.249.130.112:27017
graylog_1 | 2017-08-08 12:36:47,254 INFO : org.mongodb.driver.cluster - Discovered replica set primary 10.249.130.112:27017
graylog_1 | 2017-08-08 12:36:47,261 INFO : org.mongodb.driver.connection - Opened connection [connectionId{localValue:4, serverValue:47}] to 10.249.130.112:27017
graylog_1 | 2017-08-08 12:36:47,589 INFO : org.elasticsearch.node - [graylog-9497e0b0-254a-4e9f-acd5-555d10a7c348] version[2.4.4], pid[1], build[fcbb46d/2017-01-03T11:33:16Z]
graylog_1 | 2017-08-08 12:36:47,589 INFO : org.elasticsearch.node - [graylog-9497e0b0-254a-4e9f-acd5-555d10a7c348] initializing ...
graylog_1 | 2017-08-08 12:36:47,596 INFO : org.elasticsearch.plugins - [graylog-9497e0b0-254a-4e9f-acd5-555d10a7c348] modules [], plugins [graylog-monitor], sites []
graylog_1 | 2017-08-08 12:36:49,183 INFO : org.elasticsearch.node - [graylog-9497e0b0-254a-4e9f-acd5-555d10a7c348] initialized
graylog_1 | 2017-08-08 12:36:49,270 INFO : org.graylog2.shared.buffers.ProcessBuffer - Initialized ProcessBuffer with ring size <65536> and wait strategy <BlockingWaitStrategy>.
graylog_1 | 2017-08-08 12:36:50,983 INFO : org.graylog2.bindings.providers.RulesEngineProvider - No static rules file loaded.
graylog_1 | 2017-08-08 12:36:51,502 INFO : org.graylog2.buffers.OutputBuffer - Initialized OutputBuffer with ring size <65536> and wait strategy <BlockingWaitStrategy>.
graylog_1 | 2017-08-08 12:36:52,408 INFO : org.graylog2.bootstrap.ServerBootstrap - Graylog server 2.2.1+4829190 starting up
graylog_1 | 2017-08-08 12:36:52,408 INFO : org.graylog2.bootstrap.ServerBootstrap - JRE: Oracle Corporation 1.8.0_72-internal on Linux 3.16.0-4-amd64
graylog_1 | 2017-08-08 12:36:52,408 INFO : org.graylog2.bootstrap.ServerBootstrap - Deployment: docker
graylog_1 | 2017-08-08 12:36:52,408 INFO : org.graylog2.bootstrap.ServerBootstrap - OS: Debian GNU/Linux 8 (jessie) (debian)
graylog_1 | 2017-08-08 12:36:52,409 INFO : org.graylog2.bootstrap.ServerBootstrap - Arch: amd64
graylog_1 | 2017-08-08 12:36:52,412 WARN : org.graylog2.shared.events.DeadEventLoggingListener - Received unhandled event of type <org.graylog2.plugin.lifecycles.Lifecycle> from event bus <AsyncEventBus{graylog-eventbus}>
graylog_1 | 2017-08-08 12:36:52,438 INFO : org.graylog2.shared.initializers.PeriodicalsService - Starting 26 periodicals ...
graylog_1 | 2017-08-08 12:36:52,439 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.periodical.ThroughputCalculator] periodical in [0s], polling every [1s].
graylog_1 | 2017-08-08 12:36:52,443 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.periodical.AlertScannerThread] periodical in [10s], polling every [60s].
graylog_1 | 2017-08-08 12:36:52,446 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.periodical.BatchedElasticSearchOutputFlushThread] periodical in [0s], polling every [1s].
graylog_1 | 2017-08-08 12:36:52,445 INFO : org.elasticsearch.node - [graylog-9497e0b0-254a-4e9f-acd5-555d10a7c348] starting ...
graylog_1 | 2017-08-08 12:36:52,448 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.periodical.ClusterHealthCheckThread] periodical in [120s], polling every [20s].
graylog_1 | 2017-08-08 12:36:52,448 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.periodical.ContentPackLoaderPeriodical] periodical, running forever.
graylog_1 | 2017-08-08 12:36:52,452 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.periodical.GarbageCollectionWarningThread] periodical, running forever.
graylog_1 | 2017-08-08 12:36:52,453 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.periodical.IndexerClusterCheckerThread] periodical in [0s], polling every [30s].
graylog_1 | 2017-08-08 12:36:52,454 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.periodical.IndexRetentionThread] periodical in [0s], polling every [300s].
graylog_1 | 2017-08-08 12:36:52,454 INFO : org.graylog2.periodical.IndexRetentionThread - Elasticsearch cluster not available, skipping index retention checks.
graylog_1 | 2017-08-08 12:36:52,454 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.periodical.IndexRotationThread] periodical in [0s], polling every [10s].
graylog_1 | 2017-08-08 12:36:52,456 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.periodical.NodePingThread] periodical in [0s], polling every [1s].
graylog_1 | 2017-08-08 12:36:52,457 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.periodical.VersionCheckThread] periodical in [300s], polling every [1800s].
graylog_1 | 2017-08-08 12:36:52,462 INFO : org.mongodb.driver.connection - Opened connection [connectionId{localValue:6, serverValue:49}] to 10.249.130.112:27017
graylog_1 | 2017-08-08 12:36:52,462 INFO : org.mongodb.driver.connection - Opened connection [connectionId{localValue:5, serverValue:48}] to 10.249.130.112:27017
graylog_1 | 2017-08-08 12:36:52,462 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.periodical.ThrottleStateUpdaterThread] periodical in [1s], polling every [1s].
graylog_1 | 2017-08-08 12:36:52,463 INFO : org.mongodb.driver.connection - Opened connection [connectionId{localValue:7, serverValue:50}] to 10.249.130.112:27017
graylog_1 | 2017-08-08 12:36:52,463 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.events.ClusterEventPeriodical] periodical in [0s], polling every [1s].
graylog_1 | 2017-08-08 12:36:52,463 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.events.ClusterEventCleanupPeriodical] periodical in [0s], polling every [86400s].
graylog_1 | 2017-08-08 12:36:52,465 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.periodical.ClusterIdGeneratorPeriodical] periodical, running forever.
graylog_1 | 2017-08-08 12:36:52,465 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.periodical.IndexRangesMigrationPeriodical] periodical, running forever.
graylog_1 | 2017-08-08 12:36:52,469 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.periodical.IndexRangesCleanupPeriodical] periodical in [15s], polling every [3600s].
graylog_1 | 2017-08-08 12:36:52,478 INFO : org.mongodb.driver.connection - Opened connection [connectionId{localValue:9, serverValue:52}] to 10.249.130.112:27017
graylog_1 | 2017-08-08 12:36:52,479 INFO : org.mongodb.driver.connection - Opened connection [connectionId{localValue:8, serverValue:51}] to 10.249.130.112:27017
graylog_1 | 2017-08-08 12:36:52,480 INFO : org.mongodb.driver.connection - Opened connection [connectionId{localValue:10, serverValue:53}] to 10.249.130.112:27017
graylog_1 | 2017-08-08 12:36:52,503 INFO : org.graylog2.periodical.IndexerClusterCheckerThread - Indexer not fully initialized yet. Skipping periodic cluster check.
graylog_1 | 2017-08-08 12:36:52,505 INFO : org.graylog2.shared.initializers.PeriodicalsService - Not starting [org.graylog2.periodical.UserPermissionMigrationPeriodical] periodical. Not configured to run on this node.
graylog_1 | 2017-08-08 12:36:52,505 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.periodical.AlarmCallbacksMigrationPeriodical] periodical, running forever.
graylog_1 | 2017-08-08 12:36:52,505 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.periodical.ConfigurationManagementPeriodical] periodical, running forever.
graylog_1 | 2017-08-08 12:36:52,512 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.periodical.LdapGroupMappingMigration] periodical, running forever.
graylog_1 | 2017-08-08 12:36:52,515 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.periodical.IndexFailuresPeriodical] periodical, running forever.
graylog_1 | 2017-08-08 12:36:52,516 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog.plugins.usagestatistics.UsageStatsNodePeriodical] periodical in [300s], polling every [21600s].
graylog_1 | 2017-08-08 12:36:52,518 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog.plugins.usagestatistics.UsageStatsClusterPeriodical] periodical in [300s], polling every [21600s].
graylog_1 | 2017-08-08 12:36:52,522 INFO : org.graylog2.shared.initializers.PeriodicalsService - Not starting [org.graylog.plugins.pipelineprocessor.periodical.LegacyDefaultStreamMigration] periodical. Not configured to run on this node.
graylog_1 | 2017-08-08 12:36:52,522 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog.plugins.collector.periodical.PurgeExpiredCollectorsThread] periodical in [0s], polling every [3600s].
graylog_1 | 2017-08-08 12:36:52,812 INFO : org.graylog2.migrations.V20161130141500_DefaultStreamRecalcIndexRanges - Cluster not connected yet, delaying migration until it is reachable.
graylog_1 | 2017-08-08 12:36:52,823 INFO : org.elasticsearch.transport - [graylog-9497e0b0-254a-4e9f-acd5-555d10a7c348] publish_address {10.249.130.112:9350}, bound_addresses {[::]:9350}
graylog_1 | 2017-08-08 12:36:52,829 INFO : org.elasticsearch.discovery - [graylog-9497e0b0-254a-4e9f-acd5-555d10a7c348] graylog/UH7JCMTjS-SD_dS4Of2S9Q
graylog_1 | 2017-08-08 12:36:53,114 INFO : org.graylog2.shared.initializers.JerseyService - Enabling CORS for HTTP endpoint
graylog_1 | 2017-08-08 12:36:55,832 WARN : org.elasticsearch.discovery - [graylog-9497e0b0-254a-4e9f-acd5-555d10a7c348] waited for 3s and no initial state was set by the discovery
graylog_1 | 2017-08-08 12:36:55,833 INFO : org.elasticsearch.node - [graylog-9497e0b0-254a-4e9f-acd5-555d10a7c348] started
graylog_1 | 2017-08-08 12:36:55,955 INFO : org.elasticsearch.cluster.service - [graylog-9497e0b0-254a-4e9f-acd5-555d10a7c348] detected_master {es-node-3}{PAO9r-f9Sv-KndRp6lb9ag}{10.249.130.116}{10.249.130.116:9300}, added {{es-node-5}{gDaUe636RhOJQjySE4SNgw}{10.249.130.117}{10.249.130.117:9300},{es-node-10}{nh6bS_WCQiyPoYULs8KfYw}{10.249.130.119}{10.249.130.119:9400},{es-node-3}{PAO9r-f9Sv-KndRp6lb9ag}{10.249.130.116}{10.249.130.116:9300},{es-node-9}{uoXTRkP5QlmcPWXTCjrapw}{10.249.130.119}{10.249.130.119:9300},{es-node-7}{zvJLtdyiR6q6WPnflGgeZA}{10.249.130.118}{10.249.130.118:9300},{es-node-4}{QqA6PWoNToGPyBKAySKFhQ}{10.249.130.116}{10.249.130.116:9400},{es-node-6}{dwZB_cQORTaoSbe6AnURhA}{10.249.130.117}{10.249.130.117:9400},{es-node-8}{roWM8VSBSAirxqkdS1JDAA}{10.249.130.118}{10.249.130.118:9400},{graylog-master}{5gxvdvbTQ4aQh--ibxoLuQ}{10.249.130.112}{10.249.130.112:9300}{data=false, master=true},{graylog-slave}{WQgDf8lgT82020liRajS3Q}{10.249.130.113}{10.249.130.113:9300}{data=false, master=false},}, reason: zen-disco-receive(from master [{es-node-3}{PAO9r-f9Sv-KndRp6lb9ag}{10.249.130.116}{10.249.130.116:9300}])
graylog_1 | 2017-08-08 12:36:55,999 INFO : org.elasticsearch.discovery.zen - [graylog-9497e0b0-254a-4e9f-acd5-555d10a7c348] updating discovery.zen.minimum_master_nodes from [-1] to [5]
graylog_1 | 2017-08-08 12:37:02,539 INFO : org.glassfish.grizzly.http.server.NetworkListener - Started listener bound to [0.0.0.0:12900]
graylog_1 | 2017-08-08 12:37:02,541 INFO : org.glassfish.grizzly.http.server.HttpServer - [HttpServer] Started.
graylog_1 | 2017-08-08 12:37:02,541 INFO : org.graylog2.shared.initializers.JerseyService - Started REST API at <http://0.0.0.0:12900/>
graylog_1 | 2017-08-08 12:37:02,543 INFO : org.graylog2.shared.initializers.JerseyService - Enabling CORS for HTTP endpoint
graylog_1 | 2017-08-08 12:37:04,655 INFO : org.glassfish.grizzly.http.server.NetworkListener - Started listener bound to [0.0.0.0:9000]
graylog_1 | 2017-08-08 12:37:04,655 INFO : org.glassfish.grizzly.http.server.HttpServer - [HttpServer-1] Started.
graylog_1 | 2017-08-08 12:37:04,656 INFO : org.graylog2.shared.initializers.JerseyService - Started Web Interface at <http://0.0.0.0:9000/>
graylog_1 | 2017-08-08 12:37:04,657 INFO : org.graylog2.shared.initializers.ServiceManagerListener - Services are healthy
graylog_1 | 2017-08-08 12:37:04,657 INFO : org.graylog2.shared.initializers.InputSetupService - Triggering launching persisted inputs, node transitioned from Uninitialized [LB:DEAD] to Running [LB:ALIVE]
graylog_1 | 2017-08-08 12:37:04,658 INFO : org.graylog2.bootstrap.ServerBootstrap - Services started, startup times in ms: {OutputSetupService [RUNNING]=2, BufferSynchronizerService [RUNNING]=3, JournalReader [RUNNING]=5, InputSetupService [RUNNING]=7, ConfigurationEtagService [RUNNING]=8, KafkaJournal [RUNNING]=9, StreamCacheService [RUNNING]=52, PeriodicalsService [RUNNING]=97, IndexerSetupService [RUNNING]=3570, JerseyService [RUNNING]=12219}
graylog_1 | 2017-08-08 12:37:04,661 INFO : org.graylog2.bootstrap.ServerBootstrap - Graylog server up and running.
graylog_1 | 2017-08-08 12:37:04,679 INFO : org.graylog2.inputs.InputStateListener - Input [GELF UDP/57a0c26cb3cb1f0001e5f845] is now STARTING
graylog_1 | 2017-08-08 12:37:04,714 WARN : org.graylog2.plugin.inputs.transports.NettyTransport - receiveBufferSize (SO_RCVBUF) for input GELFUDPInput{title=logstash, type=org.graylog2.inputs.gelf.udp.GELFUDPInput, nodeId=null} should be 262144 but is 212992.
graylog_1 | 2017-08-08 12:37:04,719 INFO : org.graylog2.inputs.InputStateListener - Input [GELF UDP/57a0c26cb3cb1f0001e5f845] is now RUNNING
No specific error on elasticsearch logs, so I don’t put them…
I tried to figure it out by removing /graylog/journal/* , it solved the issue but no for a long time. Now, nothing arrives to elastic cluster
I tried to restart graylog container, mongo too, restarted docker service but no luck, still no out to elastic cluster
Anybody has the same issue ? Have you some advice to find out what 's going wrong ?
Thanks for your help !