Graylog want start

asalma · July 30, 2018, 8:15am

Hi,

Please I can’t start graylog and I have this warn :

2018-07-30T10:08:16.081+02:00 WARN  [NettyTransport] receiveBufferSize (SO_RCVBUF) for input BeatsInput{title=frghcslnetv04, type=org.graylog.plugins.beats.BeatsInput, nodeId=82702595-8116-4676-a17d-ea606560cb75} should be 1048576 but is 212992.
2018-07-30T10:08:16.088+02:00 INFO  [InputStateListener] Input [Beats/5b27800b526a9b0c14ef077d] is now RUNNING
2018-07-30T10:08:16.101+02:00 INFO  [KafkaJournal] Read offset 284769557 before start of log at 290327808, starting to read from the beginning of the journal.
2018-07-30T10:08:23.229+02:00 INFO  [KafkaJournal] Read offset 284769557 before start of log at 290327808, starting to read from the beginning of the journal.
2018-07-30T10:10:00.628+02:00 ERROR [Cluster] Couldn't read cluster health for indices [graylog_*] (Could not connect to http://127.0.0.1:9200)
2018-07-30T10:10:00.628+02:00 INFO  [IndexerClusterCheckerThread] Indexer not fully initialized yet. Skipping periodic cluster check.
2018-07-30T10:11:30.584+02:00 WARN  [KafkaJournal] Journal utilization (108.0%) has gone over 95%.
2018-07-30T10:11:30.586+02:00 INFO  [KafkaJournal] Journal usage is 108.00% (threshold 100%), changing load balancer status from ALIVE to THROTTLED
2018-07-30T10:12:30.591+02:00 WARN  [KafkaJournal] Journal utilization (136.0%) has gone over 95%.
2018-07-30T10:13:30.578+02:00 WARN  [KafkaJournal] Journal utilization (137.0%) has gone over 95%.
2018-07-30T10:14:30.583+02:00 WARN  [KafkaJournal] Journal utilization (136.0%) has gone over 95%.
2018-07-30T10:15:30.584+02:00 WARN  [KafkaJournal] Journal utilization (135.0%) has gone over 95%.

How can I solve this ?

achrafbsecure · July 30, 2018, 10:45am

hi salma
can you show us the graylog status??
and see

github.com/Graylog2/graylog2-server

Elasticsearch cluster not available, skipping index retention checks

opened 03:43PM - 25 Jul 17 UTC

closed 06:39AM - 26 Jul 17 UTC

toddlers

## Current Behavior Graylog not able to connect to elasticsearch as I see in th…e logs ``` 2017-07-25T15:27:08.292Z INFO [node] [graylog-a2c36082-47d7-446d-acbd-84e010df1fcd] starting ... 2017-07-25T15:27:08.294Z INFO [Periodicals] Starting [org.graylog2.periodical.AlertScannerThread] periodical in [10s], polling every [60s]. 2017-07-25T15:27:08.294Z INFO [Periodicals] Starting [org.graylog2.periodical.BatchedElasticSearchOutputFlushThread] periodical in [0s], polling every [1s]. 2017-07-25T15:27:08.295Z INFO [Periodicals] Starting [org.graylog2.periodical.ClusterHealthCheckThread] periodical in [120s], polling every [20s]. 2017-07-25T15:27:08.296Z INFO [Periodicals] Starting [org.graylog2.periodical.ContentPackLoaderPeriodical] periodical, running forever. 2017-07-25T15:27:08.296Z INFO [Periodicals] Starting [org.graylog2.periodical.GarbageCollectionWarningThread] periodical, running forever. 2017-07-25T15:27:08.296Z INFO [Periodicals] Starting [org.graylog2.periodical.IndexerClusterCheckerThread] periodical in [0s], polling every [30s]. 2017-07-25T15:27:08.297Z INFO [Periodicals] Starting [org.graylog2.periodical.IndexRetentionThread] periodical in [0s], polling every [300s]. 2017-07-25T15:27:08.297Z INFO [Periodicals] Starting [org.graylog2.periodical.IndexRotationThread] periodical in [0s], polling every [10s]. 2017-07-25T15:27:08.297Z INFO [IndexRetentionThread] Elasticsearch cluster not available, skipping index retention checks. 2017-07-25T15:27:08.299Z INFO [Periodicals] Starting [org.graylog2.periodical.NodePingThread] periodical in [0s], polling every [1s]. 2017-07-25T15:27:08.300Z INFO [Periodicals] Starting [org.graylog2.periodical.VersionCheckThread] periodical in [300s], polling every [1800s]. 2017-07-25T15:27:08.301Z INFO [Periodicals] Starting [org.graylog2.periodical.ThrottleStateUpdaterThread] periodical in [1s], polling every [1s]. 2017-07-25T15:27:08.302Z INFO [Periodicals] Starting [org.graylog2.events.ClusterEventPeriodical] periodical in [0s], polling every [1s]. 2017-07-25T15:27:08.302Z INFO [Periodicals] Starting [org.graylog2.events.ClusterEventCleanupPeriodical] periodical in [0s], polling every [86400s]. 2017-07-25T15:27:08.303Z INFO [Periodicals] Starting [org.graylog2.periodical.ClusterIdGeneratorPeriodical] periodical, running forever. 2017-07-25T15:27:08.303Z INFO [Periodicals] Starting [org.graylog2.periodical.IndexRangesMigrationPeriodical] periodical, running forever. 2017-07-25T15:27:08.303Z INFO [Periodicals] Starting [org.graylog2.periodical.IndexRangesCleanupPeriodical] periodical in [15s], polling every [3600s]. 2017-07-25T15:27:08.304Z INFO [connection] Opened connection [connectionId{localValue:3, serverValue:330}] to 127.0.0.1:27017 2017-07-25T15:27:08.304Z INFO [connection] Opened connection [connectionId{localValue:5, serverValue:332}] to 127.0.0.1:27017 2017-07-25T15:27:08.304Z INFO [connection] Opened connection [connectionId{localValue:6, serverValue:333}] to 127.0.0.1:27017 2017-07-25T15:27:08.304Z INFO [connection] Opened connection [connectionId{localValue:4, serverValue:331}] to 127.0.0.1:27017 2017-07-25T15:27:08.307Z INFO [connection] Opened connection [connectionId{localValue:7, serverValue:334}] to 127.0.0.1:27017 2017-07-25T15:27:08.313Z INFO [connection] Opened connection [connectionId{localValue:8, serverValue:335}] to 127.0.0.1:27017 2017-07-25T15:27:08.313Z INFO [connection] Opened connection [connectionId{localValue:9, serverValue:336}] to 127.0.0.1:27017 2017-07-25T15:27:08.334Z INFO [PeriodicalsService] Not starting [org.graylog2.periodical.UserPermissionMigrationPeriodical] periodical. Not configured to run on this node. 2017-07-25T15:27:08.334Z INFO [Periodicals] Starting [org.graylog2.periodical.AlarmCallbacksMigrationPeriodical] periodical, running forever. 2017-07-25T15:27:08.335Z INFO [Periodicals] Starting [org.graylog2.periodical.ConfigurationManagementPeriodical] periodical, running forever. 2017-07-25T15:27:08.340Z INFO [Periodicals] Starting [org.graylog2.periodical.LdapGroupMappingMigration] periodical, running forever. 2017-07-25T15:27:08.340Z INFO [Periodicals] Starting [org.graylog2.periodical.IndexFailuresPeriodical] periodical, running forever. 2017-07-25T15:27:08.341Z INFO [Periodicals] Starting [org.graylog.plugins.usagestatistics.UsageStatsNodePeriodical] periodical in [300s], polling every [21600s]. 2017-07-25T15:27:08.341Z INFO [Periodicals] Starting [org.graylog.plugins.usagestatistics.UsageStatsClusterPeriodical] periodical in [300s], polling every [21600s]. 2017-07-25T15:27:08.343Z INFO [Periodicals] Starting [org.graylog.plugins.pipelineprocessor.periodical.LegacyDefaultStreamMigration] periodical, running forever. 2017-07-25T15:27:08.344Z INFO [Periodicals] Starting [org.graylog.plugins.collector.periodical.PurgeExpiredCollectorsThread] periodical in [0s], polling every [3600s]. 2017-07-25T15:27:08.428Z INFO [LegacyDefaultStreamMigration] Legacy default stream has no connections, no migration needed. 2017-07-25T15:27:08.444Z INFO [IndexerClusterCheckerThread] Indexer not fully initialized yet. Skipping periodic cluster check. 2017-07-25T15:27:08.509Z INFO [V20161130141500_DefaultStreamRecalcIndexRanges] Cluster not connected yet, delaying migration until it is reachable. 2017-07-25T15:27:08.566Z INFO [transport] [graylog-a2c36082-47d7-446d-acbd-84e010df1fcd] publish_address {127.0.0.1:9350}, bound_addresses {127.0.0.1:9350} 2017-07-25T15:27:08.570Z INFO [discovery] [graylog-a2c36082-47d7-446d-acbd-84e010df1fcd] graylog/5dFjDDDvRimPW9VOdogNQg 2017-07-25T15:27:08.684Z INFO [JerseyService] Enabling CORS for HTTP endpoint 2017-07-25T15:27:11.572Z WARN [discovery] [graylog-a2c36082-47d7-446d-acbd-84e010df1fcd] waited for 3s and no initial state was set by the discovery 2017-07-25T15:27:11.573Z INFO [node] [graylog-a2c36082-47d7-446d-acbd-84e010df1fcd] started 2017-07-25T15:27:11.626Z INFO [service] [graylog-a2c36082-47d7-446d-acbd-84e010df1fcd] detected_master {graylog}{alHd4ZyiROS7paFPZjU8sA}{94.130.32.23}{94.130.32.23:9300}{master=true}, added {{graylog}{alHd4ZyiROS7paFPZjU8sA}{94.130.32.23}{94.130.32.23:9300}{master=true},}, reason: zen-disco-receive(from master [{graylog}{alHd4ZyiROS7paFPZjU8sA}{94.130.32.23}{94.130.32.23:9300}{master=true}]) 2017-07-25T15:27:16.583Z INFO [NetworkListener] Started listener bound to [0.0.0.0:9000] 2017-07-25T15:27:16.584Z INFO [HttpServer] [HttpServer] Started. 2017-07-25T15:27:16.584Z INFO [JerseyService] Started REST API at <http://0.0.0.0:9000/api/> 2017-07-25T15:27:16.584Z INFO [JerseyService] Started Web Interface at <http://0.0.0.0:9000/> 2017-07-25T15:27:16.585Z INFO [ServiceManagerListener] Services are healthy 2017-07-25T15:27:16.586Z INFO [ServerBootstrap] Services started, startup times in ms: {InputSetupService [RUNNING]=8, KafkaJournal [RUNNING]=11, JournalReader [RUNNING]=11, OutputSetupService [RUNNING]=11, BufferSynchronizerService [RUNNING]=15, ConfigurationEtagService [RUNNING]=18, StreamCacheService [RUNNING]=130, PeriodicalsService [RUNNING]=142, IndexerSetupService [RUNNING]=3347, JerseyService [RUNNING]=8296} 2017-07-25T15:27:16.586Z INFO [InputSetupService] Triggering launching persisted inputs, node transitioned from Uninitialized [LB:DEAD] to Running [LB:ALIVE] 2017-07-25T15:27:16.594Z INFO [ServerBootstrap] Graylog server up and running. 2017-07-25T15:27:16.606Z INFO [InputStateListener] Input [GELF UDP/59774ed1dc3aaa50d9cc7d39] is now STARTING 2017-07-25T15:27:16.640Z WARN [NettyTransport] receiveBufferSize (SO_RCVBUF) for input GELFUDPInput{title=sao, type=org.graylog2.inputs.gelf.udp.GELFUDPInput, nodeId=null} should be 262144 but is 212992. 2017-07-25T15:27:16.642Z INFO [InputStateListener] Input [GELF UDP/59774ed1dc3aaa50d9cc7d39] is now RUNNING 2017-07-25T15:27:17.968Z DEBUG [OffsetIndex] Adding index entry 61454 => 32693296 to 00000000000000039096.index. 2017-07-25T15:27:18.054Z DEBUG [OffsetIndex] Adding index entry 61459 => 32700617 to 00000000000000039096.index. ``` ## Context I am sending some logs to UDP input of graylog . I can see the input is coming as the message count increasing , but there is no output . When I check in the logs its says `2017-07-25T15:27:08.297Z INFO [IndexRetentionThread] Elasticsearch cluster not available, skipping index retention checks.` ## Configurations are: # Everything is running on the same box # graylog server conf ``` is_master = True node_id_file = /etc/graylog/server/node-id password_secret =*********** root_username = admin root_password_sha2 = **************** root_email = root_timezone = UTC plugin_dir = /usr/share/graylog-server/plugin rest_listen_uri = http://0.0.0.0:9000/api/ rest_enable_cors = True rest_enable_gzip = True rest_enable_tls = False rest_tls_cert_file = /path/to/graylog.crt rest_tls_key_file = /path/to/graylog.key rest_tls_key_password = secret rest_max_header_size = 8192 rest_max_initial_line_length = 4096 rest_thread_pool_size = 16 web_enable = True web_listen_uri = http://0.0.0.0:9000/ web_enable_cors = True web_enable_gzip = True web_enable_tls = False web_tls_cert_file = web_tls_key_file = web_tls_key_password = web_max_header_size = 8192 web_max_initial_line_length = 4096 web_thread_pool_size = 16 rotation_strategy = count elasticsearch_max_docs_per_index = 20000000 elasticsearch_max_size_per_index = 1073741824 elasticsearch_max_time_per_index = 1d elasticsearch_disable_version_check = True no_retention = False elasticsearch_max_number_of_indices = 30 retention_strategy = delete elasticsearch_shards = 4 elasticsearch_replicas = 0 elasticsearch_index_prefix = graylog elasticsearch_template_name = graylog-internal allow_leading_wildcard_searches = False allow_highlighting = False elasticsearch_cluster_name = graylog elasticsearch_node_name_prefix = graylog- elasticsearch_discovery_zen_ping_unicast_hosts = 127.0.0.1:9300 elasticsearch_node_master = false elasticsearch_node_data = false elasticsearch_transport_tcp_port = 9350 elasticsearch_http_enabled = False elasticsearch_cluster_discovery_timeout = 5000 elasticsearch_network_host = elasticsearch_network_bind_host = elasticsearch_network_publish_host = elasticsearch_discovery_initial_state_timeout = 3s elasticsearch_analyzer = standard elasticsearch_request_timeout = 1m index_ranges_cleanup_interval = 1h output_batch_size = 25 output_flush_interval = 1 output_fault_count_threshold = 5 output_fault_penalty_seconds = 30 processbuffer_processors = 5 outputbuffer_processors = 3 outputbuffer_processor_keep_alive_time = 5000 outputbuffer_processor_threads_core_pool_size = 3 outputbuffer_processor_threads_max_pool_size = 30 udp_recvbuffer_sizes = 1048576 processor_wait_strategy = blocking ring_size = 65536 inputbuffer_ring_size = 65536 inputbuffer_processors = 2 inputbuffer_wait_strategy = blocking message_journal_enabled = True message_journal_dir = /var/lib/graylog-server/journal message_journal_max_age = 12h message_journal_max_size = 5gb message_journal_flush_age = 1m message_journal_flush_interval = 1000000 message_journal_segment_age = 1h message_journal_segment_size = 100mb async_eventbus_processors = 2 lb_recognition_period_seconds = 3 lb_throttle_threshold_percentage = 95 stream_processing_timeout = 2000 stream_processing_max_faults = 3 alert_check_interval = 60 output_module_timeout = 10000 stale_master_timeout = 2000 shutdown_timeout = 30000 mongodb_uri = mongodb://127.0.0.1:27017/graylog mongodb_max_connections = 100 mongodb_threads_allowed_to_block_multiplier = 5 rules_file = transport_email_enabled = False transport_email_hostname = transport_email_port = 587 transport_email_use_auth = True transport_email_use_tls = True transport_email_use_ssl = True transport_email_auth_username = transport_email_auth_password = transport_email_subject_prefix = [graylog] transport_email_from_email = transport_email_web_interface_url = http_connect_timeout = 5s http_read_timeout = 10s http_write_timeout = 10s disable_index_optimization = True index_optimization_max_num_segments = 1 gc_warning_threshold = 1s ldap_connection_timeout = 2000 disable_sigar = False dashboard_widget_default_cache_time = 10s content_packs_loader_enabled = True content_packs_dir = /usr/share/graylog-server/contentpacks content_packs_auto_load = proxied_requests_thread_pool_size = 32 ``` # elasticsearch ``` bootstrap.mlockall: false cluster.name: graylog discovery.zen.ping.multicast.enabled: false discovery.zen.ping.unicast.hosts: 127.0.0.1:9300 http.port: 9200 network.host: 0.0.0.0 node.data: true node.master: true node.name: graylog transport.tcp.port: 9300 path.conf: /etc/elasticsearch/graylog path.data: /var/lib/elasticsearch/graylog.example-graylog path.work: /tmp/elasticsearch/graylog.example-graylog path.logs: /var/log/elasticsearch/graylog.example-graylog ``` * when I check elasticsearch health that's green and graylog interface also shows everything good. ``` curl localhost:9200/_cluster/health?pretty=true { "cluster_name" : "graylog", "status" : "green", "timed_out" : false, "number_of_nodes" : 2, "number_of_data_nodes" : 1, "active_primary_shards" : 4, "active_shards" : 4, "relocating_shards" : 0, "initializing_shards" : 0, "unassigned_shards" : 0, "delayed_unassigned_shards" : 0, "number_of_pending_tasks" : 0, "number_of_in_flight_fetch" : 0, "task_max_waiting_in_queue_millis" : 0, "active_shards_percent_as_number" : 100.0 } ``` * I tried with sending a sample message as well, Elasticsearch works. ``` curl -X POST 'http://localhost:9200/tutorial/helloworld/1' -d '{ "message": "Hello World!" }' {"_index":"tutorial","_type":"helloworld","_id":"1","_version":1,"_shards":{"total":1,"successful":1,"failed":0},"created":true}root@graylog:/tmp# root@graylog:/tmp# curl -X GET 'http://localhost:9200/tutorial/helloworld/1' {"_index":"tutorial","_type":"helloworld","_id":"1","_version":1,"found":true,"_source":{ "message": "Hello World!" }}root@graylog:/tmp# ``` * Graylog Version: ``` graylog-server 2.2.3-1 ``` * Elasticsearch Version: ``` ii elasticsearch 2.4.3 ``` * MongoDB Version: ``` ii mongodb-org 3.2.15 amd64 MongoDB open source document-oriented database system (metapackage) ii mongodb-org-mongos 2.6.12 amd64 MongoDB sharded cluster query router ii mongodb-org-server 3.2.15 amd64 MongoDB database server ii mongodb-org-shell 3.2.15 amd64 MongoDB shell client ii mongodb-org-tools 2.6.12 amd64 MongoDB tools ``` * Operating System: ``` Distributor ID: Ubuntu Description: Ubuntu 16.04.2 LTS Release: 16.04 Codename: xenial ``` I would be glad to provide more information if any needed .

github.com/Graylog2/graylog2-server

ERROR [IndexerSetupService] Could not connect to Elasticsearch: Connection refused

opened 11:25AM - 14 Mar 16 UTC

closed 12:18PM - 14 Mar 16 UTC

mshar039

question

### Problem description Could not connect to Elasticsearch after 2 weeks of suc…cessful run. I was able to run the graylog and elasticsearch services properly. After feeding logs file to the elasticsearch through logstash, cluster health became yellow and soon after that graylog released the elasticsearch cluster. elasticsearch/elasticsearch.yml ``` cluster.name: graylog2 node.master: true node.data: true bootstrap.mlockall: true network.host: 127.0.0.1 discovery.zen.ping.multicast.enabled: false discovery.zen.ping.unicast.hosts: ["127.0.0.1:9300"] ``` graylog/server/server.conf ``` rest_listen_uri = http://127.0.0.1:12900/ elasticsearch_discovery_zen_ping_multicast_enabled = false elasticsearch_discovery_zen_ping_unicast_hosts = 127.0.0.1:9300 ``` ### Environment - Graylog Version: v 1.3 - Elasticsearch Version: v 1.7.3 - Logstash Version: v 2.2.2 - Operating System: CentOS 7 ``` 2016-03-14T04:27:05.417+05:30 INFO [CmdLineTool] Loaded plugins: [Anonymous Usage Statistics 1.2.1 [org.graylog.plugins.usagestatistics.UsageStatsPlugin]] 2016-03-14T04:27:05.498+05:30 INFO [CmdLineTool] Running with JVM arguments: -Xms1g -Xmx1g -XX:NewRatio=1 -XX:+ResizeTLAB -XX:+UseConcMarkSweepGC -XX:+CMSConcurrentMTEnabled -XX:+CMSClassUnloadingEnabled -XX:+UseParNewGC -XX:-OmitStackTraceInFastThrow -Xms2G -Xmx4G -Dlog4j.configuration=file:///etc/graylog/server/log4j.xml -Djava.library.path=/usr/share/graylog-server/lib/sigar -Dgraylog2.installation_source=rpm 2016-03-14T04:27:07.485+05:30 INFO [InputBufferImpl] Message journal is enabled. 2016-03-14T04:27:07.667+05:30 INFO [LogManager] Loading logs. 2016-03-14T04:27:07.727+05:30 INFO [LogManager] Logs loading complete. 2016-03-14T04:27:07.728+05:30 INFO [KafkaJournal] Initialized Kafka based journal at /var/lib/graylog-server/journal 2016-03-14T04:27:07.739+05:30 INFO [InputBufferImpl] Initialized InputBufferImpl with ring size <65536> and wait strategy <BlockingWaitStrategy>, running 2 parallel message handlers. 2016-03-14T04:27:08.006+05:30 INFO [NodeId] Node ID: dbc313e5-3eba-4cda-b8b1-6707b04a93d8 2016-03-14T04:27:08.234+05:30 INFO [node] [graylog2-server] version[1.7.3], pid[333], build[05d4530/2015-10-15T09:14:17Z] 2016-03-14T04:27:08.234+05:30 INFO [node] [graylog2-server] initializing ... 2016-03-14T04:27:08.304+05:30 INFO [plugins] [graylog2-server] loaded [graylog-monitor], sites [] 2016-03-14T04:27:10.127+05:30 INFO [node] [graylog2-server] initialized 2016-03-14T04:27:10.174+05:30 INFO [Version] HV000001: Hibernate Validator 5.2.2.Final 2016-03-14T04:27:10.302+05:30 INFO [ProcessBuffer] Initialized ProcessBuffer with ring size <65536> and wait strategy <BlockingWaitStrategy>. 2016-03-14T04:27:11.682+05:30 INFO [RulesEngineProvider] No static rules file loaded. 2016-03-14T04:27:11.730+05:30 INFO [OutputBuffer] Initialized OutputBuffer with ring size <65536> and wait strategy <BlockingWaitStrategy>. 2016-03-14T04:27:12.441+05:30 INFO [ServerBootstrap] Graylog server 1.3.3 (0fda9dc) starting up 2016-03-14T04:27:12.441+05:30 INFO [ServerBootstrap] JRE: Oracle Corporation 1.8.0_71 on Linux 3.10.0-327.10.1.el7.x86_64 2016-03-14T04:27:12.441+05:30 INFO [ServerBootstrap] Deployment: rpm 2016-03-14T04:27:12.441+05:30 INFO [ServerBootstrap] OS: CentOS Linux release 7.2.1511 (Core) 2016-03-14T04:27:12.441+05:30 INFO [ServerBootstrap] Arch: amd64 2016-03-14T04:27:12.560+05:30 INFO [PeriodicalsService] Starting 24 periodicals ... 2016-03-14T04:27:12.564+05:30 INFO [Periodicals] Starting [org.graylog2.periodical.ThroughputCalculator] periodical in [0s], polling every [1s]. 2016-03-14T04:27:12.572+05:30 INFO [Periodicals] Starting [org.graylog2.periodical.AlertScannerThread] periodical in [10s], polling every [60s]. 2016-03-14T04:27:12.579+05:30 INFO [Periodicals] Starting [org.graylog2.periodical.BatchedElasticSearchOutputFlushThread] periodical in [0s], polling every [1s]. 2016-03-14T04:27:12.579+05:30 INFO [Periodicals] Starting [org.graylog2.periodical.ClusterHealthCheckThread] periodical in [0s], polling every [20s]. 2016-03-14T04:27:12.581+05:30 INFO [node] [graylog2-server] starting ... 2016-03-14T04:27:12.584+05:30 INFO [Periodicals] Starting [org.graylog2.periodical.ContentPackLoaderPeriodical] periodical, running forever. 2016-03-14T04:27:12.585+05:30 INFO [Periodicals] Starting [org.graylog2.periodical.DeadLetterThread] periodical, running forever. 2016-03-14T04:27:12.586+05:30 INFO [Periodicals] Starting [org.graylog2.periodical.GarbageCollectionWarningThread] periodical, running forever. 2016-03-14T04:27:12.586+05:30 INFO [Periodicals] Starting [org.graylog2.periodical.IndexerClusterCheckerThread] periodical in [0s], polling every [30s]. 2016-03-14T04:27:12.588+05:30 INFO [Periodicals] Starting [org.graylog2.periodical.IndexRetentionThread] periodical in [0s], polling every [300s]. 2016-03-14T04:27:12.589+05:30 INFO [Periodicals] Starting [org.graylog2.periodical.IndexRotationThread] periodical in [0s], polling every [10s]. 2016-03-14T04:27:12.589+05:30 INFO [IndexRetentionThread] Elasticsearch cluster not available, skipping index retention checks. 2016-03-14T04:27:12.589+05:30 INFO [Periodicals] Starting [org.graylog2.periodical.NodePingThread] periodical in [0s], polling every [1s]. 2016-03-14T04:27:12.590+05:30 INFO [Periodicals] Starting [org.graylog2.periodical.StreamThroughputCounterManagerThread] periodical in [0s], polling every [1s]. 2016-03-14T04:27:12.598+05:30 INFO [Periodicals] Starting [org.graylog2.periodical.VersionCheckThread] periodical in [300s], polling every [1800s]. 2016-03-14T04:27:12.598+05:30 INFO [Periodicals] Starting [org.graylog2.periodical.ThrottleStateUpdaterThread] periodical in [1s], polling every [1s]. 2016-03-14T04:27:12.599+05:30 INFO [Periodicals] Starting [org.graylog2.events.ClusterEventPeriodical] periodical in [0s], polling every [1s]. 2016-03-14T04:27:12.599+05:30 INFO [Periodicals] Starting [org.graylog2.events.ClusterEventCleanupPeriodical] periodical in [0s], polling every [300s]. 2016-03-14T04:27:12.599+05:30 INFO [Periodicals] Starting [org.graylog2.periodical.ClusterIdGeneratorPeriodical] periodical, running forever. 2016-03-14T04:27:12.600+05:30 INFO [Periodicals] Starting [org.graylog2.periodical.PurgeExpiredCollectorsThread] periodical in [0s], polling every [3600s]. 2016-03-14T04:27:12.600+05:30 INFO [Periodicals] Starting [org.graylog2.periodical.IndexRangesMigrationPeriodical] periodical, running forever. 2016-03-14T04:27:12.601+05:30 INFO [Periodicals] Starting [org.graylog2.periodical.IndexRangesCleanupPeriodical] periodical in [15s], polling every [3600s]. 2016-03-14T04:27:12.907+05:30 INFO [IndexerClusterCheckerThread] Indexer not fully initialized yet. Skipping periodic cluster check. 2016-03-14T04:27:12.942+05:30 INFO [transport] [graylog2-server] bound_address {inet[/127.0.0.1:9350]}, publish_address {inet[/127.0.0.1:9350]} 2016-03-14T04:27:12.993+05:30 INFO [discovery] [graylog2-server] graylog2/fUMzabR0Rk-rlztj1tDFTg 2016-03-14T04:27:13.033+05:30 INFO [PeriodicalsService] Not starting [org.graylog2.periodical.UserPermissionMigrationPeriodical] periodical. Not configured to run on this node. 2016-03-14T04:27:13.033+05:30 INFO [Periodicals] Starting [org.graylog2.periodical.AlarmCallbacksMigrationPeriodical] periodical, running forever. 2016-03-14T04:27:13.036+05:30 INFO [Periodicals] Starting [org.graylog.plugins.usagestatistics.UsageStatsNodePeriodical] periodical in [300s], polling every [21600s]. 2016-03-14T04:27:13.036+05:30 INFO [Periodicals] Starting [org.graylog.plugins.usagestatistics.UsageStatsClusterPeriodical] periodical in [300s], polling every [21600s]. 2016-03-14T04:27:15.995+05:30 WARN [discovery] [graylog2-server] waited for 3s and no initial state was set by the discovery 2016-03-14T04:27:15.995+05:30 INFO [node] [graylog2-server] started 2016-03-14T04:27:16.860+05:30 INFO [RestApiService] Adding security context factory: <org.graylog2.security.ShiroSecurityContextFactory@1c59b0b> 2016-03-14T04:27:16.879+05:30 INFO [RestApiService] Started REST API at <http://127.0.0.1:12900/> 2016-03-14T04:27:20.998+05:30 INFO [IndexerSetupService] Checking Elasticsearch HTTP API at http://127.0.0.1:9200/ 2016-03-14T04:27:21.208+05:30 ERROR [IndexerSetupService] Could not connect to Elasticsearch: Connection refused 2016-03-14T04:27:21.210+05:30 WARN [IndexerSetupService] Could not connect to Elasticsearch 2016-03-14T04:27:21.210+05:30 INFO [IndexerSetupService] If you're using multicast, check that it is working in your network and that Elasticsearch is accessible. Also check that the cluster name setting is correct. 2016-03-14T04:27:21.210+05:30 INFO [IndexerSetupService] See http://docs.graylog.org/en/1.3/pages/configuring_es.html for details. 2016-03-14T04:27:21.211+05:30 INFO [ServiceManagerListener] Services are healthy 2016-03-14T04:27:21.211+05:30 INFO [InputSetupService] Triggering launching persisted inputs, node transitioned from Uninitialized [LB:DEAD] to Running [LB:ALIVE] 2016-03-14T04:27:21.212+05:30 INFO [ServerBootstrap] Services started, startup times in ms: {InputSetupService [RUNNING]=10, KafkaJournal [RUNNING]=13, OutputSetupService [RUNNING]=23, BufferSynchronizerService [RUNNING]=24, DashboardRegistryService [RUNNING]=30, MetricsReporterService [RUNNING]=30, JournalReader [RUNNING]=74, PeriodicalsService [RUNNING]=481, RestApiService [RUNNING]=4321, IndexerSetupService [RUNNING]=8643} 2016-03-14T04:27:21.267+05:30 INFO [ServerBootstrap] Graylog server up and running. 2016-03-14T04:27:21.271+05:30 INFO [InputStateListener] Input [GELF TCP/56e08fbe327c4c10e139577c] is now STARTING 2016-03-14T04:27:21.300+05:30 INFO [InputStateListener] Input [GELF TCP/56e08fbe327c4c10e139577c] is now RUNNING 2016-03-14T04:27:22.352+05:30 INFO [AbstractValidatingSessionManager] Enabling session validation scheduler... 2016-03-14T04:27:27.601+05:30 INFO [IndexRangesCleanupPeriodical] Skipping index range cleanup because the Elasticsearch cluster is unreachable or unhealthy 2016-03-14T04:28:12.590+05:30 INFO [IndexerClusterCheckerThread] Indexer not fully initialized yet. Skipping periodic cluster check. 2016-03-14T04:28:42.593+05:30 INFO [IndexerClusterCheckerThread] Indexer not fully initialized yet. Skipping periodic cluster check. 2016-03-14T04:29:12.595+05:30 INFO [IndexerClusterCheckerThread] Indexer not fully initialized yet. Skipping periodic cluster check. 2016-03-14T04:29:42.597+05:30 INFO [IndexerClusterCheckerThread] Indexer not fully initialized yet. Skipping periodic cluster check. 2016-03-14T04:30:12.600+05:30 INFO [IndexerClusterCheckerThread] Indexer not fully initialized yet. Skipping periodic cluster check. ``` I am new to working with tools and linux environment. Please help me with this.

asalma · July 30, 2018, 12:04pm

[root@frghcslnetv11 ~]# systemctl status graylog-server.service
● graylog-server.service - Graylog server
   Loaded: loaded (/usr/lib/systemd/system/graylog-server.service; enabled; vendor preset: disabled)
   Active: active (running) since Mon 2018-07-30 11:53:20 CEST; 2h 10min ago
     Docs: http://docs.graylog.org/
 Main PID: 23215 (graylog-server)
    Tasks: 122
   CGroup: /system.slice/graylog-server.service
           ├─23215 /bin/sh /usr/share/graylog-server/bin/graylog-server
           └─23216 /usr/bin/java -Xms4g -Xmx4g -XX:NewRatio=1 -server -XX:+ResizeTLAB -XX:+UseConcMarkSweepGC -XX:+CMSConcurrentMTEnabled -XX:+CMSClassUnloadingEnabled -XX:+U...

Jul 30 11:53:20 frghcslnetv11 systemd[1]: Started Graylog server.
Jul 30 11:53:20 frghcslnetv11 systemd[1]: Starting Graylog server...

asalma · July 30, 2018, 12:16pm

On Elasticsearch logs, I have this warn :

[2018-07-30T10:29:18,258][WARN ][o.e.i.e.Engine           ] [9hSWSDU] [graylog_6][3] failed to rollback writer on close
java.nio.file.NoSuchFileException: /data/elasticsearch/nodes/0/indices/S7Pj25k1Rj6bark5lKr-OQ/3/index
        at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86) ~[?:?]
        at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) ~[?:?]
        at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) ~[?:?]
        at sun.nio.fs.UnixFileSystemProvider.newDirectoryStream(UnixFileSystemProvider.java:427) ~[?:?]
        at java.nio.file.Files.newDirectoryStream(Files.java:457) ~[?:1.8.0_171]
        at org.apache.lucene.store.FSDirectory.listAll(FSDirectory.java:215) ~[lucene-core-6.6.1.jar:6.6.1 9aa465a89b64ff2dabe7b4d50c472de32c298683 - varunthacker - 2017-08-29 21:54:39]
        at org.apache.lucene.store.FSDirectory.listAll(FSDirectory.java:234) ~[lucene-core-6.6.1.jar:6.6.1 9aa465a89b64ff2dabe7b4d50c472de32c298683 - varunthacker - 2017-08-29 21:54:39]
        at org.apache.lucene.store.FilterDirectory.listAll(FilterDirectory.java:57) ~[lucene-core-6.6.1.jar:6.6.1 9aa465a89b64ff2dabe7b4d50c472de32c298683 - varunthacker - 2017-08-29 21:54:39]
        at org.apache.lucene.store.FilterDirectory.listAll(FilterDirectory.java:57) ~[lucene-core-6.6.1.jar:6.6.1 9aa465a89b64ff2dabe7b4d50c472de32c298683 - varunthacker - 2017-08-29 21:54:39]
        at org.apache.lucene.store.FilterDirectory.listAll(FilterDirectory.java:57) ~[lucene-core-6.6.1.jar:6.6.1 9aa465a89b64ff2dabe7b4d50c472de32c298683 - varunthacker - 2017-08-29 21:54:39]
        at org.apache.lucene.index.IndexFileDeleter.refresh(IndexFileDeleter.java:429) ~[lucene-core-6.6.1.jar:6.6.1 9aa465a89b64ff2dabe7b4d50c472de32c298683 - varunthacker - 2017-08-29 21:54:39]
        at org.apache.lucene.index.IndexWriter.rollbackInternalNoCommit(IndexWriter.java:2249) ~[lucene-core-6.6.1.jar:6.6.1 9aa465a89b64ff2dabe7b4d50c472de32c298683 - varunthacker - 2017-08-29 21:54:39]
        at org.apache.lucene.index.IndexWriter.rollbackInternal(IndexWriter.java:2193) ~[lucene-core-6.6.1.jar:6.6.1 9aa465a89b64ff2dabe7b4d50c472de32c298683 - varunthacker - 2017-08-29 21:54:39]
        at org.apache.lucene.index.IndexWriter.rollback(IndexWriter.java:2186) ~[lucene-core-6.6.1.jar:6.6.1 9aa465a89b64ff2dabe7b4d50c472de32c298683 - varunthacker - 2017-08-29 21:54:39]
        at org.elasticsearch.index.engine.InternalEngine.closeNoLock(InternalEngine.java:1342) ~[elasticsearch-5.6.10.jar:5.6.10]
        at org.elasticsearch.index.engine.Engine.close(Engine.java:1252) ~[elasticsearch-5.6.10.jar:5.6.10]
        at org.elasticsearch.index.engine.Engine.flushAndClose(Engine.java:1239) ~[elasticsearch-5.6.10.jar:5.6.10]
        at org.elasticsearch.index.shard.IndexShard.close(IndexShard.java:920) ~[elasticsearch-5.6.10.jar:5.6.10]
        at org.elasticsearch.index.IndexService.closeShard(IndexService.java:407) ~[elasticsearch-5.6.10.jar:5.6.10]
        at org.elasticsearch.index.IndexService.removeShard(IndexService.java:390) ~[elasticsearch-5.6.10.jar:5.6.10]
        at org.elasticsearch.index.IndexService.close(IndexService.java:246) ~[elasticsearch-5.6.10.jar:5.6.10]
        at org.elasticsearch.indices.IndicesService.removeIndex(IndicesService.java:539) ~[elasticsearch-5.6.10.jar:5.6.10]
        at org.elasticsearch.indices.IndicesService.lambda$doStop$2(IndicesService.java:240) ~[elasticsearch-5.6.10.jar:5.6.10]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_171]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_171]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_171]

achrafbsecure · July 30, 2018, 12:49pm

did u check ur the capacity of ur disque ??

asalma · July 31, 2018, 12:00pm

I add more CPU and RAM and now I have those errors :

[root@frghcslnetv11 graylog-server]# tail -30 server.log
2018-07-31T13:47:49.502+02:00 ERROR [NettyTransport] Error in Input [Beats/5b27800b526a9b0c14ef077d] (channel [id: 0x2e830dd8, /172.16.250.20:35448 => /172.16.250.19:5044])
java.lang.OutOfMemoryError: Java heap space
2018-07-31T13:43:40.321+02:00 ERROR [NettyTransport] Error in Input [Beats/5b27800b526a9b0c14ef077d] (channel [id: 0x3c3c31d6, /172.16.250.20:35148 :> /172.16.250.19:5044])
java.lang.OutOfMemoryError: Java heap space
2018-07-31T13:47:49.502+02:00 ERROR [NettyTransport] Error in Input [Beats/5b27800b526a9b0c14ef077d] (channel [id: 0x23cfa1c7, /172.16.250.20:35597 :> /172.16.250.19:5044])
java.lang.OutOfMemoryError: Java heap space
2018-07-31T13:48:08.175+02:00 ERROR [NettyTransport] Error in Input [Beats/5b27800b526a9b0c14ef077d] (channel [id: 0x2e830dd8, /172.16.250.20:35448 :> /172.16.250.19:5044])
java.lang.OutOfMemoryError: Java heap space
2018-07-31T13:47:49.502+02:00 ERROR [NettyTransport] Error in Input [Beats/5b27800b526a9b0c14ef077d] (channel [id: 0xebd88267, /172.16.250.20:39069 :> /172.16.250.19:5044])
java.lang.OutOfMemoryError: Java heap space
2018-07-31T13:44:31.316+02:00 ERROR [NettyTransport] Error in Input [Beats/5b27800b526a9b0c14ef077d] (channel [id: 0xf6ab20c3, /172.16.250.20:36621 :> /172.16.250.19:5044])
java.lang.OutOfMemoryError: Java heap space
2018-07-31T13:50:42.483+02:00 ERROR [NettyTransport] Error in Input [Beats/5b27800b526a9b0c14ef077d] (channel [id: 0x3ef16c26, /172.16.250.20:38603 => /172.16.250.19:5044])
java.lang.OutOfMemoryError: Java heap space
2018-07-31T13:51:09.470+02:00 ERROR [NettyTransport] Error in Input [Beats/5b27800b526a9b0c14ef077d] (channel [id: 0xae75fdd2, /172.16.250.20:41738 => /172.16.250.19:5044])
java.lang.OutOfMemoryError: Java heap space
2018-07-31T13:52:49.141+02:00 ERROR [NettyTransport] Error in Input [Beats/5b27800b526a9b0c14ef077d] (channel [id: 0x9bae2490, /172.16.250.20:34734 => /172.16.250.19:5044])
java.lang.OutOfMemoryError: Java heap space
2018-07-31T13:43:58.206+02:00 ERROR [NettyTransport] Error in Input [Beats/5b27800b526a9b0c14ef077d] (channel [id: 0x73c99663, /172.16.250.20:37480 :> /172.16.250.19:5044])
java.lang.OutOfMemoryError: Java heap space
2018-07-31T13:49:51.012+02:00 ERROR [NettyTransport] Error in Input [Beats/5b27800b526a9b0c14ef077d] (channel [id: 0x45b634d5, /172.16.250.20:38234 :> /172.16.250.19:5044])
java.lang.OutOfMemoryError: Java heap space
2018-07-31T13:52:08.194+02:00 ERROR [NettyTransport] Error in Input [Beats/5b27800b526a9b0c14ef077d] (channel [id: 0xcfb1f15a, /172.16.250.20:39186 :> /172.16.250.19:5044])
java.lang.OutOfMemoryError: Java heap space
2018-07-31T13:56:27.694+02:00 ERROR [NettyTransport] Error in Input [Beats/5b27800b526a9b0c14ef077d] (channel [id: 0x4367245d, /172.16.250.20:34741 => /172.16.250.19:5044])
java.lang.OutOfMemoryError: Java heap space
2018-07-31T13:54:10.592+02:00 ERROR [NettyTransport] Error in Input [Beats/5b27800b526a9b0c14ef077d] (channel [id: 0x12020977, /172.16.250.20:39104 => /172.16.250.19:5044])
java.lang.OutOfMemoryError: Java heap space
2018-07-31T13:53:34.624+02:00 ERROR [NettyTransport] Error in Input [Beats/5b27800b526a9b0c14ef077d] (channel [id: 0x9bae2490, /172.16.250.20:34734 :> /172.16.250.19:5044])
java.lang.OutOfMemoryError: Java heap space

achrafbsecure · July 31, 2018, 12:50pm

ur virtual machine haven’t enough space to run graylog i think check ur disk by using the command line df-v if u see that ur disck is totally check the directory /var/log (df -h ) and delete the logs and i think it wil works

asalma · July 31, 2018, 2:03pm

[root@frghcslnetv11 server]# df -h
Filesystem                 Size  Used Avail Use% Mounted on
/dev/mapper/vgroot-root     22G  9.9G   12G  47% /
devtmpfs                   3.8G     0  3.8G   0% /dev
tmpfs                      3.9G     0  3.9G   0% /dev/shm
tmpfs                      3.9G   21M  3.8G   1% /run
tmpfs                      3.9G     0  3.9G   0% /sys/fs/cgroup
/dev/sda1                 1014M  261M  754M  26% /boot
/dev/mapper/vgdata-lvdata  196G  4.7G  183G   3% /data
tmpfs                      781M     0  781M   0% /run/user/6012
tmpfs                      781M     0  781M   0% /run/user/0

And I don’t know why there is data on the buffer and nothing goeas out on elasticsearch ?

achrafbsecure · July 31, 2018, 2:12pm

did you check the health of your cluster if it is green or not ??

asalma · August 1, 2018, 7:38am

Yes the cluster is green

achrafbsecure · August 1, 2018, 7:57am

you can check this link

hada lah

system · August 15, 2018, 7:57am

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Logs not appear instantly in graylog search Graylog Central (peer support)	6	1176	July 30, 2019
Graylog INPUT Beat Graylog Central (peer support) sidecar , nxlog , filebeat-linux , filebeat-windows , winlogbeat , nodatanx , nosendlogfblx , clfblx	10	5837	July 31, 2020
ERROR [NettyTransport] Error in Input [Beats/] Graylog Central (peer support)	2	482	March 4, 2019
Could not get journal information when clicking on system/nodes Graylog Central (peer support)	3	1116	March 17, 2020
Sending logs from filebeat to kafka and kafka to graylog Graylog Central (peer support)	8	3017	October 30, 2019

Graylog want start

Related topics