Graylog 2.3.1: "Show received message" loading forever

Hi

Need help on pointing out where I did wrong.
I looked at similar search loading error but not able to fix my own problem.

Search loading taking forever

I am planning to upgrade from graylog 2.2.3(all-in-one,centos 7) to 2.3.1.
Not sure about the success of this upgrade path. I created another VM instance from centos doc to refresh my memory about mongod,elasticsearch and graylog.

  • Here is the problem when I do system->input->syslog tcp input->show received message. I have one Unix host sending small amount syslog message into this new graylog.

image

my config info with private info masked.

  • OS info
[root@syslog02 mongodb]# rpm -qa |egrep 'mongodb-org|elasticsearch|graylog'
graylog-server-2.3.1-1.noarch
mongodb-org-shell-3.2.17-1.el7.x86_64
mongodb-org-3.2.17-1.el7.x86_64
elasticsearch-5.6.3-1.noarch
mongodb-org-mongos-3.2.17-1.el7.x86_64
mongodb-org-server-3.2.17-1.el7.x86_64
graylog-2.3-repository-1-5.noarch
mongodb-org-tools-3.2.17-1.el7.x86_64
[root@syslog02 mongodb]# cat /etc/redhat-release
CentOS Linux release 7.4.1708 (Core)
[root@syslog02 mongodb]#

  • Graylog 2.3.1 info
    • /var/log/graylog-server/server.log
2017-10-14T08:27:07.856-04:00 INFO  [CmdLineTool] Loaded plugin: Elastic Beats Input 2.3.1 [org.graylog.plugins.beats.BeatsInputPlugin]
2017-10-14T08:27:07.858-04:00 INFO  [CmdLineTool] Loaded plugin: Collector 2.3.1 [org.graylog.plugins.collector.CollectorPlugin]
2017-10-14T08:27:07.859-04:00 INFO  [CmdLineTool] Loaded plugin: Enterprise Integration Plugin 2.3.1 [org.graylog.plugins.enterprise_integration.EnterpriseIntegrationPlugin]
2017-10-14T08:27:07.859-04:00 INFO  [CmdLineTool] Loaded plugin: MapWidgetPlugin 2.3.1 [org.graylog.plugins.map.MapWidgetPlugin]
2017-10-14T08:27:07.867-04:00 INFO  [CmdLineTool] Loaded plugin: Pipeline Processor Plugin 2.3.1 [org.graylog.plugins.pipelineprocessor.ProcessorPlugin]
2017-10-14T08:27:07.868-04:00 INFO  [CmdLineTool] Loaded plugin: Anonymous Usage Statistics 2.3.1 [org.graylog.plugins.usagestatistics.UsageStatsPlugin]
2017-10-14T08:27:08.085-04:00 INFO  [CmdLineTool] Running with JVM arguments: -Xms1g -Xmx1g -XX:NewRatio=1 -XX:+ResizeTLAB -XX:+UseConcMarkSweepGC -XX:+CMSConcurrentMTEnabled -XX:+CMSClassUnloadingEnabled -XX:+UseParNewGC -XX:-OmitStackTraceInFastThrow -Dlog4j.configurationFile=file:///etc/graylog/server/log4j2.xml -Djava.library.path=/usr/share/graylog-server/lib/sigar -Dgraylog2.installation_source=rpm
2017-10-14T08:27:08.270-04:00 INFO  [Version] HV000001: Hibernate Validator null
2017-10-14T08:27:10.199-04:00 INFO  [InputBufferImpl] Message journal is enabled.
2017-10-14T08:27:10.220-04:00 INFO  [NodeId] Node ID: 651c724e-ef23-4983-81cf-e4a9836585bf
2017-10-14T08:27:10.399-04:00 INFO  [LogManager] Loading logs.
2017-10-14T08:27:10.445-04:00 INFO  [LogManager] Logs loading complete.
2017-10-14T08:27:10.445-04:00 INFO  [KafkaJournal] Initialized Kafka based journal at /var/lib/graylog-server/journal
2017-10-14T08:27:10.458-04:00 INFO  [InputBufferImpl] Initialized InputBufferImpl with ring size <65536> and wait strategy <BlockingWaitStrategy>, running 2 parallel message handlers.
2017-10-14T08:27:10.476-04:00 INFO  [cluster] Cluster created with settings {hosts=[localhost:27017], mode=SINGLE, requiredClusterType=UNKNOWN, serverSelectionTimeout='30000 ms', maxWaitQueueSize=5000}
2017-10-14T08:27:10.515-04:00 INFO  [cluster] No server chosen by ReadPreferenceServerSelector{readPreference=primary} from cluster description ClusterDescription{type=UNKNOWN, connectionMode=SINGLE, serverDescriptions=[ServerDescription{address=localhost:27017, type=UNKNOWN, state=CONNECTING}]}. Waiting for 30000 ms before timing out
2017-10-14T08:27:10.532-04:00 INFO  [connection] Opened connection [connectionId{localValue:1, serverValue:26}] to localhost:27017
2017-10-14T08:27:10.534-04:00 INFO  [cluster] Monitor thread successfully connected to server with description ServerDescription{address=localhost:27017, type=STANDALONE, state=CONNECTED, ok=true, version=ServerVersion{versionList=[3, 2, 17]}, minWireVersion=0, maxWireVersion=4, maxDocumentSize=16777216, roundTripTimeNanos=430366}
2017-10-14T08:27:10.539-04:00 INFO  [connection] Opened connection [connectionId{localValue:2, serverValue:27}] to localhost:27017
2017-10-14T08:27:10.824-04:00 INFO  [AbstractJestClient] Setting server pool to a list of 1 servers: [http://100.65.184.12:9200]
2017-10-14T08:27:10.825-04:00 INFO  [JestClientFactory] Using multi thread/connection supporting pooling connection manager
2017-10-14T08:27:10.894-04:00 INFO  [JestClientFactory] Using custom ObjectMapper instance
2017-10-14T08:27:10.894-04:00 INFO  [JestClientFactory] Node Discovery disabled...
2017-10-14T08:27:10.894-04:00 INFO  [JestClientFactory] Idle connection reaping disabled...
2017-10-14T08:27:11.115-04:00 INFO  [ProcessBuffer] Initialized ProcessBuffer with ring size <65536> and wait strategy <BlockingWaitStrategy>.
2017-10-14T08:27:12.550-04:00 INFO  [RulesEngineProvider] No static rules file loaded.
2017-10-14T08:27:12.698-04:00 INFO  [OutputBuffer] Initialized OutputBuffer with ring size <65536> and wait strategy <BlockingWaitStrategy>.
2017-10-14T08:27:12.945-04:00 INFO  [ServerBootstrap] Graylog server 2.3.1+9f2c6ef starting up
2017-10-14T08:27:12.945-04:00 INFO  [ServerBootstrap] JRE: Oracle Corporation 1.8.0_144 on Linux 3.10.0-693.2.2.el7.x86_64
2017-10-14T08:27:12.945-04:00 INFO  [ServerBootstrap] Deployment: rpm
2017-10-14T08:27:12.945-04:00 INFO  [ServerBootstrap] OS: CentOS Linux 7 (Core) (centos)
2017-10-14T08:27:12.946-04:00 INFO  [ServerBootstrap] Arch: amd64
2017-10-14T08:27:12.948-04:00 WARN  [DeadEventLoggingListener] Received unhandled event of type <org.graylog2.plugin.lifecycles.Lifecycle> from event bus <AsyncEventBus{graylog-eventbus}>
2017-10-14T08:27:12.982-04:00 INFO  [PeriodicalsService] Starting 26 periodicals ...
2017-10-14T08:27:12.982-04:00 INFO  [Periodicals] Starting [org.graylog2.periodical.ThroughputCalculator] periodical in [0s], polling every [1s].
2017-10-14T08:27:12.983-04:00 INFO  [Periodicals] Starting [org.graylog2.periodical.AlertScannerThread] periodical in [10s], polling every [60s].
2017-10-14T08:27:12.986-04:00 INFO  [Periodicals] Starting [org.graylog2.periodical.BatchedElasticSearchOutputFlushThread] periodical in [0s], polling every [1s].
2017-10-14T08:27:12.994-04:00 INFO  [Periodicals] Starting [org.graylog2.periodical.ClusterHealthCheckThread] periodical in [120s], polling every [20s].
2017-10-14T08:27:12.994-04:00 INFO  [Periodicals] Starting [org.graylog2.periodical.ContentPackLoaderPeriodical] periodical, running forever.
2017-10-14T08:27:12.994-04:00 INFO  [Periodicals] Starting [org.graylog2.periodical.GarbageCollectionWarningThread] periodical, running forever.
2017-10-14T08:27:12.995-04:00 INFO  [Periodicals] Starting [org.graylog2.periodical.IndexerClusterCheckerThread] periodical in [0s], polling every [30s].
2017-10-14T08:27:12.996-04:00 INFO  [Periodicals] Starting [org.graylog2.periodical.IndexRetentionThread] periodical in [0s], polling every [300s].
2017-10-14T08:27:12.996-04:00 INFO  [Periodicals] Starting [org.graylog2.periodical.IndexRotationThread] periodical in [0s], polling every [10s].
2017-10-14T08:27:12.997-04:00 INFO  [Periodicals] Starting [org.graylog2.periodical.NodePingThread] periodical in [0s], polling every [1s].
2017-10-14T08:27:12.997-04:00 INFO  [Periodicals] Starting [org.graylog2.periodical.VersionCheckThread] periodical in [300s], polling every [1800s].
2017-10-14T08:27:12.997-04:00 INFO  [Periodicals] Starting [org.graylog2.periodical.ThrottleStateUpdaterThread] periodical in [1s], polling every [1s].
2017-10-14T08:27:12.998-04:00 INFO  [Periodicals] Starting [org.graylog2.events.ClusterEventPeriodical] periodical in [0s], polling every [1s].
2017-10-14T08:27:12.998-04:00 INFO  [Periodicals] Starting [org.graylog2.events.ClusterEventCleanupPeriodical] periodical in [0s], polling every [86400s].
2017-10-14T08:27:12.998-04:00 INFO  [Periodicals] Starting [org.graylog2.periodical.ClusterIdGeneratorPeriodical] periodical, running forever.
2017-10-14T08:27:12.998-04:00 INFO  [Periodicals] Starting [org.graylog2.periodical.IndexRangesMigrationPeriodical] periodical, running forever.
2017-10-14T08:27:12.999-04:00 INFO  [Periodicals] Starting [org.graylog2.periodical.IndexRangesCleanupPeriodical] periodical in [15s], polling every [3600s].
2017-10-14T08:27:13.003-04:00 INFO  [connection] Opened connection [connectionId{localValue:3, serverValue:28}] to localhost:27017
2017-10-14T08:27:13.003-04:00 INFO  [connection] Opened connection [connectionId{localValue:4, serverValue:29}] to localhost:27017
2017-10-14T08:27:13.004-04:00 INFO  [connection] Opened connection [connectionId{localValue:5, serverValue:30}] to localhost:27017
2017-10-14T08:27:13.011-04:00 INFO  [connection] Opened connection [connectionId{localValue:6, serverValue:31}] to localhost:27017
2017-10-14T08:27:13.013-04:00 INFO  [connection] Opened connection [connectionId{localValue:7, serverValue:32}] to localhost:27017
2017-10-14T08:27:13.014-04:00 INFO  [connection] Opened connection [connectionId{localValue:9, serverValue:34}] to localhost:27017
2017-10-14T08:27:13.014-04:00 INFO  [connection] Opened connection [connectionId{localValue:10, serverValue:35}] to localhost:27017
2017-10-14T08:27:13.018-04:00 INFO  [connection] Opened connection [connectionId{localValue:8, serverValue:33}] to localhost:27017
2017-10-14T08:27:13.021-04:00 INFO  [connection] Opened connection [connectionId{localValue:11, serverValue:36}] to localhost:27017
2017-10-14T08:27:13.023-04:00 INFO  [PeriodicalsService] Not starting [org.graylog2.periodical.UserPermissionMigrationPeriodical] periodical. Not configured to run on this node.
2017-10-14T08:27:13.023-04:00 INFO  [Periodicals] Starting [org.graylog2.periodical.AlarmCallbacksMigrationPeriodical] periodical, running forever.
2017-10-14T08:27:13.023-04:00 INFO  [Periodicals] Starting [org.graylog2.periodical.ConfigurationManagementPeriodical] periodical, running forever.
2017-10-14T08:27:13.038-04:00 INFO  [PeriodicalsService] Not starting [org.graylog2.periodical.LdapGroupMappingMigration] periodical. Not configured to run on this node.
2017-10-14T08:27:13.039-04:00 INFO  [Periodicals] Starting [org.graylog2.periodical.IndexFailuresPeriodical] periodical, running forever.
2017-10-14T08:27:13.039-04:00 INFO  [Periodicals] Starting [org.graylog.plugins.usagestatistics.UsageStatsNodePeriodical] periodical in [300s], polling every [21600s].
2017-10-14T08:27:13.039-04:00 INFO  [Periodicals] Starting [org.graylog.plugins.usagestatistics.UsageStatsClusterPeriodical] periodical in [300s], polling every [21600s].
2017-10-14T08:27:13.041-04:00 INFO  [Periodicals] Starting [org.graylog.plugins.pipelineprocessor.periodical.LegacyDefaultStreamMigration] periodical, running forever.
2017-10-14T08:27:13.041-04:00 INFO  [Periodicals] Starting [org.graylog.plugins.collector.periodical.PurgeExpiredCollectorsThread] periodical in [0s], polling every [3600s].
2017-10-14T08:27:13.043-04:00 INFO  [LegacyDefaultStreamMigration] Legacy default stream has no connections, no migration needed.
2017-10-14T08:27:13.343-04:00 INFO  [JerseyService] Enabling CORS for HTTP endpoint
2017-10-14T08:27:23.170-04:00 INFO  [NetworkListener] Started listener bound to [100.65.184.12:9000]
2017-10-14T08:27:23.171-04:00 INFO  [HttpServer] [HttpServer] Started.
2017-10-14T08:27:23.172-04:00 INFO  [JerseyService] Started REST API at <http://100.65.184.12:9000/api/>
2017-10-14T08:27:23.172-04:00 INFO  [JerseyService] Started Web Interface at <http://100.65.184.12:9000/>
2017-10-14T08:27:23.173-04:00 INFO  [ServiceManagerListener] Services are healthy
2017-10-14T08:27:23.174-04:00 INFO  [InputSetupService] Triggering launching persisted inputs, node transitioned from Uninitialized [LB:DEAD] to Running [LB:ALIVE]
2017-10-14T08:27:23.174-04:00 INFO  [ServerBootstrap] Services started, startup times in ms: {InputSetupService [RUNNING]=7, OutputSetupService [RUNNING]=14, BufferSynchronizerService [RUNNING]=15, KafkaJournal [RUNNING]=17, ConfigurationEtagService [RUNNING]=46, StreamCacheService [RUNNING]=46, LookupTableService [RUNNING]=50, JournalReader [RUNNING]=52, PeriodicalsService [RUNNING]=64, JerseyService [RUNNING]=10192}
2017-10-14T08:27:23.180-04:00 INFO  [ServerBootstrap] Graylog server up and running.
2017-10-14T08:27:23.198-04:00 INFO  [InputStateListener] Input [Syslog TCP/59e1017d6b80680c12bcf5de] is now STARTING
2017-10-14T08:27:23.200-04:00 INFO  [InputStateListener] Input [Syslog UDP/59e107456b80680c12bcfc27] is now STARTING
2017-10-14T08:27:23.268-04:00 WARN  [NettyTransport] receiveBufferSize (SO_RCVBUF) for input SyslogTCPInput{title=Syslog TCP, type=org.graylog2.inputs.syslog.tcp.SyslogTCPInput, nodeId=651c724e-ef23-4983-81cf-e4a9836585bf} should be 1048576 but is 212992.
2017-10-14T08:27:23.270-04:00 WARN  [NettyTransport] receiveBufferSize (SO_RCVBUF) for input SyslogUDPInput{title=Syslog UDP, type=org.graylog2.inputs.syslog.udp.SyslogUDPInput, nodeId=651c724e-ef23-4983-81cf-e4a9836585bf} should be 262144 but is 212992.
2017-10-14T08:27:23.270-04:00 INFO  [InputStateListener] Input [Syslog TCP/59e1017d6b80680c12bcf5de] is now RUNNING
2017-10-14T08:27:23.271-04:00 INFO  [InputStateListener] Input [Syslog UDP/59e107456b80680c12bcfc27] is now RUNNING
  • graylog conf
cat server.conf.txt
is_master = true
node_id_file = /etc/graylog/server/node-id
password_secret =R4rrI4pRqP3RZp1k3oBOxxx
root_username = admin
root_password_sha2 =41dfa2fddc7e5fxxx
plugin_dir = /usr/share/graylog-server/plugin
rest_listen_uri = http://10.65.184.12:9000/api/
web_listen_uri = http://10.65.184.12:9000/
elasticsearch_hosts = http://10.65.184.12:9200
rotation_strategy = count
elasticsearch_max_docs_per_index = 20000000
elasticsearch_max_number_of_indices = 20
retention_strategy = delete
elasticsearch_shards = 4
elasticsearch_replicas = 0
elasticsearch_index_prefix = graylog
allow_leading_wildcard_searches = false
allow_highlighting = false
elasticsearch_analyzer = standard
output_batch_size = 500
output_flush_interval = 1
output_fault_count_threshold = 5
output_fault_penalty_seconds = 30
processbuffer_processors = 5
outputbuffer_processors = 3
processor_wait_strategy = blocking
ring_size = 65536
inputbuffer_ring_size = 65536
inputbuffer_processors = 2
inputbuffer_wait_strategy = blocking
message_journal_enabled = true
message_journal_dir = /var/lib/graylog-server/journal
lb_recognition_period_seconds = 3
mongodb_uri = mongodb://localhost/graylog
mongodb_max_connections = 1000
mongodb_threads_allowed_to_block_multiplier = 5
content_packs_dir = /usr/share/graylog-server/contentpacks
content_packs_auto_load = grok-patterns.json
proxied_requests_thread_pool_size = 32
  • Elasticsearch info
    • elasticsearch conf
cluster.name:  graylog
node.name:  syslog02
network.host:  10.65.184.12
  • elasticsearch log
x8:45:47,614][INFO ][o.e.n.Node               ] [syslog02] initializing ...
x8:45:47,708][INFO ][o.e.e.NodeEnvironment    ] [syslog02] using [1] data paths, mounts [[/pub (/dev/mapper/data-d01)]], net usable_space [279.8gb], net total_space [295.1gb], spins? [possibly], types [ext4]
x8:45:47,708][INFO ][o.e.e.NodeEnvironment    ] [syslog02] heap size [1.9gb], compressed ordinary object pointers [true]
x8:45:47,721][INFO ][o.e.n.Node               ] [syslog02] node name [syslog02], node ID [vIqQThEJQny3KxVICtTJVg]
x8:45:47,721][INFO ][o.e.n.Node               ] [syslog02] version[5.6.3], pid[23402], build[1a2f265/2017-10-06T20:33:39.012Z], OS[Linux/3.10.0-693.2.2.el7.x86_64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_144/25.144-b01]
x8:45:47,721][INFO ][o.e.n.Node               ] [syslog02] JVM arguments [-Xms2g, -Xmx2g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -Djdk.io.permissionsUseCanonicalPath=true, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j.skipJansi=true, -XX:+HeapDumpOnOutOfMemoryError, -Des.path.home=/usr/share/elasticsearch]
x8:45:48,537][INFO ][o.e.p.PluginsService     ] [syslog02] loaded module [aggs-matrix-stats]
x8:45:48,538][INFO ][o.e.p.PluginsService     ] [syslog02] loaded module [ingest-common]
x8:45:48,538][INFO ][o.e.p.PluginsService     ] [syslog02] loaded module [lang-expression]
x8:45:48,538][INFO ][o.e.p.PluginsService     ] [syslog02] loaded module [lang-groovy]
x8:45:48,538][INFO ][o.e.p.PluginsService     ] [syslog02] loaded module [lang-mustache]
x8:45:48,538][INFO ][o.e.p.PluginsService     ] [syslog02] loaded module [lang-painless]
x8:45:48,538][INFO ][o.e.p.PluginsService     ] [syslog02] loaded module [parent-join]
x8:45:48,538][INFO ][o.e.p.PluginsService     ] [syslog02] loaded module [percolator]
x8:45:48,538][INFO ][o.e.p.PluginsService     ] [syslog02] loaded module [reindex]
x8:45:48,538][INFO ][o.e.p.PluginsService     ] [syslog02] loaded module [transport-netty3]
x8:45:48,538][INFO ][o.e.p.PluginsService     ] [syslog02] loaded module [transport-netty4]
x8:45:48,539][INFO ][o.e.p.PluginsService     ] [syslog02] no plugins loaded
x8:45:49,901][INFO ][o.e.d.DiscoveryModule    ] [syslog02] using discovery type [zen]
x8:45:50,396][INFO ][o.e.n.Node               ] [syslog02] initialized
x8:45:50,396][INFO ][o.e.n.Node               ] [syslog02] starting ...
x8:45:50,536][INFO ][o.e.t.TransportService   ] [syslog02] publish_address {10.65.184.12:9300}, bound_addresses {10.65.184.12:9300}
x8:45:50,545][INFO ][o.e.b.BootstrapChecks    ] [syslog02] bound or publishing to a non-loopback or non-link-local address, enforcing bootstrap checks
x8:45:53,590][INFO ][o.e.c.s.ClusterService   ] [syslog02] new_master {syslog02}{vIqQThEJQny3KxVICtTJVg}{fNN9hOt6SLCepklv35Z7rQ}{10.65.184.12}{10.65.184.12:9300}, reason: zen-disco-elected-as-master ([0] nodes joined)
x8:45:53,608][INFO ][o.e.h.n.Netty4HttpServerTransport] [syslog02] publish_address {10.65.184.12:9200}, bound_addresses {10.65.184.12:9200}
x8:45:53,608][INFO ][o.e.n.Node               ] [syslog02] started
x8:45:53,760][INFO ][o.e.g.GatewayService     ] [syslog02] recovered [1] indices into cluster_state
x8:45:53,976][INFO ][o.e.c.r.a.AllocationService] [syslog02] Cluster health status changed from [RED] to [GREEN] (reason: [shards started [[graylog_0][3], [graylog_0][2], [graylog_0][1]] ...]).

  • mongodb info
    • mongodb log
x04:50:08.897-0400 I CONTROL  [main] ***** SERVER RESTARTED *****
x04:50:08.921-0400 I CONTROL  [initandlisten] MongoDB starting : pid=1234 port=27017 dbpath=/var/lib/mongo 64-bit host=va32lsyslog02
x04:50:08.921-0400 I CONTROL  [initandlisten] db version v3.2.17
x04:50:08.921-0400 I CONTROL  [initandlisten] git version: 186656d79574f7dfe0831a7e7821292ab380f667
x04:50:08.921-0400 I CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.0.1e-fips 11 Feb 2013
x04:50:08.921-0400 I CONTROL  [initandlisten] allocator: tcmalloc
x04:50:08.921-0400 I CONTROL  [initandlisten] modules: none
x04:50:08.921-0400 I CONTROL  [initandlisten] build environment:
x04:50:08.921-0400 I CONTROL  [initandlisten]     distmod: rhel70
x04:50:08.921-0400 I CONTROL  [initandlisten]     distarch: x86_64
x04:50:08.921-0400 I CONTROL  [initandlisten]     target_arch: x86_64
x04:50:08.921-0400 I CONTROL  [initandlisten] options: { config: "/etc/mongod.conf", net: { bindIp: "127.0.0.1", port: 27017 }, processManagement: { fork: true, pidFilePath: "/var/run/mongodb/mongod.pid" }, storage: { dbPath: "/var/lib/mongo", journal: { enabled: true } }, systemLog: { destination: "file", logAppend: true, path: "/var/log/mongodb/mongod.log" } }
x04:50:08.949-0400 I -        [initandlisten] Detected data files in /var/lib/mongo created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
x04:50:08.949-0400 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=8G,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
x04:50:09.511-0400 I CONTROL  [initandlisten]
x04:50:09.511-0400 I CONTROL  [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/enabled is 'always'.
x04:50:09.511-0400 I CONTROL  [initandlisten] **        We suggest setting it to 'never'
x04:50:09.511-0400 I CONTROL  [initandlisten]
x04:50:09.511-0400 I CONTROL  [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/defrag is 'always'.
x04:50:09.511-0400 I CONTROL  [initandlisten] **        We suggest setting it to 'never'
x04:50:09.511-0400 I CONTROL  [initandlisten]
x04:50:09.511-0400 I CONTROL  [initandlisten] ** WARNING: soft rlimits too low. rlimits set to 4096 processes, 64000 files. Number of processes should be at least 32000 : 0.5 times number of files.
x04:50:09.511-0400 I CONTROL  [initandlisten]
x04:50:09.546-0400 I FTDC     [initandlisten] Initializing full-time diagnostic data capture with directory '/var/lib/mongo/diagnostic.data'
x04:50:09.547-0400 I NETWORK  [initandlisten] waiting for connections on port 27017
x04:50:09.547-0400 I NETWORK  [HostnameCanonicalizationWorker] Starting hostname canonicalization worker
x04:50:15.332-0400 I NETWORK  [initandlisten] connection accepted from 127.0.0.1:54308 #1 (1 connection now open)
<snipped>
x04:50:18.006-0400 I NETWORK  [initandlisten] connection accepted from 127.0.0.1:54334 #9 (9 connections now open)
<snipped>
x07:38:41.016-0400 I NETWORK  [conn6] end connection 127.0.0.1:54328 (8 connections now open)
x07:39:13.863-0400 I NETWORK  [initandlisten] connection accepted from 127.0.0.1:55012 #16 (7 connections now open)
x08:17:09.875-0400 I NETWORK  [conn11] end connection 127.0.0.1:55002 (6 connections now open)
<snipped>
x08:17:09.875-0400 I NETWORK  [conn16] end connection 127.0.0.1:55012 (6 connections now open)
x08:17:56.689-0400 I NETWORK  [initandlisten] connection accepted from 127.0.0.1:55156 #17 (1 connection now open)
<snipped>
x08:17:59.180-0400 I NETWORK  [initandlisten] connection accepted from 127.0.0.1:55172 #25 (9 connections now open)
x08:26:53.176-0400 I NETWORK  [conn22] end connection 127.0.0.1:55166 (8 connections now open)
<snipped>
x08:26:53.176-0400 I NETWORK  [conn19] end connection 127.0.0.1:55160 (8 connections now open)
x08:27:10.516-0400 I NETWORK  [initandlisten] connection accepted from 127.0.0.1:55192 #26 (1 connection now open)
<snipped>
x08:27:13.014-0400 I NETWORK  [initandlisten] connection accepted from 127.0.0.1:55212 #36 (11 connections now open)
  • mongodb /etc/mogod.conf
systemLog:
  destination: file
  logAppend: true
  path: /var/log/mongodb/mongod.log
storage:
  dbPath: /var/lib/mongo
  journal:
    enabled: true
processManagement:
  fork: true  # fork and run in background
  pidFilePath: /var/run/mongodb/mongod.pid  # location of pidfile
net:
  port: 27017
  bindIp: 127.0.0.1  # Listen to local interface only, comment to listen on all interfaces.

Health check

  • Elasticsearch
[root@syslog02 ~]# curl -XGET 'http://10.65.184.12:9200/_cluster/health?pretty=true'
{
  "cluster_name" : "graylog",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 1,
  "number_of_data_nodes" : 1,
  "active_primary_shards" : 4,
  "active_shards" : 4,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 100.0
}
[root@syslog02 ~]#

1 Like

Hi

I was able to bite the bullet and upgraded cenots 7.2 + graylog 2.2.3 to centos 7.4+ graylog 2.3.1.

Please ignore this issue, I added the node that had issue as one of 4 nodes graylog elasticsearch cluster. Following is the screenshot from Elasticsearch Head Chrome Plugin.

image

1 Like

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.