I’m working on getting a new install setup currently with one server running MongoDB and Greylog Server and another running ElasticSearch. I installed everything on the two servers, set the ElasticSearch server to listen on the correct IP, and added the ElasticSearch server to the Graylog server.conf file. Graylog and Elastic services are both running and from the Graylog server I can run curl http://:9200 and get a response. I can login to the Graylog interface but when I click on search I get:
While retrieving data for this widget, the following error(s) occurred:
- Connection refused.
This is the log from right after I restarted the graylog-server service:
2021-04-21T20:50:16.344Z INFO [CmdLineTool] Loaded plugin: AWS plugins 4.0.6 [org.graylog.aws.AWSPlugin]
2021-04-21T20:50:16.347Z INFO [CmdLineTool] Loaded plugin: Enterprise Integrations 4.0.6 [org.graylog.enterprise.integrations.EnterpriseIntegrationsPlugin]
2021-04-21T20:50:16.347Z INFO [CmdLineTool] Loaded plugin: Integrations 4.0.6 [org.graylog.integrations.IntegrationsPlugin]
2021-04-21T20:50:16.348Z INFO [CmdLineTool] Loaded plugin: Collector 4.0.6 [org.graylog.plugins.collector.CollectorPlugin]
2021-04-21T20:50:16.349Z INFO [CmdLineTool] Loaded plugin: Graylog Enterprise 4.0.6 [org.graylog.plugins.enterprise.EnterprisePlugin]
2021-04-21T20:50:16.349Z INFO [CmdLineTool] Loaded plugin: Threat Intelligence Plugin 4.0.6 [org.graylog.plugins.threatintel.ThreatIntelPlugin]
2021-04-21T20:50:16.349Z INFO [CmdLineTool] Loaded plugin: Elasticsearch 6 Support 4.0.6+40b7be5 [org.graylog.storage.elasticsearch6.Elasticsearch6Plugin]
2021-04-21T20:50:16.350Z INFO [CmdLineTool] Loaded plugin: Elasticsearch 7 Support 4.0.6+40b7be5 [org.graylog.storage.elasticsearch7.Elasticsearch7Plugin]
2021-04-21T20:50:16.536Z INFO [CmdLineTool] Running with JVM arguments: -Xms1g -Xmx1g -XX:NewRatio=1 -XX:+ResizeTLAB -XX:-OmitStackTraceInFastThrow -Djdk.tls.acknowledgeCloseNotify=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSConcurrentMTEnabled -XX:+CMSClassUnloadingEnabled -Dlog4j.configurationFile=file:///etc/graylog/server/log4j2.xml -Djava.library.path=/usr/share/graylog-server/lib/sigar -Dgraylog2.installation_source=deb
2021-04-21T20:50:16.712Z INFO [Version] HV000001: Hibernate Validator null
2021-04-21T20:50:19.362Z INFO [InputBufferImpl] Message journal is enabled.
2021-04-21T20:50:19.379Z INFO [NodeId] Node ID: 320fe684-2155-4685-93ea-b8eaa2c8caf8
2021-04-21T20:50:19.543Z INFO [LogManager] Loading logs.
2021-04-21T20:50:19.569Z WARN [Log] Found a corrupted index file, /var/lib/graylog-server/journal/messagejournal-0/00000000000000000000.index, deleting and rebuilding index…
2021-04-21T20:50:19.595Z INFO [LogManager] Logs loading complete.
2021-04-21T20:50:19.601Z INFO [KafkaJournal] Initialized Kafka based journal at /var/lib/graylog-server/journal
2021-04-21T20:50:19.618Z INFO [cluster] Cluster created with settings {hosts=[localhost:27017], mode=SINGLE, requiredClusterType=UNKNOWN, serverSelectionTimeout=‘30000 ms’, maxWaitQueueSize=5000}
2021-04-21T20:50:19.655Z INFO [cluster] Cluster description not yet available. Waiting for 30000 ms before timing out
2021-04-21T20:50:19.684Z INFO [connection] Opened connection [connectionId{localValue:1, serverValue:18}] to localhost:27017
2021-04-21T20:50:19.691Z INFO [cluster] Monitor thread successfully connected to server with description ServerDescription{address=localhost:27017, type=STANDALONE, state=CONNECTED, ok=true, version=ServerVersion{versionList=[4, 0, 24]}, minWireVersion=0, maxWireVersion=7, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=4107756}
2021-04-21T20:50:19.700Z INFO [connection] Opened connection [connectionId{localValue:2, serverValue:19}] to localhost:27017
2021-04-21T20:50:19.862Z INFO [InputBufferImpl] Initialized InputBufferImpl with ring size <65536> and wait strategy , running 2 parallel message handlers.
2021-04-21T20:50:20.017Z INFO [ElasticsearchVersionProvider] Elasticsearch version set to 7.0.0 - disabling version probe.
2021-04-21T20:50:20.658Z INFO [ProcessBuffer] Initialized ProcessBuffer with ring size <65536> and wait strategy .
2021-04-21T20:50:20.809Z WARN [GeoIpResolverEngine] GeoIP database file does not exist: /etc/graylog/server/GeoLite2-City.mmdb
2021-04-21T20:50:20.815Z INFO [OutputBuffer] Initialized OutputBuffer with ring size <65536> and wait strategy .
2021-04-21T20:50:20.832Z WARN [GeoIpResolverEngine] GeoIP database file does not exist: /etc/graylog/server/GeoLite2-City.mmdb
2021-04-21T20:50:20.838Z INFO [connection] Opened connection [connectionId{localValue:3, serverValue:20}] to localhost:27017
2021-04-21T20:50:20.843Z WARN [GeoIpResolverEngine] GeoIP database file does not exist: /etc/graylog/server/GeoLite2-City.mmdb
2021-04-21T20:50:20.855Z WARN [GeoIpResolverEngine] GeoIP database file does not exist: /etc/graylog/server/GeoLite2-City.mmdb
2021-04-21T20:50:20.868Z WARN [GeoIpResolverEngine] GeoIP database file does not exist: /etc/graylog/server/GeoLite2-City.mmdb
2021-04-21T20:50:21.305Z ERROR [AuditLogger] Unable to write audit log entry because there is no valid license.
2021-04-21T20:50:21.307Z INFO [ServerBootstrap] Graylog server 4.0.6+40b7be5 starting up
2021-04-21T20:50:21.307Z INFO [ServerBootstrap] JRE: Private Build 1.8.0_282 on Linux 4.15.0-142-generic
2021-04-21T20:50:21.307Z INFO [ServerBootstrap] Deployment: deb
2021-04-21T20:50:21.307Z INFO [ServerBootstrap] OS: Ubuntu 18.04.5 LTS (bionic)
2021-04-21T20:50:21.307Z INFO [ServerBootstrap] Arch: amd64
2021-04-21T20:50:21.333Z INFO [PeriodicalsService] Starting 36 periodicals …
2021-04-21T20:50:21.334Z INFO [Periodicals] Starting [org.graylog2.periodical.ThroughputCalculator] periodical in [0s], polling every [1s].
2021-04-21T20:50:21.337Z INFO [Periodicals] Starting [org.graylog.plugins.pipelineprocessor.periodical.LegacyDefaultStreamMigration] periodical, running forever.
2021-04-21T20:50:21.344Z INFO [PeriodicalsService] Not starting [org.graylog2.periodical.AlertScannerThread] periodical. Not configured to run on this node.
2021-04-21T20:50:21.345Z INFO [LegacyDefaultStreamMigration] Legacy default stream has no connections, no migration needed.
2021-04-21T20:50:21.344Z INFO [Periodicals] Starting [org.graylog2.periodical.BatchedElasticSearchOutputFlushThread] periodical in [0s], polling every [1s].
2021-04-21T20:50:21.350Z INFO [Periodicals] Starting [org.graylog2.periodical.ClusterHealthCheckThread] periodical in [120s], polling every [20s].
2021-04-21T20:50:21.352Z INFO [PeriodicalsService] Not starting [org.graylog2.periodical.ContentPackLoaderPeriodical] periodical. Not configured to run on this node.
2021-04-21T20:50:21.353Z INFO [Periodicals] Starting [org.graylog2.periodical.GarbageCollectionWarningThread] periodical, running forever.
2021-04-21T20:50:21.353Z INFO [Periodicals] Starting [org.graylog2.periodical.IndexerClusterCheckerThread] periodical in [0s], polling every [30s].
2021-04-21T20:50:21.353Z INFO [Periodicals] Starting [org.graylog2.periodical.IndexRetentionThread] periodical in [0s], polling every [300s].
2021-04-21T20:50:21.354Z INFO [Periodicals] Starting [org.graylog2.periodical.IndexRotationThread] periodical in [0s], polling every [10s].
2021-04-21T20:50:21.354Z INFO [Periodicals] Starting [org.graylog2.periodical.NodePingThread] periodical in [0s], polling every [1s].
2021-04-21T20:50:21.355Z INFO [Periodicals] Starting [org.graylog2.periodical.VersionCheckThread] periodical in [300s], polling every [1800s].
2021-04-21T20:50:21.355Z INFO [Periodicals] Starting [org.graylog2.periodical.ThrottleStateUpdaterThread] periodical in [1s], polling every [1s].
2021-04-21T20:50:21.355Z INFO [Periodicals] Starting [org.graylog2.events.ClusterEventPeriodical] periodical in [0s], polling every [1s].
2021-04-21T20:50:21.356Z INFO [Periodicals] Starting [org.graylog2.events.ClusterEventCleanupPeriodical] periodical in [0s], polling every [86400s].
2021-04-21T20:50:21.356Z INFO [Periodicals] Starting [org.graylog2.periodical.ClusterIdGeneratorPeriodical] periodical, running forever.
2021-04-21T20:50:21.356Z INFO [Periodicals] Starting [org.graylog2.periodical.IndexRangesMigrationPeriodical] periodical, running forever.
2021-04-21T20:50:21.357Z INFO [Periodicals] Starting [org.graylog2.periodical.IndexRangesCleanupPeriodical] periodical in [15s], polling every [3600s].
2021-04-21T20:50:21.360Z INFO [connection] Opened connection [connectionId{localValue:4, serverValue:21}] to localhost:27017
2021-04-21T20:50:21.367Z INFO [PeriodicalsService] Not starting [org.graylog2.periodical.UserPermissionMigrationPeriodical] periodical. Not configured to run on this node.
2021-04-21T20:50:21.367Z INFO [Periodicals] Starting [org.graylog2.periodical.ConfigurationManagementPeriodical] periodical, running forever.
2021-04-21T20:50:21.374Z INFO [Periodicals] Starting [org.graylog2.periodical.IndexFailuresPeriodical] periodical, running forever.
2021-04-21T20:50:21.376Z INFO [connection] Opened connection [connectionId{localValue:8, serverValue:25}] to localhost:27017
2021-04-21T20:50:21.377Z INFO [connection] Opened connection [connectionId{localValue:5, serverValue:22}] to localhost:27017
2021-04-21T20:50:21.377Z INFO [connection] Opened connection [connectionId{localValue:6, serverValue:23}] to localhost:27017
2021-04-21T20:50:21.377Z INFO [connection] Opened connection [connectionId{localValue:7, serverValue:24}] to localhost:27017
2021-04-21T20:50:21.379Z INFO [Periodicals] Starting [org.graylog2.periodical.TrafficCounterCalculator] periodical in [0s], polling every [1s].
2021-04-21T20:50:21.386Z INFO [Periodicals] Starting [org.graylog2.indexer.fieldtypes.IndexFieldTypePollerPeriodical] periodical in [0s], polling every [3600s].
2021-04-21T20:50:21.386Z INFO [Periodicals] Starting [org.graylog.scheduler.periodicals.ScheduleTriggerCleanUp] periodical in [120s], polling every [86400s].
2021-04-21T20:50:21.393Z INFO [Periodicals] Starting [org.graylog2.periodical.ESVersionCheckPeriodical] periodical in [0s], polling every [30s].
2021-04-21T20:50:21.396Z INFO [Periodicals] Starting [org.graylog.plugins.sidecar.periodical.PurgeExpiredSidecarsThread] periodical in [0s], polling every [600s].
2021-04-21T20:50:21.396Z INFO [Periodicals] Starting [org.graylog.plugins.sidecar.periodical.PurgeExpiredConfigurationUploads] periodical in [0s], polling every [600s].
2021-04-21T20:50:21.397Z INFO [Periodicals] Starting [org.graylog.plugins.views.search.db.SearchesCleanUpJob] periodical in [3600s], polling every [28800s].
2021-04-21T20:50:21.398Z INFO [Periodicals] Starting [org.graylog.events.periodicals.EventNotificationStatusCleanUp] periodical in [120s], polling every [86400s].
2021-04-21T20:50:21.573Z INFO [Periodicals] Starting [org.graylog.plugins.collector.periodical.PurgeExpiredCollectorsThread] periodical in [0s], polling every [3600s].
2021-04-21T20:50:21.573Z INFO [Periodicals] Starting [org.graylog.plugins.license.LicenseManagerPeriodical] periodical in [0s], polling every [300s].
2021-04-21T20:50:21.575Z INFO [Periodicals] Starting [org.graylog.plugins.license.LicenseReportPeriodical] periodical in [300s], polling every [3600s].
2021-04-21T20:50:21.576Z INFO [Periodicals] Starting [org.graylog.plugins.license.StagedLicenseInstallerPeriodical] periodical, running forever.
2021-04-21T20:50:21.576Z INFO [Periodicals] Starting [org.graylog.plugins.auditlog.mongodb.MongoAuditLogPeriodical] periodical in [0s], polling every [3600s].
2021-04-21T20:50:21.577Z INFO [Periodicals] Starting [org.graylog.plugins.report.scheduler.ReportPeriodical] periodical in [120s], polling every [60s].
2021-04-21T20:50:21.577Z ERROR [MongoAuditLogPeriodical] Not running cleanup for auditlog entries in MongoDB because there is no valid license.
2021-04-21T20:50:21.577Z INFO [Periodicals] Starting [org.graylog.plugins.report.service.ChromeDriverCleanupPeriodical] periodical in [60s], polling every [180s].
2021-04-21T20:50:21.717Z ERROR [IndexerClusterCheckerThread] Uncaught exception in periodical
org.graylog.shaded.elasticsearch7.org.elasticsearch.ElasticsearchException: An error occurred:
at org.graylog.storage.elasticsearch7.ElasticsearchClient.exceptionFrom(ElasticsearchClient.java:136) ~[?:?]
at org.graylog.storage.elasticsearch7.ElasticsearchClient.execute(ElasticsearchClient.java:99) ~[?:?]
at org.graylog.storage.elasticsearch7.ElasticsearchClient.execute(ElasticsearchClient.java:92) ~[?:?]
at org.graylog.storage.elasticsearch7.ClusterAdapterES7.indicesExist(ClusterAdapterES7.java:289) ~[?:?]
at org.graylog.storage.elasticsearch7.ClusterAdapterES7.clusterHealth(ClusterAdapterES7.java:268) ~[?:?]
at org.graylog.storage.elasticsearch7.ClusterAdapterES7.health(ClusterAdapterES7.java:81) ~[?:?]
at org.graylog2.indexer.cluster.Cluster.health(Cluster.java:70) ~[graylog.jar:?]
at org.graylog2.periodical.IndexerClusterCheckerThread.doRun(IndexerClusterCheckerThread.java:58) ~[graylog.jar:?]
at org.graylog2.plugin.periodical.Periodical.run(Periodical.java:77) [graylog.jar:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_282]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_282]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_282]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [?:1.8.0_282]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_282]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_282]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_282]
Caused by: java.net.ConnectException: Connection refused
at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.RestClient.extractAndWrapCause(RestClient.java:849) ~[?:?]
at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.RestClient.performRequest(RestClient.java:259) ~[?:?]
at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.RestClient.performRequest(RestClient.java:246) ~[?:?]
at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1613) ~[?:?]
at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1598) ~[?:?]
at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.IndicesClient.exists(IndicesClient.java:974) ~[?:?]
at org.graylog.storage.elasticsearch7.ClusterAdapterES7.lambda$indicesExist$13(ClusterAdapterES7.java:289) ~[?:?]
at org.graylog.storage.elasticsearch7.ElasticsearchClient.execute(ElasticsearchClient.java:97) ~[?:?]
… 14 more
Caused by: java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:1.8.0_282]
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:716) ~[?:1.8.0_282]
at org.graylog.shaded.elasticsearch7.org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvent(DefaultConnectingIOReactor.java:174) ~[?:?]
at org.graylog.shaded.elasticsearch7.org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvents(DefaultConnectingIOReactor.java:148) ~[?:?]
at org.graylog.shaded.elasticsearch7.org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:351) ~[?:?]
at org.graylog.shaded.elasticsearch7.org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.execute(PoolingNHttpClientConnectionManager.java:221) ~[?:?]
at org.graylog.shaded.elasticsearch7.org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase$1.run(CloseableHttpAsyncClientBase.java:64) ~[?:?]
… 1 more
2021-04-21T20:50:21.717Z ERROR [ConfigurationManagementPeriodical] Error while running migration <V20161130141500_DefaultStreamRecalcIndexRanges{2016-11-30T14:15:00Z}>
org.graylog.shaded.elasticsearch7.org.elasticsearch.ElasticsearchException: An error occurred:
at org.graylog.storage.elasticsearch7.ElasticsearchClient.exceptionFrom(ElasticsearchClient.java:136) ~[?:?]
at org.graylog.storage.elasticsearch7.ElasticsearchClient.execute(ElasticsearchClient.java:99) ~[?:?]
at org.graylog.storage.elasticsearch7.ElasticsearchClient.execute(ElasticsearchClient.java:92) ~[?:?]
at org.graylog.storage.elasticsearch7.ClusterAdapterES7.isConnected(ClusterAdapterES7.java:161) ~[?:?]
at org.graylog2.indexer.cluster.Cluster.isConnected(Cluster.java:115) ~[graylog.jar:?]
at org.graylog2.migrations.V20161130141500_DefaultStreamRecalcIndexRanges.upgrade(V20161130141500_DefaultStreamRecalcIndexRanges.java:84) ~[graylog.jar:?]
at org.graylog2.periodical.ConfigurationManagementPeriodical.doRun(ConfigurationManagementPeriodical.java:42) [graylog.jar:?]
at org.graylog2.plugin.periodical.Periodical.run(Periodical.java:77) [graylog.jar:?]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_282]
Caused by: java.net.ConnectException: Connection refused
at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.RestClient.extractAndWrapCause(RestClient.java:849) ~[?:?]
at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.RestClient.performRequest(RestClient.java:259) ~[?:?]
at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.RestClient.performRequest(RestClient.java:246) ~[?:?]
at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1613) ~[?:?]
at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1583) ~[?:?]
at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1553) ~[?:?]
at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.ClusterClient.health(ClusterClient.java:130) ~[?:?]
at org.graylog.storage.elasticsearch7.ClusterAdapterES7.lambda$isConnected$9(ClusterAdapterES7.java:161) ~[?:?]
at org.graylog.storage.elasticsearch7.ElasticsearchClient.execute(ElasticsearchClient.java:97) ~[?:?]
… 7 more
Any ideas what could be going on here?