Server currently unavailable 2

Elvis · January 13, 2021, 6:05pm

I just installed graylog on a vm with 8GB Ram.

i am getting error “We are experiencing problems connecting to the Graylog server running on http://127.0.0.1:9000/api/ . Please verify that the server is healthy and working correctly.”

server
{
listen 80;
listen :80 ipv6only=on;
server_name servername.com;
access_log /var/log/nginx/graylog.access.log;

location / {
  proxy_set_header Host $http_host;
  proxy_set_header X-Forwarded-Host $host;
  proxy_set_header X-Forwarded-Server $host;
  proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
  proxy_set_header X-Graylog-Server-URL http://$server_name/;
  proxy_pass       http://127.0.0.1:9000;
}

}

Graylog server is up and running
MongoDB is connected and running with no errors, i suspect something in config file,
Any advice will be appreciated

-Elvis

aaronsachs · January 13, 2021, 8:46pm

What’s the result of running curl -s 127.0.0.1:9000? Also, seeing your server.conf and any associated logs from Graylog would help.

Elvis · January 13, 2021, 10:03pm

I get a below: Please note that i removed enterprise-plugins

root@servername:~# curl -s 127.0.0.1:9000

Graylog Web Interface

<script src="http://127.0.0.1:9000/assets/vendor.594b2a39cb22b445205e.js"></script>

<script src="http://127.0.0.1:9000/assets/polyfill.a41ce78e1beaf9cde903.js"></script>

<script src="http://127.0.0.1:9000/assets/builtins.a41ce78e1beaf9cde903.js"></script>

<script src="http://127.0.0.1:9000/assets/plugin/org.graylog.plugins.threatintel.ThreatIntelPlugin/plugin.org.graylog.plugins.threatintel.ThreatIntelPlugin.3ffe487c5932d920ffec.js"></script>

<script src="http://127.0.0.1:9000/assets/plugin/org.graylog.plugins.collector.CollectorPlugin/plugin.org.graylog.plugins.collector.CollectorPlugin.4d259ce7c45134eff0fb.js"></script>

<script src="http://127.0.0.1:9000/assets/plugin/org.graylog.integrations.IntegrationsPlugin/plugin.org.graylog.integrations.IntegrationsPlugin.e951c9a4e4cd6c24e9df.js"></script>

<script src="http://127.0.0.1:9000/assets/plugin/org.graylog.aws.AWSPlugin/plugin.org.graylog.aws.AWSPlugin.a03644a3671e94e1f717.js"></script>

<script src="http://127.0.0.1:9000/assets/app.a41ce78e1beaf9cde903.js"></script>

root@servername:~#

aaronsachs · January 14, 2021, 12:15am

Hi there, thanks for that, but what about your sever.conf and logs? Also, what version are you running?

Elvis · January 14, 2021, 10:06am

I am running graylog 4.0

is_master = true
node_id_file = /etc/graylog/server/node-id
password_secret = xxxxx
root_password_sha2 = xxxxx
bin_dir = /usr/share/graylog-server/bin
data_dir = /var/lib/graylog-server
plugin_dir = /usr/share/graylog-server/plugin

http_bind_address = 127.0.0.1:9000
http_publish_uri = http://127.0.0.1:9000/
http_enable_cors = true
rotation_strategy = count
elasticsearch_max_docs_per_index = 20000000
elasticsearch_max_number_of_indices = 20
retention_strategy = delete
elasticsearch_shards = 4
elasticsearch_replicas = 0
elasticsearch_index_prefix = graylog
allow_leading_wildcard_searches = false
allow_highlighting = false
elasticsearch_analyzer = standard
output_batch_size = 500
output_flush_interval = 1
output_fault_count_threshold = 5
output_fault_penalty_seconds = 30
processbuffer_processors = 5
outputbuffer_processors = 3
processor_wait_strategy = blocking
ring_size = 65536

inputbuffer_ring_size = 65536
inputbuffer_processors = 2
inputbuffer_wait_strategy = blocking
message_journal_enabled = true
message_journal_dir = /var/lib/graylog-server/journal
lb_recognition_period_seconds = 3
mongodb_uri = mongodb://localhost/graylog
mongodb_max_connections = 1000
mongodb_threads_allowed_to_block_multiplier = 5
proxied_requests_thread_pool_size = 32

When i restart service and trace logs
…
2021-01-14T09:57:51.143Z INFO [ServiceManagerListener] Services are healthy
2021-01-14T09:57:51.150Z INFO [InputSetupService] Triggering launching persisted inputs, node transitioned from Uninitialized [LB:DEAD] to Running [LB:ALIVE]
2021-01-14T09:57:51.152Z INFO [ServerBootstrap] Graylog server up and running.

Elvis · January 14, 2021, 10:36am

Logs:

root@servername:~# sudo systemctl restart graylog-server && tail -n 0 -f /var/log/graylog-server/server.log
2021-01-14T09:57:18.890Z INFO [CmdLineTool] Loaded plugin: AWS plugins 4.0.1 [org.graylog.aws.AWSPlugin]
2021-01-14T09:57:18.894Z INFO [CmdLineTool] Loaded plugin: Integrations 4.0.1 [org.graylog.integrations.IntegrationsPlugin]
2021-01-14T09:57:18.898Z INFO [CmdLineTool] Loaded plugin: Collector 4.0.1 [org.graylog.plugins.collector.CollectorPlugin]
2021-01-14T09:57:18.899Z INFO [CmdLineTool] Loaded plugin: Threat Intelligence Plugin 4.0.1 [org.graylog.plugins.threatintel.ThreatIntelPlugin]
2021-01-14T09:57:18.900Z INFO [CmdLineTool] Loaded plugin: Elasticsearch 6 Support 4.0.1+6a0cc0b [org.graylog.storage.elasticsearch6.Elasticsearch6Plugin]
2021-01-14T09:57:18.900Z INFO [CmdLineTool] Loaded plugin: Elasticsearch 7 Support 4.0.1+6a0cc0b [org.graylog.storage.elasticsearch7.Elasticsearch7Plugin]
2021-01-14T09:57:19.161Z INFO [CmdLineTool] Running with JVM arguments: -Xms1g -Xmx1g -XX:NewRatio=1 -XX:+ResizeTLAB -XX:+UseConcMarkSweepGC -XX:+CMSConcurrentMTEnabled -XX:+CMSClassUnloadingEnabled -XX:-OmitStackTraceInFastThrow -Djdk.tls.acknowledgeCloseNotify=true -Dlog4j.configurationFile=file:///etc/graylog/server/log4j2.xml -Djava.library.path=/usr/share/graylog-server/lib/sigar -Dgraylog2.installation_source=deb
2021-01-14T09:57:19.528Z INFO [Version] HV000001: Hibernate Validator null
2021-01-14T09:57:24.130Z INFO [InputBufferImpl] Message journal is enabled.
2021-01-14T09:57:24.154Z INFO [NodeId] Node ID: 861550b3-5ac7-47c2-b6d1-b83cbf9d7f10
2021-01-14T09:57:24.439Z INFO [LogManager] Loading logs.
2021-01-14T09:57:24.531Z INFO [LogManager] Logs loading complete.
2021-01-14T09:57:24.533Z INFO [KafkaJournal] Initialized Kafka based journal at /var/lib/graylog-server/journal
2021-01-14T09:57:24.568Z INFO [cluster] Cluster created with settings {hosts=[localhost:27017], mode=SINGLE, requiredClusterType=UNKNOWN, serverSelectionTimeout=‘30000 ms’, maxWaitQueueSize=5000}
2021-01-14T09:57:24.644Z INFO [cluster] Cluster description not yet available. Waiting for 30000 ms before timing out
2021-01-14T09:57:24.687Z INFO [connection] Opened connection [connectionId{localValue:1, serverValue:187}] to localhost:27017
2021-01-14T09:57:24.695Z INFO [cluster] Monitor thread successfully connected to server with description ServerDescription{address=localhost:27017, type=STANDALONE, state=CONNECTED, ok=true, version=ServerVersion{versionList=[4, 2, 11]}, minWireVersion=0, maxWireVersion=8, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=4016278}
2021-01-14T09:57:24.717Z INFO [connection] Opened connection [connectionId{localValue:2, serverValue:188}] to localhost:27017
2021-01-14T09:57:25.031Z INFO [InputBufferImpl] Initialized InputBufferImpl with ring size <65536> and wait strategy , running 2 parallel message handlers.
2021-01-14T09:57:25.740Z INFO [ElasticsearchVersionProvider] Elasticsearch cluster is running v7.10.1
2021-01-14T09:57:26.974Z INFO [ProcessBuffer] Initialized ProcessBuffer with ring size <65536> and wait strategy .
2021-01-14T09:57:27.123Z WARN [GeoIpResolverEngine] GeoIP database file does not exist: /etc/graylog/server/GeoLite2-City.mmdb
2021-01-14T09:57:27.136Z INFO [OutputBuffer] Initialized OutputBuffer with ring size <65536> and wait strategy .
2021-01-14T09:57:27.161Z INFO [connection] Opened connection [connectionId{localValue:3, serverValue:189}] to localhost:27017
2021-01-14T09:57:27.203Z WARN [GeoIpResolverEngine] GeoIP database file does not exist: /etc/graylog/server/GeoLite2-City.mmdb
2021-01-14T09:57:27.233Z WARN [GeoIpResolverEngine] GeoIP database file does not exist: /etc/graylog/server/GeoLite2-City.mmdb
2021-01-14T09:57:27.269Z WARN [GeoIpResolverEngine] GeoIP database file does not exist: /etc/graylog/server/GeoLite2-City.mmdb
2021-01-14T09:57:27.305Z WARN [GeoIpResolverEngine] GeoIP database file does not exist: /etc/graylog/server/GeoLite2-City.mmdb
2021-01-14T09:57:28.166Z INFO [ServerBootstrap] Graylog server 4.0.1+6a0cc0b starting up
2021-01-14T09:57:28.168Z INFO [ServerBootstrap] JRE: Debian 11.0.9.1 on Linux 4.19.0-13-amd64
2021-01-14T09:57:28.169Z INFO [ServerBootstrap] Deployment: deb
2021-01-14T09:57:28.169Z INFO [ServerBootstrap] OS: Debian GNU/Linux 10 (buster) (debian)
2021-01-14T09:57:28.169Z INFO [ServerBootstrap] Arch: amd64
2021-01-14T09:57:28.225Z INFO [PeriodicalsService] Starting 30 periodicals …
2021-01-14T09:57:28.225Z INFO [Periodicals] Starting [org.graylog2.periodical.ThroughputCalculator] periodical in [0s], polling every [1s].
2021-01-14T09:57:28.245Z INFO [Periodicals] Starting [org.graylog.plugins.pipelineprocessor.periodical.LegacyDefaultStreamMigration] periodical, running forever.
2021-01-14T09:57:28.273Z INFO [PeriodicalsService] Not starting [org.graylog2.periodical.AlertScannerThread] periodical. Not configured to run on this node.
2021-01-14T09:57:28.273Z INFO [Periodicals] Starting [org.graylog2.periodical.BatchedElasticSearchOutputFlushThread] periodical in [0s], polling every [1s].
2021-01-14T09:57:28.275Z INFO [Periodicals] Starting [org.graylog2.periodical.ClusterHealthCheckThread] periodical in [120s], polling every [20s].
2021-01-14T09:57:28.275Z INFO [PeriodicalsService] Not starting [org.graylog2.periodical.ContentPackLoaderPeriodical] periodical. Not configured to run on this node.
2021-01-14T09:57:28.275Z INFO [Periodicals] Starting [org.graylog2.periodical.GarbageCollectionWarningThread] periodical, running forever.
2021-01-14T09:57:28.276Z INFO [Periodicals] Starting [org.graylog2.periodical.IndexerClusterCheckerThread] periodical in [0s], polling every [30s].
2021-01-14T09:57:28.277Z INFO [Periodicals] Starting [org.graylog2.periodical.IndexRetentionThread] periodical in [0s], polling every [300s].
2021-01-14T09:57:28.283Z INFO [Periodicals] Starting [org.graylog2.periodical.IndexRotationThread] periodical in [0s], polling every [10s].
2021-01-14T09:57:28.289Z INFO [Periodicals] Starting [org.graylog2.periodical.NodePingThread] periodical in [0s], polling every [1s].
2021-01-14T09:57:28.297Z INFO [Periodicals] Starting [org.graylog2.periodical.VersionCheckThread] periodical in [300s], polling every [1800s].
2021-01-14T09:57:28.298Z INFO [Periodicals] Starting [org.graylog2.periodical.ThrottleStateUpdaterThread] periodical in [1s], polling every [1s].
2021-01-14T09:57:28.298Z INFO [Periodicals] Starting [org.graylog2.events.ClusterEventPeriodical] periodical in [0s], polling every [1s].
2021-01-14T09:57:28.300Z INFO [Periodicals] Starting [org.graylog2.events.ClusterEventCleanupPeriodical] periodical in [0s], polling every [86400s].
2021-01-14T09:57:28.303Z INFO [Periodicals] Starting [org.graylog2.periodical.ClusterIdGeneratorPeriodical] periodical, running forever.
2021-01-14T09:57:28.309Z INFO [Periodicals] Starting [org.graylog2.periodical.IndexRangesMigrationPeriodical] periodical, running forever.
2021-01-14T09:57:28.309Z INFO [Periodicals] Starting [org.graylog2.periodical.IndexRangesCleanupPeriodical] periodical in [15s], polling every [3600s].
2021-01-14T09:57:28.330Z INFO [LegacyDefaultStreamMigration] Legacy default stream has no connections, no migration needed.
2021-01-14T09:57:28.345Z INFO [connection] Opened connection [connectionId{localValue:4, serverValue:190}] to localhost:27017
2021-01-14T09:57:28.346Z INFO [connection] Opened connection [connectionId{localValue:8, serverValue:194}] to localhost:27017
2021-01-14T09:57:28.346Z INFO [connection] Opened connection [connectionId{localValue:5, serverValue:191}] to localhost:27017
2021-01-14T09:57:28.347Z INFO [connection] Opened connection [connectionId{localValue:6, serverValue:192}] to localhost:27017
2021-01-14T09:57:28.348Z INFO [connection] Opened connection [connectionId{localValue:10, serverValue:196}] to localhost:27017
2021-01-14T09:57:28.348Z INFO [connection] Opened connection [connectionId{localValue:7, serverValue:193}] to localhost:27017
2021-01-14T09:57:28.352Z INFO [PeriodicalsService] Not starting [org.graylog2.periodical.UserPermissionMigrationPeriodical] periodical. Not configured to run on this node.
2021-01-14T09:57:28.352Z INFO [Periodicals] Starting [org.graylog2.periodical.ConfigurationManagementPeriodical] periodical, running forever.
2021-01-14T09:57:28.353Z INFO [connection] Opened connection [connectionId{localValue:9, serverValue:195}] to localhost:27017
2021-01-14T09:57:28.362Z INFO [Periodicals] Starting [org.graylog2.periodical.IndexFailuresPeriodical] periodical, running forever.
2021-01-14T09:57:28.362Z INFO [Periodicals] Starting [org.graylog2.periodical.TrafficCounterCalculator] periodical in [0s], polling every [1s].
2021-01-14T09:57:28.402Z INFO [Periodicals] Starting [org.graylog2.indexer.fieldtypes.IndexFieldTypePollerPeriodical] periodical in [0s], polling every [3600s].
2021-01-14T09:57:28.411Z INFO [Periodicals] Starting [org.graylog.scheduler.periodicals.ScheduleTriggerCleanUp] periodical in [120s], polling every [86400s].
2021-01-14T09:57:28.413Z INFO [Periodicals] Starting [org.graylog2.periodical.ESVersionCheckPeriodical] periodical in [0s], polling every [30s].
2021-01-14T09:57:28.417Z INFO [Periodicals] Starting [org.graylog.plugins.sidecar.periodical.PurgeExpiredSidecarsThread] periodical in [0s], polling every [600s].
2021-01-14T09:57:28.425Z INFO [Periodicals] Starting [org.graylog.plugins.sidecar.periodical.PurgeExpiredConfigurationUploads] periodical in [0s], polling every [600s].
2021-01-14T09:57:28.429Z INFO [Periodicals] Starting [org.graylog.plugins.views.search.db.SearchesCleanUpJob] periodical in [3600s], polling every [28800s].
2021-01-14T09:57:28.435Z INFO [Periodicals] Starting [org.graylog.events.periodicals.EventNotificationStatusCleanUp] periodical in [120s], polling every [86400s].
2021-01-14T09:57:28.440Z INFO [Periodicals] Starting [org.graylog.plugins.collector.periodical.PurgeExpiredCollectorsThread] periodical in [0s], polling every [3600s].
2021-01-14T09:57:29.220Z INFO [JerseyService] Enabling CORS for HTTP endpoint
2021-01-14T09:57:51.135Z INFO [JerseyService] Started REST API at <127.0.0.1:9000>
2021-01-14T09:57:51.136Z INFO [ServerBootstrap] Services started, startup times in ms: {JobSchedulerService [RUNNING]=46, EtagService [RUNNING]=46, OutputSetupService [RUNNING]=49, BufferSynchronizerService [RUNNING]=50, GracefulShutdownService [RUNNING]=62, InputSetupService [RUNNING]=70, ConfigurationEtagService [RUNNING]=102, UrlWhitelistService [RUNNING]=113, KafkaJournal [RUNNING]=114, JournalReader [RUNNING]=114, MongoDBProcessingStatusRecorderService [RUNNING]=135, LookupTableService [RUNNING]=180, PeriodicalsService [RUNNING]=275, StreamCacheService [RUNNING]=284, JerseyService [RUNNING]=22937}
2021-01-14T09:57:51.143Z INFO [ServiceManagerListener] Services are healthy
2021-01-14T09:57:51.150Z INFO [InputSetupService] Triggering launching persisted inputs, node transitioned from Uninitialized [LB:DEAD] to Running [LB:ALIVE]
2021-01-14T09:57:51.152Z INFO [ServerBootstrap] Graylog server up and running.
^C

dickinsonzach · January 14, 2021, 1:17pm

I had to add port 9000 to the local OS firewall

sudo firewall-cmd --add-port=9000/tcp --permanent

On CentOS 8

Thank you, Zach.

Elvis · January 14, 2021, 2:15pm

I am on Debian, not sure if this is valid form me.

Thank you for the tip.

Elvis · January 14, 2021, 2:25pm

I looked it up and it seems like Debian has a local firewall-cmd too, so i installed that and added port and restarted graylog server. but its still giving same error

root@servername:~# sudo firewall-cmd --add-port=9000/tcp --permanent
success

2021-01-14T14:22:25.424Z INFO [JerseyService] Enabling CORS for HTTP endpoint
2021-01-14T14:22:47.125Z INFO [JerseyService] Started REST API at <127.0.0.1:9000>
2021-01-14T14:22:47.126Z INFO [ServerBootstrap] Services started, startup times in ms: {JobSchedulerService [RUNNING]=84, EtagService [RUNNING]=89, OutputSetupService [RUNNING]=90, GracefulShutdownService [RUNNING]=91, BufferSynchronizerService [RUNNING]=92, InputSetupService [RUNNING]=98, ConfigurationEtagService [RUNNING]=133, UrlWhitelistService [RUNNING]=145, JournalReader [RUNNING]=146, KafkaJournal [RUNNING]=146, MongoDBProcessingStatusRecorderService [RUNNING]=152, PeriodicalsService [RUNNING]=193, LookupTableService [RUNNING]=209, StreamCacheService [RUNNING]=228, JerseyService [RUNNING]=22693}
2021-01-14T14:22:47.129Z INFO [ServiceManagerListener] Services are healthy
2021-01-14T14:22:47.135Z INFO [InputSetupService] Triggering launching persisted inputs, node transitioned from Uninitialized [LB:DEAD] to Running [LB:ALIVE]
2021-01-14T14:22:47.137Z INFO [ServerBootstrap] Graylog server up and running.

Elvis · January 14, 2021, 8:33pm

Here is my Elasticsearch Logs:

root@servername:~# tail -f /var/log/elasticsearch/graylog.log
[2021-01-14T20:27:12,997][INFO ][o.e.t.TransportService ] [servername] publish_address {127.0.0.1:9300}, bound_addresses {[::1]:9300}, {127.0.0.1:9300}
[2021-01-14T20:27:13,426][WARN ][o.e.b.BootstrapChecks ] [servername] the default discovery settings are unsuitable for production use; at least one of [discovery.seed_hosts, discovery.seed_providers, cluster.initial_master_nodes] must be configured
[2021-01-14T20:27:13,430][INFO ][o.e.c.c.Coordinator ] [servername] cluster UUID [g7kLQ20jTw-4I3YOa5ubNw]
[2021-01-14T20:27:13,437][INFO ][o.e.c.c.ClusterBootstrapService] [servername] no discovery configuration found, will perform best-effort cluster bootstrapping after [3s] unless existing master is discovered
[2021-01-14T20:27:13,561][INFO ][o.e.c.s.MasterService ] [servername] elected-as-master ([1] nodes joined)[{servername}{39E8DpBzQ9-Y_LsoQyTWgA}{FdZ9t748QNWUgGvH0CdRGA}{127.0.0.1}{127.0.0.1:9300}{dimr} elect leader, BECOME_MASTER_TASK, FINISH_ELECTION], term: 12, version: 68, delta: master node changed {previous , current [{servername}{39E8DpBzQ9-Y_LsoQyTWgA}{FdZ9t748QNWUgGvH0CdRGA}{127.0.0.1}{127.0.0.1:9300}{dimr}]}
[2021-01-14T20:27:13,646][INFO ][o.e.c.s.ClusterApplierService] [svx-esp-glog] master node changed {previous , current [{servername}{39E8DpBzQ9-Y_LsoQyTWgA}{FdZ9t748QNWUgGvH0CdRGA}{127.0.0.1}{127.0.0.1:9300}{dimr}]}, term: 12, version: 68, reason: Publication{term=12, version=68}
[2021-01-14T20:27:13,681][INFO ][o.e.h.AbstractHttpServerTransport] [servername] publish_address {127.0.0.1:9200}, bound_addresses {[::1]:9200}, {127.0.0.1:9200}
[2021-01-14T20:27:13,684][INFO ][o.e.n.Node ] [servername] started
[2021-01-14T20:27:13,808][INFO ][o.e.g.GatewayService ] [servername] recovered [3] indices into cluster_state
[2021-01-14T20:27:15,533][INFO ][o.e.c.r.a.AllocationService] [servername] Cluster health status changed from [RED] to [GREEN] (reason: [shards started [[graylog_0][3], [graylog_0][2], [graylog_0][0]]]).

system · January 28, 2021, 8:34pm

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Can't connect to graylog service Graylog Central (peer support)	7	3482	November 8, 2018
Graylog has stopped working - Server Currently unavailable Graylog Central (peer support)	3	964	March 10, 2020
Server currently unavailable for Ubuntu 16.04 Graylog Central (peer support)	19	2301	May 30, 2018
New setup for graylog the web interface is not working Graylog Central (peer support)	2	3218	June 21, 2017
Graylog logs indicate no errors and server running; web site not responsive Graylog Central (peer support)	11	8010	February 12, 2018

Server currently unavailable 2

Related topics