Server currently unavailable 2

I just installed graylog on a vm with 8GB Ram.

i am getting error “We are experiencing problems connecting to the Graylog server running on http://127.0.0.1:9000/api/ . Please verify that the server is healthy and working correctly.”

server
{
listen 80;
listen :80 ipv6only=on;
server_name servername.com;
access_log /var/log/nginx/graylog.access.log;

location / {
  proxy_set_header Host $http_host;
  proxy_set_header X-Forwarded-Host $host;
  proxy_set_header X-Forwarded-Server $host;
  proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
  proxy_set_header X-Graylog-Server-URL http://$server_name/;
  proxy_pass       http://127.0.0.1:9000;
}

}

Graylog server is up and running
MongoDB is connected and running with no errors, i suspect something in config file,
Any advice will be appreciated

-Elvis

What’s the result of running curl -s 127.0.0.1:9000? Also, seeing your server.conf and any associated logs from Graylog would help.

I get a below: Please note that i removed enterprise-plugins

root@servername:~# curl -s 127.0.0.1:9000

Graylog Web Interface
<script src="http://127.0.0.1:9000/assets/vendor.594b2a39cb22b445205e.js"></script>

<script src="http://127.0.0.1:9000/assets/polyfill.a41ce78e1beaf9cde903.js"></script>

<script src="http://127.0.0.1:9000/assets/builtins.a41ce78e1beaf9cde903.js"></script>

<script src="http://127.0.0.1:9000/assets/plugin/org.graylog.plugins.threatintel.ThreatIntelPlugin/plugin.org.graylog.plugins.threatintel.ThreatIntelPlugin.3ffe487c5932d920ffec.js"></script>

<script src="http://127.0.0.1:9000/assets/plugin/org.graylog.plugins.collector.CollectorPlugin/plugin.org.graylog.plugins.collector.CollectorPlugin.4d259ce7c45134eff0fb.js"></script>

<script src="http://127.0.0.1:9000/assets/plugin/org.graylog.integrations.IntegrationsPlugin/plugin.org.graylog.integrations.IntegrationsPlugin.e951c9a4e4cd6c24e9df.js"></script>

<script src="http://127.0.0.1:9000/assets/plugin/org.graylog.aws.AWSPlugin/plugin.org.graylog.aws.AWSPlugin.a03644a3671e94e1f717.js"></script>

<script src="http://127.0.0.1:9000/assets/app.a41ce78e1beaf9cde903.js"></script>
root@servername:~#

Hi there, thanks for that, but what about your sever.conf and logs? Also, what version are you running?

I am running graylog 4.0

is_master = true
node_id_file = /etc/graylog/server/node-id
password_secret = xxxxx
root_password_sha2 = xxxxx
bin_dir = /usr/share/graylog-server/bin
data_dir = /var/lib/graylog-server
plugin_dir = /usr/share/graylog-server/plugin

http_bind_address = 127.0.0.1:9000
http_publish_uri = http://127.0.0.1:9000/
http_enable_cors = true
rotation_strategy = count
elasticsearch_max_docs_per_index = 20000000
elasticsearch_max_number_of_indices = 20
retention_strategy = delete
elasticsearch_shards = 4
elasticsearch_replicas = 0
elasticsearch_index_prefix = graylog
allow_leading_wildcard_searches = false
allow_highlighting = false
elasticsearch_analyzer = standard
output_batch_size = 500
output_flush_interval = 1
output_fault_count_threshold = 5
output_fault_penalty_seconds = 30
processbuffer_processors = 5
outputbuffer_processors = 3
processor_wait_strategy = blocking
ring_size = 65536

inputbuffer_ring_size = 65536
inputbuffer_processors = 2
inputbuffer_wait_strategy = blocking
message_journal_enabled = true
message_journal_dir = /var/lib/graylog-server/journal
lb_recognition_period_seconds = 3
mongodb_uri = mongodb://localhost/graylog
mongodb_max_connections = 1000
mongodb_threads_allowed_to_block_multiplier = 5
proxied_requests_thread_pool_size = 32

When i restart service and trace logs

2021-01-14T09:57:51.143Z INFO [ServiceManagerListener] Services are healthy
2021-01-14T09:57:51.150Z INFO [InputSetupService] Triggering launching persisted inputs, node transitioned from Uninitialized [LB:DEAD] to Running [LB:ALIVE]
2021-01-14T09:57:51.152Z INFO [ServerBootstrap] Graylog server up and running.

Logs:

root@servername:~# sudo systemctl restart graylog-server && tail -n 0 -f /var/log/graylog-server/server.log
2021-01-14T09:57:18.890Z INFO [CmdLineTool] Loaded plugin: AWS plugins 4.0.1 [org.graylog.aws.AWSPlugin]
2021-01-14T09:57:18.894Z INFO [CmdLineTool] Loaded plugin: Integrations 4.0.1 [org.graylog.integrations.IntegrationsPlugin]
2021-01-14T09:57:18.898Z INFO [CmdLineTool] Loaded plugin: Collector 4.0.1 [org.graylog.plugins.collector.CollectorPlugin]
2021-01-14T09:57:18.899Z INFO [CmdLineTool] Loaded plugin: Threat Intelligence Plugin 4.0.1 [org.graylog.plugins.threatintel.ThreatIntelPlugin]
2021-01-14T09:57:18.900Z INFO [CmdLineTool] Loaded plugin: Elasticsearch 6 Support 4.0.1+6a0cc0b [org.graylog.storage.elasticsearch6.Elasticsearch6Plugin]
2021-01-14T09:57:18.900Z INFO [CmdLineTool] Loaded plugin: Elasticsearch 7 Support 4.0.1+6a0cc0b [org.graylog.storage.elasticsearch7.Elasticsearch7Plugin]
2021-01-14T09:57:19.161Z INFO [CmdLineTool] Running with JVM arguments: -Xms1g -Xmx1g -XX:NewRatio=1 -XX:+ResizeTLAB -XX:+UseConcMarkSweepGC -XX:+CMSConcurrentMTEnabled -XX:+CMSClassUnloadingEnabled -XX:-OmitStackTraceInFastThrow -Djdk.tls.acknowledgeCloseNotify=true -Dlog4j.configurationFile=file:///etc/graylog/server/log4j2.xml -Djava.library.path=/usr/share/graylog-server/lib/sigar -Dgraylog2.installation_source=deb
2021-01-14T09:57:19.528Z INFO [Version] HV000001: Hibernate Validator null
2021-01-14T09:57:24.130Z INFO [InputBufferImpl] Message journal is enabled.
2021-01-14T09:57:24.154Z INFO [NodeId] Node ID: 861550b3-5ac7-47c2-b6d1-b83cbf9d7f10
2021-01-14T09:57:24.439Z INFO [LogManager] Loading logs.
2021-01-14T09:57:24.531Z INFO [LogManager] Logs loading complete.
2021-01-14T09:57:24.533Z INFO [KafkaJournal] Initialized Kafka based journal at /var/lib/graylog-server/journal
2021-01-14T09:57:24.568Z INFO [cluster] Cluster created with settings {hosts=[localhost:27017], mode=SINGLE, requiredClusterType=UNKNOWN, serverSelectionTimeout=‘30000 ms’, maxWaitQueueSize=5000}
2021-01-14T09:57:24.644Z INFO [cluster] Cluster description not yet available. Waiting for 30000 ms before timing out
2021-01-14T09:57:24.687Z INFO [connection] Opened connection [connectionId{localValue:1, serverValue:187}] to localhost:27017
2021-01-14T09:57:24.695Z INFO [cluster] Monitor thread successfully connected to server with description ServerDescription{address=localhost:27017, type=STANDALONE, state=CONNECTED, ok=true, version=ServerVersion{versionList=[4, 2, 11]}, minWireVersion=0, maxWireVersion=8, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=4016278}
2021-01-14T09:57:24.717Z INFO [connection] Opened connection [connectionId{localValue:2, serverValue:188}] to localhost:27017
2021-01-14T09:57:25.031Z INFO [InputBufferImpl] Initialized InputBufferImpl with ring size <65536> and wait strategy , running 2 parallel message handlers.
2021-01-14T09:57:25.740Z INFO [ElasticsearchVersionProvider] Elasticsearch cluster is running v7.10.1
2021-01-14T09:57:26.974Z INFO [ProcessBuffer] Initialized ProcessBuffer with ring size <65536> and wait strategy .
2021-01-14T09:57:27.123Z WARN [GeoIpResolverEngine] GeoIP database file does not exist: /etc/graylog/server/GeoLite2-City.mmdb
2021-01-14T09:57:27.136Z INFO [OutputBuffer] Initialized OutputBuffer with ring size <65536> and wait strategy .
2021-01-14T09:57:27.161Z INFO [connection] Opened connection [connectionId{localValue:3, serverValue:189}] to localhost:27017
2021-01-14T09:57:27.203Z WARN [GeoIpResolverEngine] GeoIP database file does not exist: /etc/graylog/server/GeoLite2-City.mmdb
2021-01-14T09:57:27.233Z WARN [GeoIpResolverEngine] GeoIP database file does not exist: /etc/graylog/server/GeoLite2-City.mmdb
2021-01-14T09:57:27.269Z WARN [GeoIpResolverEngine] GeoIP database file does not exist: /etc/graylog/server/GeoLite2-City.mmdb
2021-01-14T09:57:27.305Z WARN [GeoIpResolverEngine] GeoIP database file does not exist: /etc/graylog/server/GeoLite2-City.mmdb
2021-01-14T09:57:28.166Z INFO [ServerBootstrap] Graylog server 4.0.1+6a0cc0b starting up
2021-01-14T09:57:28.168Z INFO [ServerBootstrap] JRE: Debian 11.0.9.1 on Linux 4.19.0-13-amd64
2021-01-14T09:57:28.169Z INFO [ServerBootstrap] Deployment: deb
2021-01-14T09:57:28.169Z INFO [ServerBootstrap] OS: Debian GNU/Linux 10 (buster) (debian)
2021-01-14T09:57:28.169Z INFO [ServerBootstrap] Arch: amd64
2021-01-14T09:57:28.225Z INFO [PeriodicalsService] Starting 30 periodicals …
2021-01-14T09:57:28.225Z INFO [Periodicals] Starting [org.graylog2.periodical.ThroughputCalculator] periodical in [0s], polling every [1s].
2021-01-14T09:57:28.245Z INFO [Periodicals] Starting [org.graylog.plugins.pipelineprocessor.periodical.LegacyDefaultStreamMigration] periodical, running forever.
2021-01-14T09:57:28.273Z INFO [PeriodicalsService] Not starting [org.graylog2.periodical.AlertScannerThread] periodical. Not configured to run on this node.
2021-01-14T09:57:28.273Z INFO [Periodicals] Starting [org.graylog2.periodical.BatchedElasticSearchOutputFlushThread] periodical in [0s], polling every [1s].
2021-01-14T09:57:28.275Z INFO [Periodicals] Starting [org.graylog2.periodical.ClusterHealthCheckThread] periodical in [120s], polling every [20s].
2021-01-14T09:57:28.275Z INFO [PeriodicalsService] Not starting [org.graylog2.periodical.ContentPackLoaderPeriodical] periodical. Not configured to run on this node.
2021-01-14T09:57:28.275Z INFO [Periodicals] Starting [org.graylog2.periodical.GarbageCollectionWarningThread] periodical, running forever.
2021-01-14T09:57:28.276Z INFO [Periodicals] Starting [org.graylog2.periodical.IndexerClusterCheckerThread] periodical in [0s], polling every [30s].
2021-01-14T09:57:28.277Z INFO [Periodicals] Starting [org.graylog2.periodical.IndexRetentionThread] periodical in [0s], polling every [300s].
2021-01-14T09:57:28.283Z INFO [Periodicals] Starting [org.graylog2.periodical.IndexRotationThread] periodical in [0s], polling every [10s].
2021-01-14T09:57:28.289Z INFO [Periodicals] Starting [org.graylog2.periodical.NodePingThread] periodical in [0s], polling every [1s].
2021-01-14T09:57:28.297Z INFO [Periodicals] Starting [org.graylog2.periodical.VersionCheckThread] periodical in [300s], polling every [1800s].
2021-01-14T09:57:28.298Z INFO [Periodicals] Starting [org.graylog2.periodical.ThrottleStateUpdaterThread] periodical in [1s], polling every [1s].
2021-01-14T09:57:28.298Z INFO [Periodicals] Starting [org.graylog2.events.ClusterEventPeriodical] periodical in [0s], polling every [1s].
2021-01-14T09:57:28.300Z INFO [Periodicals] Starting [org.graylog2.events.ClusterEventCleanupPeriodical] periodical in [0s], polling every [86400s].
2021-01-14T09:57:28.303Z INFO [Periodicals] Starting [org.graylog2.periodical.ClusterIdGeneratorPeriodical] periodical, running forever.
2021-01-14T09:57:28.309Z INFO [Periodicals] Starting [org.graylog2.periodical.IndexRangesMigrationPeriodical] periodical, running forever.
2021-01-14T09:57:28.309Z INFO [Periodicals] Starting [org.graylog2.periodical.IndexRangesCleanupPeriodical] periodical in [15s], polling every [3600s].
2021-01-14T09:57:28.330Z INFO [LegacyDefaultStreamMigration] Legacy default stream has no connections, no migration needed.
2021-01-14T09:57:28.345Z INFO [connection] Opened connection [connectionId{localValue:4, serverValue:190}] to localhost:27017
2021-01-14T09:57:28.346Z INFO [connection] Opened connection [connectionId{localValue:8, serverValue:194}] to localhost:27017
2021-01-14T09:57:28.346Z INFO [connection] Opened connection [connectionId{localValue:5, serverValue:191}] to localhost:27017
2021-01-14T09:57:28.347Z INFO [connection] Opened connection [connectionId{localValue:6, serverValue:192}] to localhost:27017
2021-01-14T09:57:28.348Z INFO [connection] Opened connection [connectionId{localValue:10, serverValue:196}] to localhost:27017
2021-01-14T09:57:28.348Z INFO [connection] Opened connection [connectionId{localValue:7, serverValue:193}] to localhost:27017
2021-01-14T09:57:28.352Z INFO [PeriodicalsService] Not starting [org.graylog2.periodical.UserPermissionMigrationPeriodical] periodical. Not configured to run on this node.
2021-01-14T09:57:28.352Z INFO [Periodicals] Starting [org.graylog2.periodical.ConfigurationManagementPeriodical] periodical, running forever.
2021-01-14T09:57:28.353Z INFO [connection] Opened connection [connectionId{localValue:9, serverValue:195}] to localhost:27017
2021-01-14T09:57:28.362Z INFO [Periodicals] Starting [org.graylog2.periodical.IndexFailuresPeriodical] periodical, running forever.
2021-01-14T09:57:28.362Z INFO [Periodicals] Starting [org.graylog2.periodical.TrafficCounterCalculator] periodical in [0s], polling every [1s].
2021-01-14T09:57:28.402Z INFO [Periodicals] Starting [org.graylog2.indexer.fieldtypes.IndexFieldTypePollerPeriodical] periodical in [0s], polling every [3600s].
2021-01-14T09:57:28.411Z INFO [Periodicals] Starting [org.graylog.scheduler.periodicals.ScheduleTriggerCleanUp] periodical in [120s], polling every [86400s].
2021-01-14T09:57:28.413Z INFO [Periodicals] Starting [org.graylog2.periodical.ESVersionCheckPeriodical] periodical in [0s], polling every [30s].
2021-01-14T09:57:28.417Z INFO [Periodicals] Starting [org.graylog.plugins.sidecar.periodical.PurgeExpiredSidecarsThread] periodical in [0s], polling every [600s].
2021-01-14T09:57:28.425Z INFO [Periodicals] Starting [org.graylog.plugins.sidecar.periodical.PurgeExpiredConfigurationUploads] periodical in [0s], polling every [600s].
2021-01-14T09:57:28.429Z INFO [Periodicals] Starting [org.graylog.plugins.views.search.db.SearchesCleanUpJob] periodical in [3600s], polling every [28800s].
2021-01-14T09:57:28.435Z INFO [Periodicals] Starting [org.graylog.events.periodicals.EventNotificationStatusCleanUp] periodical in [120s], polling every [86400s].
2021-01-14T09:57:28.440Z INFO [Periodicals] Starting [org.graylog.plugins.collector.periodical.PurgeExpiredCollectorsThread] periodical in [0s], polling every [3600s].
2021-01-14T09:57:29.220Z INFO [JerseyService] Enabling CORS for HTTP endpoint
2021-01-14T09:57:51.135Z INFO [JerseyService] Started REST API at <127.0.0.1:9000>
2021-01-14T09:57:51.136Z INFO [ServerBootstrap] Services started, startup times in ms: {JobSchedulerService [RUNNING]=46, EtagService [RUNNING]=46, OutputSetupService [RUNNING]=49, BufferSynchronizerService [RUNNING]=50, GracefulShutdownService [RUNNING]=62, InputSetupService [RUNNING]=70, ConfigurationEtagService [RUNNING]=102, UrlWhitelistService [RUNNING]=113, KafkaJournal [RUNNING]=114, JournalReader [RUNNING]=114, MongoDBProcessingStatusRecorderService [RUNNING]=135, LookupTableService [RUNNING]=180, PeriodicalsService [RUNNING]=275, StreamCacheService [RUNNING]=284, JerseyService [RUNNING]=22937}
2021-01-14T09:57:51.143Z INFO [ServiceManagerListener] Services are healthy
2021-01-14T09:57:51.150Z INFO [InputSetupService] Triggering launching persisted inputs, node transitioned from Uninitialized [LB:DEAD] to Running [LB:ALIVE]
2021-01-14T09:57:51.152Z INFO [ServerBootstrap] Graylog server up and running.
^C

I had to add port 9000 to the local OS firewall

sudo firewall-cmd --add-port=9000/tcp --permanent

On CentOS 8

Thank you, Zach.

I am on Debian, not sure if this is valid form me.

Thank you for the tip.

I looked it up and it seems like Debian has a local firewall-cmd too, so i installed that and added port and restarted graylog server. but its still giving same error

root@servername:~# sudo firewall-cmd --add-port=9000/tcp --permanent
success

2021-01-14T14:22:25.424Z INFO [JerseyService] Enabling CORS for HTTP endpoint
2021-01-14T14:22:47.125Z INFO [JerseyService] Started REST API at <127.0.0.1:9000>
2021-01-14T14:22:47.126Z INFO [ServerBootstrap] Services started, startup times in ms: {JobSchedulerService [RUNNING]=84, EtagService [RUNNING]=89, OutputSetupService [RUNNING]=90, GracefulShutdownService [RUNNING]=91, BufferSynchronizerService [RUNNING]=92, InputSetupService [RUNNING]=98, ConfigurationEtagService [RUNNING]=133, UrlWhitelistService [RUNNING]=145, JournalReader [RUNNING]=146, KafkaJournal [RUNNING]=146, MongoDBProcessingStatusRecorderService [RUNNING]=152, PeriodicalsService [RUNNING]=193, LookupTableService [RUNNING]=209, StreamCacheService [RUNNING]=228, JerseyService [RUNNING]=22693}
2021-01-14T14:22:47.129Z INFO [ServiceManagerListener] Services are healthy
2021-01-14T14:22:47.135Z INFO [InputSetupService] Triggering launching persisted inputs, node transitioned from Uninitialized [LB:DEAD] to Running [LB:ALIVE]
2021-01-14T14:22:47.137Z INFO [ServerBootstrap] Graylog server up and running.

Here is my Elasticsearch Logs:

root@servername:~# tail -f /var/log/elasticsearch/graylog.log
[2021-01-14T20:27:12,997][INFO ][o.e.t.TransportService ] [servername] publish_address {127.0.0.1:9300}, bound_addresses {[::1]:9300}, {127.0.0.1:9300}
[2021-01-14T20:27:13,426][WARN ][o.e.b.BootstrapChecks ] [servername] the default discovery settings are unsuitable for production use; at least one of [discovery.seed_hosts, discovery.seed_providers, cluster.initial_master_nodes] must be configured
[2021-01-14T20:27:13,430][INFO ][o.e.c.c.Coordinator ] [servername] cluster UUID [g7kLQ20jTw-4I3YOa5ubNw]
[2021-01-14T20:27:13,437][INFO ][o.e.c.c.ClusterBootstrapService] [servername] no discovery configuration found, will perform best-effort cluster bootstrapping after [3s] unless existing master is discovered
[2021-01-14T20:27:13,561][INFO ][o.e.c.s.MasterService ] [servername] elected-as-master ([1] nodes joined)[{servername}{39E8DpBzQ9-Y_LsoQyTWgA}{FdZ9t748QNWUgGvH0CdRGA}{127.0.0.1}{127.0.0.1:9300}{dimr} elect leader, BECOME_MASTER_TASK, FINISH_ELECTION], term: 12, version: 68, delta: master node changed {previous , current [{servername}{39E8DpBzQ9-Y_LsoQyTWgA}{FdZ9t748QNWUgGvH0CdRGA}{127.0.0.1}{127.0.0.1:9300}{dimr}]}
[2021-01-14T20:27:13,646][INFO ][o.e.c.s.ClusterApplierService] [svx-esp-glog] master node changed {previous , current [{servername}{39E8DpBzQ9-Y_LsoQyTWgA}{FdZ9t748QNWUgGvH0CdRGA}{127.0.0.1}{127.0.0.1:9300}{dimr}]}, term: 12, version: 68, reason: Publication{term=12, version=68}
[2021-01-14T20:27:13,681][INFO ][o.e.h.AbstractHttpServerTransport] [servername] publish_address {127.0.0.1:9200}, bound_addresses {[::1]:9200}, {127.0.0.1:9200}
[2021-01-14T20:27:13,684][INFO ][o.e.n.Node ] [servername] started
[2021-01-14T20:27:13,808][INFO ][o.e.g.GatewayService ] [servername] recovered [3] indices into cluster_state
[2021-01-14T20:27:15,533][INFO ][o.e.c.r.a.AllocationService] [servername] Cluster health status changed from [RED] to [GREEN] (reason: [shards started [[graylog_0][3], [graylog_0][2], [graylog_0][0]]]).