API URL Wrong Unable to call http://api/system

[jkavenaugh (James Kavenaugh)]

Before you post: Your responses to these questions will help the community help you. Please complete this template if you’re asking a support question.
Don’t forget to select tags to help index your topic!

1. Describe your incident:
Inputs won’t start. The log is filled with Unable to call http://api/system warnings.
Clicking on node 8964c162 / localhost shows me the monkey. Error getting data.

2. Describe your environment:

• OS Information:
  Ubuntu 20.04.4
• Package Version:
  4.2.7-1
• Service logs, configurations, and environment variables:
  Unable to call h**://api/system/inputstates/623b45dc20fad36bfa76229d on node <8964c162-c032-4db5-880f-82e3258b9a05>: Connect timed out

nginx
server
{
listen 443 ssl http2;
server_name graylog.datanat.com;
ssl_certificate /etc/ssl/certs/localhost.crt;
ssl_certificate_key /etc/ssl/private/localhost.key;
add_header ‘Content-Security-Policy’ ‘upgrade-insecure-requests’;
location /
{
proxy_set_header Host $http_host;
proxy_set_header X-Forwarded-Host $host;
proxy_set_header X-Forwarded-Server $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Graylog-Server-URL https://$server_name/;
proxy_pass h**://10.141.0.64:9000;
}
}

server.conf
http_bind_address = 10.141.0.64:9000
http_publish_uri = h**://$http_bind_address/
http_external_uri = h**://$http_publish_address/
3. What steps have you already taken to try and solve the problem?
I originally had http_bind_address set to 127.0.0.1:9000. Changed it to the network IP 10.141.64:9000. Tried accessing the web server directly without going through NGINX. Re-installed the whole server.
I tried setting http_external_uri = 10.141.0.64:9000

4. How can the community help?
What is causing the server to access h**://api/system/inputstates instead of h**://10.141.0.64/api/system/inputstates ?

h** = http, the forum wouldn’t let me use more than two http://

Thanks!

[gsmith]

Hello && Welcome

I’m unsure exactly what’s going on. From what I can understand is the Graylog server is not work at all, is this correct?

If you could us the markdown for posting Configuration files and logs that would be great, you can find more information here on “HowTo”.

How to Post a Question in the Community that Gets Responses

This platform is made with love for community discussions on the open source tool Graylog, it components and usage. Here’s a Graylog support-inspired template (thank you, @aaronsachs ) that’ll get responses: Description of your problem <!-- Use this section to describe the problem that you're encountering. Please include any screenshots or recordings of the problem you're running into.--> Description of steps you’ve taken to attempt to solve the issue <!-- Use this section to provide detail…

If you can give us a great amount of details what’s happening that would be great.

Showing your whole configuration file would be nice. NOTE: remove personal information when posting.
Example:

cat /etc/graylog/server/server.conf | egrep -v "^\s*(#|$)"

Output

[root@graylog ingest-attachment]# cat /etc/graylog/server/server.conf | egrep -v "^\s*(#|$)"
is_master = true
node_id_file = /etc/graylog/server/node-id
password_secret =en22EBT17elgGTUJgbD
root_password_sha2 =2df7fef0a4171ea6
root_email = "greg.smith@domain.com"
root_timezone = America/Chicago
bin_dir = /usr/share/graylog-server/bin
data_dir = /var/lib/graylog-server
plugin_dir = /usr/share/graylog-server/plugin
http_bind_address = graylog.domain.com:9000
http_publish_uri = https://graylog.domain.com:9000/
http_enable_cors = true
http_enable_tls = true
http_tls_cert_file = /etc/pki/tls/certs/graylog/graylog-certificate.pem
http_tls_key_file = /etc/pki/tls/certs/graylog/graylog-key.pem
http_tls_key_password = secret
elasticsearch_hosts = http://10.10.10.10:9200

Question: Did this Graylog server work before with/without Nginx?

[jkavenaugh (James Kavenaugh)]

Thank you for responding.

From what I can understand is the Graylog server is not work at all, is this correct?

The web portion works. When the web portion tries to access the API, it fails. It appears to be trying to access the API without using an IP + port number in the URL.

Question: Did this Graylog server work before with/without Nginx?

This is a new build. Disabling Nginx and accessing http://10.141.0.64:9000/ directly also causes the issue.

server.conf

is_master = true
node_id_file = /etc/graylog/server/node-id
password_secret = ****
root_password_sha2 = ****
root_timezone = EST
bin_dir = /usr/share/graylog-server/bin
data_dir = /var/lib/graylog-server
plugin_dir = /usr/share/graylog-server/plugin
http_bind_address = 10.141.0.64:9000
http_publish_uri = http://$http_bind_address/
http_external_uri = http://10.141.0.64:9000/
http_enable_cors = true
http_enable_gzip = true
http_max_header_size = 8192
http_thread_pool_size = 16
http_enable_tls = false
trusted_proxies = 127.0.0.1/32, 0:0:0:0:0:0:0:1/128
rotation_strategy = count
elasticsearch_max_docs_per_index = 20000000
elasticsearch_max_number_of_indices = 20
retention_strategy = delete
elasticsearch_shards = 4
elasticsearch_replicas = 0
elasticsearch_index_prefix = graylog
allow_leading_wildcard_searches = false
allow_highlighting = false
elasticsearch_analyzer = standard
output_batch_size = 500
output_flush_interval = 1
output_fault_count_threshold = 5
output_fault_penalty_seconds = 30
processbuffer_processors = 5
outputbuffer_processors = 3
processor_wait_strategy = blocking
ring_size = 65536
inputbuffer_ring_size = 65536
inputbuffer_processors = 2
inputbuffer_wait_strategy = blocking
message_journal_enabled = true
message_journal_dir = /var/lib/graylog-server/journal
lb_recognition_period_seconds = 3
mongodb_uri = mongodb://localhost/graylog
mongodb_max_connections = 1000
mongodb_threads_allowed_to_block_multiplier = 5
proxied_requests_thread_pool_size = 32

server.log

2022-03-24T13:41:42.070-04:00 INFO  [ImmutableFeatureFlagsCollector] Following feature flags are used: {}
2022-03-24T13:41:44.067-04:00 INFO  [CmdLineTool] Loaded plugin: AWS plugins 4.2.7 [org.graylog.aws.AWSPlugin]
2022-03-24T13:41:44.068-04:00 INFO  [CmdLineTool] Loaded plugin: Enterprise Integrations 4.2.7 [org.graylog.enterprise.integrations.EnterpriseIntegrationsPlugin]
2022-03-24T13:41:44.068-04:00 INFO  [CmdLineTool] Loaded plugin: Integrations 4.2.7 [org.graylog.integrations.IntegrationsPlugin]
2022-03-24T13:41:44.069-04:00 INFO  [CmdLineTool] Loaded plugin: Collector 4.2.7 [org.graylog.plugins.collector.CollectorPlugin]
2022-03-24T13:41:44.069-04:00 INFO  [CmdLineTool] Loaded plugin: Graylog Enterprise 4.2.7 [org.graylog.plugins.enterprise.EnterprisePlugin]
2022-03-24T13:41:44.070-04:00 INFO  [CmdLineTool] Loaded plugin: Graylog Enterprise (ES6 Support) 4.2.7 [org.graylog.plugins.enterprise.org.graylog.plugins.enterprise.es6.EnterpriseES6Plugin]
2022-03-24T13:41:44.071-04:00 INFO  [CmdLineTool] Loaded plugin: Graylog Enterprise (ES7 Support) 4.2.7 [org.graylog.plugins.enterprise.org.graylog.plugins.enterprise.es7.EnterpriseES7Plugin]
2022-03-24T13:41:44.071-04:00 INFO  [CmdLineTool] Loaded plugin: Threat Intelligence Plugin 4.2.7 [org.graylog.plugins.threatintel.ThreatIntelPlugin]
2022-03-24T13:41:44.071-04:00 INFO  [CmdLineTool] Loaded plugin: Elasticsearch 6 Support 4.2.7+879e651 [org.graylog.storage.elasticsearch6.Elasticsearch6Plugin]
2022-03-24T13:41:44.072-04:00 INFO  [CmdLineTool] Loaded plugin: Elasticsearch 7 Support 4.2.7+879e651 [org.graylog.storage.elasticsearch7.Elasticsearch7Plugin]
2022-03-24T13:41:44.110-04:00 INFO  [CmdLineTool] Running with JVM arguments: -Xms1g -Xmx1g -XX:NewRatio=1 -XX:+ResizeTLAB -XX:-OmitStackTraceInFastThrow -Djdk.tls.acknowledgeCloseNotify=true -Dlog4j2.formatMsgNoLookups=true -Dlog4j.configurationFile=file:///etc/graylog/server/log4j2.xml -Djava.library.path=/usr/share/graylog-server/lib/sigar -Dgraylog2.installation_source=deb
2022-03-24T13:41:44.280-04:00 INFO  [Version] HV000001: Hibernate Validator null
2022-03-24T13:41:47.630-04:00 INFO  [InputBufferImpl] Message journal is enabled.
2022-03-24T13:41:47.658-04:00 INFO  [NodeId] Node ID: 8964c162-c032-4db5-880f-82e3258b9a05
2022-03-24T13:41:47.991-04:00 INFO  [LogManager] Loading logs.
2022-03-24T13:41:48.084-04:00 INFO  [LogManager] Logs loading complete.
2022-03-24T13:41:48.089-04:00 INFO  [LocalKafkaJournal] Initialized Kafka based journal at /var/lib/graylog-server/journal
2022-03-24T13:41:48.125-04:00 INFO  [cluster] Cluster created with settings {hosts=[localhost:27017], mode=SINGLE, requiredClusterType=UNKNOWN, serverSelectionTimeout='30000 ms', maxWaitQueueSize=5000}
2022-03-24T13:41:48.168-04:00 INFO  [cluster] Cluster description not yet available. Waiting for 30000 ms before timing out
2022-03-24T13:41:48.443-04:00 INFO  [connection] Opened connection [connectionId{localValue:1, serverValue:206}] to localhost:27017
2022-03-24T13:41:48.447-04:00 INFO  [cluster] Monitor thread successfully connected to server with description ServerDescription{address=localhost:27017, type=STANDALONE, state=CONNECTED, ok=true, version=ServerVersion{versionList=[4, 0, 28]}, minWireVersion=0, maxWireVersion=7, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=2921619}
2022-03-24T13:41:48.463-04:00 INFO  [connection] Opened connection [connectionId{localValue:2, serverValue:207}] to localhost:27017
2022-03-24T13:41:49.211-04:00 INFO  [InputBufferImpl] Initialized InputBufferImpl with ring size <65536> and wait strategy <BlockingWaitStrategy>, running 2 parallel message handlers.
2022-03-24T13:41:49.772-04:00 INFO  [ElasticsearchVersionProvider] Elasticsearch cluster is running v7.10.2
2022-03-24T13:41:50.900-04:00 INFO  [ProcessBuffer] Initialized ProcessBuffer with ring size <65536> and wait strategy <BlockingWaitStrategy>.
2022-03-24T13:41:50.959-04:00 WARN  [GeoIpResolverEngine] GeoIP database file does not exist: /etc/graylog/server/GeoLite2-City.mmdb
2022-03-24T13:41:50.966-04:00 INFO  [OutputBuffer] Initialized OutputBuffer with ring size <65536> and wait strategy <BlockingWaitStrategy>.
2022-03-24T13:41:50.973-04:00 INFO  [connection] Opened connection [connectionId{localValue:3, serverValue:208}] to localhost:27017
2022-03-24T13:41:50.989-04:00 WARN  [GeoIpResolverEngine] GeoIP database file does not exist: /etc/graylog/server/GeoLite2-City.mmdb
2022-03-24T13:41:51.012-04:00 WARN  [GeoIpResolverEngine] GeoIP database file does not exist: /etc/graylog/server/GeoLite2-City.mmdb
2022-03-24T13:41:51.031-04:00 WARN  [GeoIpResolverEngine] GeoIP database file does not exist: /etc/graylog/server/GeoLite2-City.mmdb
2022-03-24T13:41:51.050-04:00 WARN  [GeoIpResolverEngine] GeoIP database file does not exist: /etc/graylog/server/GeoLite2-City.mmdb
2022-03-24T13:41:51.563-04:00 ERROR [AuditLogger] Unable to write audit log entry because there is no valid license.
2022-03-24T13:41:51.565-04:00 INFO  [ServerBootstrap] Graylog server 4.2.7+879e651 starting up
2022-03-24T13:41:51.565-04:00 INFO  [ServerBootstrap] JRE: Private Build 17.0.2 on Linux 5.4.0-99-generic
2022-03-24T13:41:51.565-04:00 INFO  [ServerBootstrap] Deployment: deb
2022-03-24T13:41:51.565-04:00 INFO  [ServerBootstrap] OS: Ubuntu 20.04.4 LTS (focal)
2022-03-24T13:41:51.565-04:00 INFO  [ServerBootstrap] Arch: amd64
2022-03-24T13:41:51.588-04:00 INFO  [PeriodicalsService] Starting 37 periodicals ...
2022-03-24T13:41:51.588-04:00 INFO  [Periodicals] Starting [org.graylog2.periodical.ThroughputCalculator] periodical in [0s], polling every [1s].
2022-03-24T13:41:51.592-04:00 INFO  [Periodicals] Starting [org.graylog.plugins.pipelineprocessor.periodical.LegacyDefaultStreamMigration] periodical, running forever.
2022-03-24T13:41:51.621-04:00 INFO  [PeriodicalsService] Not starting [org.graylog2.periodical.AlertScannerThread] periodical. Not configured to run on this node.
2022-03-24T13:41:51.622-04:00 INFO  [Periodicals] Starting [org.graylog2.periodical.BatchedElasticSearchOutputFlushThread] periodical in [0s], polling every [1s].
2022-03-24T13:41:51.624-04:00 INFO  [LegacyDefaultStreamMigration] Legacy default stream has no connections, no migration needed.
2022-03-24T13:41:51.632-04:00 INFO  [Periodicals] Starting [org.graylog2.periodical.ClusterHealthCheckThread] periodical in [120s], polling every [20s].
2022-03-24T13:41:51.637-04:00 INFO  [PeriodicalsService] Not starting [org.graylog2.periodical.ContentPackLoaderPeriodical] periodical. Not configured to run on this node.
2022-03-24T13:41:51.637-04:00 INFO  [Periodicals] Starting [org.graylog2.periodical.GarbageCollectionWarningThread] periodical, running forever.
2022-03-24T13:41:51.638-04:00 INFO  [Periodicals] Starting [org.graylog2.periodical.IndexerClusterCheckerThread] periodical in [0s], polling every [30s].
2022-03-24T13:41:51.653-04:00 INFO  [LookupTableService] Data Adapter watchlist-mongo/623b40e120fad36bfa761d19 [@84bcffe] STARTING
2022-03-24T13:41:51.654-04:00 INFO  [LookupTableService] Data Adapter watchlist-mongo/623b40e120fad36bfa761d19 [@84bcffe] RUNNING
2022-03-24T13:41:51.669-04:00 INFO  [Periodicals] Starting [org.graylog2.periodical.IndexRetentionThread] periodical in [0s], polling every [300s].
2022-03-24T13:41:51.674-04:00 INFO  [Periodicals] Starting [org.graylog2.periodical.IndexRotationThread] periodical in [0s], polling every [10s].
2022-03-24T13:41:51.675-04:00 INFO  [Periodicals] Starting [org.graylog2.periodical.NodePingThread] periodical in [0s], polling every [1s].
2022-03-24T13:41:51.676-04:00 INFO  [Periodicals] Starting [org.graylog2.periodical.VersionCheckThread] periodical in [300s], polling every [1800s].
2022-03-24T13:41:51.680-04:00 INFO  [Periodicals] Starting [org.graylog2.periodical.ThrottleStateUpdaterThread] periodical in [1s], polling every [1s].
2022-03-24T13:41:51.684-04:00 INFO  [Periodicals] Starting [org.graylog2.events.ClusterEventPeriodical] periodical in [0s], polling every [1s].
2022-03-24T13:41:51.687-04:00 INFO  [Periodicals] Starting [org.graylog2.events.ClusterEventCleanupPeriodical] periodical in [0s], polling every [86400s].
2022-03-24T13:41:51.688-04:00 INFO  [Periodicals] Starting [org.graylog2.periodical.ClusterIdGeneratorPeriodical] periodical, running forever.
2022-03-24T13:41:51.688-04:00 INFO  [Periodicals] Starting [org.graylog2.periodical.IndexRangesMigrationPeriodical] periodical, running forever.
2022-03-24T13:41:51.690-04:00 INFO  [Periodicals] Starting [org.graylog2.periodical.IndexRangesCleanupPeriodical] periodical in [15s], polling every [3600s].
2022-03-24T13:41:51.693-04:00 INFO  [LookupTableService] Cache watchlist-cache/623b40e120fad36bfa761d1b [@467aa05d] STARTING
2022-03-24T13:41:51.694-04:00 INFO  [LookupTableService] Cache watchlist-cache/623b40e120fad36bfa761d1b [@467aa05d] RUNNING
2022-03-24T13:41:51.702-04:00 INFO  [connection] Opened connection [connectionId{localValue:4, serverValue:209}] to localhost:27017
2022-03-24T13:41:51.717-04:00 INFO  [PeriodicalsService] Not starting [org.graylog2.periodical.UserPermissionMigrationPeriodical] periodical. Not configured to run on this node.
2022-03-24T13:41:51.718-04:00 INFO  [Periodicals] Starting [org.graylog2.periodical.ConfigurationManagementPeriodical] periodical, running forever.
2022-03-24T13:41:51.718-04:00 INFO  [Periodicals] Starting [org.graylog2.periodical.TrafficCounterCalculator] periodical in [0s], polling every [1s].
2022-03-24T13:41:51.722-04:00 INFO  [LookupTableService] Starting lookup table watchlist/623b40e120fad36bfa761d1d [@569cd649] using cache watchlist-cache/623b40e120fad36bfa761d1b [@467aa05d], data adapter watchlist-mongo/623b40e120fad36bfa761d19 [@84bcffe]
2022-03-24T13:41:51.724-04:00 INFO  [Periodicals] Starting [org.graylog2.indexer.fieldtypes.IndexFieldTypePollerPeriodical] periodical in [0s], polling every [3600s].
2022-03-24T13:41:51.727-04:00 INFO  [Periodicals] Starting [org.graylog.scheduler.periodicals.ScheduleTriggerCleanUp] periodical in [120s], polling every [86400s].
2022-03-24T13:41:51.727-04:00 INFO  [Periodicals] Starting [org.graylog2.periodical.ESVersionCheckPeriodical] periodical in [0s], polling every [30s].
2022-03-24T13:41:51.728-04:00 INFO  [Periodicals] Starting [org.graylog.plugins.sidecar.periodical.PurgeExpiredSidecarsThread] periodical in [0s], polling every [600s].
2022-03-24T13:41:51.728-04:00 INFO  [Periodicals] Starting [org.graylog.plugins.sidecar.periodical.PurgeExpiredConfigurationUploads] periodical in [0s], polling every [600s].
2022-03-24T13:41:51.728-04:00 INFO  [Periodicals] Starting [org.graylog.plugins.views.search.db.SearchesCleanUpJob] periodical in [3600s], polling every [28800s].
2022-03-24T13:41:51.733-04:00 INFO  [Periodicals] Starting [org.graylog.events.periodicals.EventNotificationStatusCleanUp] periodical in [120s], polling every [86400s].
2022-03-24T13:41:51.734-04:00 INFO  [Periodicals] Starting [org.graylog.plugins.collector.periodical.PurgeExpiredCollectorsThread] periodical in [0s], polling every [3600s].
2022-03-24T13:41:51.734-04:00 INFO  [Periodicals] Starting [org.graylog.plugins.forwarder.ForwarderStatePeriodical] periodical in [0s], polling every [15s].
2022-03-24T13:41:51.734-04:00 INFO  [Periodicals] Starting [org.graylog.plugins.license.LicenseManagerPeriodical] periodical in [0s], polling every [300s].
2022-03-24T13:41:51.735-04:00 INFO  [Periodicals] Starting [org.graylog.plugins.license.LicenseReportPeriodical] periodical in [300s], polling every [3600s].
2022-03-24T13:41:51.735-04:00 INFO  [Periodicals] Starting [org.graylog.plugins.license.StagedLicenseInstallerPeriodical] periodical, running forever.
2022-03-24T13:41:51.736-04:00 INFO  [connection] Opened connection [connectionId{localValue:5, serverValue:210}] to localhost:27017
2022-03-24T13:41:51.737-04:00 INFO  [connection] Opened connection [connectionId{localValue:6, serverValue:211}] to localhost:27017
2022-03-24T13:41:51.738-04:00 INFO  [Periodicals] Starting [org.graylog.plugins.auditlog.mongodb.MongoAuditLogPeriodical] periodical in [0s], polling every [3600s].
2022-03-24T13:41:51.738-04:00 INFO  [Periodicals] Starting [org.graylog.plugins.report.scheduler.ReportPeriodical] periodical in [120s], polling every [60s].
2022-03-24T13:41:51.738-04:00 INFO  [Periodicals] Starting [org.graylog.plugins.report.service.ChromeDriverCleanupPeriodical] periodical in [60s], polling every [180s].
2022-03-24T13:41:51.738-04:00 ERROR [MongoAuditLogPeriodical] Not running cleanup for auditlog entries in MongoDB because there is no valid license.
2022-03-24T13:41:51.739-04:00 INFO  [Periodicals] Starting [org.graylog.plugins.files.CleanupPeriodical] periodical in [0s], polling every [86400s].
2022-03-24T13:41:52.013-04:00 INFO  [JerseyService] Enabling CORS for HTTP endpoint
2022-03-24T13:41:55.213-04:00 INFO  [JerseyService] Started REST API at <10.141.0.64:9000>
2022-03-24T13:41:55.213-04:00 INFO  [ServiceManagerListener] Services are healthy
2022-03-24T13:41:55.216-04:00 INFO  [InputSetupService] Triggering launching persisted inputs, node transitioned from Uninitialized [LB:DEAD] to Running [LB:ALIVE]
2022-03-24T13:41:55.216-04:00 INFO  [ServerBootstrap] Services started, startup times in ms: {GracefulShutdownService [RUNNING]=1, BufferSynchronizerService [RUNNING]=1, FailureHandlingService [RUNNING]=1, UrlWhitelistService [RUNNING]=1, LocalKafkaMessageQueueWriter [RUNNING]=4, LocalKafkaMessageQueueReader [RUNNING]=4, JobSchedulerService [RUNNING]=8, OutputSetupService [RUNNING]=9, EtagService [RUNNING]=9, DevelopmentDirectoryObserverService [RUNNING]=10, InputSetupService [RUNNING]=14, PrometheusExporter [RUNNING]=15, UserSessionTerminationService [RUNNING]=16, StreamCacheService [RUNNING]=16, ConfigurationEtagService [RUNNING]=16, ProcessingConfigurationManager [RUNNING]=22, MongoDBProcessingStatusRecorderService [RUNNING]=34, LocalKafkaJournal [RUNNING]=42, LookupTableService [RUNNING]=119, PeriodicalsService [RUNNING]=160, JerseyService [RUNNING]=3627}
2022-03-24T13:41:55.227-04:00 INFO  [ServerBootstrap] Graylog server up and running.
2022-03-24T13:41:55.228-04:00 ERROR [AuditLogger] Unable to write audit log entry because there is no valid license.
2022-03-24T13:41:55.239-04:00 INFO  [InputStateListener] Input [Raw/Plaintext UDP/623b45dc20fad36bfa76229d] is now STARTING
2022-03-24T13:41:55.477-04:00 INFO  [InputStateListener] Input [Raw/Plaintext UDP/623b45dc20fad36bfa76229d] is now RUNNING
2022-03-24T13:43:12.371-04:00 WARN  [TeamSyncProvisionerAction] No valid enterprise license found. Teams synchronisation is disabled.
2022-03-24T13:43:12.442-04:00 ERROR [AuditLogger] Unable to write audit log entry because there is no valid license.
2022-03-24T13:43:20.334-04:00 WARN  [ProxiedResource] Unable to call http://api/system/metrics/multiple on node <8964c162-c032-4db5-880f-82e3258b9a05>: Connect timed out

2022-03-24T13:49:26.864-04:00 WARN  [ProxiedResource] Unable to call http://api/system/metrics/multiple on node <8964c162-c032-4db5-880f-82e3258b9a05>: Connect timed out
2022-03-24T13:49:31.954-04:00 WARN  [ProxiedResource] Unable to call http://api/system/inputstates on node <8964c162-c032-4db5-880f-82e3258b9a05>: Connect timed out
2022-03-24T13:49:32.808-04:00 WARN  [ProxiedResource] Unable to call http://api/system/metrics/multiple on node <8964c162-c032-4db5-880f-82e3258b9a05>: Connect timed out
2022-03-24T13:49:33.250-04:00 WARN  [ProxiedResource] Unable to call http://api/system/inputstates/623b45dc20fad36bfa76229d on node <8964c162-c032-4db5-880f-82e3258b9a05>: Connect timed out
2022-03-24T13:49:33.253-04:00 ERROR [AuditLogger] Unable to write audit log entry because there is no valid license.
2022-03-24T13:49:36.998-04:00 WARN  [ProxiedResource] Unable to call http://api/system/inputstates on node <8964c162-c032-4db5-880f-82e3258b9a05>: Connect timed out
2022-03-24T13:49:38.802-04:00 WARN  [ProxiedResource] Unable to call http://api/system/metrics/multiple on node <8964c162-c032-4db5-880f-82e3258b9a05>: Connect timed out
2022-03-24T13:49:42.034-04:00 WARN  [ProxiedResource] Unable to call http://api/system/inputstates on node <8964c162-c032-4db5-880f-82e3258b9a05>: Connect timed out
2022-03-24T13:49:44.798-04:00 WARN  [ProxiedResource] Unable to call http://api/system/metrics/multiple on node <8964c162-c032-4db5-880f-82e3258b9a05>: Connect timed out
2022-03-24T13:49:47.074-04:00 WARN  [ProxiedResource] Unable to call http://api/system/inputstates on node <8964c162-c032-4db5-880f-82e3258b9a05>: Connect timed out
2022-03-24T13:49:51.047-04:00 WARN  [ProxiedResource] Unable to call http://api/system/metrics/multiple on node <8964c162-c032-4db5-880f-82e3258b9a05>: Connect timed out
2022-03-24T13:49:52.114-04:00 WARN  [ProxiedResource] Unable to call http://api/system/inputstates on node <8964c162-c032-4db5-880f-82e3258b9a05>: Connect timed out
2022-03-24T13:49:56.817-04:00 WARN  [ProxiedResource] Unable to call http://api/system/metrics/multiple on node <8964c162-c032-4db5-880f-82e3258b9a05>: Connect timed out
2022-03-24T13:49:57.154-04:00 WARN  [ProxiedResource] Unable to call http://api/system/inputstates on node <8964c162-c032-4db5-880f-82e3258b9a05>: Connect timed out
2022-03-24T13:50:01.922-04:00 WARN  [ProxiedResource] Unable to call http://api/system/inputstates on node <8964c162-c032-4db5-880f-82e3258b9a05>: Connect timed out
2022-03-24T13:50:02.202-04:00 WARN  [ProxiedResource] Unable to call http://api/system/inputstates on node <8964c162-c032-4db5-880f-82e3258b9a05>: Connect timed out
2022-03-24T13:50:02.819-04:00 WARN  [ProxiedResource] Unable to call http://api/system/metrics/multiple on node <8964c162-c032-4db5-880f-82e3258b9a05>: Connect timed out
2022-03-24T13:50:05.926-04:00 WARN  [ProxiedResource] Unable to call http://api/system/inputstates on node <8964c162-c032-4db5-880f-82e3258b9a05>: Connect timed out
2022-03-24T13:50:07.246-04:00 WARN  [ProxiedResource] Unable to call http://api/system/inputstates on node <8964c162-c032-4db5-880f-82e3258b9a05>: Connect timed out
2022-03-24T13:50:08.814-04:00 WARN  [ProxiedResource] Unable to call http://api/system/metrics/multiple on node <8964c162-c032-4db5-880f-82e3258b9a05>: Connect timed out
2022-03-24T13:50:10.058-04:00 WARN  [ProxiedResource] Unable to call http://api/system/inputstates on node <8964c162-c032-4db5-880f-82e3258b9a05>: Connect timed out
2022-03-24T13:50:12.058-04:00 WARN  [ProxiedResource] Unable to call http://api/system/inputstates on node <8964c162-c032-4db5-880f-82e3258b9a05>: Connect timed out
2022-03-24T13:50:12.290-04:00 WARN  [ProxiedResource] Unable to call http://api/system/inputstates on node <8964c162-c032-4db5-880f-82e3258b9a05>: Connect timed out
2022-03-24T13:50:14.798-04:00 WARN  [ProxiedResource] Unable to call http://api/system/metrics/multiple on node <8964c162-c032-4db5-880f-82e3258b9a05>: Connect timed out
2022-03-24T13:50:15.927-04:00 WARN  [ProxiedResource] Unable to call http://api/system/inputstates on node <8964c162-c032-4db5-880f-82e3258b9a05>: Connect timed out
2022-03-24T13:50:17.330-04:00 WARN  [ProxiedResource] Unable to call http://api/system/inputstates on node <8964c162-c032-4db5-880f-82e3258b9a05>: Connect timed out
2022-03-24T13:50:19.934-04:00 WARN  [ProxiedResource] Unable to call http://api/system/inputstates on node <8964c162-c032-4db5-880f-82e3258b9a05>: Connect timed out
2022-03-24T13:50:21.062-04:00 WARN  [ProxiedResource] Unable to call http://api/system/metrics/multiple on node <8964c162-c032-4db5-880f-82e3258b9a05>: Connect timed out
2022-03-24T13:50:21.938-04:00 WARN  [ProxiedResource] Unable to call http://api/system/inputstates on node <8964c162-c032-4db5-880f-82e3258b9a05>: Connect timed out
2022-03-24T13:50:22.366-04:00 WARN  [ProxiedResource] Unable to call http://api/system/inputstates on node <8964c162-c032-4db5-880f-82e3258b9a05>: Connect timed out
2022-03-24T13:50:25.930-04:00 WARN  [ProxiedResource] Unable to call http://api/system/inputstates on node <8964c162-c032-4db5-880f-82e3258b9a05>: Connect timed out
2022-03-24T13:50:27.054-04:00 WARN  [ProxiedResource] Unable to call http://api/system/metrics/multiple on node <8964c162-c032-4db5-880f-82e3258b9a05>: Connect timed out

[gsmith]

Hello,

I have a suggestion you can try.

First you can comment these two line out

#http_publish_uri = http://$http_bind_address/
#http_external_uri = http://10.141.0.64:9000/

Second add/uncomment this line.

elasticsearch_hosts = http://127.0.0.1:9200

Make sure you elasticsearch yaml file is correct.

cluster.name: graylog
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: 127.0.0.1
http.port: 9200
action.auto_create_index: false
discovery.type: single-node

If you have to make adjustments to either file, the service will have to be restarted. If you do have to make adjustment to both files I personal would stop Graylog service and restart elasticsearch. Once elasticsearch is running without issues Then start Graylog service and tail -f Graylogs log file to insure there are no problems. If there are issues, posting them here would help, Don’t forget to remove personal info.

EDIT: I forgot to mention check Firewall/Selinux/Apparmor. Insure the permissions in Graylog’s directory are correct. This maybe a no brainer but I have seen Graylog unable to access its own files.

Check Elasticsearch Health.

curl -XGET http://127.0.0.1:9200/_cluster/health?pretty=true

Check status on these two services, sometimes there could be a error /warning that could help troubleshoot.

root# systemctl status mongod
root# systemctl status graylog-server

I just noticed you tried using HTTPS, On that note make sure your URL is correct and you trying to connect http://10.141.0.64:9000 also you /etc/hosts file is correct.

[jkavenaugh (James Kavenaugh)]

Oh… thank you so much! I commented those lines out and restarted. It’s now working fine.

[gsmith]

Glad it worked out for you, If you could mark this as resolved for future issues that would be great