Graylog - Failed to call API on node <node>, cause: timeout (duration: 5006ms)

Graylog Central (peer support)
1

Before you post: Your responses to these questions will help the community help you. Please complete this template if you’re asking a support question.
Don’t forget to select tags to help index your topic!

1. Describe your incident:
I have Graylog setup completely but I am having a constant issue with the API timing out. Everything is local and I have tried setting everything to direct ip or even 127.0.0.1 but constantly getting this timeout. I did as well set the location in my /etc/hosts file and still experiencing the same thing. When I log into graylog and look at the node, it will be there then disappear because the api connection timesout then comes back. At a loss on what to do at this point.

2. Describe your environment:

  • OS Information: RHEL 8.9

  • Package Version: 5.2

  • Service logs, configurations, and environment variables:
    Cannot provide logs / configs as this is an airgapped network / environment.

3. What steps have you already taken to try and solve the problem?
I have tried setting to a domain, tried setting to direct ip (example: 192.168.12.69) even tried 127.0.0.1

4. How can the community help?
Anything to look for specifically that could help.

Helpful Posting Tips: Tips for Posting Questions that Get Answers [Hold down CTRL and link on link to open tips documents in a separate tab]

2

Hey @jbradley

By chance did you see this post?

3

I did see that post and my settings do match. I’ll see what I can do to get some logs over.

First off, everything else in graylog seems to be working, my sidecar is reporting all though no logs have been sent over to graylog as of yet.

Here is more detailed, I guess error

ERROR (AnyExceptionClassMapper) Unhandled exception in REST resource
java.io.InterruptedIOException: timeout
Caused by: java.io.IOExecption: Cancelled

WARN [ProxiedResource] Failed to call API on node , cause: timeout (duration: 5004 ms)

I have set my java to /bin/java in the /etc/sysconfig/graylog-server based on doing which java that way it would solve another issue…im not sure what to do to resolve this.

SELinux is disabled,
Firewall is disabled
everything set to 127.0.0.1

4

Hey @jbradley

Can you show your configurations?

5

@gsmith

Here is what I have for configurations

Here is my OpenSearch config that is based exactly on the documentation.

cluster.name: graylog
node.name: ${HOSTNAME}
path.data: /var/lib/opensearch
path.logs: /var/log/opensearch
discovery.type: single-node
network.host: 0.0.0.0
action.auto_create_index: false
plugins.security.disabled: true
indices.query.bool.max_clause_count: 32768

Here is my graylog config

is_leader = true
node_id_file = /etc/graylog/server/node-id
password_secret = xxx
root_password_sha2 = yyy 
bin_dir = /usr/share/graylog-server/bin
data_dir = /var/lib/graylog-server
plugin_dir = /usr/share/graylog-server/plugin
http_bind_address = 0.0.0.0:9000
rotation_strategy = count
elasticsearch_max_docs_per_index = 20000000
elasticsearch_max_number_of_indices = 20
retention_strategy = delete
elasticsearch_shards = 4
elasticsearch_replicas = 0
elasticsearch_index_prefix = graylog
allow_leading_wildcard_searches = false
allow_highlighting = false
elasticsearch_analyzer = standard
output_batch_size = 500
output_flush_interval = 1
output_fault_count_threshold = 5
output_fault_penalty_seconds = 30
processbuffer_processors = 5
outputbuffer_processors = 3
processor_wait_strategy = blocking
ring_size = 65536
inputbuffer_ring_size = 65536
inputbuffer_processors = 2
inputbuffer_wait_strategy = blocking
message_journal_enabled = true
message_journal_dir = /var/lib/graylog-server/journal
lb_recognition_period_seconds = 3
mongodb_uri = mongodb://127.0.0.1/graylog
mongodb_max_connections = 1000
mongodb_threads_allowed_to_block_multiplier = 5
proxied_requests_thread_pool_size = 32

Pretty standard based on what I have seen. http_bind address I have tried 127.0.0.1, 0.0.0.0, 192.168.12.69 everything and still get this API timeout where it times out, comes back, times out, comes back.

Im pulling my hair out on this.

6

As an update, I do have the rsyslog stuff from the server itself being ingested into gitlab which is great but my Windows Sidecar is not working, I am not sure if its cause the API keeps essentially crashing or what. I just do not understand how the API timesout to itself

7

Hey,

Check Graylog Sidecar log file on that Windows device. See if you can find anything that would pertain to this issue.

Check your resources ( CPU, Memory, etc…)

8

You configuration looks good.

9

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.