WARN ProxiedRessource

Hello,

I have some problems configuring graylog and elasticsearch,

Graylog System information is unavailable and jvm 500 couldn’t load histogram data when I check api

My graylog log show this :

WARN [ProxiedResource] Unable to call http://GraylogIP:9000/api/system on node <e???-GraylogNode-???-???f>
java.net.SocketTimeoutException: timeout
at okio.Okio$4.newTimeoutException(Okio.java:230) ~[graylog.jar:?]
at okio.AsyncTimeout.exit(AsyncTimeout.java:285) ~[graylog.jar:?]
at okio.AsyncTimeout$2.read(AsyncTimeout.java:241) ~[graylog.jar:?]
at okio.RealBufferedSource.indexOf(RealBufferedSource.java:345) ~[graylog.jar:?]
at okio.RealBufferedSource.readUtf8LineStrict(RealBufferedSource.java:217) ~[graylog.jar:?]
at okio.RealBufferedSource.readUtf8LineStrict(RealBufferedSource.java:211) ~[graylog.jar:?]
at okhttp3.internal.http1.Http1Codec.readResponseHeaders(Http1Codec.java:189) ~[graylog.jar:?]
at okhttp3.internal.http.CallServerInterceptor.intercept(CallServerInterceptor.java:75) ~[graylog.jar:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[graylog.jar:?]
at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:45) ~[graylog.jar:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[graylog.jar:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67) ~[graylog.jar:?]
at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:93) ~[graylog.jar:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[graylog.jar:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67) ~[graylog.jar:?]
at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93) ~[graylog.jar:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[graylog.jar:?]
at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:120) ~[graylog.jar:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[graylog.jar:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67) ~[graylog.jar:?]
at org.graylog2.rest.RemoteInterfaceProvider.lambda$get$0(RemoteInterfaceProvider.java:59) ~[graylog.jar:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[graylog.jar:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67) ~[graylog.jar:?]
at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:185) ~[graylog.jar:?]
at okhttp3.RealCall.execute(RealCall.java:69) ~[graylog.jar:?]
at retrofit2.OkHttpCall.execute(OkHttpCall.java:180) ~[graylog.jar:?]
at org.graylog2.shared.rest.resources.ProxiedResource.lambda$getForAllNodes$0(ProxiedResource.java:76) ~[graylog.jar:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_151]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_151]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_151]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_151]
Caused by: java.net.SocketException: Socket closed
at java.net.SocketInputStream.read(SocketInputStream.java:204) ~[?:1.8.0_151]
at java.net.SocketInputStream.read(SocketInputStream.java:141) ~[?:1.8.0_151]
at okio.Okio$2.read(Okio.java:139) ~[graylog.jar:?]
at okio.AsyncTimeout$2.read(AsyncTimeout.java:237) ~[graylog.jar:?]
… 28 more

Here is my graylog server conf

is_master = true
node_id_file = /etc/graylog/server/node-id
password_secret = mypass
root_username = myname
root_password_sha2 = mysha2pass
root_email = mymail
root_timezone = Europe/Paris
plugin_dir = /usr/share/graylog-server/plugin
rest_listen_uri = http://0.0.0.0:9000/api
rest_enable_cors = true
web_listen_uri = http://0.0.0.0:9000/
elasticsearch_hosts = http://IPelaticsearch:9200, http://IPels:9200, http://IPels:9200
rotation_strategy = count
elasticsearch_max_docs_per_index = 20000000
elasticsearch_max_number_of_indices = 20
retention_strategy = delete
elasticsearch_shards = 4
elasticsearch_replicas = 1
elasticsearch_index_prefix = graylog
allow_leading_wildcard_searches = true
allow_highlighting = false
elasticsearch_analyzer = standard
output_batch_size = 500
output_flush_interval = 1
output_fault_count_threshold = 5
output_fault_penalty_seconds = 30
processbuffer_processors = 8
outputbuffer_processors = 5
processor_wait_strategy = blocking
ring_size = 65536
inputbuffer_ring_size = 65536
inputbuffer_processors = 2
inputbuffer_wait_strategy = blocking
message_journal_enabled = true
message_journal_dir = /var/lib/graylog-server/journal
lb_recognition_period_seconds = 3
mongodb_uri = mongodb://localhost/graylog
mongodb_max_connections = 1000
mongodb_threads_allowed_to_block_multiplier = 5
content_packs_dir = /usr/share/graylog-server/contentpacks
content_packs_auto_load = grok-patterns.json
proxied_requests_thread_pool_size = 32

the other params are commented by “#”

Graylog 2.3.2 is running on CentOS7 with mongodb on the same virtual machine (vsphere)
There is 2 Master node Elasticsearch 5.5.2 and 1 slave.

Tell me if you need more informations,

I’ll be grateful if anyone can help

Regards

Is http://GraylogIP:9000/api/system reachable for the Graylog node itself?
What’s the output of the following command on the machine running Graylog?

# curl -i http://GraylogIP:9000/api/system

On the browser; http://GraylogIP:9000/api/system is reachable but I have to login with my graylog web ID and Password each time I try to connect

On graylog server the command return :
image

Thanks a lot

Is there only this one error message in the logs of your Graylog node or is this reproducible?

I don’t know if it’s reproductible but yes it’s the only error message I have, it happens just after the gelf input started in the log

In that case it’s possible that only this single request timed out and it’s nothing to worry about.

Keep an eye on the logs of your Graylog nodes and open a new topic if you have more questions.

After serverals reboot of all my servers, log are still the same,

The point is, I can’t see the Graylog Info
image

And when I click on the blue text, it shows this

Any ideas?

you might carefully read http://docs.graylog.org/en/2.3/pages/configuration/server.conf.html#general and special the comment for rest_transport_uri:

REST API transport address. Defaults to the value of rest_listen_uri. Exception: If rest_listen_uri is set to a wildcard IP address (0.0.0.0) the first non-loopback IPv4 system address is used.

Thanks jan

but, before I try several change on my configuration, no one of my server.conf IP addresses was on 0.0.0.0

I changed rest_transport_uri to http://GraylogIP:9000/api and the problem is still the same.

On the wait to read you,

Regards

the configuration you provided to us included wildcards

rest_listen_uri = http://0.0.0.0:9000/api
web_listen_uri = http://0.0.0.0:9000/

Sorry if you modify the provided information in a way that it does not represent the environment, nobody can help you.

My apologies, that’s a mistake I should saw,

I correct that, so my last conf is :

web_listen_uri = http://GraylogIP:9000/
rest_listen_uri = http://GraylogIP:9000/api
#rest_transport_uri = not set

The information are still unavailable

With API

Is it possible that a bug happened when I converted graylog from virtual box to vmware?

Graylog seems like it can’t connect to itself

@Manuu

you name the problem yourself:

Graylog seems like it can’t connect to itself

now you need to find the reason for that. Did you checked if SELinux or any kind of firewall is the reason?

SElinux is removed, and the firewall-cmd disabled

I’m trying to solve this issue since 2 weeks, I’m lost now, I don’t know what kind of test I can do or what params I can add…

I did not like to play ping pong with you about all possible reasons why Graylog is not able to connect to itself. You need to find that yourself as we all can only guess the reason, but you are able to check.

Yes of course I understand that,

But I was just asking if maybe there is an hidden authentication in the graylog protocol to connect to itself or something else which required attention

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.