Inputs show failed, but ports are open and logs are coming in

hi, i have a new install of Graylog 4.2.7 on RHEL 8.5 (Graylog 4.2.7+879e651 on servername (Red Hat, Inc. 1.8.0_322 on Linux 4.18.0-348.20.1.el8_5.x86_64))

I have created several inputs and all fail when i try to start them

…however…a netstat shows the ports open and graylog is accepting logs on those ports.

It does not seem to matter where the ports are, i have tried low and high ports and all my inputs are doing this… and i cannot find any logs with information in them.

can someone point me in the right direction, not sure where to look

thanks in advance

Hello && Welcome

I might be able to help.

When you try to start this input, what does Graylog logs show?

Default file locations

Insure Graylog status is good.

systemctl status graylog-server

Insure elasticsearch is good.

curl -XGET http://localhost:9200/_cluster/health?pretty=true

Insure MongoDb is good.

systemctl status mongod

Check permission on your Graylog directory

ls -al /etc/graylog

Showing graylog configuration file would be appreciated also.

cat /etc/graylog/server/server.conf | egrep -v "^\s*(#|$)"

Hope that helps

h, thanks for taking the time! so i noticed that last night graylog upgraded to 4.28. i also noticed that i had no log in /var/log/graylog so i decided to bounce the server. when it came back up, i had a log and there are some errors in it.
i also have the info you requested, the services seem good (all 3), the log shows the listeners starting without error, but there is a different error in the logs. you mentioned permissions on /etc/graylog…they are set to root…is that wrong? should that be the graylog user?
here is all the info

ls -la /etc/graylog
total 12
drwxr-xr-x.   3 root root   20 Apr  8 10:01 .
drwxr-xr-x. 154 root root 8192 Apr 13 07:17 ..
drwxr-xr-x.   2 root root   84 Apr 13 06:31 server

2022-04-13T07:17:45.867-04:00 INFO  [ServerBootstrap] Graylog server up and running.
2022-04-13T07:17:45.871-04:00 INFO  [InputLauncher] Launching input [Syslog UDP/Switch-Input/6250508ee1cc6671f2599791] - desired state is RUNNING
2022-04-13T07:17:45.872-04:00 INFO  [InputLauncher] Launching input [Beats/Windows Events/62546a7fe1cc6671f25e0c1c] - desired state is RUNNING
2022-04-13T07:17:45.875-04:00 INFO  [InputLauncher] Launching input [Syslog UDP/Routers/62557c27e1cc6671f25f3664] - desired state is RUNNING
2022-04-13T07:17:45.877-04:00 INFO  [InputStateListener] Input [Raw/Plaintext UDP/62504a68e1cc6671f25990b7] is now STARTING
2022-04-13T07:17:45.880-04:00 INFO  [InputStateListener] Input [Syslog UDP/6250508ee1cc6671f2599791] is now STARTING
2022-04-13T07:17:45.881-04:00 INFO  [InputStateListener] Input [Beats/62546a7fe1cc6671f25e0c1c] is now STARTING
2022-04-13T07:17:45.883-04:00 INFO  [InputStateListener] Input [Syslog UDP/62557c27e1cc6671f25f3664] is now STARTING
2022-04-13T07:17:46.102-04:00 INFO  [InputStateListener] Input [Beats/62546a7fe1cc6671f25e0c1c] is now RUNNING
2022-04-13T07:17:46.160-04:00 WARN  [AbstractTcpTransport] receiveBufferSize (SO_RCVBUF) for input Beats2Input{title=Windows Events,, nodeId=null} (channel [id: 0x8f42a896, L:/0:0:0:0:0:0:0:0%0:5044]) should be >= 1048576 but is 425984.
2022-04-13T07:17:46.177-04:00 INFO  [InputStateListener] Input [Syslog UDP/6250508ee1cc6671f2599791] is now RUNNING
2022-04-13T07:17:46.180-04:00 INFO  [InputStateListener] Input [Syslog UDP/62557c27e1cc6671f25f3664] is now RUNNING
2022-04-13T07:17:46.183-04:00 INFO  [InputStateListener] Input [Raw/Plaintext UDP/62504a68e1cc6671f25990b7] is now RUNNING
2022-04-13T07:17:52.565-04:00 ERROR [IndexRotationThread] Couldn't point deflector to a new index
java.lang.RuntimeException: Unable to extract count from response.
        at ~[?:?]
        at org.graylog2.indexer.indices.Indices.numberOfMessages( ~[graylog.jar:?]
        at org.graylog2.indexer.rotation.strategies.MessageCountRotationStrategy.shouldRotate( ~[graylog.jar:?]
        at org.graylog2.indexer.rotation.strategies.MessageCountRotationStrategy.shouldRotate( ~[graylog.jar:?]
        at org.graylog2.indexer.rotation.strategies.AbstractRotationStrategy.rotate( ~[graylog.jar:?]

curl -XGET http://localhost:9200/_cluster/health?pretty=true
  "cluster_name" : "graylog",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 1,
  "number_of_data_nodes" : 1,
  "active_primary_shards" : 80,
  "active_shards" : 80,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 100.0

cat /etc/graylog/server/server.conf | egrep -v "^\s*(#|$)"
is_master = true
node_id_file = /etc/graylog/server/node-id
password_secret = <removed>
root_password_sha2 = <removed>
root_email = "itsupport@company.local"
root_timezone = America/New_York
bin_dir = /usr/share/graylog-server/bin
data_dir = /opt/graylog-server
plugin_dir = /usr/share/graylog-server/plugin
http_bind_address =
http_publish_uri =
http_enable_tls = true
http_tls_cert_file = /etc/pki/tls/certs/
http_tls_key_file = /etc/pki/tls/private/
rotation_strategy = count
elasticsearch_max_docs_per_index = 20000000
elasticsearch_max_number_of_indices = 20
retention_strategy = delete
elasticsearch_shards = 4
elasticsearch_replicas = 0
elasticsearch_index_prefix = graylog
allow_leading_wildcard_searches = false
allow_highlighting = false
elasticsearch_analyzer = standard
output_batch_size = 500
output_flush_interval = 1
output_fault_count_threshold = 5
output_fault_penalty_seconds = 30
processbuffer_processors = 5
outputbuffer_processors = 3
processor_wait_strategy = blocking
ring_size = 65536
inputbuffer_ring_size = 65536
inputbuffer_processors = 2
inputbuffer_wait_strategy = blocking
message_journal_enabled = true
message_journal_dir = /var/lib/graylog-server/journal
lb_recognition_period_seconds = 3
mongodb_uri = mongodb://localhost/graylog
mongodb_max_connections = 1000
mongodb_threads_allowed_to_block_multiplier = 5
proxied_requests_thread_pool_size = 32

on a guess, i chown’d the /etc/graylog subdir to graylog:graylog and rebooted…that did not help

ok, i think i am close to a solution…i have configured graylog for https and that seems to work…except, when it calls itself, it seems to not accept my self signed cert

2022-04-13T10:03:18.513-04:00 WARN  [ProxiedResource] Unable to call https://lnb-graylog.lyonsbank.local:9000/api/system/inputstates on node <f86dbf9f-2179-4e44-91f4-2e10f416b008>: Hostname not verified:
    certificate: sha256/CEL8Mn5mpFvDY/NBLbco0Wen5NULerAjjSRdvr3okPg=
    DN:, OU=IT, O=org, L=city, ST=New York, C=US
    subjectAltNames: []

i added my cert to /etc/pki/ca-trust/source/anchors and ran update-ca-trust extract but it seems not to like the cert…any clue on how to make graylog accept this cert?..i think this is the issue


Thanks for the added info. I found a couple configuration I’m unsure of.

Here Is my lab GL, because of testing I don’t use Localhost/ for ES connections. Take notice on http_publish_uri section.

http_bind_address =
http_publish_uri =
http_enable_cors = true
http_enable_tls = true
http_tls_cert_file = /etc/pki/tls/certs/graylog/graylog-certificate.pem
http_tls_key_file = /etc/pki/tls/certs/graylog/graylog-key.pem
http_tls_key_password = secret
elasticsearch_hosts =

I would need to see the steps taken or documentation used to create your certificates to help you further on that…

You can do a test to be 100% sure that its your certificate cause all these issue.
Set you Graylog configuration back to a HTTP connection.

Comment the following lines out.

http_bind_address =
# http_publish_uri =
# http_enable_tls = true
# http_tls_cert_file = /etc/pki/tls/certs/
# http_tls_key_file = /etc/pki/tls/private/

Resart Graylog service

The use the following to logon.

If you able to do that and everything works, we can look into your certificates and configuration for HTTPS.

Hope that helps

1 Like


ok, fixed!!! so, thanks for all your help on this… I had made these mistakes:

  1. i had
http_publish_uri =

but should have bee this

http_publish_uri =
  1. i skipped over the part in the docs ( Using HTTPS - Configuring Graylog) where you need to create a san cert and include BOTH the FQDN AND the IP of the server.

fixing these two things got me working

thanks again for your help!

1 Like

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.