Inputs show failed, but ports are open and logs are coming in

Graylog Central (peer support)
1

hi, i have a new install of Graylog 4.2.7 on RHEL 8.5 (Graylog 4.2.7+879e651 on servername (Red Hat, Inc. 1.8.0_322 on Linux 4.18.0-348.20.1.el8_5.x86_64))

I have created several inputs and all fail when i try to start them

image
image707×67 8.68 KB

…however…a netstat shows the ports open and graylog is accepting logs on those ports.

It does not seem to matter where the ports are, i have tried low and high ports and all my inputs are doing this… and i cannot find any logs with information in them.

can someone point me in the right direction, not sure where to look

thanks in advance

2

Hello && Welcome

I might be able to help.

When you try to start this input, what does Graylog logs show?

Default file locations

Insure Graylog status is good.

systemctl status graylog-server

Insure elasticsearch is good.

curl -XGET http://localhost:9200/_cluster/health?pretty=true

Insure MongoDb is good.

systemctl status mongod

Check permission on your Graylog directory

ls -al /etc/graylog

Showing graylog configuration file would be appreciated also.

cat /etc/graylog/server/server.conf | egrep -v "^\s*(#|$)"

Hope that helps

3

h, thanks for taking the time! so i noticed that last night graylog upgraded to 4.28. i also noticed that i had no log in /var/log/graylog so i decided to bounce the server. when it came back up, i had a log and there are some errors in it.
i also have the info you requested, the services seem good (all 3), the log shows the listeners starting without error, but there is a different error in the logs. you mentioned permissions on /etc/graylog…they are set to root…is that wrong? should that be the graylog user?
here is all the info

ls -la /etc/graylog
total 12
drwxr-xr-x.   3 root root   20 Apr  8 10:01 .
drwxr-xr-x. 154 root root 8192 Apr 13 07:17 ..
drwxr-xr-x.   2 root root   84 Apr 13 06:31 server


2022-04-13T07:17:45.867-04:00 INFO  [ServerBootstrap] Graylog server up and running.
2022-04-13T07:17:45.871-04:00 INFO  [InputLauncher] Launching input [Syslog UDP/Switch-Input/6250508ee1cc6671f2599791] - desired state is RUNNING
2022-04-13T07:17:45.872-04:00 INFO  [InputLauncher] Launching input [Beats/Windows Events/62546a7fe1cc6671f25e0c1c] - desired state is RUNNING
2022-04-13T07:17:45.875-04:00 INFO  [InputLauncher] Launching input [Syslog UDP/Routers/62557c27e1cc6671f25f3664] - desired state is RUNNING
2022-04-13T07:17:45.877-04:00 INFO  [InputStateListener] Input [Raw/Plaintext UDP/62504a68e1cc6671f25990b7] is now STARTING
2022-04-13T07:17:45.880-04:00 INFO  [InputStateListener] Input [Syslog UDP/6250508ee1cc6671f2599791] is now STARTING
2022-04-13T07:17:45.881-04:00 INFO  [InputStateListener] Input [Beats/62546a7fe1cc6671f25e0c1c] is now STARTING
2022-04-13T07:17:45.883-04:00 INFO  [InputStateListener] Input [Syslog UDP/62557c27e1cc6671f25f3664] is now STARTING
2022-04-13T07:17:46.102-04:00 INFO  [InputStateListener] Input [Beats/62546a7fe1cc6671f25e0c1c] is now RUNNING
2022-04-13T07:17:46.160-04:00 WARN  [AbstractTcpTransport] receiveBufferSize (SO_RCVBUF) for input Beats2Input{title=Windows Events, type=org.graylog.plugins.beats.Beats2Input, nodeId=null} (channel [id: 0x8f42a896, L:/0:0:0:0:0:0:0:0%0:5044]) should be >= 1048576 but is 425984.
2022-04-13T07:17:46.177-04:00 INFO  [InputStateListener] Input [Syslog UDP/6250508ee1cc6671f2599791] is now RUNNING
2022-04-13T07:17:46.180-04:00 INFO  [InputStateListener] Input [Syslog UDP/62557c27e1cc6671f25f3664] is now RUNNING
2022-04-13T07:17:46.183-04:00 INFO  [InputStateListener] Input [Raw/Plaintext UDP/62504a68e1cc6671f25990b7] is now RUNNING
2022-04-13T07:17:52.565-04:00 ERROR [IndexRotationThread] Couldn't point deflector to a new index
java.lang.RuntimeException: Unable to extract count from response.
        at org.graylog.storage.elasticsearch7.IndicesAdapterES7.numberOfMessages(IndicesAdapterES7.java:265) ~[?:?]
        at org.graylog2.indexer.indices.Indices.numberOfMessages(Indices.java:113) ~[graylog.jar:?]
        at org.graylog2.indexer.rotation.strategies.MessageCountRotationStrategy.shouldRotate(MessageCountRotationStrategy.java:68) ~[graylog.jar:?]
        at org.graylog2.indexer.rotation.strategies.MessageCountRotationStrategy.shouldRotate(MessageCountRotationStrategy.java:34) ~[graylog.jar:?]
        at org.graylog2.indexer.rotation.strategies.AbstractRotationStrategy.rotate(AbstractRotationStrategy.java:71) ~[graylog.jar:?]


curl -XGET http://localhost:9200/_cluster/health?pretty=true
{
  "cluster_name" : "graylog",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 1,
  "number_of_data_nodes" : 1,
  "active_primary_shards" : 80,
  "active_shards" : 80,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 100.0


cat /etc/graylog/server/server.conf | egrep -v "^\s*(#|$)"
is_master = true
node_id_file = /etc/graylog/server/node-id
password_secret = <removed>
root_password_sha2 = <removed>
root_email = "itsupport@company.local"
root_timezone = America/New_York
bin_dir = /usr/share/graylog-server/bin
data_dir = /opt/graylog-server
plugin_dir = /usr/share/graylog-server/plugin
http_bind_address = 10.40.1.250:9000
http_publish_uri = http://lnb-graylog.company.local:9000/
http_enable_tls = true
http_tls_cert_file = /etc/pki/tls/certs/lnb-graylog.company.local.crt
http_tls_key_file = /etc/pki/tls/private/lnb-graylog.company.local-npwkey.pem
rotation_strategy = count
elasticsearch_max_docs_per_index = 20000000
elasticsearch_max_number_of_indices = 20
retention_strategy = delete
elasticsearch_shards = 4
elasticsearch_replicas = 0
elasticsearch_index_prefix = graylog
allow_leading_wildcard_searches = false
allow_highlighting = false
elasticsearch_analyzer = standard
output_batch_size = 500
output_flush_interval = 1
output_fault_count_threshold = 5
output_fault_penalty_seconds = 30
processbuffer_processors = 5
outputbuffer_processors = 3
processor_wait_strategy = blocking
ring_size = 65536
inputbuffer_ring_size = 65536
inputbuffer_processors = 2
inputbuffer_wait_strategy = blocking
message_journal_enabled = true
message_journal_dir = /var/lib/graylog-server/journal
lb_recognition_period_seconds = 3
mongodb_uri = mongodb://localhost/graylog
mongodb_max_connections = 1000
mongodb_threads_allowed_to_block_multiplier = 5
proxied_requests_thread_pool_size = 32
4

on a guess, i chown’d the /etc/graylog subdir to graylog:graylog and rebooted…that did not help

5

ok, i think i am close to a solution…i have configured graylog for https and that seems to work…except, when it calls itself, it seems to not accept my self signed cert

2022-04-13T10:03:18.513-04:00 WARN  [ProxiedResource] Unable to call https://lnb-graylog.lyonsbank.local:9000/api/system/inputstates on node <f86dbf9f-2179-4e44-91f4-2e10f416b008>: Hostname lnb-graylog.company.local not verified:
    certificate: sha256/CEL8Mn5mpFvDY/NBLbco0Wen5NULerAjjSRdvr3okPg=
    DN: CN=lnb-graylog.company.local, OU=IT, O=org, L=city, ST=New York, C=US
    subjectAltNames: []

i added my cert to /etc/pki/ca-trust/source/anchors and ran update-ca-trust extract but it seems not to like the cert…any clue on how to make graylog accept this cert?..i think this is the issue

6

Hello,

Thanks for the added info. I found a couple configuration I’m unsure of.

Here Is my lab GL, because of testing I don’t use Localhost/127.0.0.1 for ES connections. Take notice on http_publish_uri section.

http_bind_address = graylog.domain.com:9000
http_publish_uri = https://graylog.domain.com:9000/
http_enable_cors = true
http_enable_tls = true
http_tls_cert_file = /etc/pki/tls/certs/graylog/graylog-certificate.pem
http_tls_key_file = /etc/pki/tls/certs/graylog/graylog-key.pem
http_tls_key_password = secret
elasticsearch_hosts = http://10.10.10.10:9200

I would need to see the steps taken or documentation used to create your certificates to help you further on that…

You can do a test to be 100% sure that its your certificate cause all these issue.
Set you Graylog configuration back to a HTTP connection.

Example:
Comment the following lines out.

http_bind_address = 10.40.1.250:9000
# http_publish_uri = http://lnb-graylog.company.local:9000/
# http_enable_tls = true
# http_tls_cert_file = /etc/pki/tls/certs/lnb-graylog.company.local.crt
# http_tls_key_file = /etc/pki/tls/private/lnb-graylog.company.local-npwkey.pem

Resart Graylog service

The use the following to logon.

http://10.40.1.250:9000

If you able to do that and everything works, we can look into your certificates and configuration for HTTPS.

Hope that helps

7

image

ok, fixed!!! so, thanks for all your help on this… I had made these mistakes:

  1. i had
http_publish_uri = http://lnb-graylog.company.local:9000/

but should have bee this

http_publish_uri = https://lnb-graylog.company.local:9000/
  1. i skipped over the part in the docs ( Using HTTPS - Configuring Graylog) where you need to create a san cert and include BOTH the FQDN AND the IP of the server.

fixing these two things got me working

thanks again for your help!