Inputs show failed, but ports are open and logs are coming in

tonyg · April 12, 2022, 9:09pm

hi, i have a new install of Graylog 4.2.7 on RHEL 8.5 (Graylog 4.2.7+879e651 on servername (Red Hat, Inc. 1.8.0_322 on Linux 4.18.0-348.20.1.el8_5.x86_64))

I have created several inputs and all fail when i try to start them

…however…a netstat shows the ports open and graylog is accepting logs on those ports.

It does not seem to matter where the ports are, i have tried low and high ports and all my inputs are doing this… and i cannot find any logs with information in them.

can someone point me in the right direction, not sure where to look

thanks in advance

gsmith · April 12, 2022, 11:18pm

Hello && Welcome

I might be able to help.

When you try to start this input, what does Graylog logs show?

Default file locations

Default file locations - Configuring Graylog

Insure Graylog status is good.

systemctl status graylog-server

Insure elasticsearch is good.

curl -XGET http://localhost:9200/_cluster/health?pretty=true

Insure MongoDb is good.

systemctl status mongod

Check permission on your Graylog directory

ls -al /etc/graylog

Showing graylog configuration file would be appreciated also.

cat /etc/graylog/server/server.conf | egrep -v "^\s*(#|$)"

Hope that helps

tonyg · April 13, 2022, 11:36am

h, thanks for taking the time! so i noticed that last night graylog upgraded to 4.28. i also noticed that i had no log in /var/log/graylog so i decided to bounce the server. when it came back up, i had a log and there are some errors in it.
i also have the info you requested, the services seem good (all 3), the log shows the listeners starting without error, but there is a different error in the logs. you mentioned permissions on /etc/graylog…they are set to root…is that wrong? should that be the graylog user?
here is all the info

ls -la /etc/graylog
total 12
drwxr-xr-x.   3 root root   20 Apr  8 10:01 .
drwxr-xr-x. 154 root root 8192 Apr 13 07:17 ..
drwxr-xr-x.   2 root root   84 Apr 13 06:31 server

2022-04-13T07:17:45.867-04:00 INFO  [ServerBootstrap] Graylog server up and running.
2022-04-13T07:17:45.871-04:00 INFO  [InputLauncher] Launching input [Syslog UDP/Switch-Input/6250508ee1cc6671f2599791] - desired state is RUNNING
2022-04-13T07:17:45.872-04:00 INFO  [InputLauncher] Launching input [Beats/Windows Events/62546a7fe1cc6671f25e0c1c] - desired state is RUNNING
2022-04-13T07:17:45.875-04:00 INFO  [InputLauncher] Launching input [Syslog UDP/Routers/62557c27e1cc6671f25f3664] - desired state is RUNNING
2022-04-13T07:17:45.877-04:00 INFO  [InputStateListener] Input [Raw/Plaintext UDP/62504a68e1cc6671f25990b7] is now STARTING
2022-04-13T07:17:45.880-04:00 INFO  [InputStateListener] Input [Syslog UDP/6250508ee1cc6671f2599791] is now STARTING
2022-04-13T07:17:45.881-04:00 INFO  [InputStateListener] Input [Beats/62546a7fe1cc6671f25e0c1c] is now STARTING
2022-04-13T07:17:45.883-04:00 INFO  [InputStateListener] Input [Syslog UDP/62557c27e1cc6671f25f3664] is now STARTING
2022-04-13T07:17:46.102-04:00 INFO  [InputStateListener] Input [Beats/62546a7fe1cc6671f25e0c1c] is now RUNNING
2022-04-13T07:17:46.160-04:00 WARN  [AbstractTcpTransport] receiveBufferSize (SO_RCVBUF) for input Beats2Input{title=Windows Events, type=org.graylog.plugins.beats.Beats2Input, nodeId=null} (channel [id: 0x8f42a896, L:/0:0:0:0:0:0:0:0%0:5044]) should be >= 1048576 but is 425984.
2022-04-13T07:17:46.177-04:00 INFO  [InputStateListener] Input [Syslog UDP/6250508ee1cc6671f2599791] is now RUNNING
2022-04-13T07:17:46.180-04:00 INFO  [InputStateListener] Input [Syslog UDP/62557c27e1cc6671f25f3664] is now RUNNING
2022-04-13T07:17:46.183-04:00 INFO  [InputStateListener] Input [Raw/Plaintext UDP/62504a68e1cc6671f25990b7] is now RUNNING
2022-04-13T07:17:52.565-04:00 ERROR [IndexRotationThread] Couldn't point deflector to a new index
java.lang.RuntimeException: Unable to extract count from response.
        at org.graylog.storage.elasticsearch7.IndicesAdapterES7.numberOfMessages(IndicesAdapterES7.java:265) ~[?:?]
        at org.graylog2.indexer.indices.Indices.numberOfMessages(Indices.java:113) ~[graylog.jar:?]
        at org.graylog2.indexer.rotation.strategies.MessageCountRotationStrategy.shouldRotate(MessageCountRotationStrategy.java:68) ~[graylog.jar:?]
        at org.graylog2.indexer.rotation.strategies.MessageCountRotationStrategy.shouldRotate(MessageCountRotationStrategy.java:34) ~[graylog.jar:?]
        at org.graylog2.indexer.rotation.strategies.AbstractRotationStrategy.rotate(AbstractRotationStrategy.java:71) ~[graylog.jar:?]

curl -XGET http://localhost:9200/_cluster/health?pretty=true
{
  "cluster_name" : "graylog",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 1,
  "number_of_data_nodes" : 1,
  "active_primary_shards" : 80,
  "active_shards" : 80,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 100.0

cat /etc/graylog/server/server.conf | egrep -v "^\s*(#|$)"
is_master = true
node_id_file = /etc/graylog/server/node-id
password_secret = <removed>
root_password_sha2 = <removed>
root_email = "itsupport@company.local"
root_timezone = America/New_York
bin_dir = /usr/share/graylog-server/bin
data_dir = /opt/graylog-server
plugin_dir = /usr/share/graylog-server/plugin
http_bind_address = 10.40.1.250:9000
http_publish_uri = http://lnb-graylog.company.local:9000/
http_enable_tls = true
http_tls_cert_file = /etc/pki/tls/certs/lnb-graylog.company.local.crt
http_tls_key_file = /etc/pki/tls/private/lnb-graylog.company.local-npwkey.pem
rotation_strategy = count
elasticsearch_max_docs_per_index = 20000000
elasticsearch_max_number_of_indices = 20
retention_strategy = delete
elasticsearch_shards = 4
elasticsearch_replicas = 0
elasticsearch_index_prefix = graylog
allow_leading_wildcard_searches = false
allow_highlighting = false
elasticsearch_analyzer = standard
output_batch_size = 500
output_flush_interval = 1
output_fault_count_threshold = 5
output_fault_penalty_seconds = 30
processbuffer_processors = 5
outputbuffer_processors = 3
processor_wait_strategy = blocking
ring_size = 65536
inputbuffer_ring_size = 65536
inputbuffer_processors = 2
inputbuffer_wait_strategy = blocking
message_journal_enabled = true
message_journal_dir = /var/lib/graylog-server/journal
lb_recognition_period_seconds = 3
mongodb_uri = mongodb://localhost/graylog
mongodb_max_connections = 1000
mongodb_threads_allowed_to_block_multiplier = 5
proxied_requests_thread_pool_size = 32

tonyg · April 13, 2022, 12:45pm

on a guess, i chown’d the /etc/graylog subdir to graylog:graylog and rebooted…that did not help

tonyg · April 13, 2022, 2:07pm

ok, i think i am close to a solution…i have configured graylog for https and that seems to work…except, when it calls itself, it seems to not accept my self signed cert

2022-04-13T10:03:18.513-04:00 WARN  [ProxiedResource] Unable to call https://lnb-graylog.lyonsbank.local:9000/api/system/inputstates on node <f86dbf9f-2179-4e44-91f4-2e10f416b008>: Hostname lnb-graylog.company.local not verified:
    certificate: sha256/CEL8Mn5mpFvDY/NBLbco0Wen5NULerAjjSRdvr3okPg=
    DN: CN=lnb-graylog.company.local, OU=IT, O=org, L=city, ST=New York, C=US
    subjectAltNames: []

i added my cert to /etc/pki/ca-trust/source/anchors and ran update-ca-trust extract but it seems not to like the cert…any clue on how to make graylog accept this cert?..i think this is the issue

gsmith · April 13, 2022, 9:37pm

Hello,

Thanks for the added info. I found a couple configuration I’m unsure of.

Here Is my lab GL, because of testing I don’t use Localhost/127.0.0.1 for ES connections. Take notice on http_publish_uri section.

http_bind_address = graylog.domain.com:9000
http_publish_uri = https://graylog.domain.com:9000/
http_enable_cors = true
http_enable_tls = true
http_tls_cert_file = /etc/pki/tls/certs/graylog/graylog-certificate.pem
http_tls_key_file = /etc/pki/tls/certs/graylog/graylog-key.pem
http_tls_key_password = secret
elasticsearch_hosts = http://10.10.10.10:9200

I would need to see the steps taken or documentation used to create your certificates to help you further on that…

You can do a test to be 100% sure that its your certificate cause all these issue.
Set you Graylog configuration back to a HTTP connection.

Example:
Comment the following lines out.

http_bind_address = 10.40.1.250:9000
# http_publish_uri = http://lnb-graylog.company.local:9000/
# http_enable_tls = true
# http_tls_cert_file = /etc/pki/tls/certs/lnb-graylog.company.local.crt
# http_tls_key_file = /etc/pki/tls/private/lnb-graylog.company.local-npwkey.pem

Resart Graylog service

The use the following to logon.

http://10.40.1.250:9000

If you able to do that and everything works, we can look into your certificates and configuration for HTTPS.

Hope that helps

tonyg · April 14, 2022, 8:02pm

ok, fixed!!! so, thanks for all your help on this… I had made these mistakes:

i had

http_publish_uri = http://lnb-graylog.company.local:9000/

but should have bee this

http_publish_uri = https://lnb-graylog.company.local:9000/

i skipped over the part in the docs ( Using HTTPS - Configuring Graylog) where you need to create a san cert and include BOTH the FQDN AND the IP of the server.

fixing these two things got me working

thanks again for your help!

system · April 28, 2022, 8:02pm

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
New install, failing (invisible) input Graylog Central (peer support)	10	2369	September 27, 2017
Graylog not able to start inputs Graylog Central (peer support)	2	395	March 20, 2019
Input start fail Graylog Central (peer support)	9	859	August 8, 2018
Input - no active connection Graylog Central (peer support)	12	1647	November 11, 2020
An input has failed to start Graylog Central (peer support)	5	6797	July 30, 2018

Inputs show failed, but ports are open and logs are coming in

Related topics