Inputs not showing as running, but they seem to be anyway

bbbart · March 8, 2021, 10:26am

Hi there,

I’ve got a fresh installation running of Graylog 4.0.5+d95b909 on Debian 10 with MongoDB 4.4.4 and Elasticsearch 7.10.2, all in a minimal setup on a simple, single server.

Everything seems to running smoothly. I defined some inputs (UDP GELF) and succeed in getting messages into the platform, do some searches, get some graphs, etc…

However, on my inputs overview page, these inputs always show up as not running. Clicking the Start input button tells me that the the request to start was sent successfully and the input would be started soon.

I’m not sure this really is a problem, but I reckoned mentioning it here wouldn’t cause any harm either. Perhaps someone else encountered the same issue and has a solution in mind?

Thanks!

dickinsonzach · March 8, 2021, 1:29pm

Are you doing SSL by any chance?

I am having a similar issue and it’s related to the not being able to all the APIs that check those states.

Run
tail -f /var/log/graylog-server/server.log

and see if you have anything like
Unable to call https://…/api/system/inputstates on node <000>

Thank you, Zach.

bbbart · March 8, 2021, 7:23pm

Hi Zach,

No, there’s no SSL involved (except for nginx reverse proxying through https for the web interface, not for the inputs).

There are no occurences of Unable to call in my /var/log/graylog-server/server.log related to this issue.

When tailing that file while clicking the start input button, the following appears:

2021-03-08T20:19:31.106+01:00 INFO  [InputStateListener] Input [GELF UDP/604401342bf721557ccf1cc3] is now STOPPING
2021-03-08T20:19:32.325+01:00 INFO  [InputStateListener] Input [GELF UDP/604401342bf721557ccf1cc3] is now STOPPED
2021-03-08T20:19:32.340+01:00 INFO  [InputStateListener] Input [GELF UDP/604401342bf721557ccf1cc3] is now TERMINATED
2021-03-08T20:19:33.537+01:00 INFO  [InputStateListener] Input [GELF UDP/604401342bf721557ccf1cc3] is now STARTING
2021-03-08T20:19:35.403+01:00 INFO  [InputStateListener] Input [GELF UDP/604401342bf721557ccf1cc3] is now RUNNING
2021-03-08T20:19:35.485+01:00 WARN  [UdpTransport] receiveBufferSize (SO_RCVBUF) for input GELFUDPInput{title=Gamemaster logs (development), type=org.graylog2.inputs.gelf.udp.GELFUDPInput, nodeId=f8c5641c-58b2-4631-bcc7-ac99b320ffa4} (channel [id: 0x936d29b5, L:/0:0:0:0:0:0:0:0%0:12200]) should be 262144 but is 425984.

Let me investigate that last line more…

bbbart · March 8, 2021, 7:35pm

ok, never mind that; this was solve by setting the sysctl value for net.core.rmem_max to 131072 but doesn’t seem related to the problem at hand.

I also checked my browser console to see if it was simply a frontend/JS issue, but nothing appeared there either.

gsmith · March 9, 2021, 1:06am

@bbbart
Did you see any errors or warnings in Elasticsearch log files? By chance could we see you INPUT configuration? How many INPUTS do you have?and are they all GELF UDP? Have you tried to create other different INPUTS (i.e. syslog UDP, RAW/PLAIN TEXT", etc…), and if so do they all show up “Not Running”. Did you restart graylog-server service and tail the graylog log file to see if any new errors/warnings showed up after creating the inputs?

This is one of mine in the lab I just randomly created.

gsmith · March 9, 2021, 3:24am

@bbbart
I tried to create you errors your having, which is the one you stated below.

I had one INPUT running already and I created another INPUT ( Linux test #2) with the same Port, by chance have you done the same thing?
Just an idea I had.

bbbart · March 9, 2021, 12:17pm

I didn’t check before, but couldn’t really find anything now. Also, when tail-following /var/log/elasticsearch/g*log while starting an input, nothing is appended to these logfiles at all.

Sure. I have configured two inputs, both GELF UDP, both showing the same behaviour (remember, they seem started, they just don’t show up as started in the web interface).

bind_address:     0.0.0.0
decompress_size_limit:     8388608
number_worker_threads:     1
override_source:     <empty>
port:     12200
recv_buffer_size:     262144

and

bind_address:     0.0.0.0
decompress_size_limit:     8388608
number_worker_threads:     1
override_source:     <empty>
port:     12201
recv_buffer_size:     262144

The only difference between the two is the port.

I had the same problem with a GELF HTTP input I created for testing. I also just created a Syslog TCP input for testing and it showed the same behaviour.

Yes. I just did this again. The only things in the Graylog log file related to inputs when restarting the server is:

[...]
2021-03-09T13:10:04.305+01:00 INFO  [InputBufferImpl] Initialized InputBufferImpl with ring size <65536> and wait strategy <BlockingWaitStrategy>, running 2 parallel message handlers.
[...]
2021-03-09T13:10:52.375+01:00 INFO  [InputSetupService] Triggering launching persisted inputs, node transitioned from Uninitialized [LB:DEAD] to Running [LB:ALIVE]
2021-03-09T13:10:52.387+01:00 INFO  [ServerBootstrap] Graylog server up and running.
[...]   
2021-03-09T13:10:52.454+01:00 INFO  [InputStateListener] Input [GELF UDP/604401342bf721557ccf1cc3] is now STARTING
2021-03-09T13:10:52.480+01:00 INFO  [InputStateListener] Input [GELF UDP/60454cb02bf721557cd08687] is now STARTING
2021-03-09T13:10:52.645+01:00 INFO  [InputStateListener] Input [GELF UDP/60454cb02bf721557cd08687] is now RUNNING
2021-03-09T13:10:52.648+01:00 INFO  [InputStateListener] Input [GELF UDP/604401342bf721557ccf1cc3] is now RUNNING

However, I think I found something interesting here. While the web interface was loading the running-status of the inputs, I by chance saw a Java error appearing in the graylog log:

org.glassfish.jersey.server.internal.process.MappableException: java.io.IOException: Connection is closed
[...]
Caused by: java.io.IOException: Connection is closed
[...]
Caused by: java.io.IOException: Broken pipe
[...]

Is this perhaps the API or something else complaining it can’t retrieve the input running status, or can’t pass that information on to the frontend requesting it? I’m of course only out on a limb here…

Thanks for your involvement, Greg!

bbbart · March 9, 2021, 12:20pm

No, I don’t. Thanks for the idea.

The problem was already there when I had only one input configured. Also, I’m not getting ‘1 failed’, but simply ‘not running’:

gsmith · March 9, 2021, 11:50pm

@bbbart
Java and I dont play well together
Correct me if I’m wrong, just wanted to sum it up.

You see log/s coming in when you click on “Show received messages” from those inputs that state, “Not running” in real time?
Your using nginx reverse proxying w/ https?
Tried different INPUTs but have the same outcome " Not Running".

Last questions

Are those inputs set for “Global” or Selected Node"?
How about permission on Graylog files?
Before you used nginx (HTTPS) did the inputs show a running state?
Do you have Selinux or Apparmor enabled?
I should have ask this before, but how about Mongdb/nginx log files did you see any errors/warnings there?

It probably would help to see your graylog configuration file, and your nginx config.
If this doesn’t help, at this point I’m not sure. If I had this issue with a fresh Graylog installation, I would remove nginx from the equation to see if everything functions as expected. Then roll into securing it. That way if something weird pops up I know for sure where the issues is from.
All I can do is offer suggestions and speculate sorry I cant be more help.
Maybe someone else can jump in help.

bbbart · March 10, 2021, 2:41pm

Yes, that’s correct. It really feels like a simple GUI issue: it’s all working as it’s supposed to (I think), but it’s just not telling me that it is.

Now, I also fond some other GUI quirks, which I find strange but don’t know about if it’s expected behaviour or not:

the ‘home page’ keeps on showing me the guide even after creating inputs, performing some searches and defining a dashboard; I would expect to see a default dashboard or something on the homepage
on the dashboard page, I always only ever get to see the predefined Sources dashboard. Those I have created myself are only accessible after searching for them, this making it necessary to know their names up front.
the inputs page keeps on showing me inputs I have deleted, or when I switch from selected node to/from global they remain where they are (in the GUI, not in the configuration).

Strange, no?

(come to think about it… doesn’t this smell like a caching problem?)

Selected node (there’s only a single node)

Which files? /var/lib/graylog-server and children are owned by graylog:graylog.

There was never a before nginx stage. But let me try that, indeed.

Nope. Neither of them.

The nginx error logs are completely empty. The mongodb logs I’m unfamiliar with and seem hard to parse for humans at first glance. However some grep-guesses for errors didn’t reveal anything useful.

I’d be happy enough to share them. How do I do that best? Upload to this forum somewhere?

Yes, great suggestion. Let me try just that!

Thank you so much for your help.

bbbart · March 10, 2021, 2:53pm

Alright, found the culprit. Accessing Graylog without passing through nginx indeed gives me a whole new user experience. A much better one, I may say.

The issue that sparked this topic, but also all the others I mentioned meanwhile are gone in direct access.

Time to play around with my nginx config.

Thank you so much for the suggestion, Greg. Much appreciated.

gsmith · March 11, 2021, 1:03am

@bbbart
Glad I could help,and good luck

system · March 25, 2021, 1:03am

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Inputs appear as not running Graylog Central (peer support) sidecar , nxlog , nodatanx	24	7854	July 18, 2017
Inputs not Running(But still getting data in),, Node unavailable "Error Getting Data?" Graylog Central (peer support)	5	895	April 13, 2022
Inputs dont run but I recive messages Graylog Central (peer support)	4	580	January 31, 2023
Can't create new inputs Graylog Central (peer support)	7	1708	July 10, 2017
Local Inputs not running and not starting Graylog Central (peer support)	9	4713	December 21, 2017

Inputs not showing as running, but they seem to be anyway

Related topics