Hi @all, I am new to Graylogs and I have been trying to troubleshoot a server running with Graylog installation but while checking on the server log I found following issues. The performing server has stopped suddenly.
Server Link: https://logs.grabroomz.com
Pasting here the last ten lines of the log file:
2020-08-31T08:53:25.305Z WARN [NettyTransport] receiveBufferSize (SO_RCVBUF) for input SyslogUDPInput{title=nginx error_log, type=org.graylog2.inputs.syslog.udp.SyslogUDPInput, nodeId=null} should be 1048576 but is 212992.
2020-08-31T08:53:25.306Z WARN [NettyTransport] receiveBufferSize (SO_RCVBUF) for input SyslogUDPInput{title=grabrooms backoffice, type=org.graylog2.inputs.syslog.udp.SyslogUDPInput, nodeId=null} should be 1048576 but is 212992.
2020-08-31T08:53:25.307Z WARN [NettyTransport] receiveBufferSize (SO_RCVBUF) for input SyslogUDPInput{title=grabrooms facebook, type=org.graylog2.inputs.syslog.udp.SyslogUDPInput, nodeId=null} should be 1048576 but is 212992.
2020-08-31T08:53:25.308Z WARN [NettyTransport] receiveBufferSize (SO_RCVBUF) for input GELFUDPInput{title=grabrooms laravel logs, type=org.graylog2.inputs.gelf.udp.GELFUDPInput, nodeId=null} should be 262144 but is 212992.
2020-08-31T08:53:25.309Z INFO [InputStateListener] Input [Syslog UDP/5a4bc0be12522a5ae7bf6fb4] is now RUNNING
2020-08-31T08:53:25.310Z INFO [InputStateListener] Input [Syslog UDP/5a5dcb44dbafb50f61c6aff0] is now RUNNING
2020-08-31T08:53:25.311Z INFO [InputStateListener] Input [GELF UDP/5bdc0900dbafb50cecb9d3e7] is now RUNNING
2020-08-31T08:53:25.312Z INFO [InputStateListener] Input [Syslog UDP/5a5dca95dbafb50f61c6af03] is now RUNNING
2020-08-31T08:53:25.313Z WARN [NettyTransport] receiveBufferSize (SO_RCVBUF) for input SyslogUDPInput{title=grabrooms api, type=org.graylog2.inputs.syslog.udp.SyslogUDPInput, nodeId=null} should be 1048576 but is 212992.
2020-08-31T08:53:25.314Z INFO [InputStateListener] Input [Syslog UDP/5a5dca3cdbafb50f61c6ae99] is now RUNNING
I see nothing in these INFO/WARN messages indicating a failure. Are there any ERROR or FATAL messages in your server logs?
Is it possible the Graylog instance is running but you are simply unable to access it remotely? What are the outputs of the following commands? They should all be active/running.
systemctl status elasticsearch
systemctl status mongod
systemctl status graylog-server
I checked output of the following commands:
systemctl status elasticsearch
systemctl status mongod
systemctl status graylog-server
All above three services are showing active/running.
I checked more from the log file as follows:
2020-08-31T08:53:13.985Z ERROR [LookupDataAdapter] Couldn't start data adapter <spamhaus-drop/5a5d9c7bdbafb52a7c82335f/@5290d497>
org.graylog.plugins.threatintel.tools.AdapterDisabledException: Spamhaus service is disabled, not starting (E)DROP adapter. To enable it please go to System / Configurations.
at org.graylog.plugins.threatintel.adapters.spamhaus.SpamhausEDROPDataAdapter.doStart(SpamhausEDROPDataAdapter.java:68) ~[?:?]
at org.graylog2.plugin.lookup.LookupDataAdapter.startUp(LookupDataAdapter.java:59) [graylog.jar:?]
at com.google.common.util.concurrent.AbstractIdleService$DelegateService$1.run(AbstractIdleService.java:62) [graylog.jar:?]
at com.google.common.util.concurrent.Callables$4.run(Callables.java:122) [graylog.jar:?]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_242]
2020-08-31T08:53:14.012Z INFO [LookupTableService] Data Adapter spamhaus-drop/5a5d9c7bdbafb52a7c82335f [@5290d497] RUNNING
2020-08-31T08:53:13.985Z ERROR [LookupDataAdapter] Couldn't start data adapter <abuse-ch-ransomware-ip/5a5d9c7bdbafb52a7c823361/@6662ee93>
org.graylog.plugins.threatintel.tools.AdapterDisabledException: Abuse.ch service is disabled, not starting adapter. To enable it please go to System / Configurations.
at org.graylog.plugins.threatintel.adapters.abusech.AbuseChRansomAdapter.doStart(AbuseChRansomAdapter.java:80) ~[?:?]
at org.graylog2.plugin.lookup.LookupDataAdapter.startUp(LookupDataAdapter.java:59) [graylog.jar:?]
at com.google.common.util.concurrent.AbstractIdleService$DelegateService$1.run(AbstractIdleService.java:62) [graylog.jar:?]
at com.google.common.util.concurrent.Callables$4.run(Callables.java:122) [graylog.jar:?]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_242]
2020-08-31T08:53:13.986Z ERROR [LookupDataAdapter] Couldn't start data adapter <tor-exit-node/5a5d9c7bdbafb52a7c82335e/@7a2d0f06>
org.graylog.plugins.threatintel.tools.AdapterDisabledException: TOR service is disabled, not starting TOR exit addresses adapter. To enable it please go to System / Configurations.
at org.graylog.plugins.threatintel.adapters.tor.TorExitNodeDataAdapter.doStart(TorExitNodeDataAdapter.java:73) ~[?:?]
at org.graylog2.plugin.lookup.LookupDataAdapter.startUp(LookupDataAdapter.java:59) [graylog.jar:?]
at com.google.common.util.concurrent.AbstractIdleService$DelegateService$1.run(AbstractIdleService.java:62) [graylog.jar:?]
at com.google.common.util.concurrent.Callables$4.run(Callables.java:122) [graylog.jar:?]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_242]
2020-08-31T08:53:13.988Z ERROR [LookupDataAdapter] Couldn't start data adapter <abuse-ch-ransomware-domains/5a5d9c7bdbafb52a7c823360/@33d5bd1a>
org.graylog.plugins.threatintel.tools.AdapterDisabledException: Abuse.ch service is disabled, not starting adapter. To enable it please go to System / Configurations.
at org.graylog.plugins.threatintel.adapters.abusech.AbuseChRansomAdapter.doStart(AbuseChRansomAdapter.java:80) ~[?:?]
at org.graylog2.plugin.lookup.LookupDataAdapter.startUp(LookupDataAdapter.java:59) [graylog.jar:?]
at com.google.common.util.concurrent.AbstractIdleService$DelegateService$1.run(AbstractIdleService.java:62) [graylog.jar:?]
at com.google.common.util.concurrent.Callables$4.run(Callables.java:122) [graylog.jar:?]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_242]
Those data adapter errors are not fatal for graylog-server, they only mean that the adapters aren’t running. You have confirmed that the core Graylog services are running. You mention it suddenly stopped working. I take that to mean that neither you nor anyone else has made any changes to Graylog (configuration changes or updates), so that information coupled with the fact that the services are running and there apparently aren’t any fatal issues in the logs leads me to suspect the issue is not Graylog. You have provided an internet facing URL – are you able to access Graylog from an internal network? Could this be a firewall configuration problem? Could there be some other issue with the host’s network connectivity?
The issue was on DNS mapping. I’ve rectified it by updating with new server IP address. But now I can see the latest logs are not showing. Can you please guide on it?
Hello @soumyajit, glad to know you solved that! When you say the latest logs are not showing, do you mean that you are not seeing new data incoming to Graylog from sources for which Graylog has previously received data? And that no configuration change has occurred in Graylog or the sources sending the data? If so, you will need to check a few things. Are the requisite inputs running? Do you see that they are receiving messages?
I would also offer that if you had to update a DNS record you may need to check that 1) your sources aren’t configured to send directly to the old IP rather than a DNS name and 2) if they are configured to send to a DNS name, that you don’t need to force them to refresh their DNS cache. It’s also possible that intermediate DNS server records have not updated because the record TTL has not yet elapsed.
Thanks for your reply. Now I can see the latest logs. But another issue is now incurring, I can not find the log records of that stipulated time when the link was down. Can I retrieve those missing logs? Can you please advise on it?
Graylog has internal disk journal of unprocessed messages: Incoming messages are written to the disk journal to ensure they are kept safe in case of a server failure. The journal also helps keeping Graylog working if any of the outputs is too slow to keep up with the message rate or whenever there is a peak in incoming messages. It makes sure that Graylog does not buffer all of those messages in main memory and avoids overly long garbage collection pauses that way.
But if log collector didn’t cache logs, it’s probably gone.
It sounds like your sources were sending to the wrong IP. In that case the logs are not in Graylog anywhere. If they still exist you will have to devise a way to resend them now that your configuration is corrected.
How do you ingest messages? What type of input do you use.
I don’t think, that is possible by default to send older logs, you have to manualy send them some way, depends on type of messages.
