I was forwarding my Suricata eve.json log file using rsyslog to send the log data to a Graylog Syslog TCP input listening on port 12201 (later changed to port 12202 for troubleshooting).
This was working until exactly midnight today (February 12, 2023). The logs just stopped. No indications as to why have been found. I am happy to provide full logs for anyone else to take a look.
2. Describe your environment:
OS Information: Ubuntu Server 22.04
Package Version: 5.0.3
Service logs, configurations, and environment variables:
RSYSLOG Config
# Forward nginx logs to graylog server
$ModLoad mmjsonparse
$ModLoad imfile
$InputFileName /var/log/suricata/eve.json
$InputFileTag @cee
$InputFileStateFile suricata.statefile
$InputFileFacility local4
$InputFileSeverity info
$InputRunFileMonitor
local4.* @@127.0.0.1:12202;RSYSLOG_SyslogProtocol23Format
& stop
A Warning in server.log
2023-02-13T14:00:41.477Z WARN [AbstractTcpTransport] receiveBufferSize (SO_RCVBUF) for input SyslogTCPInput{title=Suricata Eve Beat, type=org.graylog2.inputs.syslog.tcp.SyslogTCPInput, nodeId=a30f2b45-6107-45e9-9bba-76d3112d6e71} (channel [id: 0xdc1b5527, L:/0.0.0.0:12202]) should be >= 2097152 but is 425984.
3. What steps have you already taken to try and solve the problem?
Deleted and re-created the Syslog TCP input
Deleted the entire journal folder and restarted
Move everything to a Syslog UDP input
Changed port numbers
Tcpdump listening on port with no protocol (no packets)
4. How can the community help?
Please help me troubleshoot how a working local input just stops working without leaving a trace.
2023-02-13T14:00:41.477Z WARN [AbstractTcpTransport] receiveBufferSize (SO_RCVBUF) for input SyslogTCPInput{title=Suricata Eve Beat, type=org.graylog2.inputs.syslog.tcp.SyslogTCPInput, nodeId=a30f2b45-6107-45e9-9bba-76d3112d6e71} (channel [id: 0xdc1b5527, L:/0.0.0.0:12202]) should be >= 2097152 but is 425984.
Since this is a warning, its just stating that “to increase maximum socket receive buffer size”
As for this, it could be a few diiferent thing, the Logs would be the best way to find out whats going on, This would include Elasticseach/Graylog logs. If posible a stack trace after starting up Graylog service would be Ideal.
Next, I see this is a Beats input and your using Rsyslog using Syslog/TCP format? If this is correct then try using a different type of input.
I will clear that warning later since it doesn’t seem to be related to the logging issue. As Beats, that is in the name of the interface only, the input is or should be Syslog TCP.
Thank you for the Logs. I see that past days looked like an issue with Connecting to mongoDb/Elasticsearch It might be from what service was started first.
Some of the logs I found were interesting, but I also noticed these were from the past couple days.
I assume all the service are working as expected?
Caused by: java.io.IOException: Broken pipe
WARN [Errors] The following warnings have been detected: WARNING: Unknown HK2 failure detected:
java.lang.IllegalStateException: Could not find an active context for org.glassfish.jersey.process.internal.RequestScoped
While attempting to resolve the dependencies of org.graylog2.rest.resources.cluster.ClusterInputStatesResource errors were found
Unable to perform operation: resolve on org.graylog2.rest.resources.cluster.ClusterInputStatesResource
ERROR [VersionProbe] Unable to retrieve version from Elasticsearch node: Failed to connect to /127.0.0.1:9200. - Connection refused.
My Appologies I thought it was a Beat Input, I mis-read this.
syslogTCPInput{title=Suricata Eve Beat,
As for
This should not have made a difference in changing ports. What could make a difference is type of log sent, to that input.
Troubleshooting:
Try Manual rotating the Index set.
Check Rsylog internal logs for any issues.
Perhape create a new INPUT with port 12202 RawPlainText /TCP
Wasnt stated if you reconfigured Rsyslog to use UDP
If the Journal Filled up double check Elasticsearch to ensure its not in Read mode. There a lot of posts here and Elasticsearch Documentation on "How and What " to do.
Ensure all the serivces are workking correct.
Since I seen MongoDb in Graylog Log’s check MongoDb logs for any Errors or warnings.
You won’t believe this, but it started working again and because I’m really bad at keeping track of all that I did, I’m not sure exactly what finally worked.
However, I do suspect it was in the Configurations section on Graylog.
I think this was the last piece I was messing with when I started seeing messages start to flow again. I moved the GeoIP Resolver to the last active processor in the list just above the AWS Name Lookup.
Ok well I guess this is solved now. It’s just a very weird event. I made no changes before it stopped originally so this will probably be one of those strange ‘ghosts in the machine’ type situations where something happened, but there is no evidence of what exactly happened.
As a follow up, I did check each service that Graylog uses and found everything running as expected.
