I thought to attack this from the other end and found how change the format of the syslog timestamp and remove the sequence number in the switch itself. I have it down to this:
<189>: Jan 28 21:58:14: %SYS-5-CONFIG_....
The only issue I see now is the extra colons around the timestamp, the message is still not being by a RAW UDP input, nor the input from the cisco content pack I installed
he @samwarez
the RAW UDP Input is like netcat listening on a network device/ip - if that does not record anything something between the sending device and Graylog is blocking the communication. Simple rule.
where do you think I got the packet captures from? If you read my posts you will see that I have documented proof that the server running graylog is receiving the inputs and that the RAW UDP input is configured and correctly processing inputs from other sources. It’s the Graylog service that is simply not reacting to those packets.
what model cisco device is it and what version of code? I’m getting logs from a bunch of Cisco devices using both raw and syslog inputs.
So lets back up…
How many inputs do you have configured? If this is your first install and the cisco switch your first input, lets delete all the inputs and just create one. If you have other inputs, then you’ll need to check them to ensure the Cisco messages are not accidentally getting routed into one of them. After creating the single input for the cisco device and starting it, I would check the server.log file to see if there are any errors in there with regards to that input. You had some listed earlier. If there are none, then the input should be in a state ready to receive messages.
The first thing that Graylog does upon receiving the message via an input is write it to the journal. Afterwards, it processes the message and outputs it to Elasticsearch for indexing where you can search for it, etc.
So if you have confirmed that the message is being sent from the switch, it is making it to the Graylog Server (tshark) and the input is up, then, you should see the message in the journal (albeit VERY briefly) before it is picked up by the processing buffer and moved through the system to be processed. You can try tailing the journal log file, (these are .log files in the /var/lib/graylog-server/journal/ directory (or perhaps a subdirectory) to see if the message is actually being successfully received by the input. then there is something going on with the input or as @jan pointed out, something in the message from Cisco that is non standard and causing Graylog to fail to receive it. Although the RAW input should work
So, check the inputs, check the server.log, check the journal.
FWIW… try a different Cisco device too if you can.
It’s a 6509 running IOS 12.2.
I am running just one input.
$ tail -n 10 /var/log/graylog-server/server.log
2020-02-03T14:41:35.254-07:00 ERROR [AuditLogger] Unable to write audit log entry because there is no valid license.
2020-02-03T14:41:35.257-07:00 INFO [InputStateListener] Input [Syslog UDP/5e305a8ad9f024060623697d] is now STOPPED
2020-02-03T14:41:36.349-07:00 ERROR [AuditLogger] Unable to write audit log entry because there is no valid license.
2020-02-03T14:41:39.644-07:00 ERROR [AuditLogger] Unable to write audit log entry because there is no valid license.
2020-02-03T14:41:39.660-07:00 ERROR [AuditLogger] Unable to write audit log entry because there is no valid license.
2020-02-03T14:41:39.680-07:00 INFO [InputStateListener] Input [Raw/Plaintext UDP/5e30a50cd9f024060623ba4f] is now STARTING
2020-02-03T14:41:39.682-07:00 WARN [UdpTransport] receiveBufferSize (SO_RCVBUF) for input RawUDPInput{title=ciscotestRAW, type=org.graylog2.inputs.raw.udp.RawUDPInput, nodeId=null} (channel [id: 0x216ef549, L:/0:0:0:0:0:0:0:0%0:15151]) should be 262144 but is 425984.
2020-02-03T14:41:39.684-07:00 WARN [UdpTransport] receiveBufferSize (SO_RCVBUF) for input RawUDPInput{title=ciscotestRAW, type=org.graylog2.inputs.raw.udp.RawUDPInput, nodeId=null} (channel [id: 0x67090de3, L:/0:0:0:0:0:0:0:0%0:15151]) should be 262144 but is 425984.
2020-02-03T14:41:39.695-07:00 ERROR [AuditLogger] Unable to write audit log entry because there is no valid license.
2020-02-03T14:41:39.696-07:00 INFO [InputStateListener] Input [Raw/Plaintext UDP/5e30a50cd9f024060623ba4f] is now RUNNING
tail -f /var/lib/graylog-server/journal/messagejournal-0/00000000000000002309.log shows no updates when I trigger a log message from the switch, it does when I do something on the linux server however.
Same issue here, fresh graylog 3.2 install, one syslog udp input (port 1514 with iptables redirect).
I see that Input sees the message from cisco (metric shows messaged being read but not written) but it ignores them.
Should I just use 3rd party plugin/contact pack like this one > https://marketplace.graylog.org/addons/92c7e8b5-8f9b-44c1-9b36-7bc41685d2fe ?
Message that he cant parse/understand - "apollo.. 6440: Feb 4 04:00:43.054: %SYS-5-CONFIG_I: Configured from console by *** on vty0 (94...*) "
as we do not have that kind of device “to play” with - can you provide a pcap of some syslog traffic from that kind of devices?
I can. Can you please tell me how can I share pcap with you (without compromising sensitive data to whole community)
Ok, so the fun part - It does work but messages are written “in the future” (+3h, and our zone is GMT +3) despite the fact both devices (graylog server and cisco 6500 router has the correct time (same time))
Now I want to look into that pcap myself.
UPD. So the question is why does graylog adds your time zone again (another +3h) to what he received from cisco cat 6500 switch.
ok, so the issue is solved for me. By default cisco 6500 sends syslog messages without timezone so graylog thinks they are UTC and ads your timezone (and you will be able to see the messages but in the future.
By default they look like this Feb 4 14:56:42.538: %SYS-5-CONFIG_I:
In order to “fix” it you need to add time zone to the messages:
conf t
service timestamps log datetime msec localtime show-timezone
After that messages will look like this - Feb 4 15:14:40.718 MSK: %SYS-5-CONFIG_I:
he @alexiff
glad that you found the solution. But sorry to say in the linked blog post exactly this is written:
As we like to joke in our region - Chukcha, however, is not a reader. Chukcha is a writer.
p.s. thanks for the article (I actually bookmarked it but forgot to read it)
Chukcha, the Chuck Norris of Russian region 
glad you find that helpful!
In my case however the timezones are all the same. What would stop a message from being written to the journal?
@samwarez did you see something in the Graylog server.log when you have a message that is not written to the journal?
The server should write when it does not accept a message because the codec fails. In addition to that - does the Elasticsearch server write any errors into the log? The Graylog journal is only short permanent storage when the message is ingest. Nearly instant the message is read again from the journal by the processing and after that written to Elasticsearch what is the permanent storage of messages.
This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.