I am trying to get gray log to receive syslogs from a cisco switch, I have it working and receiving data from other sources, but I can’t get the switch to work. I have checked via tshark that the packets are getting to the graylog server, but no messages show up in the web interface.
The server was setup on Centos 7 using the documentation here: http://docs.graylog.org/en/3.1/pages/installation/os/centos.html
cat /etc/os-release shows that its CentOS Linux release 7.7.1908 (Core)
$ yum info graylog-server
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
* base: mirrors.tripadvisor.com
* epel: mirror.colorado.edu
* extras: centos.s.uw.edu
* updates: centos.den.host-engine.com
Installed Packages
Name : graylog-server
Arch : noarch
Version : 3.1.4
Release : 1
Size : 126 M
Repo : installed
From repo : graylog
Summary : Graylog server
URL : https://www.graylog.org/
License : GPLv3
Description : Graylog server
$ cat /etc/graylog/server/server.conf
is_master = true
node_id_file = /etc/graylog/server/node-id
password_secret = *A big long super secret string*
root_username = admin
root_password_sha2 = *Another big long super secret string*
bin_dir = /usr/share/graylog-server/bin
data_dir = /var/lib/graylog-server
plugin_dir = /usr/share/graylog-server/plugin
http_bind_address = 192.168.10.26:9000
rotation_strategy = count
elasticsearch_max_docs_per_index = 20000000
elasticsearch_max_number_of_indices = 20
retention_strategy = delete
elasticsearch_shards = 4
elasticsearch_replicas = 0
elasticsearch_index_prefix = graylog
allow_leading_wildcard_searches = false
allow_highlighting = false
elasticsearch_analyzer = standard
output_batch_size = 500
output_flush_interval = 1
output_fault_count_threshold = 5
output_fault_penalty_seconds = 30
processbuffer_processors = 5
outputbuffer_processors = 3
processor_wait_strategy = blocking
ring_size = 65536
inputbuffer_ring_size = 65536
inputbuffer_processors = 2
inputbuffer_wait_strategy = blocking
message_journal_enabled = true
message_journal_dir = /var/lib/graylog-server/journal
lb_recognition_period_seconds = 3
mongodb_uri = mongodb://localhost/graylog
mongodb_max_connections = 1000
mongodb_threads_allowed_to_block_multiplier = 5
proxied_requests_thread_pool_size = 32
(Commented lines filtered out for brevity)
My input setup:
I have also tried with a Raw/Plaintext UDP input with the same results.
TShark:
$ tshark -i ens160 -f "dst port 15150" -w capture-output.pcap Capturing on 'ens160' 5
I bounce a port on the switch to generate log entries
$ tshark -r capture-output.pcap 1 0.000000000 *Switch IP* -> *Graylog IP* UDP 154 Source port: 63486 Destination port: 15150 2 1.004095516 *Switch IP* -> *Graylog IP* UDP 157 Source port: 63486 Destination port: 15150 3 3.600196975 *Switch IP* -> *Graylog IP* UDP 136 Source port: 63486 Destination port: 15150 4 4.628186956 *Switch IP* -> *Graylog IP* UDP 139 Source port: 63486 Destination port: 15150 5 7.880187322 *Switch IP* -> *Graylog IP* UDP 144 Source port: 63486 Destination port: 15150
Firewall and selinux are disabled
$ sestatus
SELinux status: disabled
$ systemctl status firewalld
● firewalld.service - firewalld - dynamic firewall daemon
Loaded: loaded (/usr/lib/systemd/system/firewalld.service; disabled; vendor preset: enabled)
Active: inactive (dead)
Docs: man:firewalld(1)
$ tail -n 100 /var/log/graylog-server/server.log
2020-01-21T09:19:44.596-07:00 INFO [HttpServer] [HttpServer] Started.
2020-01-21T09:19:44.596-07:00 INFO [JerseyService] Started REST API at <192.168.10.26:9000>
2020-01-21T09:19:44.597-07:00 INFO [ServiceManagerListener] Services are healthy
2020-01-21T09:19:44.598-07:00 INFO [ServerBootstrap] Services started, startup times in ms: {OutputSetupService [RUNNING]=47, GracefulShutdownService [RUNNING]=53, BufferSynchronizerService [RUNNING]=70, InputSetupService [RUNNING]=130, KafkaJournal [RUNNING]=167, JobSchedulerService [RUNNING]=180, ConfigurationEtagService [RUNNING]=180, EtagService [RUNNING]=180, UrlWhitelistService [RUNNING]=184, JournalReader [RUNNING]=186, StreamCacheService [RUNNING]=261, LookupTableService [RUNNING]=262, MongoDBProcessingStatusRecorderService [RUNNING]=267, PeriodicalsService [RUNNING]=452, JerseyService [RUNNING]=28303}
2020-01-21T09:19:44.603-07:00 INFO [InputSetupService] Triggering launching persisted inputs, node transitioned from Uninitialized [LB:DEAD] to Running [LB:ALIVE]
2020-01-21T09:19:44.631-07:00 INFO [ServerBootstrap] Graylog server up and running.
2020-01-21T09:19:44.632-07:00 ERROR [AuditLogger] Unable to write audit log entry because there is no valid license.
2020-01-21T09:19:44.653-07:00 INFO [InputStateListener] Input [Syslog UDP/5e25e9cbd9f024274dac03e9] is now STARTING
2020-01-21T09:19:44.673-07:00 INFO [InputStateListener] Input [Raw/Plaintext UDP/5e25f4bcd9f024274dac0ff1] is now STARTING
2020-01-21T09:19:44.680-07:00 INFO [InputStateListener] Input [Syslog UDP/5e25f775d9f024274dac12ea] is now STARTING
2020-01-21T09:19:44.682-07:00 INFO [InputStateListener] Input [Raw/Plaintext UDP/5e25eecdd9f024274dac0965] is now STARTING
2020-01-21T09:19:44.780-07:00 WARN [UdpTransport] receiveBufferSize (SO_RCVBUF) for input SyslogUDPInput{title=CISCO_IOS_SYSLOG, type=org.graylog2.inputs.syslog.udp.SyslogUDPInput, nodeId=null} (channel [id: 0x67d6ccbf, L:/0:0:0:0:0:0:0:0%0:15150]) should be 262144 but is 425984.
2020-01-21T09:19:44.782-07:00 WARN [UdpTransport] receiveBufferSize (SO_RCVBUF) for input SyslogUDPInput{title=syslog test, type=org.graylog2.inputs.syslog.udp.SyslogUDPInput, nodeId=null} (channel [id: 0x55021083, L:/0:0:0:0:0:0:0:0%0:15150]) should be 262144 but is 425984.
2020-01-21T09:19:44.782-07:00 WARN [UdpTransport] receiveBufferSize (SO_RCVBUF) for input RawUDPInput{title=cisco test, type=org.graylog2.inputs.raw.udp.RawUDPInput, nodeId=null} (channel [id: 0x205bf7ee, L:/0:0:0:0:0:0:0:0%0:15150]) should be 262144 but is 425984.
2020-01-21T09:19:44.783-07:00 WARN [UdpTransport] receiveBufferSize (SO_RCVBUF) for input RawUDPInput{title=test input, type=org.graylog2.inputs.raw.udp.RawUDPInput, nodeId=null} (channel [id: 0x52ed780a, L:/0:0:0:0:0:0:0:0%0:12201]) should be 262144 but is 425984.
2020-01-21T09:19:44.789-07:00 WARN [UdpTransport] receiveBufferSize (SO_RCVBUF) for input SyslogUDPInput{title=CISCO_IOS_SYSLOG, type=org.graylog2.inputs.syslog.udp.SyslogUDPInput, nodeId=null} (channel [id: 0xc910e9c4, L:/0:0:0:0:0:0:0:0%0:15150]) should be 262144 but is 425984.
2020-01-21T09:19:44.795-07:00 WARN [UdpTransport] receiveBufferSize (SO_RCVBUF) for input SyslogUDPInput{title=syslog test, type=org.graylog2.inputs.syslog.udp.SyslogUDPInput, nodeId=null} (channel [id: 0x7c8e7d80, L:/0:0:0:0:0:0:0:0%0:15150]) should be 262144 but is 425984.
2020-01-21T09:19:44.796-07:00 WARN [UdpTransport] receiveBufferSize (SO_RCVBUF) for input RawUDPInput{title=cisco test, type=org.graylog2.inputs.raw.udp.RawUDPInput, nodeId=null} (channel [id: 0xef7bf911, L:/0:0:0:0:0:0:0:0%0:15150]) should be 262144 but is 425984.
2020-01-21T09:19:44.799-07:00 WARN [UdpTransport] receiveBufferSize (SO_RCVBUF) for input RawUDPInput{title=test input, type=org.graylog2.inputs.raw.udp.RawUDPInput, nodeId=null} (channel [id: 0x1638788b, L:/0:0:0:0:0:0:0:0%0:12201]) should be 262144 but is 425984.
2020-01-21T09:19:44.801-07:00 INFO [InputStateListener] Input [Syslog UDP/5e25e9cbd9f024274dac03e9] is now RUNNING
2020-01-21T09:19:44.803-07:00 INFO [InputStateListener] Input [Raw/Plaintext UDP/5e25f4bcd9f024274dac0ff1] is now RUNNING
2020-01-21T09:19:44.806-07:00 INFO [InputStateListener] Input [Raw/Plaintext UDP/5e25eecdd9f024274dac0965] is now RUNNING
2020-01-21T09:19:44.808-07:00 INFO [InputStateListener] Input [Syslog UDP/5e25f775d9f024274dac12ea] is now RUNNING
2020-01-21T09:23:46.102-07:00 ERROR [AuditLogger] Unable to write audit log entry because there is no valid license.
2020-01-21T09:29:15.564-07:00 ERROR [AuditLogger] Unable to write audit log entry because there is no valid license.
2020-01-21T09:29:16.390-07:00 INFO [connection] Opened connection [connectionId{localValue:8, serverValue:8}] to localhost:27017
2020-01-21T09:29:16.391-07:00 INFO [connection] Opened connection [connectionId{localValue:13, serverValue:13}] to localhost:27017
2020-01-21T09:29:16.391-07:00 INFO [connection] Opened connection [connectionId{localValue:10, serverValue:10}] to localhost:27017
2020-01-21T09:29:16.397-07:00 INFO [InputStateListener] Input [Syslog UDP/5e25f775d9f024274dac12ea] is now STOPPING
2020-01-21T09:29:16.397-07:00 INFO [connection] Opened connection [connectionId{localValue:11, serverValue:11}] to localhost:27017
2020-01-21T09:29:16.400-07:00 INFO [connection] Opened connection [connectionId{localValue:14, serverValue:14}] to localhost:27017
2020-01-21T09:29:16.404-07:00 INFO [connection] Opened connection [connectionId{localValue:12, serverValue:12}] to localhost:27017
2020-01-21T09:29:16.405-07:00 INFO [connection] Opened connection [connectionId{localValue:9, serverValue:9}] to localhost:27017
2020-01-21T09:29:16.412-07:00 INFO [InputStateListener] Input [Syslog UDP/5e25f775d9f024274dac12ea] is now STOPPED
2020-01-21T09:29:16.413-07:00 INFO [InputStateListener] Input [Syslog UDP/5e25f775d9f024274dac12ea] is now STARTING
2020-01-21T09:29:16.414-07:00 INFO [InputStateListener] Input [Syslog UDP/5e25f775d9f024274dac12ea] is now TERMINATED
2020-01-21T09:29:16.411-07:00 WARN [UdpTransport] Failed to start channel for input SyslogUDPInput{title=syslog test, type=org.graylog2.inputs.syslog.udp.SyslogUDPInput, nodeId=null}
io.netty.channel.unix.Errors$NativeIoException: bind(..) failed: Permission denied
2020-01-21T09:29:16.426-07:00 ERROR [InputLauncher] The [org.graylog2.inputs.syslog.udp.SyslogUDPInput] input with ID <5e25f775d9f024274dac12ea> misfired. Reason: bind(..) failed: Permission denied.
org.graylog2.plugin.inputs.MisfireException: org.graylog2.plugin.inputs.MisfireException: io.netty.channel.unix.Errors$NativeIoException: bind(..) failed: Permission denied
at org.graylog2.plugin.inputs.MessageInput.launch(MessageInput.java:158) ~[graylog.jar:?]
at org.graylog2.shared.inputs.InputLauncher$1.run(InputLauncher.java:84) [graylog.jar:?]
at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:181) [graylog.jar:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_232]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_232]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_232]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_232]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_232]
Caused by: org.graylog2.plugin.inputs.MisfireException: io.netty.channel.unix.Errors$NativeIoException: bind(..) failed: Permission denied
at org.graylog2.inputs.transports.UdpTransport.launch(UdpTransport.java:135) ~[graylog.jar:?]
at org.graylog2.plugin.inputs.MessageInput.launch(MessageInput.java:155) ~[graylog.jar:?]
... 7 more
Caused by: io.netty.channel.unix.Errors$NativeIoException: bind(..) failed: Permission denied
2020-01-21T09:29:16.436-07:00 ERROR [AuditLogger] Unable to write audit log entry because there is no valid license.
2020-01-21T09:29:16.436-07:00 INFO [InputStateListener] Input [Syslog UDP/5e25f775d9f024274dac12ea] is now FAILED
2020-01-21T09:29:57.554-07:00 ERROR [AuditLogger] Unable to write audit log entry because there is no valid license.
2020-01-21T09:29:57.560-07:00 ERROR [AuditLogger] Unable to write audit log entry because there is no valid license.
2020-01-21T09:29:57.561-07:00 INFO [InputStateListener] Input [Syslog UDP/5e25f775d9f024274dac12ea] is now STOPPING
2020-01-21T09:29:57.563-07:00 INFO [InputStateListener] Input [Syslog UDP/5e25f775d9f024274dac12ea] is now STOPPED
2020-01-21T09:29:57.564-07:00 INFO [InputStateListener] Input [Syslog UDP/5e25f775d9f024274dac12ea] is now STARTING
2020-01-21T09:29:57.565-07:00 INFO [InputStateListener] Input [Syslog UDP/5e25f775d9f024274dac12ea] is now TERMINATED
2020-01-21T09:29:57.568-07:00 WARN [UdpTransport] Failed to start channel for input SyslogUDPInput{title=syslog test, type=org.graylog2.inputs.syslog.udp.SyslogUDPInput, nodeId=null}
io.netty.channel.unix.Errors$NativeIoException: bind(..) failed: Permission denied
2020-01-21T09:29:57.569-07:00 ERROR [InputLauncher] The [org.graylog2.inputs.syslog.udp.SyslogUDPInput] input with ID <5e25f775d9f024274dac12ea> misfired. Reason: bind(..) failed: Permission denied.
org.graylog2.plugin.inputs.MisfireException: org.graylog2.plugin.inputs.MisfireException: io.netty.channel.unix.Errors$NativeIoException: bind(..) failed: Permission denied
at org.graylog2.plugin.inputs.MessageInput.launch(MessageInput.java:158) ~[graylog.jar:?]
at org.graylog2.shared.inputs.InputLauncher$1.run(InputLauncher.java:84) [graylog.jar:?]
at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:181) [graylog.jar:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_232]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_232]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_232]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_232]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_232]
Caused by: org.graylog2.plugin.inputs.MisfireException: io.netty.channel.unix.Errors$NativeIoException: bind(..) failed: Permission denied
at org.graylog2.inputs.transports.UdpTransport.launch(UdpTransport.java:135) ~[graylog.jar:?]
at org.graylog2.plugin.inputs.MessageInput.launch(MessageInput.java:155) ~[graylog.jar:?]
... 7 more
Caused by: io.netty.channel.unix.Errors$NativeIoException: bind(..) failed: Permission denied
2020-01-21T09:29:57.572-07:00 INFO [InputStateListener] Input [Syslog UDP/5e25f775d9f024274dac12ea] is now FAILED
2020-01-21T09:30:09.546-07:00 ERROR [AuditLogger] Unable to write audit log entry because there is no valid license.
2020-01-21T09:30:10.364-07:00 INFO [InputStateListener] Input [Syslog UDP/5e25f775d9f024274dac12ea] is now STOPPING
2020-01-21T09:30:10.365-07:00 INFO [InputStateListener] Input [Syslog UDP/5e25f775d9f024274dac12ea] is now TERMINATED
2020-01-21T09:30:10.367-07:00 INFO [InputStateListener] Input [Syslog UDP/5e25f775d9f024274dac12ea] is now STOPPED
2020-01-21T09:30:28.796-07:00 ERROR [AuditLogger] Unable to write audit log entry because there is no valid license.
2020-01-21T09:30:28.804-07:00 INFO [InputStateListener] Input [Syslog UDP/5e25f775d9f024274dac12ea] is now STARTING
2020-01-21T09:30:28.805-07:00 ERROR [AuditLogger] Unable to write audit log entry because there is no valid license.
2020-01-21T09:30:28.809-07:00 WARN [UdpTransport] receiveBufferSize (SO_RCVBUF) for input SyslogUDPInput{title=syslog test, type=org.graylog2.inputs.syslog.udp.SyslogUDPInput, nodeId=null} (channel [id: 0xd35e1112, L:/0:0:0:0:0:0:0:0%0:15150]) should be 262144 but is 425984.
2020-01-21T09:30:28.812-07:00 WARN [UdpTransport] receiveBufferSize (SO_RCVBUF) for input SyslogUDPInput{title=syslog test, type=org.graylog2.inputs.syslog.udp.SyslogUDPInput, nodeId=null} (channel [id: 0x852f5492, L:/0:0:0:0:0:0:0:0%0:15150]) should be 262144 but is 425984.
2020-01-21T09:30:28.814-07:00 INFO [InputStateListener] Input [Syslog UDP/5e25f775d9f024274dac12ea] is now RUNNING
2020-01-21T09:30:56.885-07:00 ERROR [AuditLogger] Unable to write audit log entry because there is no valid license.
2020-01-21T09:30:56.890-07:00 INFO [InputStateListener] Input [Syslog UDP/5e25e9cbd9f024274dac03e9] is now STOPPING
2020-01-21T09:30:56.890-07:00 INFO [InputStateListener] Input [Syslog UDP/5e25e9cbd9f024274dac03e9] is now TERMINATED
2020-01-21T09:30:56.891-07:00 ERROR [AuditLogger] Unable to write audit log entry because there is no valid license.
2020-01-21T09:30:56.891-07:00 INFO [InputStateListener] Input [Syslog UDP/5e25e9cbd9f024274dac03e9] is now STOPPED
2020-01-21T09:30:57.540-07:00 ERROR [AuditLogger] Unable to write audit log entry because there is no valid license.
2020-01-21T09:30:57.541-07:00 INFO [InputStateListener] Input [Raw/Plaintext UDP/5e25f4bcd9f024274dac0ff1] is now STOPPING
2020-01-21T09:30:57.543-07:00 ERROR [AuditLogger] Unable to write audit log entry because there is no valid license.
2020-01-21T09:30:57.544-07:00 INFO [InputStateListener] Input [Raw/Plaintext UDP/5e25f4bcd9f024274dac0ff1] is now STOPPED
2020-01-21T09:30:57.545-07:00 INFO [InputStateListener] Input [Raw/Plaintext UDP/5e25f4bcd9f024274dac0ff1] is now TERMINATED
2020-01-21T10:19:16.685-07:00 ERROR [MongoAuditLogPeriodical] Not running cleanup for auditlog entries in MongoDB because there is no valid license.
I see some errors and warnings in the log file but I don’t know how to fix them or if they are even related.
What am I missing here?