Hello,
1. Describe your incident: Note: posting this topic after reviewing countless similar issues on the blog, but could not find a solution. This is includes internet searches
New graylog 5.0.8 server setup.
- able to create a new input; however, nothing seems to be getting into graylog
- Had some issues with using TCP port 514 and looking through the forum found a solution and instead created an input with a custom port “9999”; which show to be listening:
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:514 0.0.0.0:* LISTEN 128921/rsyslogd
tcp 0 0 0.0.0.0:37993 0.0.0.0:* LISTEN -
tcp 0 0 127.0.0.1:27017 0.0.0.0:* LISTEN 821/mongod
tcp 0 0 0.0.0.0:54703 0.0.0.0:* LISTEN 3160/rpc.statd
tcp 0 0 0.0.0.0:111 0.0.0.0:* LISTEN 1/init
tcp 0 0 127.0.0.53:53 0.0.0.0:* LISTEN 796/systemd-resolve
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 878/sshd: /usr/sbin
tcp 0 0 127.0.0.1:6010 0.0.0.0:* LISTEN 122647/sshd: testuser@
tcp6 0 0 :::514 :::* LISTEN 128921/rsyslogd
tcp6 0 0 grelogserverIP:9000 :::* LISTEN 128453/java
tcp6 0 0 grelogserverIP:9999 :::* LISTEN 128453/java
tcp6 0 0 :::111 :::* LISTEN 1/init
tcp6 0 0 grelogserverIP:9200 :::* LISTEN 6236/java
tcp6 0 0 :::33107 :::* LISTEN -
tcp6 0 0 :::42419 :::* LISTEN 3160/rpc.statd
tcp6 0 0 grelogserverIP:9300 :::* LISTEN 6236/java
Note: that I have rsyslog server runnin on the graylog server using tcp port 514. #note sure if
greylog needs rsyslog server to be running. You can see the port is listening.
Just for good measure I added all of the ports to the firewall to ensure there was nothing getting blocked.
On the Graylog server, I’ve checked the Graylog and opensearch logs and there are not errors.
On the client: centos 7
I setup the rsyslog client config as follows:
*.* @@<GrayLogServerIP>:9999;RSYSLOG_SyslogProtocol23Format
On the client: a tcpump host graylogserver shows data being sent:
15:00:27.950623 IP greylogCentosClient.38168 > greylogServer.9999: Flags [P.], seq 1738:1858, ack 1, win 229, options [nop,nop,TS val 1400446060 ecr 1602721358], length 120
15:00:27.951043 IP greylogServer.9999 > greylogCentosClient.38168: Flags [.], ack 1858, win 18528, options [nop,nop,TS val 1602744313 ecr 1400446060], length 0
15:00:27.956031 IP greylogCentosClient.38168 > greylogServer.9999: Flags [P.], seq 1858:2042, ack 1, win 229, options [nop,nop,TS val 1400446065 ecr 1602744313], length 184
15:00:27.956354 IP greylogServer.9999 > greylogCentosClient.38168: Flags [.], ack 2042, win 18796, options [nop,nop,TS val 1602744318 ecr 1400446065], length 0
15:00:28.217229 IP greylogCentosClient.38168 > greylogServer.9999: Flags [P.], seq 2042:2162, ack 1, win 229, options [nop,nop,TS val 1400446326 ecr 1602744318], length 120
15:00:28.217603 IP greylogServer.9999 > greylogCentosClient.38168: Flags [.], ack 2162, win 18796, options [nop,nop,TS val 1602744579 ecr 1400446326], length 0
On the Graylog Server side:
# provides TCP syslog reception
module(load="imtcp")
input(type="imtcp" port="514")
# check opensearch data with curl
curl -XGET http://graylogserver:9200/_cluster/health?pretty=true
{
"cluster_name" : "graylogCluster",
"status" : "green",
"timed_out" : false,
"number_of_nodes" : 1,
"number_of_data_nodes" : 1,
"discovered_master" : true,
"discovered_cluster_manager" : true,
"active_primary_shards" : 13,
"active_shards" : 13,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0,
"task_max_waiting_in_queue_millis" : 0,
"active_shards_percent_as_number" : 100.0
}
# check with ps on graylog server that process is running:
ps|grep opensearch
opensea+ 6236 1 1 Jun26 ? 01:12:29 /usr/share/opensearch/jdk/bin/java -Xshare:auto
-Dopensearch.networkaddress.cache.ttl=60 -Dopensearch.networkaddress.cache.negative.ttl=10
-XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true
-XX:-OmitStackTraceInFastThrow -XX:+ShowCodeDetailsInExceptionMessages -Dio.netty.noUnsafe=true
-Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0
-Dio.netty.allocator.numDirectArenas=0 -Dlog4j.shutdownHookEnabled=false
-Dlog4j2.disable.jmx=true -Djava.locale.providers=SPI,COMPAT -Xms15g -Xmx15g -XX:+UseG1GC
-XX:G1ReservePercent=25 -XX:InitiatingHeapOccupancyPercent=30
-Djava.io.tmpdir=/tmp/opensearch-5283596445488769283 -XX:+HeapDumpOnOutOfMemoryError
-XX:HeapDumpPath=/var/lib/opensearch -XX:ErrorFile=/var/log/opensearch/hs_err_pid%p.log
-Xlog:gc*,gc+age=trace,safepoint:file=/var/log/opensearch/gc.log:utctime,pid,tags:filecount=32,filesize=64m
-Dclk.tck=100 -Djdk.attach.allowAttachSelf=true
-Djava.security.policy=file:///etc/opensearch/opensearch-performance-analyzer/opensearch_security.policy
--add-opens=jdk.attach/sun.tools.attach=ALL-UNNAMED -XX:MaxDirectMemorySize=8053063680
-Dopensearch.path.home=/usr/share/opensearch -Dopensearch.path.conf=/etc/opensearch
-Dopensearch.distribution.type=deb -Dopensearch.bundled_jdk=true
-cp /usr/share/opensearch/lib/* org.opensearch.bootstrap.OpenSearch -p /var/run/opensearch/opensearch.pid
--quiet
# Graylog server Input config:
allow_override_date: true
bind_address: graylogIP
charset_name: UTF-8
expand_structured_data: false
force_rdns: false
max_message_size: 2097152
number_worker_threads: 4
override_source: <empty>
port: 9999
recv_buffer_size: 1048576
store_full_message: true
tcp_keepalive: false
tls_cert_file: <empty>
tls_client_auth: disabled
tls_client_auth_cert_file: <empty>
tls_enable: false
tls_key_file: <empty>
tls_key_password:********
use_null_delimiter: false
Throughput / Metrics
1 minute average rate: 0 msg/s
Network IO: 0B 0B (total: 6.2KiB 0B )
Active connections: 1 (1 total)
Empty messages discarded: 0
# Time Configuration
User user:
2023-06-30 15:15:45 -04:00
Your web browser:
2023-06-30 15:15:45 -04:00
Graylog server:
# graylog server log tail:
tail tail -f /var/log/graylog-server/server.log
==> /var/log/graylog-server/server.log <==
2023-06-30T13:01:52.984-04:00 INFO [InputStateListener] Input [Syslog TCP/649f08ad004b40480f621124] is now TERMINATED
2023-06-30T13:04:27.175-04:00 INFO [InputStateListener] Input [Syslog TCP/649f0b1b004b40480f62163c] is now STARTING
2023-06-30T13:04:27.183-04:00 INFO [InputStateListener] Input [Syslog TCP/649f0b1b004b40480f62163c] is now RUNNING
2023-06-30T13:04:27.188-04:00 WARN [AbstractTcpTransport] receiveBufferSize (SO_RCVBUF) for input
SyslogTCPInput{title=RSYSLOG-TCP-9999, type=org.graylog2.inputs.syslog.tcp.SyslogTCPInput,
nodeId=b1a0850f-8b19-4fa4-85b2-3c21ead149bd} (channel [id: 0xe11e3a34, L:/greylogserverIP:9999])
should be >= 1048576 but is 425984.
2023-06-30T14:35:24.786-04:00 INFO [InputStateListener] Input [Syslog TCP/649f0b1b004b40480f62163c] is now STOPPING
2023-06-30T14:35:24.793-04:00 INFO [InputStateListener] Input [Syslog TCP/649f0b1b004b40480f62163c] is now STOPPED
2023-06-30T14:35:24.794-04:00 INFO [InputStateListener] Input [Syslog TCP/649f0b1b004b40480f62163c] is now TERMINATED
2023-06-30T14:35:24.795-04:00 INFO [InputStateListener] Input [Syslog TCP/649f0b1b004b40480f62163c] is now STARTING
2023-06-30T14:35:24.797-04:00 WARN [AbstractTcpTransport] receiveBufferSize (SO_RCVBUF) for input
SyslogTCPInput{title=RSYSLOG-TCP-9999, type=org.graylog2.inputs.syslog.tcp.SyslogTCPInput,
nodeId=b1a0850f-8b19-4fa4-85b2-3c21ead149bd} (channel [id: 0x712c8382, L:/grelogserverIP:9999])
should be >= 1048576 but is 425984.
2023-06-30T14:35:24.799-04:00 INFO [InputStateListener] Input [Syslog TCP/649f0b1b004b40480f62163c] is now RUNNING
# opensearh log tail:
tail -f /var/log/opensearch/graylogCluster.log
[2023-06-30T14:39:45,981][INFO ][o.o.j.s.JobSweeper ] [grelogserverName] Running full sweep
[2023-06-30T14:44:45,981][INFO ][o.o.j.s.JobSweeper ] [grelogserverName] Running full sweep
[2023-06-30T14:49:45,982][INFO ][o.o.j.s.JobSweeper ] [grelogserverName] Running full sweep
[2023-06-30T14:54:45,983][INFO ][o.o.j.s.JobSweeper ] [grelogserverName] Running full sweep
[2023-06-30T14:59:45,983][INFO ][o.o.j.s.JobSweeper ] [grelogserverName] Running full sweep
[2023-06-30T15:04:45,984][INFO ][o.o.j.s.JobSweeper ] [grelogserverName] Running full sweep
[2023-06-30T15:09:45,985][INFO ][o.o.j.s.JobSweeper ] [grelogserverName] Running full sweep
[2023-06-30T15:14:45,985][INFO ][o.o.j.s.JobSweeper ] [grelogserverName] Running full sweep
[2023-06-30T15:19:45,986][INFO ][o.o.j.s.JobSweeper ] [grelogserverName] Running full sweep
[2023-06-30T15:24:45,987][INFO ][o.o.j.s.JobSweeper ] [grelogserverName] Running full sweep
# graylog server process status
systemctl status graylog-server.service
● graylog-server.service - Graylog server
Loaded: loaded (/lib/systemd/system/graylog-server.service; enabled; vendor preset: enabled)
Active: active (running) since Fri 2023-06-30 12:50:04 EDT; 2h 37min ago
Docs: http://docs.graylog.org/
Main PID: 128449 (graylog-server)
Tasks: 298 (limit: 45613)
Memory: 1.2G
CGroup: /system.slice/graylog-server.service
├─128449 /bin/sh /usr/share/graylog-server/bin/graylog-server
└─128453 /usr/share/graylog-server/jvm/bin/java -Xms1g -Xmx1g -server -XX:+UseG1GC -XX:-OmitStackTraceInFastThrow -Djdk.tls.a>
Jun 30 12:50:04 grelogserverName systemd[1]: Started Graylog server.
# opensearch process status
systemctl status opensearch
● opensearch.service - OpenSearch
Loaded: loaded (/lib/systemd/system/opensearch.service; enabled; vendor preset: enabled)
Active: active (running) since Mon 2023-06-26 18:39:45 EDT; 3 days ago
Docs: https://opensearch.org/
Main PID: 6236 (java)
Tasks: 83 (limit: 45613)
Memory: 15.9G
CGroup: /system.slice/opensearch.service
└─6236 /usr/share/opensearch/jdk/bin/java -Xshare:auto -Dopensearch.networkaddress.cache.ttl=60 -Dopensearch.networkaddress.c>
Jun 30 00:04:45 grelogserverName systemd-entrypoint[6236]: at org.opensearch.jobscheduler.sweeper.JobSweeper.lambda$initBackgroundSweep>
Jun 30 00:04:45 grelogserverName systemd-entrypoint[6236]: at org.opensearch.threadpool.Scheduler$ReschedulingRunnable.doRun(Scheduler.>
Jun 30 00:04:45 grelogserverName systemd-entrypoint[6236]: at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbst>
Jun 30 00:04:45 grelogserverName systemd-entrypoint[6236]: at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnab>
Jun 30 00:04:45 grelogserverName systemd-entrypoint[6236]: at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.j>
Jun 30 00:04:45 grelogserverName systemd-entrypoint[6236]: at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
Jun 30 00:04:45 grelogserverName systemd-entrypoint[6236]: at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutur>
Jun 30 00:04:45 grelogserverName systemd-entrypoint[6236]: at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExe>
Jun 30 00:04:45 grelogserverName systemd-entrypoint[6236]: at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolEx>
Jun 30 00:04:45 grelogserverName systemd-entrypoint[6236]: at java.base/java.lang.Thread.run(Thread.java:833)
2. Describe your environment:
- OS Information:
OS: Ubuntu Server 20.04
VM: 4 CPUs, 32G RAM - Package Version:
Graylog 5.0.8
Opensearch 2.8.0
mongo 6.0.6
4. How can the community help?
Looking for ideas on where to look to get rsyslog messages received in graylog
