Everything is fine, containers are starting, I can log-in and after creating a tcp/syslog input and sending logs to there I can see in/out messages, but when I search logs arrived here I don’t see any
gl2_source_input:63330ca202478d03a678973c
I don’t see any errors in logs, and when I just simply nc’d to the input port (localhost:5140) I can see my message. The logs are forwarded from rsyslog.conf like this
Target=“localhost” Port=“5140” Protocol=“tcp”
It is a fedora35 machine with the latest updates
Any idea what piece is missing? Until now, I mainly used GL3 and w/o docker, but I have never seen something like this before
Perhaps I can help, Normally when everything is working correct but you don’t see the logs on the search GUI it could mean that Date/Time, Time Zone is off.
First things to check are:
System/Overview make sure the User: Web Browser: and Graylog Server: have sync’d date/time.
Check the remote device for TimeZone/Date/time
You could expand you search criteria from 5 -10 minutes to all messages to this would give a better idea what’s going on.
With Docker ensure you have the Time Zone configured in Graylog container
EX:
environment:
# Container time Zone
- TZ=America/Chicago
Thank you. This was my first thought, but if I expand the search to ‘all’, I don’t see any, so I don’t think this is a time zone issue. Also when I put raw text by telnetting the port I can see my text. This is very strange and I have never seen this before in my v3/bare-metal environments
Yes and when I use nc localhost 5140 and put some text, it can be seen on the UI immediately. Also, I can see GL processing in/out logs, but nothing is displayed in the stream. I also can see how many messages are processed per second… I’m sure this is something trivial, but I can’t figure out
Is an extractor set or have stream rules been set?
Is the Message processors order correct (System/Configurations/Message Processors Configuration)?
First, Message Filter Chain, then Pipeline Processor
Correct me if I’m wrong,
When using Rsyslog you see messages coming in but unable to see them on GUI ( delay) ?
But when you run the command nc you can see those message in the GUI immediately?
If this is correct can you display your full Rsyslog config? if not have you tried a different log shipper?
Nxlog is not bad for testing purposes on Linux servers.
Yes, this is the case. The rsyslog.conf is nothing fancy, but maybe you are right, this is something mysteriously wrong with it… I may try replacing it with syslog-ng which is much closer to my
action(type="omfwd"
# # An on-disk queue is created for this action. If the remote host is
# # down, messages are spooled to disk and sent when it is up again.
queue.filename="fwdRule1" # unique name prefix for spool files
queue.maxdiskspace="1g" # 1gb space limit (use as much as possible)
queue.saveonshutdown="on" # save messages to disk on shutdown
queue.type="LinkedList" # run asynchronously
action.resumeRetryCount="-1" # infinite retries if host is down
# # Remote Logging (we use TCP for reliable delivery)
# # remote_host is: name/ip, e.g. 192.168.0.1, port optional e.g. 10514
Target="localhost" Port="5140" Protocol="tcp")
so when the above config sends logs to GL it shown in the in/out counter and in the stream counter, but cannot be searched
when I nc localhost 5140 and put some text, it is shown immediately
Another example why should I first remove rsyslog and install syslog-ng… after replacing rsyslog with syslog-ng and configure properly, it is working perfectly… maybe this new kind of config
Target="localhost" Port="5140" Protocol="tcp"
is working differently than the old one
*.*@8.8.8.8:5140
Thanks for the help and please evangelize syslog-ng because it is much more elegant and the config is much more readable for decades
UPDATE
It was not syslog-ng /rsyslog related because after a few minutes, I had the same situation (logs coming and counting, but nothing is displayed). I also found the process buffer was 100% full… I suppose this is some kind of docker internal trick / limit, but I have no time to play with it, so I consider it is a bad idea to try using GL in containers. At lest to me
L: