Logs are coming, but the stream is empty

Hi,

This is a very basic config and for the last few hours, I’m trying to figure out why it is not working…

I’m trying to put together some docker-based Graylog instances using the “official” docker-compose config from here docker-compose/docker-compose.yml at main · Graylog2/docker-compose · GitHub

Everything is fine, containers are starting, I can log-in and after creating a tcp/syslog input and sending logs to there I can see in/out messages, but when I search logs arrived here I don’t see any

gl2_source_input:63330ca202478d03a678973c

I don’t see any errors in logs, and when I just simply nc’d to the input port (localhost:5140) I can see my message. The logs are forwarded from rsyslog.conf like this

Target=“localhost” Port=“5140” Protocol=“tcp”

It is a fedora35 machine with the latest updates

Any idea what piece is missing? Until now, I mainly used GL3 and w/o docker, but I have never seen something like this before

Thank you and sorry for the newbie question :slight_smile:

L:

Hello @vladx

Perhaps I can help, Normally when everything is working correct but you don’t see the logs on the search GUI it could mean that Date/Time, Time Zone is off.

First things to check are:

  1. System/Overview make sure the User: Web Browser: and Graylog Server: have sync’d date/time.
  2. Check the remote device for TimeZone/Date/time
  3. You could expand you search criteria from 5 -10 minutes to all messages to this would give a better idea what’s going on.
  4. With Docker ensure you have the Time Zone configured in Graylog container

EX:

environment:
      # Container time Zone
      - TZ=America/Chicago

Hi,

Thank you. This was my first thought, but if I expand the search to ‘all’, I don’t see any, so I don’t think this is a time zone issue. Also when I put raw text by telnetting the port I can see my text. This is very strange and I have never seen this before in my v3/bare-metal environments

L:

Hey @vladx ,

the GL input has been created correctly for TCP?

Yes and when I use nc localhost 5140 and put some text, it can be seen on the UI immediately. Also, I can see GL processing in/out logs, but nothing is displayed in the stream. I also can see how many messages are processed per second… I’m sure this is something trivial, but I can’t figure out

L:

okay, i understand.

The source input id is correct?

Is an extractor set or have stream rules been set?
Is the Message processors order correct (System/Configurations/Message Processors Configuration)?
First, Message Filter Chain, then Pipeline Processor

Hello,

Correct me if I’m wrong,
When using Rsyslog you see messages coming in but unable to see them on GUI ( delay) ?
But when you run the command nc you can see those message in the GUI immediately?
If this is correct can you display your full Rsyslog config? if not have you tried a different log shipper?

Nxlog is not bad for testing purposes on Linux servers.

Yes, this is the case. The rsyslog.conf is nothing fancy, but maybe you are right, this is something mysteriously wrong with it… I may try replacing it with syslog-ng which is much closer to my :heart: :slight_smile:

action(type="omfwd"
# # An on-disk queue is created for this action. If the remote host is
# # down, messages are spooled to disk and sent when it is up again.
queue.filename="fwdRule1"       # unique name prefix for spool files
queue.maxdiskspace="1g"         # 1gb space limit (use as much as possible)
queue.saveonshutdown="on"       # save messages to disk on shutdown
queue.type="LinkedList"         # run asynchronously
action.resumeRetryCount="-1"    # infinite retries if host is down
# # Remote Logging (we use TCP for reliable delivery)
# # remote_host is: name/ip, e.g. 192.168.0.1, port optional e.g. 10514
Target="localhost" Port="5140" Protocol="tcp")

so when the above config sends logs to GL it shown in the in/out counter and in the stream counter, but cannot be searched

when I nc localhost 5140 and put some text, it is shown immediately

L:

1 Like

Hello,

Just touching base here, not sure if this will help, but here is a configuration on my CentOS 7 using Rsyslog.

[root@graylog bin]# cat /etc/rsyslog.conf | egrep -v "^\s*(#|$)"
$ModLoad imuxsock # provides support for local system logging (e.g. via logger command)
$ModLoad imjournal # provides access to the systemd journal
$ModLoad imudp
$WorkDirectory /var/lib/rsyslog
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
$IncludeConfig /etc/rsyslog.d/*.conf
$OmitLocalLogging on
$IMJournalStateFile imjournal.state
kern.debug /var/log/firewall.log
*.info;mail.none;authpriv.none;cron.none                /var/log/messages
authpriv.*                                              /var/log/secure
mail.*                                                  -/var/log/maillog
cron.*                                                  /var/log/cron
*.emerg                                                 :omusrmsg:*
uucp,news.crit                                          /var/log/spooler
local7.*                                                /var/log/boot.log
*.*@8.8.8.8:5140
[root@graylog bin]#

Another example why should I first remove rsyslog and install syslog-ng… after replacing rsyslog with syslog-ng and configure properly, it is working perfectly… maybe this new kind of config

Target="localhost" Port="5140" Protocol="tcp"

is working differently than the old one

*.*@8.8.8.8:5140

Thanks for the help and please evangelize syslog-ng because it is much more elegant and the config is much more readable for decades :slight_smile:

source s_tcp514_legacy {
    network(
        keep_hostname(yes)
        keep_timestamp(yes)
        use_fqdn(yes)
        time_zone("Europe/Budapest")
        ip("0.0.0.0")
        port(514)
        transport (tcp)
        max-connections (10000)
        dynamic-window-size (1000000)
    );
};

destination d_graylog {
   syslog(
        "localhost"
        port(5140)
        suppress(0)
        #throttle(200)
        frac_digits(0)
        log_fifo_size(500000)
        time_zone("Europe/Budapest")
        persist_name("graylog_5140")
        transport(tcp)
          disk-buffer(
            #mem-buf-length(10000)
            disk-buf-size(50000000000)
            reliable(yes)
            dir("/HDD4TB/syslog/diskbuffer")
          )
        );
};

log {
  source (s_sys);
  filter (f_nodebug);
  destination (d_graylog);
  flags (flow-control);
};

L:

1 Like

UPDATE
It was not syslog-ng /rsyslog related because after a few minutes, I had the same situation (logs coming and counting, but nothing is displayed). I also found the process buffer was 100% full… I suppose this is some kind of docker internal trick / limit, but I have no time to play with it, so I consider it is a bad idea to try using GL in containers. At lest to me :slight_smile:
L:

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.