Graylog Sidecar on Rocky Linux is shipping 3 out of 4 logs

Graylog Central (peer support)
,
1

Before you post: Your responses to these questions will help the community help you. Please complete this template if you’re asking a support question.
Don’t forget to select tags to help index your topic!

1. Describe your incident:
I upgraded from CentOS 6.x to Rocky Linux 8.6. I installed a new version Graylog-Sidecar and started shipping logs. I can see 3 of the 4 logs in Graylog. But I cannot see the 4th log. This is my incident.

2. Describe your environment:

  • OS Information:
    → Rocky Linux 8.6
  • Package Version:
    → Graylog 4.
  • Service logs, configurations, and environment variables:
    No errors in service logs. All looks normal.

3. What steps have you already taken to try and solve the problem?
So far I have restarted the sidecar on the client. I have also renamed the target logfiles in the Graylog configuration. Restarted sidecar. Still no logs. However, yesterday after playing with the sidecar host, I saw a spike of logs at that time. Then log flow stopped again.

4. How can the community help?

Are there other ways to debug this strange issue? Has anyone experienced the same problem or similar?

Helpful Posting Tips: Tips for Posting Questions that Get Answers [Hold down CTRL and link on link to open tips documents in a separate tab]

2

Hello @ofentselogger

See If I understand this correct. On your node with the GL-Sidecar you have 4 logs that gets shipped BUT on Graylog-Server you only see 3 of those 4 log?

Check Date/Time , Time zone on Graylog Server and all clients sending logs.
Check Elasticsearch Logs
Check MongoDb Logs
Check Graylog Sidecar logs

Check if Graylog is in Debugging mode, if not you can enable as the following:

check file /etc/graylog/server/log4j2.xml

This may give a better idea how-to.

SUPPLYING EXTERNAL LOGGING CONFIGURATION

3

Thank you @gsmith

These pointers are perfect for me. Is there a means to get the sidecar logs into debug mode? The sidecar logs just show filebeat start and filebeat stop and an occasional failure to report to Graylog server yesterday due to a network issue.

I am running:
Graylog 4.2.9
filebeat 7.10.0-1
graylog-sidecar 1.2.0-1

4

Hello,

Not that I know of, couple areas I do look at when i have issue are:

  • Graylog-sidecar Log file /var/log/graylog-sidecar/
  • journalctl -xeu graylog-sidecar
  • Graylog GUI in Sidecar Status. This is located under Overview and click on the Side shown.

This statement indicates there is a commination problem between Client & Server.
Check list:

  • Firewall/s
  • Apparmor/SElinux
  • IPtables/Firewalld
  • Graylog-sidecar Configuration is incorrect
5

Hi @gsmith
I thank you for all the diagnostic pointers. I have used some of these pointers to debug the issue. Luckily I had a 2nd client I collect logs from. It worked fine. So I was able to make back to back comparisons in order to debug the issue. It seems like the configuration on server side had a fault. I do not know the issue but there may have been a typo.
Now I can collect the logs I need. Thank you.

1 Like
6

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.