Logs are not shipped to graylog4.3.2 by filebeat

[mina]

Before you post: Your responses to these questions will help the community help you. Please complete this template if you’re asking a support question.
Don’t forget to select tags to help index your topic!

1. Describe your incident:
i have installed sidecar1.1.0 and filebeat 7.16.0 in my vm successfully and i can see running status of sidecare in my graylog which is running in another vm . i have defined beat input with port 5044 as well
i set configuration to ship log from /var/log/*log for test but there is no log coming to graylog due to in filebeat log there is no error and i have activated debug i do not know filebeat is not sending log or it is sending but graylog can not process it ? i run tcpdump

2. Describe your environment:

• OS Information:
  OS : Ubuntu 18.04.6 LTS ----which filebeat and sidecare have been installed
  i run graylog is based on container
• Package Version:

elasticsearch:7.14.1
graylog:4.3.2
mongo:4.4.10

• Service logs, configurations, and environment variables:
  filebeat log as follows:
  2021-12-14T15:23:47.300+0100 DEBUG [input.filestream] filestream/filestream.go:131 End of file reached: /var/log/ubuntu-advantage-timer.log; Backoff now. {“id”: “204AB73A26FB6EDC”, “source”: “filestream::.global::native::2440-2049”, “path”: “/var/log/ubuntu-advantage-timer.log”, “state-id”: “native::2440-2049”}
  2021-12-14T15:23:47.300+0100 DEBUG [input.filestream] filestream/filestream.go:131 End of file reached: /var/log/cloud-init-output.log; Backoff now. {“id”: “204AB73A26FB6EDC”, “source”: “filestream::.global::native::63407-2049”, “path”: “/var/log/cloud-init-output.log”, “state-id”: “native::63407-2049”}
  2021-12-14T15:23:47.300+0100 DEBUG [input.filestream] filestream/filestream.go:131 End of file reached: /var/log/kern.log; Backoff now. {“id”: “204AB73A26FB6EDC”, “source”: “filestream::.global::native::63460-2049”, “path”: “/var/log/kern.log”, “state-id”: “native::63460-2049”}
  2021-12-14T15:23:47.300+0100 DEBUG [input.filestream] filestream/filestream.go:131 End of file reached: /var/log/ubuntu-advantage.log; Backoff now. {“id”: “204AB73A26FB6EDC”, “source”: “filestream::.global::native::1990-2049”, “path”: “/var/log/ubuntu-advantage.log”, “state-id”: “native::1990-2049”}
  2021-12-14T15:23:47.301+0100 DEBUG [processors] processing/processors.go:203 Publish event: {
  “@timestamp”: “2021-12-14T14:23:47.300Z”,
  “@metadata”: {
  “beat”: “filebeat”,
  “type”: “_doc”,
  “version”: “7.16.0”
  },
  “host”: {
  “name”: “aqnodets1”
  },
  “agent”: {
  “name”: “aqnodets1”,
  “type”: “filebeat”,
  “version”: “7.16.0”,
  “hostname”: “aqnodets1”,
  “ephemeral_id”: “2ee4b2a6-ce27-42fd-9d26-17ad8fc94de8”,
  “id”: “20869de9-df73-4b48-b4d7-853f59cb527a”
  },
  “log”: {
  “file”: {
  “path”: “/var/log/mina.log”
  },
  “offset”: 246
  },
  “message”: “hi mina”,
  “input”: {
  “type”: “filestream”
  },
  “collector_node_id”: “aqnodets1”,
  “gl2_source_collector”: “c51aa97d-2226-42c3-9e32-cf48a1ff67d4”,
  “ecs”: {
  “version”: “1.12.0”
  }
  }
  2021-12-14T15:23:47.301+0100 DEBUG [input.filestream] filestream/filestream.go:131 End of file reached: /var/log/mina.log; Backoff now. {“id”: “204AB73A26FB6EDC”, “source”: “filestream::.global::native::71611-2049”, “path”: “/var/log/mina.log”, “state-id”: “native::71611-2049”}
  2021-12-14T15:23:47.976+0100 DEBUG [input.filestream] filestream/filestream.go:131 End of file reached: /var/log/mail.log; Backoff now. {“id”: “204AB73A26FB6EDC”, “source”: “filestream::.global::native::62181-2049”, “path”: “/var/log/mail.log”, “state-id”: “native::62181-2049”}
  2021-12-14T15:23:47.976+0100 DEBUG [input.filestream] filestream/filestream.go:131 End of file reached: /var/log/fail2ban.log; Backoff now. {“id”: “204AB73A26FB6EDC”, “source”: “filestream::.global::native::61415-2049”, “path”: “/var/log/fail2ban.log”, “state-id”: “native::61415-2049”}
  2021-12-14T15:23:48.305+0100 DEBUG [input.filestream] filestream/filestream.go:131 End of file reached: /var/log/mina.log; Backoff now. {“id”: “204AB73A26FB6EDC”, “source”: “filestream::.global::native::71611-2049”, “path”: “/var/log/mina.log”, “state-id”: “native::71611-2049”}
  2021-12-14T15:23:48.306+0100 DEBUG [logstash] logstash/async.go:172 1 events out of 1 events sent to logstash host 192.168.33.106:5044. Continue sending
  2021-12-14T15:23:48.308+0100 DEBUG [publisher] memqueue/ackloop.go:160 ackloop: receive ack [60: 0, 1]
  2021-12-14T15:23:48.308+0100 DEBUG [publisher] memqueue/eventloop.go:535 broker ACK events: count=1, start-seq=3, end-seq=3

2021-12-14T15:23:48.308+0100 DEBUG [acker] beater/acker.go:64 stateless ack {“count”: 1}
2021-12-14T15:23:48.308+0100 DEBUG [publisher] memqueue/ackloop.go:128 ackloop: return ack to broker loop:1
2021-12-14T15:23:48.308+0100 DEBUG [publisher] memqueue/ackloop.go:131 ackloop: done send ack
2021-12-14T15:23:48.318+0100 DEBUG [input.filestream] filestream/filestream.go:131 End of file reached: /var/log/alternatives.log; Backoff now. {“id”: “204AB73A26FB6EDC”, “source”: “filestream::.global::native::71572-2049”, “path”: “/var/log/alternatives.log”, “state-id”: “native::71572-2049”}
2021-12-14T15:23:48.319+0100 DEBUG [input.filestream] filestream/filestream.go:131 End of file reached: /var/log/dpkg.log; Backoff now. {“id”: “204AB73A26FB6EDC”, “source”: “filestream::.global::native::63335-2049”, “path”: “/var/log/dpkg.log”, “state-id”: “native::63335-2049”}
2021-12-14T15:23:48.319+0100 DEBUG [input.filestream] filestream/filestream.go:131 End of file reached: /var/log/apport.log; Backoff now. {“id”: “204AB73A26FB6EDC”, “source”: “filestream::.global::native::71618-2049”, “path”: "/var/log/apport.

tcpdump:
Graylog IP adrs :192.168.33.106
server that farebeat is installed on it :192.168.33.104
Running as user “root” and group “root”. This could be dangerous.
1 0.000000 192.168.33.104 → 192.168.33.106 TCP 66 40004 → 5044 [ACK] Seq=1 Ack=1 Win=502 Len=0 TSval=3600304295 TSecr=2683767161
2 0.000429 192.168.33.106 → 192.168.33.104 TCP 66 [TCP ACKed unseen segment] 5044 → 40004 [ACK] Seq=1 Ack=2 Win=17992 Len=0 TSval=2683782265 TSecr=3600243955
3 15.104970 192.168.33.104 → 192.168.33.106 TCP 66 [TCP Dup ACK 1#1] 40004 → 5044 [ACK] Seq=1 Ack=1 Win=502 Len=0 TSval=3600319400 TSecr=2683782265
4 15.105357 192.168.33.106 → 192.168.33.104 TCP 66 [TCP Dup ACK 2#1] [TCP ACKed unseen segment] 5044 → 40004 [ACK] Seq=1 Ack=2 Win=17992 Len=0 TSval=2683797370 TSecr=3600243955
5 30.208433 192.168.33.104 → 192.168.33.106 TCP 66 [TCP Dup ACK 1#2] 40004 → 5044 [ACK] Seq=1 Ack=1 Win=502 Len=0 TSval=3600334503 TSecr=2683797370
6 30.209046 192.168.33.106 → 192.168.33.104 TCP 66 [TCP Dup ACK 2#2] [TCP ACKed unseen segment] 5044 → 40004 [ACK] Seq=1 Ack=2 Win=17992 Len=0 TSval=2683812474 TSecr=3600243955
7 45.313469 192.168.33.104 → 192.168.33.106 TCP 66 [TCP Dup ACK 1#3] 40004 → 5044 [ACK] Seq=1 Ack=1 Win=502 Len=0 TSval=3600349608 TSecr=2683812474
8 45.313770 192.168.33.106 → 192.168.33.104 TCP 66 [TCP Dup ACK 2#3] [TCP ACKed unseen segment] 5044 → 40004 [ACK] Seq=1 Ack=2 Win=17992 Len=0 TSval=2683827579 TSecr=3600243955
9 60.422802 192.168.33.104 → 192.168.33.106 TCP 66 [TCP Dup ACK 1#4] 40004 → 5044 [ACK] Seq=1 Ack=1 Win=502 Len=0 TSval=3600364717 TSecr=2683827579
10 60.423053 192.168.33.106 → 192.168.33.104 TCP 66 [TCP Dup ACK 2#4] [TCP ACKed unseen segment] 5044 → 40004 [ACK] Seq=1 Ack=2 Win=17992 Len=0 TSval=2683842688 TSecr=3600243955
11 64.736825 192.168.33.104 → 192.168.33.106 TCP 425 [TCP Previous segment not captured] 40004 → 5044 [PSH, ACK] Seq=2 Ack=1 Win=502 Len=359 TSval=3600369031 TSecr=2683842688
12 64.737639 192.168.33.106 → 192.168.33.104 TCP 72 [TCP ACKed unseen segment] 5044 → 40004 [PSH, ACK] Seq=1 Ack=361 Win=18260 Len=6 TSval=2683847003 TSecr=3600369031
13 64.737652 192.168.33.104 → 192.168.33.106 TCP 66 40004 → 5044 [ACK] Seq=361 Ack=7 Win=502 Len=0 TSval=3600369032 TSecr=2683847003
14 79.872205 192.168.33.104 → 192.168.33.106 TCP 66 [TCP Keep-Alive] 40004 → 5044 [ACK] Seq=360 Ack=7 Win=502 Len=0 TSval=3600384166 TSecr=2683847003
15 79.872709 192.168.33.106 → 192.168.33.104 TCP 66 [TCP Keep-Alive ACK] 5044 → 40004 [ACK] Seq=7 Ack=361 Win=18260 Len=0 TSval=2683862138 TSecr=3600369032

3. What steps have you already taken to try and solve the problem?
check log of filebeat ,graylog , run tcpdump

4. How can the community help?

Helpful Posting Tips: Tips for Posting Questions that Get Answers [Hold down CTRL and link on link to open tips documents in a separate tab]

[tmacgbay]

Could you post the filebeat configuration you are applying from Graylog to the ubuntu machine?

Also - please use the forum tools, specifically </> and apply it to the code and log files posted so that it formatted readable. Lastly of note, the most recent supported version of Elasticsearch is 7.10

Here is an example of my working linux based filebeat configuration with formatting goodness… note that these files are sensitive to spacing:

# Needed for Graylog
fields_under_root: true
fields.collector_node_id: ${sidecar.nodeName}
fields.gl2_source_collector: ${sidecar.nodeId}

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/*.log
  ignore_older: 2h
  tags:
    - linux
#############   field marker for shunting with stream rules
  fields:
    Osys: linux

output.logstash:
   hosts: 
   - ${user.BeatsInput}

path:
  data: /var/cache/graylog-sidecar/filebeat/data
  logs: /var/log/graylog-sidecar

[mina]

@tmacgbay

Hi @tmacgbay, thanks for reply , please find filebeat configuration as follow :

# Needed for Graylog
fields_under_root: true
fields.collector_node_id: ${sidecar.nodeName}
fields.gl2_source_collector: ${sidecar.nodeId}

filebeat.inputs:
- type: filestream
  enabled: true
  paths:
    - /var/log/*.log
  #close_removed: false
  #clean_removed : false

output.logstash:
   hosts: ["192.168.33.106:5044"]
path:
  data: /var/lib/graylog-sidecar/collectors/filebeat/data
  logs: /var/lib/graylog-sidecar/collectors/filebeat/log
  
logging.level: debug

[tmacgbay]

Your beats version is also newer than the 7.10 supported Elastic… It is probably not the issue but I notice that you are using “filestream” rather than “log” in your configuration… something to examine/test. Check to see if Sidecar is managing the filebeat on your Ubuntu machine… The sidecar service should be running and that will start up the filebeat process for you…if you have filebeat running as a service then it won’t report back properly to sidecar.

$ sudo systemctl status graylog-sidecar
● graylog-sidecar.service - Wrapper service for Graylog controlled collector
     Loaded: loaded (/etc/systemd/system/graylog-sidecar.service; enabled; vendor preset: enabled)
     Active: active (running) since Mon 2021-12-13 08:21:29 EST; 2 days ago
   Main PID: 638 (graylog-sidecar)
      Tasks: 22 (limit: 6934)
     Memory: 43.0M
     CGroup: /system.slice/graylog-sidecar.service
             ├─ 638 /usr/bin/graylog-sidecar
             └─1657 /usr/share/filebeat/bin/filebeat -c /var/lib/graylog-sidecar/generated/filebeat.conf

Dec 13 08:21:29 CMG-NET01 systemd[1]: Started Wrapper service for Graylog controlled collector.
Dec 13 08:21:32 CMG-NET01 graylog-sidecar[638]: time="2021-12-13T08:21:32-05:00" level=info msg="Using node-id: e05a3f00-6e4a-4b66-8995-343a59>
Dec 13 08:21:32 CMG-NET01 graylog-sidecar[638]: time="2021-12-13T08:21:32-05:00" level=info msg="No node name was configured, falling back to >
Dec 13 08:21:32 CMG-NET01 graylog-sidecar[638]: time="2021-12-13T08:21:32-05:00" level=info msg="Starting signal distributor"
Dec 13 08:21:42 CMG-NET01 graylog-sidecar[638]: time="2021-12-13T08:21:42-05:00" level=info msg="Adding process runner for: filebeat"
Dec 13 08:21:42 CMG-NET01 graylog-sidecar[638]: time="2021-12-13T08:21:42-05:00" level=info msg="[filebeat] Configuration change detected, rew>
Dec 13 08:21:54 CMG-NET01 graylog-sidecar[638]: time="2021-12-13T08:21:54-05:00" level=info msg="[filebeat] Starting (exec driver)"

$ ps -aef | grep beat
bobby   1657     638  0 Dec13 ?        00:01:41 /usr/share/filebeat/bin/filebeat -c /var/lib/graylog-sidecar/generated/filebeat.conf

What is your sidecar.yml setup look like? Here is mine:
(note the command that removes comments)

/etc/graylog/sidecar$ cat /etc/graylog/sidecar/sidecar.yml | egrep -v "^\s*(#|$)"
server_url: "http://LittleGray:9000/api/"
server_api_token: "so stinkin secret"
node_id: "file:/etc/graylog/sidecar/node-id"
update_interval: 10
tls_skip_verify: true
cache_path: "/var/cache/graylog-sidecar"
log_path: "/var/log/graylog-sidecar"
log_rotate_max_file_size: "10MiB"
log_rotate_keep_files: 10
collector_configuration_directory: "/var/lib/graylog-sidecar/generated"
collector_binaries_accesslist:
  - "/usr/share/filebeat/bin/filebeat"

[mina]

Hi @tmacgbay
please find required log as follows:
in new version of filebeat log type is deprecated because of that i am using filestream

systemctl status graylog-sidecar
● graylog-sidecar.service - Wrapper service for Graylog controlled collector
   Loaded: loaded (/etc/systemd/system/graylog-sidecar.service; enabled; vendor preset: enabled)
   Active: active (running) since Wed 2021-12-15 18:13:20 CET; 1min 44s ago
 Main PID: 22118 (graylog-sidecar)
    Tasks: 20 (limit: 4696)
   CGroup: /system.slice/graylog-sidecar.service
           ├─22118 /usr/bin/graylog-sidecar
           └─22171 /usr/share/filebeat/bin/filebeat -c /var/lib/graylog-sidecar/generated/filebeat.conf

Dec 15 18:13:20 aqnodets1 systemd[1]: Started Wrapper service for Graylog controlled collector.
Dec 15 18:13:20 aqnodets1 graylog-sidecar[22118]: time="2021-12-15T18:13:20+01:00" level=info msg="Using node-id: c51aa97d-2226-42c3-9e32-cf48a1ff67d4"
Dec 15 18:13:20 aqnodets1 graylog-sidecar[22118]: time="2021-12-15T18:13:20+01:00" level=info msg="No node name was configured, falling back to hostname"
Dec 15 18:13:20 aqnodets1 graylog-sidecar[22118]: time="2021-12-15T18:13:20+01:00" level=info msg="Starting signal distributor"
Dec 15 18:13:30 aqnodets1 graylog-sidecar[22118]: time="2021-12-15T18:13:30+01:00" level=info msg="Adding process runner for: filebeat"
Dec 15 18:13:30 aqnodets1 graylog-sidecar[22118]: time="2021-12-15T18:13:30+01:00" level=info msg="[filebeat] Configuration change detected, rewriting configuration file."
Dec 15 18:13:31 aqnodets1 graylog-sidecar[22118]: time="2021-12-15T18:13:31+01:00" level=info msg="[filebeat] Starting (exec driver)"
-----------------------------------------------------------------------------
 $ cat /var/lib/graylog-sidecar/generated/filebeat.conf
# Needed for Graylog
fields_under_root: true
fields.collector_node_id: aqnodets1
fields.gl2_source_collector: c51aa97d-2226-42c3-9e32-cf48a1ff67d4

filebeat.inputs:
- type: filestream
  enabled: true
  paths:
    - /var/log/*.log
  #close_removed: false
  #clean_removed : false

output.logstash:
   hosts: ["192.168.33.106:5044"]
path:
  data: /var/lib/graylog-sidecar/collectors/filebeat/data
  logs: /var/lib/graylog-sidecar/collectors/filebeat/log

logging.level: debug
---------------------------------------------------------------------------
ps -aef | grep beat
root     22171 22118  0 18:13 ?        00:00:00 /usr/share/filebeat/bin/filebeat -c /var/lib/graylog-sidecar/generated/filebeat.conf
root     22300  7262  0 18:14 pts/0    00:00:00 tail -f filebeat
root     22585 21239  0 18:17 pts/3    00:00:00 grep beat
--------------------------------------------------------------------------------------------
 $ cat /etc/graylog/sidecar/sidecar.yml | egrep -v "^\s*(#|$)"
server_url: "http://192.168.33.106:9000/api/"
server_api_token: "*********my token************"
tls_skip_verify: true

here i defined my input which

problem is that there is no error and i can not do troubleshooting whether filebeat is sending log or not ? if filebeat is sending why graylog can not process there is no error regarding can not process …
in filebeat log which i have already share harvesting is done but i do not know the steps after harvesting should be something that show us logs are sent or not ?
thanks in advance

[mina]

here you can check sidecar is running

[mina]

from here i can restart filebeat and it is working fine

[mina]

here you can see collector status :

sorry i am new user in graylog can not send all images in one message

[tmacgbay]

@mina

new version of filebeat log type is deprecated because of that i am using filestream

I get that… but since Graylog only supports up to Elasticsearch 7.10 and you are using Elasticsearch version 7.14 and their filebeat 7.16 there is a very small chance that using filestream may be your issue. I think it is too far removed from where your issue lies but I would hate to skip correcting it just to find out it conflicted with sidecar or something else. I don’t see any other issues in your configuration…

You checked the filebeat and sidecar logs at /var/lib/graylog-sidecar/collectors/filebeat/log? fell free to post formatted snippets.

[gsmith]

Hello

Just chiming in. I have to agree with @tmacgbay your version maybe a culprit.

@mina Can I ask what log files have you checked that would pertain to this issue?
Have you checked the firewall on either Graylog or your remote device?
Did you execute a tcpdump on your Graylog server?

Last Question I see this. Please look at the IP Address marked in red.

Then I see this in another post

@mina

output.logstash:
   hosts: ["192.168.33.106:5044"]

These are two different networks. Unless you have Port forwarding.

EDIT: So your Graylog server is on a 192.168.33.0 and your Remote Node is on a 10.0.2.0.
Something seams not to be right. If your routing traffic from one subnet to another make sure your allowing traffic on port 5044 with it.

[mina]

ok today i will downgrade Elasticsearch to 7.10 to check it . kindly please find filebeat logs and graylog-sidecar logs as follows:

/var/lib/graylog-sidecar/collectors/filebeat/log/filebeat:
2021-12-16T07:33:50.318+0100    DEBUG   [input.filestream]      filestream/filestream.go:131    End of file reached: /var/log/ubuntu-advantage.log; Backoff now. {"id": "204AB73A26FB6EDC", "source": "filestream::.global::native::1990-2049", "path": "/var/log/ubuntu-advantage.log", "state-id": "native::1990-2049"}
2021-12-16T07:33:50.318+0100    DEBUG   [input.filestream]      filestream/filestream.go:131    End of file reached: /var/log/fail2ban.log; Backoff now. {"id": "204AB73A26FB6EDC", "source": "filestream::.global::native::61415-2049", "path": "/var/log/fail2ban.log", "state-id": "native::61415-2049"}
2021-12-16T07:33:50.318+0100    DEBUG   [processors]    processing/processors.go:203    Publish event: {
  "@timestamp": "2021-12-16T06:33:50.318Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "_doc",
    "version": "7.16.0"
  },
  "log": {
    "offset": 278,
    "file": {
      "path": "/var/log/mina.log"
    }
  },
  "message": "hi mina",
  "input": {
    "type": "filestream"
  },
  "gl2_source_collector": "c51aa97d-2226-42c3-9e32-cf48a1ff67d4",
  "collector_node_id": "aqnodets1",
  "ecs": {
    "version": "1.12.0"
  },
  "host": {
    "name": "aqnodets1"
  },
  "agent": {
    "ephemeral_id": "3af42661-306e-4d4a-bf70-86f23d691267",
    "id": "20869de9-df73-4b48-b4d7-853f59cb527a",
    "name": "aqnodets1",
    "type": "filebeat",
    "version": "7.16.0",
    "hostname": "aqnodets1"
  }
}
2021-12-16T07:33:50.318+0100    DEBUG   [input.filestream]      filestream/filestream.go:131    End of file reached: /var/log/mina.log; Backoff now.     {"id": "204AB73A26FB6EDC", "source": "filestream::.global::native::71611-2049", "path": "/var/log/mina.log", "state-id": "native::71611-2049"}
2021-12-16T07:33:51.061+0100    INFO    [file_watcher]  filestream/fswatch.go:137       Start next scan
2021-12-16T07:33:51.062+0100    DEBUG   [file_watcher]  filestream/fswatch.go:204       Found 13 paths
2021-12-16T07:33:51.062+0100    DEBUG   [input.filestream]      filestream/prospector.go:164    File /var/log/mina.log has been updated  {"id": "204AB73A26FB6EDC", "prospector": "file_prospector", "operation": "write", "source_name": "native::71611-2049", "os_id": "71611-2049", "new_path": "/var/log/mina.log", "old_path": "/var/log/mina.log"}
2021-12-16T07:33:51.062+0100    DEBUG   [input.filestream]      input-logfile/harvester.go:145  Starting harvester for file     {"id": "204AB73A26FB6EDC", "source": "filestream::.global::native::71611-2049"}
2021-12-16T07:33:51.062+0100    DEBUG   [input.filestream]      input-logfile/harvester.go:181  Stopped harvester for file      {"id": "204AB73A26FB6EDC", "source": "filestream::.global::native::71611-2049"}
2021-12-16T07:33:51.319+0100    DEBUG   [input.filestream]      filestream/filestream.go:131    End of file reached: /var/log/mina.log; Backoff now.     {"id": "204AB73A26FB6EDC", "source": "filestream::.global::native::71611-2049", "path": "/var/log/mina.log", "state-id": "native::71611-2049"}
2021-12-16T07:33:51.319+0100    DEBUG   [logstash]      logstash/async.go:172   1 events out of 1 events sent to logstash host 192.168.33.106:5044. Continue sending
2021-12-16T07:33:51.320+0100    DEBUG   [publisher]     memqueue/ackloop.go:160 ackloop: receive ack [146: 0, 1]
2021-12-16T07:33:51.320+0100    DEBUG   [publisher]     memqueue/eventloop.go:535       broker ACK events: count=1, start-seq=4, end-seq=4

2021-12-16T07:33:51.320+0100    DEBUG   [acker] beater/acker.go:64      stateless ack   {"count": 1}
2021-12-16T07:33:51.320+0100    DEBUG   [publisher]     memqueue/ackloop.go:128 ackloop: return ack to broker loop:1
2021-12-16T07:33:51.320+0100    DEBUG   [publisher]     memqueue/ackloop.go:131 ackloop:  done send ack
2021-12-16T07:33:53.323+0100    DEBUG   [input.filestream]      filestream/filestream.go:131    End of file reached: /var/log/mina.log; Backoff now.     {"id": "204AB73A26FB6EDC", "source": "filestream::.global::native::71611-2049", "path": "/var/log/mina.log", "state-id": "native::71611-2049"}
2021-12-16T07:33:55.031+0100    DEBUG   [input.filestream]      filestream/filestream.go:131    End of file reached: /var/log/ubuntu-advantage-timer.log; Backoff now.   {"id": "204AB73A26FB6EDC", "source": "filestream::.global::native::2440-2049", "path": "/var/log/ubuntu-advantage-timer.log", "state-id": "native::2440-2049"}
2021-12-16T07:33:55.331+0100    DEBUG   [input.filestream]      filestream/filestream.go:131    End of file reached: /var/log/kern.log; Backoff now.     {"id": "204AB73A26FB6EDC", "source": "filestream::.global::native::63460-2049", "path": "/var/log/kern.log", "state-id": "native::63460-2049"}

/var/log/graylog-sidecar $ cat sidecar.log
time="2021-12-15T18:13:19+01:00" level=info msg="Stopping signal distributor"
time="2021-12-15T18:13:19+01:00" level=info msg="[filebeat] Stopping"
time="2021-12-15T18:13:20+01:00" level=info msg="Starting signal distributor"
time="2021-12-15T18:13:30+01:00" level=info msg="Adding process runner for: filebeat"
time="2021-12-15T18:13:30+01:00" level=info msg="[filebeat] Configuration change detected, rewriting configuration file."
time="2021-12-15T18:13:31+01:00" level=info msg="[filebeat] Starting (exec driver)"

[mina]

Hi @gsmith ,
i am running graylog, elasticsearch and monogo db by docker because of that in Node details ip address of container is shown, i have exposed port 5044 . IP address of graylog,elastic,mongo is 192.168.33.106

CONTAINER ID   IMAGE                                                                          COMMAND                  CREATED        STATUS                    PORTS                                                                                                                                                                                                                                                                                                                                      NAMES
a8aea14fe647   /docker-images/graylog:latest-secure         "tini -- /docker-ent…"   41 hours ago   Up 41 hours (unhealthy)   192.168.33.106:80->80/tcp, 192.168.33.106:80->80/udp, 192.168.33.106:1514->1514/tcp, 192.168.33.106:1514->1514/udp, 192.168.33.106:5044->5044/tcp, 192.168.33.106:5044->5044/udp, 192.168.33.106:9000->9000/tcp, 192.168.33.106:9000->9000/udp, 192.168.33.106:12201-12202->12201-12202/tcp, 192.168.33.106:12201-12202->12201-12202/udp   graylog-67ab6474-dd3d-6857-31f9-d3c3e0d84498
8bb62de4bfe0   mongo                                                                          "docker-entrypoint.s…"   41 hours ago   Up 41 hours               192.168.33.106:27017->27017/tcp, 192.168.33.106:27017->27017/udp                                                                                                                                                                                                                                                                           mongo-67ab6474-dd3d-6857-31f9-d3c3e0d84498
01d21b5353a9   docker-images/elasticsearch:latest-secure   "/bin/tini -- /usr/l…"   41 hours ago   Up 41 hours               192.168.33.106:9200->9200/tcp, 192.168.33.106:9200->9200/udp, 192.168.33.106:9300->9300/tcp, 192.168.33.106:9300->9300/udp                                                                                                                                                                                                                 elasticsearch-67ab6474-dd3d-6857-31f9-d3c3e0d84498

please find tcp dump which i run in graylog server as follow:

$tcpdump -i eth1 port 5044 -vv 
tcpdump: listening on eth1, link-type EN10MB (Ethernet), capture size 262144 bytes
08:00:59.268033 IP (tos 0x0, ttl 64, id 28398, offset 0, flags [DF], proto TCP (6), length 52)
    192.168.33.104.36228 > b**.prd.vagrant.5044: Flags [.], cksum 0x2980 (correct), seq 1027534967, ack 2885675946, win 502, options [nop,nop,TS val 3672040902 ecr 470631384], length 0
08:00:59.270516 IP (tos 0x0, ttl 63, id 64817, offset 0, flags [DF], proto TCP (6), length 52)
    b**.prd.vagrant.5044 > 192.168.33.104.36228: Flags [.], cksum 0xc449 (incorrect -> 0x9865), seq 1, ack 1, win 52883, options [nop,nop,TS val 470646486 ecr 3672010564], length 0
08:01:07.317028 IP (tos 0x0, ttl 64, id 28399, offset 0, flags [DF], proto TCP (6), length 419)
    192.168.33.104.36228 > b**.prd.vagrant.5044: Flags [P.], cksum 0x5bc0 (correct), seq 1:368, ack 1, win 502, options [nop,nop,TS val 3672048955 ecr 470646486], length 367
08:01:07.317364 IP (tos 0x0, ttl 63, id 64818, offset 0, flags [DF], proto TCP (6), length 58)
    b**.prd.vagrant.5044 > 192.168.33.104.36228: Flags [P.], cksum 0xc44f (incorrect -> 0xaf3f), seq 1:7, ack 368, win 52883, options [nop,nop,TS val 470654533 ecr 3672048955], length 6
08:01:07.317540 IP (tos 0x0, ttl 64, id 28400, offset 0, flags [DF], proto TCP (6), length 52)
    192.168.33.104.36228 > b**.prd.vagrant.5044: Flags [.], cksum 0xae26 (correct), seq 368, ack 7, win 502, options [nop,nop,TS val 3672048956 ecr 470654533], length 0
08:01:22.545113 IP (tos 0x0, ttl 64, id 28401, offset 0, flags [DF], proto TCP (6), length 52)
    192.168.33.104.36228 > b**.prd.vagrant.5044: Flags [.], cksum 0x72a4 (correct), seq 367, ack 7, win 502, options [nop,nop,TS val 3672064191 ecr 470654533], length 0
08:01:22.545226 IP (tos 0x0, ttl 63, id 64819, offset 0, flags [DF], proto TCP (6), length 52)
    b**.prd.vagrant.5044 > 192.168.33.104.36228: Flags [.], cksum 0xc449 (incorrect -> 0xa60d), seq 7, ack 368, win 52883, options [nop,nop,TS val 470669760 ecr 3672048956], length 0
08:01:23.937813 IP (tos 0x0, ttl 64, id 28402, offset 0, flags [DF], proto TCP (6), length 476)
    192.168.33.104.36228 > b**.prd.vagrant.5044: Flags [P.], cksum 0x3b16 (correct), seq 368:792, ack 7, win 502, options [nop,nop,TS val 3672065584 ecr 470669760], length 424
08:01:23.938807 IP (tos 0x0, ttl 63, id 64820, offset 0, flags [DF], proto TCP (6), length 58)
    b**.prd.vagrant.5044 > 192.168.33.104.36228: Flags [P.], cksum 0xc44f (incorrect -> 0x2baf), seq 7:13, ack 792, win 52883, options [nop,nop,TS val 470671154 ecr 3672065584], length 6
08:01:23.939791 IP (tos 0x0, ttl 64, id 28403, offset 0, flags [DF], proto TCP (6), length 52)
    192.168.33.104.36228 > b**.prd.vagrant.5044: Flags [.], cksum 0x2a95 (correct), seq 792, ack 13, win 502, options [nop,nop,TS val 3672065586 ecr 470671154], length 0
^C
10 packets captured
10 packets received by filter
0 packets dropped by kernel
Note : b**.prd.vagrant ---> i have changed the name of graylog host name for security reason 

today i will downgrade elasticsearch version to 7.10 to check it based on @tmacgbay advise

[mina]

Dear @tmacgbay @gsmith thanks for your support i just have downgraded elasticsearch to 7.10 and all logs shipped to graylog successfully .

[tmacgbay]

Great to hear! Thanks for marking the answer too!

[gsmith]

Good Job , Thanks for posting your resolution.