Graylog docker with syslog-ng via tcp uses internal docker ip

Graylog Central (peer support)
1

I am using docker-compose to deploy graylog:
IP from graylog host starts with 10.128
IP from graylog-docker on graylog host has internal docker ip 192.168.64.3

[docker-compose.yml]
version: '2.2'
services:

      graylog:
        image: graylog/graylog:4.0.1
        hostname: graylog1-docker
        container_name: graylog
        restart: always
        environment:
          - GRAYLOG_HTTP_ENABLE_TLS=true
          - GRAYLOG_HTTP_TLS_CERT_FILE=/etc/ssl/certs/graylog-node1.pem
          - GRAYLOG_HTTP_TLS_KEY_FILE=/etc/ssl/private/graylog-node1.key
          - GRAYLOG_HTTP_EXTERNAL_URI=https://graylog-node1-fqdn:9000/
          - GRAYLOG_HTTP_PUBLISH_URI=https://graylog-node1-fqdn:9000/
          - GRAYLOG_ELASTICSEARCH_HOSTS=3 nodes defined and working
          - GRAYLOG_MONGODB_URI=replicaset defined and working
          
        networks:
          - graylog
        volumes:
          - /etc/graylog/graylog_journal:/usr/share/graylog/data/journal
        ports:
          # HTTPS
          - 9000:9000
          # Syslog TCP
          - 5140:5140
          - 5141:5141
          - 5142:5142
          - 5143:5143
          - 5144:5144
          - 5145:5145
          - 5146:5146
          - 5147:5147
          - 5148:5148
          - 5149:5149
          # Syslog UDP
          - 5140:5140/udp
          - 514:514/udp
          - 5141:5141/udp
          - 5142:5142/udp
          - 5143:5143/udp
          - 5144:5144/udp
          - 5145:5145/udp
          - 5146:5146/udp
          - 5147:5147/udp
          - 5148:5148/udp
          - 5149:5149/udp
          # GELF TCP
          - 12201:12201
          # GELF UDP
          - 12201:12201/udp
[/docker-compose.yml]

I configured a external vm and configured syslog-ng to send to graylog.
Graylog collects data successfully but it is very slowly.
On the syslog-ng host I see the following logs very often:
systemd-journal[337]: Forwarding to syslog missed 12 messages.
So I mad a tcp dump on the graylog host where the docker runs.
[tcpdump]

11:32:52.154627 IP (tos 0x0, ttl 63, id 18494, offset 0, flags [DF], proto TCP (6), length 52)
syslog-ng.fqdn.36854 > 192.168.64.3.5141: Flags [.], cksum 0x65f4 (correct), ack 1, win 229, options [nop,nop,TS val 5745664 ecr 1220618454], length 0
11:32:52.154766 IP (tos 0x0, ttl 64, id 900, offset 0, flags [DF], proto TCP (6), length 52)
192.168.64.3.5141 > syslog-ng.fqdn.36854: Flags [.], cksum 0x8be1 (incorrect → 0x49ea), ack 2916, win 19064, options [nop,nop,TS val 1220678614 ecr 5084031], length 0
11:32:52.154774 IP (tos 0x0, ttl 64, id 900, offset 0, flags [DF], proto TCP (6), length 52)
192.168.64.3.5141 > syslog-ng.fqdn.36854: Flags [.], cksum 0x8be1 (incorrect → 0x49ea), ack 2916, win 19064, options [nop,nop,TS val 1220678614 ecr 5084031], length 0
11:32:52.154789 IP (tos 0x0, ttl 63, id 900, offset 0, flags [DF], proto TCP (6), length 52)
graylog-fqdn.5141 > syslog-ng.fqdn.36854: Flags [.], cksum 0x168f (incorrect → 0xbf3c), ack 2916, win 19064, options [nop,nop,TS val 1220678614 ecr 5084031], length 0
11:33:38.681019 IP (tos 0x0, ttl 64, id 11487, offset 0, flags [DF], proto TCP (6), length 52)
syslog-ng.fqdn.45521 > graylog-fqdn.5141: Flags [.], cksum 0xc16c (correct), ack 1, win 229, options [nop,nop,TS val 2672946176 ecr 1115955523], length 0
11:33:38.681046 IP (tos 0x0, ttl 63, id 11487, offset 0, flags [DF], proto TCP (6), length 52)
syslog-ng.fqdn.45521 > 192.168.64.3.5141: Flags [.], cksum 0x4c1a (correct), ack 1, win 229, options [nop,nop,TS val 2672946176 ecr 1115955523], length 0
[/tcpdump]

So the problem is that some packets are dropped because the syslog-ng host tries to send to 192.168 which is not working as the syslog-ng need to write to 10.128 or the graylog-fqdn.

My question is:
How can I correctly configure the graylog-docker-compose to use the right ip address and not the internal docker ip address. I assume that the tcp connections breaks all the time.

The configured Input in graylog looks like this:
> allow_override_date:

 true
bind_address:
 0.0.0.0
expand_structured_data:
 false
force_rdns:
 false
max_message_size:
 2097152
number_worker_threads:
 8
override_source:
 <empty>
port:
 5141
recv_buffer_size:
 1048573
store_full_message:
 false
tcp_keepalive:
 true
tls_cert_file:
 <empty>
tls_client_auth:
 disabled
tls_client_auth_cert_file:
 <empty>
tls_enable:
 false
tls_key_file:
 admin
tls_key_password:
********
use_null_delimiter:
 false

Thank you very much

2

Does anyone know if I can use "iptables -SNAT " to rewrite source ip (docker internal ip) to the docker host ip for e.g. syslog tls connections?

Any help would be fine.
Thanks in advanced

3

I’ve noticed you don’t have the GRAYLOG_HTTP_BIND_ADDRESS environment variable in your Docker config. Can you try adding that with the IP that you expect to reach the container at? You’ll also want to ensure you add GRAYLOG_HTTP_ENABLE_CORS=true.

4

Dear aaronsachs,

thank you for your reply.
With my given setup (docker-compose) I can reach the WebUI and the API-Browser as well.
The only problem I have is that when a syslog-ng machine talks via syslog with the graylog server over TCP. The docker always send the internal docker ip instead of the host ip.
I added GRAYLOG_HTTP_ENABLE_CORS=true without any success. Also if I set the GRAYLOG_HTTP_BIND_ADDRESS to 127.0.0.1:9000 or to the host ip I cant reach the webinterface anymore.

Lets say my graylog FQDN is graylog-node.mysite.com with ip address 10.128.1.2:
Graylog Environmet Variables:

Nginx:
And I use nginx to redirect 443 with proxypass to https://graylog-node.mysite.com:9000. (which is working correctly)

location /
{
proxy_set_header Host $http_host;
proxy_set_header X-Forwarded-Host $host;
proxy_set_header X-Forwarded-Server $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Graylog-Server-URL https://$server_name/;
#proxy_pass http://127.0.0.1:9000;
proxy_pass https://graylog-node.mysite.com:9000;
}

The only problem I think is that the syslog-server established a connection to graylog-node.mysite.com with the ip 10.128.1.2 → graylog answers in the TCP Frame with his internal ip address 192.168.64.3 and the syslog server send syslog messages in that TCP Connection with destination IP 192.168.64.3 which leads to missing messages. Sometimes the messages come in graylog but there are too many mismatches like

syslog-server.mysite.com.39192 > graylog-node.mysite.com.5141: Flags [.], cksum 0xfc8a (correct), ack 3463358576, win 229, options [nop,nop,TS val 85527296 ecr 1300399506], length 0
09:42:33.341500 IP (tos 0x0, ttl 63, id 55686, offset 0, flags [DF], proto TCP (6), length 52)

syslog-server.mysite.com.39192 > 192.168.64.3.5141: Flags [.], cksum 0x8738 (correct), ack 3463358576, win 229, options [nop,nop,TS val 85527296 ecr 1300399506], length 0

192.168.64.3.5141 > syslog-server.mysite.com.39192: Flags [.], cksum 0x8be1 (incorrect → 0x1e1e), ack 1, win 16384, options [nop,nop,TS val 1300459666 ecr 85346816], length 0
09:42:33.341540 IP (tos 0x0, ttl 64, id 45536, offset 0, flags [DF], proto TCP (6), length 52)

What do I do wrong?
Any help is appreciated.
Thank you.

5

I have tested now the docker-compose option network_mode: host and now the communication
syslog <-> graylog uses only the correct fqdn and host ip. I checked with tcpdump and every message which comes from graylog → syslog is incorrect. The messages from syslog → graylog are always correct.

6

Does no one can help me with that?

7

Dear community,

i have a similar issue like hollowdew.
Setup with graylog, elasticsearch and mongodb in docker-compose.
When connecting rsyslog server with graylog via tcp, the connection is initiated with the correct ip but the acknowledge from the graylog server uses the internal docker ip address instead of the host ip.
How can I handle this issue?

Thanks in advanced!
Cheers

8

Hmmmmm :thinking: so what does your syslog-ng config look like? I feel like I’m missing something here, but I may not also be reproducing the problem. I’ll walk through my test case.

Docker Compose

version: '3'
services:
  # MongoDB: https://hub.docker.com/_/mongo/
  mongo:
    image: mongo:4
    container_name: mongodb
    networks:
      - graylog
  # Elasticsearch: https://www.elastic.co/guide/en/elasticsearch/reference/6.x/docker.html
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch-oss:7.10.0
    container_name: elasticsearch
    environment:
      - http.host=0.0.0.0
      - transport.host=localhost
      - network.host=0.0.0.0
      - "ES_JAVA_OPTS=-Xms1g -Xmx1g"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    deploy:
      resources:
        limits:
          memory: 1g
    ports:
      - 9200:9200
    networks:
      - graylog
  # Graylog: https://hub.docker.com/r/graylog/graylog/
  graylog:
    image: graylog/graylog-enterprise:4.0.3
    container_name: graylog
    environment:
      - GRAYLOG_PASSWORD_SECRET=<REDACTED>
      - GRAYLOG_ROOT_PASSWORD_SHA2=<REDACTED>
      - GRAYLOG_HTTP_EXTERNAL_URI=http://127.0.0.1:9000/
      - GRAYLOG_ELASTICSEARCH_VERSION=7
      - GRAYLOG_HTTP_ENABLE_CORS=true
    networks:
      - graylog
    depends_on:
      - mongo
      - elasticsearch
    ports:
      # Graylog web interface and REST API
      - 9000:9000
      # Syslog TCP
      - 1514:1514
      # Syslog UDP
      - 1514:1514/udp
      # GELF TCP
      - 12201:12201
      # GELF UDP
      - 12201:12201/udp
networks:
  graylog:
    driver: bridge

Syslog TCP Input Configuration

allow_override_date: true
bind_address: 0.0.0.0
expand_structured_data: false
force_rdns: false
max_message_size: 2097152
number_worker_threads: 8
override_source: <empty>
port: 1514
recv_buffer_size: 1048576
store_full_message: false
tcp_keepalive: false
tls_cert_file: <empty>
tls_client_auth: disabled
tls_client_auth_cert_file: <empty>
tls_enable: false
tls_key_file: <empty>
tls_key_password:********
use_null_delimiter: false

Sending Test Messages to Graylog

This is probably where I’m not replicating this exactly, but I’m using a fake log generator to send some syslog messages to the container:

flog -f rfc3164 -l -d 1s -s 10s | nc localhost 1514

Which is clearly sending messages to Graylog to the tune of 1 event/s:

image
image3410×1662 360 KB

So showing your syslog-ng config would be useful. I’d also try to use the flog command that I have up above to try and isolate the issue–is this actually a Graylog problem? Is the problem on the syslog-ng end? That should help narrow it down, I’d think.

9

Dear aaronsachs,

thank you very much for your reply. I am totaly lost here.
So my syslog-ng version is: syslog-ng-3.29.1-1.el7.x86_64 with CentOS 7
My config is untouched except of syslog-ng.conf:

@version:3.29
@include “scl.conf”

syslog-ng configuration file.

This should behave pretty much like the original syslog on RedHat. But

it could be configured a lot smarter.

See syslog-ng(8) and syslog-ng.conf(5) for more information.

Note: it also sources additional configuration files (*.conf)

located in /etc/syslog-ng/conf.d/

options {
flush_lines (0);
time_reopen (10);
log_fifo_size (1000);
chain_hostnames (off);
use_dns (no);
use_fqdn (no);
create_dirs (no);
keep_hostname (yes);
};

source s_sys {
system();
internal();
# udp(ip(0.0.0.0) port(514));
};

destination d_graylog_auth_udp {
network(
“10.128.1.2”
port(5142)
transport(udp)
);
};

destination d_graylog_auth {
network(
“10.128.1.2”
port(5141)
transport(tcp)
);
};

destination d_cons { file(“/dev/console”); };
destination d_mesg { file(“/var/log/messages”); };
destination d_auth { file(“/var/log/secure”); };
destination d_mail { file(“/var/log/maillog” flush_lines(10)); };
destination d_spol { file(“/var/log/spooler”); };
destination d_boot { file(“/var/log/boot.log”); };
destination d_cron { file(“/var/log/cron”); };
destination d_kern { file(“/var/log/kern”); };
destination d_mlal { usertty(“*”); };

filter f_kernel { facility(kern); };
filter f_default { level(info…emerg) and
not (facility(mail)
or facility(authpriv)
or facility(cron)); };
filter f_auth { facility(authpriv); };
filter f_mail { facility(mail); };
filter f_emergency { level(emerg); };
filter f_news { facility(uucp) or
(facility(news)
and level(crit…emerg)); };
filter f_boot { facility(local7); };
filter f_cron { facility(cron); };

#log { source(s_sys); filter(f_kernel); destination(d_cons); };
log { source(s_sys); filter(f_kernel); destination(d_kern); };
log { source(s_sys); filter(f_default); destination(d_mesg); };
log { source(s_sys); filter(f_auth); destination(d_auth); };
log { source(s_sys); filter(f_mail); destination(d_mail); };
log { source(s_sys); filter(f_emergency); destination(d_mlal); };
log { source(s_sys); filter(f_news); destination(d_spol); };
log { source(s_sys); filter(f_boot); destination(d_boot); };
log { source(s_sys); filter(f_cron); destination(d_cron); };
log { source(s_sys); filter(f_auth); destination(d_graylog_auth); };
log { source(s_sys); filter(f_auth); destination(d_graylog_auth_udp); };

Source additional configuration files (.conf extension only)

@include “/etc/syslog-ng/conf.d/*.conf”

On the machine where syslog-ng is running I see forwarding 11 missed messages to graylog
On graylogs input where TCP/5141 is defined I see this:

Throughput / Metrics

1 minute average rate: 0 msg/s
Network IO: 0B 0B (total: 7.9KiB 0B )

So the 7.9KiB is increasing every time I login on the syslog-ng server. But there are no logs when I click on “Show received messages” even if I click all messages and/or select all streams.

Best regards and thanks in advanced. :slight_smile:

10

Ok–so is 10.128.1.2 the IP of the Docker host, or the IP that you’ve assigned to the container?

11

10.128.1.2 is the ip of the VM where graylog docker is running.

12

Also when I use UDP Input with syslog-ng it is working like a charm. Also if I use GELF via syslog-ng. Only the input syslog tcp is not wokring with graylog and syslog-ng.

Best regards
Niko

13

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.