# Graylog docker with syslog-ng via tcp uses internal docker ip

**URL:** https://community.graylog.org/t/graylog-docker-with-syslog-ng-via-tcp-uses-internal-docker-ip/18848
**Category:** Graylog Central (peer support)
**Created:** [February 18, 2021, 11:00am UTC](https://community.graylog.org/t/graylog-docker-with-syslog-ng-via-tcp-uses-internal-docker-ip/18848 "2021-02-18T11:00:52Z")
**Posts on this page:** 13
**Page:** 1

<div class="post-metadata">

### Author: ![hollowdew](https://avatars.discourse-cdn.com/v4/letter/h/4bbf92/32.png) [@hollowdew](https://community.graylog.org/u/hollowdew)
#### Post date: [February 18, 2021, 11:00am UTC](https://community.graylog.org/t/graylog-docker-with-syslog-ng-via-tcp-uses-internal-docker-ip/18848/1 "2021-02-18T11:00:52Z")

</div>

I am using docker-compose to deploy graylog:  
IP from graylog host starts with 10.128  
IP from graylog-docker on graylog host has internal docker ip 192.168.64.3

```
[docker-compose.yml]
version: '2.2'
services:

      graylog:
        image: graylog/graylog:4.0.1
        hostname: graylog1-docker
        container_name: graylog
        restart: always
        environment:
          - GRAYLOG_HTTP_ENABLE_TLS=true
          - GRAYLOG_HTTP_TLS_CERT_FILE=/etc/ssl/certs/graylog-node1.pem
          - GRAYLOG_HTTP_TLS_KEY_FILE=/etc/ssl/private/graylog-node1.key
          - GRAYLOG_HTTP_EXTERNAL_URI=https://graylog-node1-fqdn:9000/
          - GRAYLOG_HTTP_PUBLISH_URI=https://graylog-node1-fqdn:9000/
          - GRAYLOG_ELASTICSEARCH_HOSTS=3 nodes defined and working
          - GRAYLOG_MONGODB_URI=replicaset defined and working
          
        networks:
          - graylog
        volumes:
          - /etc/graylog/graylog_journal:/usr/share/graylog/data/journal
        ports:
          # HTTPS
          - 9000:9000
          # Syslog TCP
          - 5140:5140
          - 5141:5141
          - 5142:5142
          - 5143:5143
          - 5144:5144
          - 5145:5145
          - 5146:5146
          - 5147:5147
          - 5148:5148
          - 5149:5149
          # Syslog UDP
          - 5140:5140/udp
          - 514:514/udp
          - 5141:5141/udp
          - 5142:5142/udp
          - 5143:5143/udp
          - 5144:5144/udp
          - 5145:5145/udp
          - 5146:5146/udp
          - 5147:5147/udp
          - 5148:5148/udp
          - 5149:5149/udp
          # GELF TCP
          - 12201:12201
          # GELF UDP
          - 12201:12201/udp
[/docker-compose.yml]

```

I configured a external vm and configured syslog-ng to send to graylog.  
Graylog collects data successfully but it is very slowly.  
On the syslog-ng host I see the following logs very often:  
`systemd-journal[337]: Forwarding to syslog missed 12 messages.`  
So I mad a tcp dump on the graylog host where the docker runs.  
[tcpdump]

> 11:32:52.154627 IP (tos 0x0, ttl 63, id 18494, offset 0, flags [DF], proto TCP (6), length 52)  
> syslog-ng.fqdn.36854 \> 192.168.64.3.5141: Flags [.], cksum 0x65f4 (correct), ack 1, win 229, options [nop,nop,TS val 5745664 ecr 1220618454], length 0  
> 11:32:52.154766 IP (tos 0x0, ttl 64, id 900, offset 0, flags [DF], proto TCP (6), length 52)  
> 192.168.64.3.5141 \> syslog-ng.fqdn.36854: Flags [.], cksum 0x8be1 (incorrect → 0x49ea), ack 2916, win 19064, options [nop,nop,TS val 1220678614 ecr 5084031], length 0  
> 11:32:52.154774 IP (tos 0x0, ttl 64, id 900, offset 0, flags [DF], proto TCP (6), length 52)  
> 192.168.64.3.5141 \> syslog-ng.fqdn.36854: Flags [.], cksum 0x8be1 (incorrect → 0x49ea), ack 2916, win 19064, options [nop,nop,TS val 1220678614 ecr 5084031], length 0  
> 11:32:52.154789 IP (tos 0x0, ttl 63, id 900, offset 0, flags [DF], proto TCP (6), length 52)  
> graylog-fqdn.5141 \> syslog-ng.fqdn.36854: Flags [.], cksum 0x168f (incorrect → 0xbf3c), ack 2916, win 19064, options [nop,nop,TS val 1220678614 ecr 5084031], length 0  
> 11:33:38.681019 IP (tos 0x0, ttl 64, id 11487, offset 0, flags [DF], proto TCP (6), length 52)  
> syslog-ng.fqdn.45521 \> graylog-fqdn.5141: Flags [.], cksum 0xc16c (correct), ack 1, win 229, options [nop,nop,TS val 2672946176 ecr 1115955523], length 0  
> 11:33:38.681046 IP (tos 0x0, ttl 63, id 11487, offset 0, flags [DF], proto TCP (6), length 52)  
> syslog-ng.fqdn.45521 \> 192.168.64.3.5141: Flags [.], cksum 0x4c1a (correct), ack 1, win 229, options [nop,nop,TS val 2672946176 ecr 1115955523], length 0  
> [/tcpdump]

So the problem is that some packets are dropped because the syslog-ng host tries to send to 192.168 which is not working as the syslog-ng need to write to 10.128 or the graylog-fqdn.

My question is:  
How can I correctly configure the graylog-docker-compose to use the right ip address and not the internal docker ip address. I assume that the tcp connections breaks all the time.

The configured Input in graylog looks like this:  
\> allow\_override\_date:

> ```
> true
> bind_address:
> 0.0.0.0
> expand_structured_data:
> false
> force_rdns:
> false
> max_message_size:
> 2097152
> number_worker_threads:
> 8
> override_source:
> <empty>
> port:
> 5141
> recv_buffer_size:
> 1048573
> store_full_message:
> false
> tcp_keepalive:
> true
> tls_cert_file:
> <empty>
> tls_client_auth:
> disabled
> tls_client_auth_cert_file:
> <empty>
> tls_enable:
> false
> tls_key_file:
> admin
> tls_key_password:
> ********
> use_null_delimiter:
> false
> 
> ```

Thank you very much

---

<div class="post-metadata">

### Author: ![hollowdew](https://avatars.discourse-cdn.com/v4/letter/h/4bbf92/32.png) [@hollowdew](https://community.graylog.org/u/hollowdew)
#### Post date: [February 18, 2021, 2:36pm UTC](https://community.graylog.org/t/graylog-docker-with-syslog-ng-via-tcp-uses-internal-docker-ip/18848/2 "2021-02-18T14:36:19Z")

</div>

Does anyone know if I can use "iptables -SNAT " to rewrite source ip (docker internal ip) to the docker host ip for e.g. syslog tls connections?

Any help would be fine.  
Thanks in advanced

---

<div class="post-metadata">

### Author: ![aaronsachs](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/aaronsachs/32/7180_2.png) [@aaronsachs](https://community.graylog.org/u/aaronsachs)
#### Post date: [February 18, 2021, 3:20pm UTC](https://community.graylog.org/t/graylog-docker-with-syslog-ng-via-tcp-uses-internal-docker-ip/18848/3 "2021-02-18T15:20:20Z")

</div>

I’ve noticed you don’t have the `GRAYLOG_HTTP_BIND_ADDRESS` environment variable in your Docker config. Can you try adding that with the IP that you expect to reach the container at? You’ll also want to ensure you add `GRAYLOG_HTTP_ENABLE_CORS=true`.

---

<div class="post-metadata">

### Author: ![hollowdew](https://avatars.discourse-cdn.com/v4/letter/h/4bbf92/32.png) [@hollowdew](https://community.graylog.org/u/hollowdew)
#### Post date: [February 19, 2021, 8:54am UTC](https://community.graylog.org/t/graylog-docker-with-syslog-ng-via-tcp-uses-internal-docker-ip/18848/4 "2021-02-19T08:54:08Z")

</div>

Dear aaronsachs,

thank you for your reply.  
With my given setup (docker-compose) I can reach the WebUI and the API-Browser as well.  
The only problem I have is that when a syslog-ng machine talks via syslog with the graylog server over TCP. The docker always send the internal docker ip instead of the host ip.  
I added `GRAYLOG_HTTP_ENABLE_CORS=true` without any success. Also if I set the `GRAYLOG_HTTP_BIND_ADDRESS` to `127.0.0.1:9000` or to the host ip I cant reach the webinterface anymore.

Lets say my graylog FQDN is [graylog-node.mysite.com](http://graylog-node.mysite.com) with ip address 10.128.1.2:  
Graylog Environmet Variables:

> - GRAYLOG\_HTTP\_ENABLE\_TLS=true
> - GRAYLOG\_HTTP\_TLS\_CERT\_FILE=/etc/ssl/certs/graylog-node.pem
> - GRAYLOG\_HTTP\_TLS\_KEY\_FILE=/etc/ssl/private/graylog-node.key
> - GRAYLOG\_HTTP\_EXTERNAL\_URI=[https://graylog-node.mysite.com:9000/](https://graylog-node.mysite.com:9000/)
> - GRAYLOG\_HTTP\_PUBLISH\_URI=[https://graylog-node.mysite.com:9000/](https://graylog-node.mysite.com:9000/)
> - GRAYLOG\_HTTP\_ENABLE\_CORS=true

Nginx:  
And I use nginx to redirect 443 with proxypass to [https://graylog-node.mysite.com:9000](https://graylog-node.mysite.com:9000). (which is working correctly)

> location /  
> {  
> proxy\_set\_header Host $http\_host;  
> proxy\_set\_header X-Forwarded-Host $host;  
> proxy\_set\_header X-Forwarded-Server $host;  
> proxy\_set\_header X-Forwarded-For $proxy\_add\_x\_forwarded\_for;  
> proxy\_set\_header X-Graylog-Server-URL https://$server\_name/;  
> #proxy\_pass[http://127.0.0.1:9000](http://127.0.0.1:9000);  
> proxy\_pass [https://graylog-node.mysite.com:9000](https://graylog-node.mysite.com:9000);  
> }

The only problem I think is that the syslog-server established a connection to [graylog-node.mysite.com](http://graylog-node.mysite.com) with the ip 10.128.1.2 → graylog answers in the TCP Frame with his internal ip address 192.168.64.3 and the syslog server send syslog messages in that TCP Connection with destination IP 192.168.64.3 which leads to missing messages. Sometimes the messages come in graylog but there are too many mismatches like

> syslog-server.mysite.com.39192 \> graylog-node.mysite.com.5141: Flags [.], cksum 0xfc8a (correct), ack 3463358576, win 229, options [nop,nop,TS val 85527296 ecr 1300399506], length 0  
> 09:42:33.341500 IP (tos 0x0, ttl 63, id 55686, offset 0, flags [DF], proto TCP (6), length 52)

> syslog-server.mysite.com.39192 \> 192.168.64.3.5141: Flags [.], cksum 0x8738 (correct), ack 3463358576, win 229, options [nop,nop,TS val 85527296 ecr 1300399506], length 0

> 192.168.64.3.5141 \> syslog-server.mysite.com.39192: Flags [.], cksum 0x8be1 (incorrect → 0x1e1e), ack 1, win 16384, options [nop,nop,TS val 1300459666 ecr 85346816], length 0  
> 09:42:33.341540 IP (tos 0x0, ttl 64, id 45536, offset 0, flags [DF], proto TCP (6), length 52)

What do I do wrong?  
Any help is appreciated.  
Thank you.

---

<div class="post-metadata">

### Author: ![hollowdew](https://avatars.discourse-cdn.com/v4/letter/h/4bbf92/32.png) [@hollowdew](https://community.graylog.org/u/hollowdew)
#### Post date: [February 19, 2021, 9:13am UTC](https://community.graylog.org/t/graylog-docker-with-syslog-ng-via-tcp-uses-internal-docker-ip/18848/5 "2021-02-19T09:13:45Z")

</div>

I have tested now the docker-compose option `network_mode: host` and now the communication  
syslog \<-\> graylog uses only the correct fqdn and host ip. I checked with tcpdump and every message which comes from graylog → syslog is incorrect. The messages from syslog → graylog are always correct.

---

<div class="post-metadata">

### Author: ![hollowdew](https://avatars.discourse-cdn.com/v4/letter/h/4bbf92/32.png) [@hollowdew](https://community.graylog.org/u/hollowdew)
#### Post date: [February 22, 2021, 11:13am UTC](https://community.graylog.org/t/graylog-docker-with-syslog-ng-via-tcp-uses-internal-docker-ip/18848/6 "2021-02-22T11:13:24Z")

</div>

Does no one can help me with that?

---

<div class="post-metadata">

### Author: ![steven101](https://avatars.discourse-cdn.com/v4/letter/s/76d3ee/32.png) [@steven101](https://community.graylog.org/u/steven101)
#### Post date: [February 22, 2021, 1:49pm UTC](https://community.graylog.org/t/graylog-docker-with-syslog-ng-via-tcp-uses-internal-docker-ip/18848/7 "2021-02-22T13:49:56Z")

</div>

Dear community,

i have a similar issue like hollowdew.  
Setup with graylog, elasticsearch and mongodb in docker-compose.  
When connecting rsyslog server with graylog via tcp, the connection is initiated with the correct ip but the acknowledge from the graylog server uses the internal docker ip address instead of the host ip.  
How can I handle this issue?

Thanks in advanced!  
Cheers

---

<div class="post-metadata">

### Author: ![aaronsachs](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/aaronsachs/32/7180_2.png) [@aaronsachs](https://community.graylog.org/u/aaronsachs)
#### Post date: [February 23, 2021, 3:29am UTC](https://community.graylog.org/t/graylog-docker-with-syslog-ng-via-tcp-uses-internal-docker-ip/18848/8 "2021-02-23T03:29:50Z")

</div>

Hmmmmm 🤔 so what does your syslog-ng config look like? I feel like I’m missing something here, but I may not also be reproducing the problem. I’ll walk through my test case.

## Docker Compose

```auto
version: '3'
services:
  # MongoDB: https://hub.docker.com/_/mongo/
  mongo:
    image: mongo:4
    container_name: mongodb
    networks:
      - graylog
  # Elasticsearch: https://www.elastic.co/guide/en/elasticsearch/reference/6.x/docker.html
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch-oss:7.10.0
    container_name: elasticsearch
    environment:
      - http.host=0.0.0.0
      - transport.host=localhost
      - network.host=0.0.0.0
      - "ES_JAVA_OPTS=-Xms1g -Xmx1g"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    deploy:
      resources:
        limits:
          memory: 1g
    ports:
      - 9200:9200
    networks:
      - graylog
  # Graylog: https://hub.docker.com/r/graylog/graylog/
  graylog:
    image: graylog/graylog-enterprise:4.0.3
    container_name: graylog
    environment:
      - GRAYLOG_PASSWORD_SECRET=<REDACTED>
      - GRAYLOG_ROOT_PASSWORD_SHA2=<REDACTED>
      - GRAYLOG_HTTP_EXTERNAL_URI=http://127.0.0.1:9000/
      - GRAYLOG_ELASTICSEARCH_VERSION=7
      - GRAYLOG_HTTP_ENABLE_CORS=true
    networks:
      - graylog
    depends_on:
      - mongo
      - elasticsearch
    ports:
      # Graylog web interface and REST API
      - 9000:9000
      # Syslog TCP
      - 1514:1514
      # Syslog UDP
      - 1514:1514/udp
      # GELF TCP
      - 12201:12201
      # GELF UDP
      - 12201:12201/udp
networks:
  graylog:
    driver: bridge

```

## Syslog TCP Input Configuration

```auto
allow_override_date: true
bind_address: 0.0.0.0
expand_structured_data: false
force_rdns: false
max_message_size: 2097152
number_worker_threads: 8
override_source: <empty>
port: 1514
recv_buffer_size: 1048576
store_full_message: false
tcp_keepalive: false
tls_cert_file: <empty>
tls_client_auth: disabled
tls_client_auth_cert_file: <empty>
tls_enable: false
tls_key_file: <empty>
tls_key_password: ********
use_null_delimiter: false

```

## Sending Test Messages to Graylog

This is probably where I’m not replicating this exactly, but I’m using a [fake log generator](https://github.com/mingrammer/flog) to send some syslog messages to the container:

```auto
flog -f rfc3164 -l -d 1s -s 10s | nc localhost 1514

```

Which is clearly sending messages to Graylog to the tune of 1 event/s:

 ![image](https://us1.discourse-cdn.com/flex016/uploads/graylog/original/2X/d/d609f97a22b719d0d9fd958e2b51fa3ca1baea9b.png)

So showing your syslog-ng config would be useful. I’d also try to use the `flog` command that I have up above to try and isolate the issue–is this actually a Graylog problem? Is the problem on the syslog-ng end? That should help narrow it down, I’d think.

---

<div class="post-metadata">

### Author: ![hollowdew](https://avatars.discourse-cdn.com/v4/letter/h/4bbf92/32.png) [@hollowdew](https://community.graylog.org/u/hollowdew)
#### Post date: [February 23, 2021, 10:21am UTC](https://community.graylog.org/t/graylog-docker-with-syslog-ng-via-tcp-uses-internal-docker-ip/18848/9 "2021-02-23T10:21:52Z")

</div>

Dear aaronsachs,

thank you very much for your reply. I am totaly lost here.  
So my syslog-ng version is: syslog-ng-3.29.1-1.el7.x86\_64 with CentOS 7  
My config is untouched except of syslog-ng.conf:

> @version:3.29  
> @include “scl.conf”
> 
> # syslog-ng configuration file.
> 
> # 
> 
> # This should behave pretty much like the original syslog on RedHat. But
> 
> # it could be configured a lot smarter.
> 
> # 
> 
> # See syslog-ng(8) and syslog-ng.conf(5) for more information.
> 
> # 
> 
> # Note: it also sources additional configuration files (\*.conf)
> 
> # located in /etc/syslog-ng/conf.d/
> 
> options {  
> flush\_lines (0);  
> time\_reopen (10);  
> log\_fifo\_size (1000);  
> chain\_hostnames (off);  
> use\_dns (no);  
> use\_fqdn (no);  
> create\_dirs (no);  
> keep\_hostname (yes);  
> };
> 
> source s\_sys {  
> system();  
> internal();  
> # udp(ip(0.0.0.0) port(514));  
> };
> 
> destination d\_graylog\_auth\_udp {  
> network(  
> “10.128.1.2”  
> port(5142)  
> transport(udp)  
> );  
> };
> 
> destination d\_graylog\_auth {  
> network(  
> “10.128.1.2”  
> port(5141)  
> transport(tcp)  
> );  
> };
> 
> destination d\_cons { file(“/dev/console”); };  
> destination d\_mesg { file(“/var/log/messages”); };  
> destination d\_auth { file(“/var/log/secure”); };  
> destination d\_mail { file(“/var/log/maillog” flush\_lines(10)); };  
> destination d\_spol { file(“/var/log/spooler”); };  
> destination d\_boot { file(“/var/log/boot.log”); };  
> destination d\_cron { file(“/var/log/cron”); };  
> destination d\_kern { file(“/var/log/kern”); };  
> destination d\_mlal { usertty(“\*”); };
> 
> filter f\_kernel { facility(kern); };  
> filter f\_default { level(info…emerg) and  
> not (facility(mail)  
> or facility(authpriv)  
> or facility(cron)); };  
> filter f\_auth { facility(authpriv); };  
> filter f\_mail { facility(mail); };  
> filter f\_emergency { level(emerg); };  
> filter f\_news { facility(uucp) or  
> (facility(news)  
> and level(crit…emerg)); };  
> filter f\_boot { facility(local7); };  
> filter f\_cron { facility(cron); };
> 
> #log { source(s\_sys); filter(f\_kernel); destination(d\_cons); };  
> log { source(s\_sys); filter(f\_kernel); destination(d\_kern); };  
> log { source(s\_sys); filter(f\_default); destination(d\_mesg); };  
> log { source(s\_sys); filter(f\_auth); destination(d\_auth); };  
> log { source(s\_sys); filter(f\_mail); destination(d\_mail); };  
> log { source(s\_sys); filter(f\_emergency); destination(d\_mlal); };  
> log { source(s\_sys); filter(f\_news); destination(d\_spol); };  
> log { source(s\_sys); filter(f\_boot); destination(d\_boot); };  
> log { source(s\_sys); filter(f\_cron); destination(d\_cron); };  
> log { source(s\_sys); filter(f\_auth); destination(d\_graylog\_auth); };  
> log { source(s\_sys); filter(f\_auth); destination(d\_graylog\_auth\_udp); };
> 
> # Source additional configuration files (.conf extension only)
> 
> @include “/etc/syslog-ng/conf.d/\*.conf”

On the machine where syslog-ng is running I see forwarding 11 missed messages to graylog  
On graylogs input where TCP/5141 is defined I see this:

> ### Throughput / Metrics
> 
> 1 minute average rate: 0 msg/s  
> Network IO: 0B 0B (total: 7.9KiB 0B )

So the 7.9KiB is increasing every time I login on the syslog-ng server. But there are no logs when I click on “Show received messages” even if I click all messages and/or select all streams.

Best regards and thanks in advanced. 🙂

---

<div class="post-metadata">

### Author: ![aaronsachs](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/aaronsachs/32/7180_2.png) [@aaronsachs](https://community.graylog.org/u/aaronsachs)
#### Post date: [February 23, 2021, 2:22pm UTC](https://community.graylog.org/t/graylog-docker-with-syslog-ng-via-tcp-uses-internal-docker-ip/18848/10 "2021-02-23T14:22:10Z")

</div>

Ok–so is `10.128.1.2` the IP of the Docker host, or the IP that you’ve assigned to the container?

---

<div class="post-metadata">

### Author: ![hollowdew](https://avatars.discourse-cdn.com/v4/letter/h/4bbf92/32.png) [@hollowdew](https://community.graylog.org/u/hollowdew)
#### Post date: [February 23, 2021, 2:33pm UTC](https://community.graylog.org/t/graylog-docker-with-syslog-ng-via-tcp-uses-internal-docker-ip/18848/11 "2021-02-23T14:33:19Z")

</div>

10.128.1.2 is the ip of the VM where graylog docker is running.

---

<div class="post-metadata">

### Author: ![hollowdew](https://avatars.discourse-cdn.com/v4/letter/h/4bbf92/32.png) [@hollowdew](https://community.graylog.org/u/hollowdew)
#### Post date: [February 23, 2021, 2:34pm UTC](https://community.graylog.org/t/graylog-docker-with-syslog-ng-via-tcp-uses-internal-docker-ip/18848/12 "2021-02-23T14:34:27Z")

</div>

Also when I use UDP Input with syslog-ng it is working like a charm. Also if I use GELF via syslog-ng. Only the input syslog tcp is not wokring with graylog and syslog-ng.

Best regards  
Niko

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/flex016/uploads/graylog/original/3X/c/7/c7c09c6b5099570133d6502b83f50ba4430de5b6.png) [@system](https://community.graylog.org/u/system)
#### Post date: [March 9, 2021, 2:35pm UTC](https://community.graylog.org/t/graylog-docker-with-syslog-ng-via-tcp-uses-internal-docker-ip/18848/13 "2021-03-09T14:35:08Z")

</div>

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.
