Graylog does not show GelfTCP and GelfUDP logs

[aramisama (Samaneh Farahmand)]

Dear all,

I have searched a lot and read almost all the topics which were related to my problem but I could find a way to solve it.
First I found that it is related to Timezone and I fix it both in Linux VMs that I am working with and in my docker-compose file.
Unfortunately, nothing has changed in Graylog UI.
I use tcpdump command and I can see logs in my VM but I can not see them in Graylog.

I really do not know what is the problem because I can see logs with Syslog and it perfectly working.

[aaronsachs]

Hi there, so if I understand the problem correctly, you’re running your Graylog deployment in Docker on a couple of Linux VMs, correct? What does your compose file look like? Also, what inputs do you have running in Graylog?

[aramisama (Samaneh Farahmand)]

This is my docker-compose file and it is working with Traefik. Inputs of graylog are : GelfTCP - gelfUdp -syslogUDP- syslogTCP.
It is working with Syslog, not Gelf. I also change the timezone of my VMs to Europe.
I also use this command for sending logs from one of my VMs to another one which graylog and traefik are working on it:

echo -n -e ‘{ “version”: “1.1”, “host”: “example.org”, “short_message”: “A short message”, “level”: 5, “_some_info”: “foo” }’“\0” | nc -u -w0 X 12201

Docker-compose file

version: '3.7'
services:
  # MongoDB: https://hub.docker.com/_/mongo/
    image: mongo:4
    volumes:
      - /mnt/Ubuntu/mongo:/data/db
    environment:
      - TZ=Europe/Berlin
    networks:
      - internal
  # Elasticsearch: https://www.elastic.co/guide/en/elasticsearch/reference/6.x/docker.html
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch-oss:7.10.0
    volumes:
      - elastic:/usr/share/elasticsearch/data
    environment:
      - http.host=0.0.0.0
      - transport.host=localhost
      - network.host=0.0.0.0
      - TZ=Europe/Berlin
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    networks:
      - internal
  # Graylog: https://hub.docker.com/r/graylog/graylog/
  graylog:
    image: graylog/graylog:4.0
    volumes:
      - /mnt/Ubuntu/graylog/data:/usr/share/graylog/data
      - /mnt/Ubuntu/graylog/config:/usr/share/graylog/data/config
    environment:
      - GRAYLOG_PASSWORD_SECRET= X
      - GRAYLOG_ROOT_PASSWORD_SHA2= X
      - GRAYLOG_HTTP_EXTERNAL_URI= X
      - TZ=Europe/Berlin
    restart: always
    depends_on:
      - mongodb
      - elasticsearch
    deploy:
      labels:
          #http
        - traefik.http.routers.graylog.service=graylog-svc
        - traefik.http.routers.graylog.rule=Host("")
        - traefik.http.services.graylog-svc.loadbalancer.server.port=9000
        - traefik.http.routers.graylog.entrypoints= web,websecure
         #tls
        - traefik.http.routers.graylog.tls=true
        - traefik.http.routers.graylog.tls.certresolver=le
         #tcp
        - traefik.tcp.routers.graylogtcp.entrypoints= syslogtcp
        - traefik.tcp.routers.graylogtcp.service=graylogtcp-svc
        - traefik.tcp.services.graylogtcp-svc.loadbalancer.server.port=1514
        - traefik.tcp.routers.graylogtcp.rule=HostSNI(`*`)
         #udp
        - traefik.udp.routers.graylogudp.service=graylogudp-svc
        - traefik.udp.routers.graylogudp.entrypoints= syslogudp
        - traefik.udp.services.graylogudp-svc.loadbalancer.server.port=1514/udp
         #GELF TCP
        - traefik.tcp.routers.grayloggelftcp.rule=HostSNI(`*`)
        - traefik.tcp.routers.grayloggelftcp.service=grayloggelftcp-svc
        - traefik.tcp.routers.grayloggelftcp.entrypoints= gelftcp
        - traefik.tcp.services.grayloggelftcp-svc.loadbalancer.server.port=12201
         #GELF UDP
        - traefik.udp.routers.grayloggelfudp.service=grayloggelfudp-svc
        - traefik.udp.routers.grayloggelfudp.entrypoints= gelfudp
        - traefik.udp.services.grayloggelfudp-svc.loadbalancer.server.port=12201/udp
        - traefik.enable=true
        - traefik.docker.network=public
        - traefik.docker.network=internal
        - traefik.http.routers.graylog.middlewares=graylog-header
        - traefik.http.middlewares.graylog-header.headers.customrequestheaders.X-Graylog-Server-URL= X
        - traefik.http.middlewares.graylog-header.headers.customresponseheaders.X-Graylog-Server-URL= X
    # Volumes for persisting data, see https://docs.docker.com/engine/admin/volumes/volumes/
    networks:
      - public
      - internal
networks:
  internal:
    external: true
  public:
    external: true
volumes:
  elastic:
    external: true

and this is my Traefik yml file:

version: '3.3'
services:
  traefik:
    image: traefik:v2.3
    ports:
      - 443:443
      - 80:80
      - 9000:9000
      - 1514:1514
      - 1514:1514/udp
      - 12201:12021
      - 12201:12201/udp
    volumes:
      - letsencrypt-certificates:/letsencrypt-certificates/
    networks:
      - internal
      - public
    command:
      - --entrypoints.web.address=:80
      - --entrypoints.websecure.address=:443
      - --entrypoints.graylogweb.address=:9000
      - --entrypoints.syslogtcp.address=:1514
      - --entrypoints.syslogudp.address=:1514/udp
      - --entrypoints.gelftcp.address=:12201
      - --entrypoints.gelfudp.address=:12201/udp
      - --providers.docker
      - --providers.docker.swarmMode=true
      - --providers.docker.endpoint=tcp://tasks.docker-socket-proxy:2375 #make secure tasks.[servicename]:[port]
      - --providers.docker.exposedbydefault=false
      - --api
      - --certificatesResolvers.le.acme.email= X
      - --certificatesResolvers.le.acme.storage=/letsencrypt-certificates/acme.json
      - --certificatesResolvers.le.acme.tlsChallenge=true
    deploy:
      labels:
        - traefik.http.middlewares.admin.basicauth.users=X
        - traefik.http.middlewares.redirect-to-https.redirectscheme.scheme=https
        - traefik.http.routers.force-https.rule=hostregexp(`{host:.+}`)
        - traefik.http.routers.force-https.entrypoints=web
        - traefik.http.routers.force-https.middlewares=redirect-to-https
        # Traefik dashboard
        - traefik.http.routers.traefik.rule=Host(" ")
        - traefik.http.routers.traefik.service=api@internal
        - traefik.http.routers.traefik.entrypoints=websecure
        - traefik.http.routers.traefik.tls.certresolver=le
        - traefik.http.routers.traefik.middlewares=admin
        - traefik.enable=true
        - traefik.docker.network=public
        - traefik.docker.network=internal
        - traefik.http.services.traefik.loadbalancer.server.port=8080
  docker-socket-proxy:
    image: tecnativa/docker-socket-proxy
    hostname: docker-socket-proxy
    environment:
      - CONTAINERS=1
      - NETWORKS=1
      - SERVICES=1
      - SWARM=1
      - TASKS=1
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
    networks:
      - internal
    deploy:
      resources:
        limits:
          cpus: '1'
          memory: 500M
      mode: replicated
      replicas: 1
volumes:
  letsencrypt-certificates:
networks:
  internal:
    external: true
  public:
    external: true

[shoothub]

Please try to extract also real messages from tcpdump, so we can check if syntax of gelf messages are OK. Also check graylog server log for errors, especially for gelf.