Quick Guide: Graylog Deployment with Ansible and Docker Swarm

[mbastide (Matthew Bastide)]

Running Graylog as part of docker swarm.

Hi,

Here is a description of how we got Graylog happy in docker swarm.
We have been running this in 2.5.4 for about a year and just upgraded to 3.1.

If you have tried to put Graylog into Docker swarm mode, you found that it does not come back gracefully. Maybe one or two nodes appear active, while the others do not display a stats and cannot be found.
This method mitigates that issue by assigning a hostname in the docker-entrypoint.sh and building a custom Dockerfile.

You will need to manually configure folder permissions to all of your mounts.
We use ansible to do this.

At the time writing this, the following user mappings work for folders:

• Elasticsearch user 1000:1000
• Mongodb user 999:999
• Graylog user 1100:1100

###Basic Components:

• A 3 node docker swarm
• Active-passive nginx load balancers on their own VMs (not in the cluster)
• Graylog Enterprise:
  • 3 elastic instances
  • 3 mongo instances
  • 3 Graylog instances for inputs
  • 1 Graylog master node for archiving and Web UI
  • portainer running within the docker swarm for checking services and reading logs.

Prework:

• Create your docker swarm
• Create an overlay network (if you want to encrypt) – https://docs.docker.com/v17.09/engine/userguide/networking/overlay-security-model/

Customizing the Dockerfile and Entrypoint.

1. Create the following:

• touch /path/to/graylog/docker-compose.yml
• touch /path/to/graylog/images/graylog/Dockerfile
• touch /path/to/graylog/images/graylog/docker-entrypoint.sh

2. Update docker-entrypoint.sh

• update docker-entrypoint.sh with the contents of https://github.com/Graylog2/graylog-docker/blob/3.1/docker-entrypoint.sh
• after #!/bin/bash added the follow:

export GRAYLOG_HTTP_PUBLISH_URI="http://`hostname`:9000/"

This line will set the service’s container id as the hostname when the service is started.
This is what prevents nodes from being “lost” when services are recovering from a shutdown.

3. Update the Dockerfile
   update /path/to/graylog/image/graylog/Dockerfile

FROM graylog/graylog:{{ GRAYLOG_VERSION }} as graylog
USER root

#DELETE plugin lines if you are not using Graylog enterprise
#GET ENTERPRISE PLUGINS
RUN apt-get update;apt-get install -y wget procps
RUN wget -O ~/plugins.gz https://downloads.graylog.org/releases/graylog-enterprise/graylog-enterprise-plugins-{{ GRAYLOG_VERSION }}.tgz
RUN tar -xf ~/plugins.gz -C ~/
RUN ls ~; mv ~/graylog-enterprise-plugins-{{ GRAYLOG_VERSION }}/plugin/* /usr/share/graylog/plugin/
#END Enterprise plugins

#copy and paste this file from graylog-docker
COPY docker-entrypoint.sh /
RUN chmod 755 /docker-entrypoint.sh

USER graylog 
ENTRYPOINT ["/docker-entrypoint.sh"]
CMD ["graylog"]

4. Update the Docker-compose file
   You may need to tweak this file to work for you.
   We setup ports on the graylog service in “host” mode to allow nginx to do the load balancing between the 3 hosts.
   TLS is also handled by the nginx load balancers.

docker-compose.yml

services:
  mongodb01a:
    image: mongo:3
    volumes:
      - /path/to/graylog/mongo/db:/data/db
    command: mongod --replSet "rs0"
    deploy:
      placement:
        constraints:
          - node.hostname == dockerhost01

  mongodb01b:
    image: mongo:3
    command: mongod -replSet "rs0"
    volumes:
      - /path/to/graylog/mongo/db:/data/db
    deploy:
      placement:
        constraints:
          - node.hostname == dockerhost02

  mongodb01c:
    image: mongo:3
    command: mongod -replSet "rs0"
    volumes:
      - /path/to/graylog/mongo/db:/data/db
    deploy:
      placement:
        constraints:
          - node.hostname == dockerhost03
          
  elasticsearch:
    image: "docker.elastic.co/elasticsearch/elasticsearch:{{ ELASTIC_VERSION }}"
    environment:
      - xpack.security.enabled=false
      - "ES_JAVA_OPTS=-Xms16528m -Xmx16528m"
      - cluster.name=graylog
      - discovery.type=zen
      - discovery.zen.minimum_master_nodes=2
      - discovery.zen.ping.unicast.hosts=elasticsearch
    volumes:
      - /path/to/graylog/elastic/:/usr/share/elasticsearch/data
    resources:
        limits:
            memory: '28G'

  graylog:
    image: "internal_docker_repo/graylog:{{GRAYLOG_VERSION }}"
    build: ./images/graylog/.
    environment:
      GRAYLOG_IS_MASTER: "false"
      GRAYLOG_ELASTICSEARCH_HOSTS: http://elasticsearch:9200
      GRAYLOG_SERVER_JAVA_OPTS: -XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap -XX:NewRatio=1 -XX:MaxMetaspaceSize=256m -server -XX:+ResizeTLAB -XX:+UseConcMarkSweepGC -XX:+CMSConcurrentMTEnabled -XX:+CMSClassUnloadingEnabled -XX:+UseParNewGC -XX:-OmitStackTraceInFastThrow
      GRAYLOG_PASSWORD_SECRET: {{ GRAYLOG_PASSWORD_SECRET }}
      GRAYLOG_ROOT_PASSWORD_SHA2: {{ GRAYLOG_ROOT_PASSWORD_SHA2 }}
      #GRAYLOG_HTTP_BIND_ADDRESS: http://127.0.0.1:9000/
      GRAYLOG_HTTP_EXTERNAL_URI: https://host.domain.com/
      GRAYLOG_MONGODB_URI: "mongodb://mongo_admin:{{ GRAYLOG_MONGO_PW }}@mongodb01a:27017,mongodb01b:27017,mongodb01c:27017/graylog?replicaSet=rs0"

    volumes:
      - /path/to/graylog/graylog/journal:/usr/share/graylog/data/journal
      - /path/to/graylog/graylog/node:/usr/share/graylog/data/config/node
      - /path/to/graylog/graylog/lookups:/usr/share/graylog/lookups

    ports:
        - target: 5515
          published: 5515
          protocol: udp
          mode: host
          
        - target: 5515
          published: 5515
          protocol: tcp
          mode: host        
        
    deploy:
      replicas: 3
      endpoint_mode: dnsrr
      resources:
        limits:
          memory: '8G'

  graylog_master:
    image: "internal_docker_repo:5000/graylog:{{ GRAYLOG_VERSION }}"
    build: ./images/graylog/.
    environment:
      GRAYLOG_IS_MASTER: "true"
      GRAYLOG_ELASTICSEARCH_HOSTS: http://elasticsearch:9200
      GRAYLOG_SERVER_JAVA_OPTS: -XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap -XX:NewRatio=1 -XX:MaxMetaspaceSize=256m -server -XX:+ResizeTLAB -XX:+UseConcMarkSweepGC -XX:+CMSConcurrentMTEnabled -XX:+CMSClassUnloadingEnabled -XX:+UseParNewGC -XX:-OmitStackTraceInFastThrow
      GRAYLOG_PASSWORD_SECRET: {{ GRAYLOG_PASSWORD_SECRET }}
      GRAYLOG_ROOT_PASSWORD_SHA2: {{ GRAYLOG_ROOT_PASSWORD_SHA2 }}
      GRAYLOG_HTTP_EXTERNAL_URI: https://host.domain.com/
      GRAYLOG_MONGODB_URI: "mongodb://mongo_admin:{{ GRAYLOG_MONGO_PW }}@mongodb01a:27017,mongodb01b:27017,mongodb01c:27017/graylog?replicaSet=rs0"

    volumes:
      - /path/to/graylog/graylog_master/lookups:/usr/share/graylog/lookups
      - /path/to/graylog/graylog_master/archive:/usr/share/graylog/data/archive
      - /path/to/graylog/graylog_master/node:/usr/share/graylog/data/config/node

    ports:
       - target: 9000
         published: 9000
         protocol: tcp
    resources:
        limits:
          memory: '8G'
        reservations:
          memory: '7.9G'
    
  cerebro:
    image: yannart/cerebro:latest
    ports:
      - 8300:9000
    deploy:
      mode: global

networks:
  default:
    external:
      name:  graylog_overlay

5. Build out your docker stack:

cd /path/to/graylog/
docker-compose build
docker-compose push
docker stack deploy graylog_stack -c docker-compose.yml

Portainer can used to visualize the health of the stack and diagnose any issues.
Cerebro is nice for visualizing the health of the elastic cluster.

Good luck!!!

ps - here is our ansible playbook if anyone finds it helpful:

- hosts: all
  become: true
  become_method: sudo
  tasks:
    - name: Add Graylog User
      user:
        name: graylog
        comment: Graylog docker user
        uid: 1100
        group: 1100
      
    - name: initial setup block
      run_once: true
      block:
        - template:
            src: ./templates/docker-compose.yml
            dest: "{{ COMPOSE_DIR }}/docker-compose.yml"
        
        - template:
            src: ./templates/Dockerfile
            dest: "{{ COMPOSE_DIR }}/images/graylog/Dockerfile"


        - template:
            src: ./templates/docker-entrypoint.sh
            dest: "{{ COMPOSE_DIR }}/images/graylog/docker-entrypoint.sh"
            mode: '0775'
            
        - name: Build and Run
          shell: |
            docker-compose build
            docker-compose push
            docker stack rm graylog_stack
            docker stack deploy graylog_stack -c docker-compose.yml
          args:
            chdir: "{{ COMPOSE_DIR }}"
    
    - name: Add Graylog's folders
      file:
        path: "/path/to/{{ item }}"
        state: directory
        mode: '0755'
        owner: graylog
        group: graylog
        
      loop:
        - graylog/
        - graylog/graylog/journal
        - graylog/graylog/node
        - graylog/graylog/lookups
        - graylog/graylog_master/node
        - graylog/graylog_master/lookups

    - name: Add Folder for gzipped archives
      file:
        path: /path/to/graylog_master/archive"
        state: directory
        mode: '0775'

    - name: Add Data folder for Elasticsearch
      file:
        path: "/path/to/graylog/elastic"
        state: directory
        mode: '0775'
        owner: 1000
        group: 1000
        

    - name: Add Data folder for Mongo
      file:
        path: "/path/to/graylog/mongo/db"
        state: directory
        mode: '0755'
        owner: 999
        group: 999


    - name: wait while the mongo cluster joins up
      pause: seconds=10

    - name: setup mongo, this was a test to force a node to be primary during setup, doesnt work well
      run_once: true
      shell: "docker exec -it $(docker ps | grep graylog_stack_mongo | awk '{ split($0,a,\" \"); print a[1]}') mongo --eval 'rs.status(); cfg = rs.conf(); cfg.members[0].priority = 1000; cfg.members[1].priority = 0.5;  cfg.members[2].priority = 0.5; rs.reconfig(cfg, { force: true });'"
      register: output
      
    - name: setup mongo, this fails a bunch, might need to manually do
      run_once: true
      shell: "docker exec -it $(docker ps | grep graylog_stack_mongo | awk '{ split($0,a,\" \"); print a[1]}') mongo --eval 'rs.initiate({ _id: \"rs0\", members: [{ _id: 1, host: \"mongodb01a:27017\" }, { _id: 2, host: \"mongodb01b:27017\" }, { _id: 3, host: \"mongodb01c:27017\" }], settings: { getLastErrorDefaults: { w: \"majority\", wtimeout: 30000 }}})'"

    - name: setup mongo user
      run_once: true
      template: 
        src: ./templates/mongo_create_user.sh
        dest: "{{ COMPOSE_DIR }}/mongo_create_user.sh"
        
    - name: run create user
      run_once: true
      shell: "{{ COMPOSE_DIR }}/mongo_create_user.sh"
      when: "'dockerhost01' in inventory_hostname"
      register: useroutput
      ignore_errors: true
      
    - debug: msg="{{ useroutput }}"
    
    - fail: msg="Mongo db not master, unable to make changes"
      when: '"not master" in useroutput.stdout'

Mongo_create_user.sh

#/bin/bash
docker exec -it $(docker ps | grep graylog_stack_mongo | awk '{ split($0,a," "); print a[1]}') mongo graylog --eval "db.createUser( { user: \"mongo_admin\", pwd: \"{{ GRAYLOG_MONGO_PW }}\", roles: [ { role: \"root\", db: \"admin\" } ] })"

[jan (Jan Doberstein)]

thank you for sharing.