Graylog_0 index not created when etcd run on separate server

gautamvij · December 12, 2017, 5:34pm

Overview

Followed this http://docs.graylog.org/en/2.3/pages/configuration/graylog_ctl.html. using aws ami from https://github.com/Graylog2/graylog2-images/tree/2.3/aws.
I am trying to setup graylog on auto scaling group. So, for that I will have to keep 3 types of servers
1- Graylog web server
2- Elasticsearch Nodes
3- Etcd

Context

1: Elasticsearch nodes needs etcd service running for their ips to be shared among other elasticsearch nodes.
2: Running sudo graylog-ctl reconfigure-as-server on graylog web server will set the ips of elastic search nodes in /opt/graylog/conf/graylog.conf with property elasticsearch_hosts.

Concluding from above,

We want etcd service up first,
then all the elastic search data nodes,
then in last graylogweb server with mongodb.

My Target

For automatic production setup,
Whenever a new instance is spawned of any type (GraylogWebserver, ElasticSearchNode, Etcd).
I will have one static ip for ETCD server that I will use in init script of " Graylog web server without etcd, Elasticsearch Node".

Steps followed:

#vm1 - To run Etcd only
#vm2 - To run graylog web server without etcd
#vm3 - To run elastic search nodes


vm1: sudo graylog-ctl reconfigure-as-server
vm1: sudo graylog-ctl stop
vm1: sudo graylog-ctl start etcd 

vm3: sudo graylog-ctl set-cluster-master <ip-of-vm1>
vm3: sudo graylog-ctl reconfigure-as-datanode

vm2: sudo graylog-ctl set-cluster-master <ip-of-vm1>
vm2: sudo graylog-ctl set-external-ip http://<public-ip-of-vm2>:9000/api
vm2: sudo graylog-ctl reconfigure-as-server
vm2: open /opt/graylog/conf/graylog.conf
vm2: update mongodb_uri which has <ip-of-vm1> to <ip-of-vm2>
vm2: sudo graylog-ctl stop
vm2: sudo graylog-ctl start graylog-server
vm2: sudo graylog-ctl start mongodb
vm2: sudo graylog-ctl start nginx

Expected Behaviour

On vm3, graylog_0 indice is created in elasticsearch.

Current Behavior

On vm3 no indice is created.

Note

Even if I manually create the graylog_0 indice from api with the same config. I get error on web server about indices not set.

curl -XPUT '<ip-of-vm3>:9200/graylog_0?pretty' -H 'Content-Type: application/json' -d'
{
  "aliases" : {
    "graylog_deflector" : { }
  },
  "mappings" : {
    "message" : {
      "dynamic_templates" : [
        {
          "internal_fields" : {
            "match" : "gl2_*",
            "mapping" : {
              "type" : "keyword"
            }
          }
        },
        {
          "store_generic" : {
            "match" : "*",
            "mapping" : {
              "index" : "not_analyzed"
            }
          }
        }
      ],
      "properties" : {
        "application_name" : {
          "type" : "keyword"
        },
        "connection_id" : {
          "type" : "long"
        },
        "connection_requests" : {
          "type" : "long"
        },
        "facility" : {
          "type" : "keyword"
        },
        "from_gelf" : {
          "type" : "keyword"
        },
        "from_syslog" : {
          "type" : "keyword"
        },
        "full_message" : {
          "type" : "text",
          "analyzer" : "standard"
        },
        "gl2_remote_ip" : {
          "type" : "keyword"
        },
        "gl2_remote_port" : {
          "type" : "keyword"
        },
        "gl2_source_input" : {
          "type" : "keyword"
        },
        "gl2_source_node" : {
          "type" : "keyword"
        },
        "http_referer" : {
          "type" : "keyword"
        },
        "http_user_agent" : {
          "type" : "keyword"
        },
        "http_version" : {
          "type" : "keyword"
        },
        "level" : {
          "type" : "long"
        },
        "message" : {
          "type" : "text",
          "analyzer" : "standard"
        },
        "millis" : {
          "type" : "float"
        },
        "process_id" : {
          "type" : "keyword"
        },
        "remote_addr" : {
          "type" : "keyword"
        },
        "remote_user" : {
          "type" : "keyword"
        },
        "request_path" : {
          "type" : "keyword"
        },
        "request_verb" : {
          "type" : "keyword"
        },
        "response_bytes" : {
          "type" : "long"
        },
        "response_status" : {
          "type" : "long"
        },
        "source" : {
          "type" : "text",
          "analyzer" : "analyzer_keyword",
          "fielddata" : true
        },
        "streams" : {
          "type" : "keyword"
        },
        "timestamp" : {
          "type" : "date",
          "format" : "yyyy-MM-dd HH:mm:ss.SSS"
        }
      }
    }
  },
  "settings" : {
    "index" : {
      "number_of_shards" : "4",
      "analysis" : {
        "analyzer" : {
          "analyzer_keyword" : {
            "filter" : "lowercase",
            "tokenizer" : "keyword"
          }
        }
      },
      "number_of_replicas" : "1"
    }
  }
}
'

My guess is there must be some db entry in mongo for the same which is not update if we create manually.

Possible Solution

(Long Term) Support for running etcd server separately in amazon ami
(Short Term) Some way of creating indice in elasticsearch and updating in mongodb as well

AWS AMI: ami-6d9ede0e

jan · December 13, 2017, 7:10am

you run that AMI in an unsupported manner. Editing the /opt/graylog/conf/graylog.conf will break your setup on next update (or run of graylog-ctl reconfigure) as this file is autogenerated.

If you want to run such a setup, i highly recommend to make a fresh installation and not using the AMI for that as this is just a showcase that you are able to look at Graylog. But it was never build to run in production.

gautamvij · December 15, 2017, 4:30am

Even if we go by the documentation, we have the same problem.
As per the documentation http://docs.graylog.org/en/2.3/pages/configuration/graylog_ctl.html#multi-vm-setup.

We cannot setup data-node first because sudo graylog-ctl set-cluster-master <ip-of-vm1> will need etcd running on master server.

After running sudo graylog-ctl reconfigure-as-server on master cluster, it will also start graylog-ctl which will also setup the graylog server. But since no data node was started and no ip was registered on etcd service. Therefore, graylog web server will not have ips of datanode set in elasticssearch.hosts.

For that we will have to re run sudo graylog-ctl reconfigure-as-server after setting up data node.

jan · December 15, 2017, 8:28am