Graylog container fails to start


(Kiran Kumar) #1

Hi,

I am trying to run Graylog on Kubernetes. I am using a mongodb and elasticsearch hosted outside Kubernetes and trying to start the containers. Although on container starting up, the graylog database is created and the node config is pushed to MongoDB. Graylog container fails to start.

It throws the following error:

2019-01-15 23:24:43,975 INFO : org.graylog2.bootstrap.ServerBootstrap - Graylog server 2.4.6+ceaa7e4 starting up 2019-01-15 23:24:43,976 INFO : org.graylog2.bootstrap.ServerBootstrap - JRE: Oracle Corporation 1.8.0_181 on Linux 4.4.148-k8s 2019-01-15 23:24:43,976 INFO : org.graylog2.bootstrap.ServerBootstrap - Deployment: docker 2019-01-15 23:24:43,976 INFO : org.graylog2.bootstrap.ServerBootstrap - OS: Debian GNU/Linux 9 (stretch) (debian) 2019-01-15 23:24:43,976 INFO : org.graylog2.bootstrap.ServerBootstrap - Arch: amd64 Exception in thread "main" com.mongodb.MongoQueryException: Query failed with error code 2 and error message 'error processing query: ns=graylog.nodesTree: $and is_master == true type == "SERVER" last_seen $gte 1547594681 $not node_id == "2568a867-4398-44bf-a740-b0a160e10a66" Sort: {} Proj: {} No query solutions' on server 172.30.17.9:27017 at com.mongodb.operation.FindOperation$1.call(FindOperation.java:720) at com.mongodb.operation.FindOperation$1.call(FindOperation.java:709) at com.mongodb.operation.OperationHelper.withConnectionSource(OperationHelper.java:433) at com.mongodb.operation.OperationHelper.withConnection(OperationHelper.java:406) at com.mongodb.operation.FindOperation.execute(FindOperation.java:709) at com.mongodb.operation.FindOperation.execute(FindOperation.java:81) at com.mongodb.Mongo.execute(Mongo.java:810) at com.mongodb.Mongo$2.execute(Mongo.java:797) at com.mongodb.DBCursor.initializeCursor(DBCursor.java:871) at com.mongodb.DBCursor.hasNext(DBCursor.java:142) at com.google.common.collect.Iterators.addAll(Iterators.java:372) at com.google.common.collect.Lists.newArrayList(Lists.java:165) at com.google.common.collect.Lists.newArrayList(Lists.java:150) at org.graylog2.database.PersistedServiceImpl.cursorToList(PersistedServiceImpl.java:121) at org.graylog2.database.PersistedServiceImpl.query(PersistedServiceImpl.java:68) at org.graylog2.database.PersistedServiceImpl.query(PersistedServiceImpl.java:72) at org.graylog2.cluster.NodeServiceImpl.isOnlyMaster(NodeServiceImpl.java:154) at org.graylog2.commands.Server.startNodeRegistration(Server.java:159) at org.graylog2.bootstrap.ServerBootstrap.startCommand(ServerBootstrap.java:120) at org.graylog2.bootstrap.CmdLineTool.run(CmdLineTool.java:209) at org.graylog2.bootstrap.Main.main(Main.java:44)

This is the statelessset using to bringup graylog containers:

apiVersion: v1
kind: ConfigMap
metadata:
name: graylog
namespace: graylog
labels:
component: graylog
data:
master-override.conf: |
is_master=true
slave-override.conf: |
is_master=false
apiVersion: apps/v1beta2
kind: StatefulSet
metadata:
name: graylog
namespace: graylog
labels:
component: graylog
spec:
selector:
matchLabels:
component: graylog
updateStrategy:
type: RollingUpdate
serviceName: graylog-svc
podManagementPolicy: OrderedReady
replicas: 1
template:
metadata:
labels:
component: graylog
spec:
initContainers:
- name: graylog-init
image: graylog/graylog:2.4
imagePullPolicy: IfNotPresent
command:
- bash
- “-c”
- |
updateMainConfigFile () {
overrideFile=$1
mainFile=$2
cat overrideFile | egrep -v "^[[:space:]]*|^[[:space:]]#" |
while read line; do
key=${line%%=
}
cat $mainFile | sed "s/^$key./$line/" > /tmp/config.tmp
mv /tmp/config.tmp mainFile done } # Retrieve all files cp -R /usr/share/graylog/data/config/* /tmp/graylog/conf/ # Override main config file if [[ {HOSTNAME##
-} == 0 ]]; then
updateMainConfigFile /tmp/config-map/master-override.conf /tmp/graylog/conf/graylog.conf
else
updateMainConfigFile /tmp/config-map/slave-override.conf /tmp/graylog/conf/graylog.conf
fi
volumeMounts:
- name: conf
mountPath: /tmp/graylog/conf
- name: config-map
mountPath: /tmp/config-map
terminationGracePeriodSeconds: 10
containers:
- name: graylog
image: graylog/graylog:2.4
imagePullPolicy: IfNotPresent
readinessProbe:
httpGet:
path: /api/system/lbstatus
port: 12900
periodSeconds: 2
initialDelaySeconds: 20
failureThreshold: 2
successThreshold: 1
timeoutSeconds: 5
ports:
- containerPort: 12900
- containerPort: 9000
volumeMounts:
- name: conf
mountPath: /usr/share/graylog/data/config
env:
- name: NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: NODE_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: “GRAYLOG_ELASTICSEARCH_HOSTS”
value: “”
- name: “GRAYLOG_ELASTICSEARCH_DISCOVERY_ENABLED”
value: “true”
- name: “GRAYLOG_MONGODB_URI”
value: “mongodb://:27017/graylog”
- name: “GRAYLOG_PASSWORD_SECRET”
value: “somepasswordpepper”
- name: “GRAYLOG_ROOT_PASSWORD_SHA2”
value: “8c6976e5b5410415bde908bd4dee15dfb167a9c873fc4bb8a81f6f2ab448a918”
- name: “GRAYLOG_WEB_ENDPOINT_URI”
value: “domain-url”
- name: “GRAYLOG_REST_TRANSPORT_URI”
value: “http://graylog-svc.graylog:9000/api/
- name: “GRAYLOG_REST_LISTEN_URI”
value: “http://graylog-svc.graylog:12900/api/
volumes:
- name: conf
emptyDir: {}
- name: config-map
configMap:
name: graylog
apiVersion: v1
kind: Service
metadata:
labels:
component: graylog
name: graylog-svc
namespace: graylog
spec:
selector:
component: graylog
ports:
port: 9000
targetPort: 9000
protocol: TCP
name: web-ui
port: 12900
targetPort: 12900
protocol: TCP
name: api
port: 5044
protocol: TCP
targetPort: 5044
name: input-beat
port: 12201
protocol: UDP
targetPort: 12201
name: input-gelf-udp
sessionAffinity: None
type: ClusterIP
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: graylog-ingress
namespace: graylog
spec:
rules:
host: mydomain-url
http:
paths:
path: /api
backend:
serviceName: graylog-svc
servicePort: 12900
path: /
backend:
serviceName: graylog-svc
servicePort: 9000

Elasticsearch version : 5.6
MongoDB version: 3.4.18


(Jan Doberstein) #2
  • name: “GRAYLOG_MONGODB_URI”
    value: “mongodb://:27017/graylog”

It looks like you redacted some parts of the configuration.

The Error indicates that your Graylog is not able to connect to MongoDB or that something is not working as it should with this part.


(Kiran Kumar) #3

I am not sure its a connection problem. Here is the complete log and it says that the connection has been established initially

2019-01-16 09:21:19,794 INFO : org.graylog2.bootstrap.CmdLineTool - Loaded plugin: AWS plugins 2.4.6 [org.graylog.aws.plugin.AWSPlugin]
2019-01-16 09:21:19,796 INFO : org.graylog2.bootstrap.CmdLineTool - Loaded plugin: Elastic Beats Input 2.4.6 [org.graylog.plugins.beats.BeatsInputPlugin]
2019-01-16 09:21:19,797 INFO : org.graylog2.bootstrap.CmdLineTool - Loaded plugin: CEF Input 2.4.6 [org.graylog.plugins.cef.CEFInputPlugin]
2019-01-16 09:21:19,797 INFO : org.graylog2.bootstrap.CmdLineTool - Loaded plugin: Collector 2.4.6 [org.graylog.plugins.collector.CollectorPlugin]
2019-01-16 09:21:19,798 INFO : org.graylog2.bootstrap.CmdLineTool - Loaded plugin: Enterprise Integration Plugin 2.4.6 [org.graylog.plugins.enterprise_integration.EnterpriseIntegrationPlugin]
2019-01-16 09:21:19,799 INFO : org.graylog2.bootstrap.CmdLineTool - Loaded plugin: MapWidgetPlugin 2.4.6 [org.graylog.plugins.map.MapWidgetPlugin]
2019-01-16 09:21:19,799 INFO : org.graylog2.bootstrap.CmdLineTool - Loaded plugin: NetFlow Plugin 2.4.6 [org.graylog.plugins.netflow.NetFlowPlugin]
2019-01-16 09:21:19,806 INFO : org.graylog2.bootstrap.CmdLineTool - Loaded plugin: Pipeline Processor Plugin 2.4.6 [org.graylog.plugins.pipelineprocessor.ProcessorPlugin]
2019-01-16 09:21:19,806 INFO : org.graylog2.bootstrap.CmdLineTool - Loaded plugin: Threat Intelligence Plugin 2.4.6 [org.graylog.plugins.threatintel.ThreatIntelPlugin]
2019-01-16 09:21:20,117 INFO : org.graylog2.bootstrap.CmdLineTool - Running with JVM arguments: -XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap -XX:NewRatio=1 -XX:MaxMetaspaceSize=256m -XX:+ResizeTLAB -XX:+UseConcMarkSweepGC -XX:+CMSConcurrentMTEnabled -XX:+CMSClassUnloadingEnabled -XX:+UseParNewGC -XX:-OmitStackTraceInFastThrow -Dlog4j.configurationFile=/usr/share/graylog/data/config/log4j2.xml -Djava.library.path=/usr/share/graylog/lib/sigar/ -Dgraylog2.installation_source=docker
2019-01-16 09:21:20,304 INFO : org.hibernate.validator.internal.util.Version - HV000001: Hibernate Validator 5.1.3.Final
2019-01-16 09:21:22,644 INFO : org.graylog2.shared.buffers.InputBufferImpl - Message journal is enabled.
2019-01-16 09:21:22,662 INFO : org.graylog2.plugin.system.NodeId - No node ID file found. Generated: dec7fa69-ae3b-488f-970d-ef707e727345
2019-01-16 09:21:22,830 INFO : kafka.log.LogManager - Loading logs.
2019-01-16 09:21:22,834 INFO : kafka.log.LogManager - Logs loading complete.
2019-01-16 09:21:22,866 INFO : kafka.log.LogManager - Created log for partition [messagejournal,0] in /usr/share/graylog/data/journal with properties {file.delete.delay.ms -> 60000, compact -> false, max.message.bytes -> 104857600, min.insync.replicas -> 1, segment.jitter.ms -> 0, index.interval.bytes -> 4096, min.cleanable.dirty.ratio -> 0.5, unclean.leader.election.enable -> true, retention.bytes -> 5368709120, delete.retention.ms -> 86400000, flush.ms -> 60000, segment.bytes -> 104857600, segment.ms -> 3600000, retention.ms -> 43200000, flush.messages -> 1000000, segment.index.bytes -> 1048576}.
2019-01-16 09:21:22,867 INFO : org.graylog2.shared.journal.KafkaJournal - Initialized Kafka based journal at /usr/share/graylog/data/journal
2019-01-16 09:21:22,883 INFO : org.graylog2.shared.buffers.InputBufferImpl - Initialized InputBufferImpl with ring size <65536> and wait strategy , running 2 parallel message handlers.
2019-01-16 09:21:22,906 INFO : org.mongodb.driver.cluster - Cluster created with settings {hosts=[172.30.17.9:27017], mode=SINGLE, requiredClusterType=UNKNOWN, serverSelectionTimeout=‘30000 ms’, maxWaitQueueSize=500}
2019-01-16 09:21:22,955 INFO : org.mongodb.driver.cluster - No server chosen by ReadPreferenceServerSelector{readPreference=primary} from cluster description ClusterDescription{type=UNKNOWN, connectionMode=SINGLE, serverDescriptions=[ServerDescription{address=172.30.17.9:27017, type=UNKNOWN, state=CONNECTING}]}. Waiting for 30000 ms before timing out
2019-01-16 09:21:22,964 INFO : org.mongodb.driver.connection - Opened connection [connectionId{localValue:1, serverValue:18}] to 172.30.17.9:27017
2019-01-16 09:21:22,967 INFO : org.mongodb.driver.cluster - Monitor thread successfully connected to server with description ServerDescription{address=172.30.17.9:27017, type=STANDALONE, state=CONNECTED, ok=true, version=ServerVersion{versionList=[3, 4, 18]}, minWireVersion=0, maxWireVersion=5, maxDocumentSize=16777216, roundTripTimeNanos=1522053}
2019-01-16 09:21:22,975 INFO : org.mongodb.driver.connection - Opened connection [connectionId{localValue:2, serverValue:19}] to 172.30.17.9:27017
2019-01-16 09:21:23,284 INFO : io.searchbox.client.AbstractJestClient - Setting server pool to a list of 1 servers: [https://vpc-graylog-ye3mra52h2mmv7fqagg3gnim54.eu-central-1.es.amazonaws.com/]
2019-01-16 09:21:23,285 INFO : io.searchbox.client.JestClientFactory - Using multi thread/connection supporting pooling connection manager
2019-01-16 09:21:23,359 INFO : io.searchbox.client.JestClientFactory - Using custom ObjectMapper instance
2019-01-16 09:21:23,359 INFO : io.searchbox.client.JestClientFactory - Node Discovery enabled…
2019-01-16 09:21:23,368 INFO : io.searchbox.client.JestClientFactory - Idle connection reaping disabled…
2019-01-16 09:21:23,654 INFO : io.searchbox.client.AbstractJestClient - Setting server pool to a list of 0 servers:
2019-01-16 09:21:23,654 WARN : io.searchbox.client.AbstractJestClient - No servers are currently available to connect.
2019-01-16 09:21:23,721 INFO : org.graylog2.shared.buffers.ProcessBuffer - Initialized ProcessBuffer with ring size <65536> and wait strategy .
2019-01-16 09:21:24,981 INFO : org.graylog2.bindings.providers.RulesEngineProvider - No static rules file loaded.
2019-01-16 09:21:25,112 WARN : org.graylog.plugins.map.geoip.GeoIpResolverEngine - GeoIP database file does not exist: /etc/graylog/server/GeoLite2-City.mmdb
2019-01-16 09:21:25,122 INFO : org.graylog2.buffers.OutputBuffer - Initialized OutputBuffer with ring size <65536> and wait strategy .
2019-01-16 09:21:25,154 WARN : org.graylog.plugins.map.geoip.GeoIpResolverEngine - GeoIP database file does not exist: /etc/graylog/server/GeoLite2-City.mmdb
2019-01-16 09:21:25,160 INFO : org.mongodb.driver.connection - Opened connection [connectionId{localValue:3, serverValue:20}] to 172.30.17.9:27017
2019-01-16 09:21:25,182 WARN : org.graylog.plugins.map.geoip.GeoIpResolverEngine - GeoIP database file does not exist: /etc/graylog/server/GeoLite2-City.mmdb
2019-01-16 09:21:25,210 WARN : org.graylog.plugins.map.geoip.GeoIpResolverEngine - GeoIP database file does not exist: /etc/graylog/server/GeoLite2-City.mmdb
2019-01-16 09:21:25,240 WARN : org.graylog.plugins.map.geoip.GeoIpResolverEngine - GeoIP database file does not exist: /etc/graylog/server/GeoLite2-City.mmdb
2019-01-16 09:21:25,586 INFO : org.graylog2.bootstrap.ServerBootstrap - Graylog server 2.4.6+ceaa7e4 starting up
2019-01-16 09:21:25,587 INFO : org.graylog2.bootstrap.ServerBootstrap - JRE: Oracle Corporation 1.8.0_181 on Linux 4.4.148-k8s
2019-01-16 09:21:25,587 INFO : org.graylog2.bootstrap.ServerBootstrap - Deployment: docker
2019-01-16 09:21:25,587 INFO : org.graylog2.bootstrap.ServerBootstrap - OS: Debian GNU/Linux 9 (stretch) (debian)
2019-01-16 09:21:25,587 INFO : org.graylog2.bootstrap.ServerBootstrap - Arch: amd64 Exception in thread “main” com.mongodb.MongoQueryException: Query failed with error code 2 and error message ‘error processing query: ns=graylog.nodesTree:
$and is_master == true
type == “SERVER”
last_seen
$gte 1547594681
$not
node_id == “dec7fa69-ae3b-488f-970d-ef707e727345”
Sort: {}
Proj: {}
No query solutions’ on server 172.30.17.9:27017
at com.mongodb.operation.FindOperation$1.call(FindOperation.java:720)
at com.mongodb.operation.FindOperation$1.call(FindOperation.java:709)
at com.mongodb.operation.OperationHelper.withConnectionSource(OperationHelper.java:433)
at com.mongodb.operation.OperationHelper.withConnection(OperationHelper.java:406)
at com.mongodb.operation.FindOperation.execute(FindOperation.java:709)
at com.mongodb.operation.FindOperation.execute(FindOperation.java:81)
at com.mongodb.Mongo.execute(Mongo.java:810)
at com.mongodb.Mongo$2.execute(Mongo.java:797)
at com.mongodb.DBCursor.initializeCursor(DBCursor.java:871)
at com.mongodb.DBCursor.hasNext(DBCursor.java:142)
at com.mongodb.DBCursor.one(DBCursor.java:680)
at com.mongodb.DBCollection.findOne(DBCollection.java:831)
at com.mongodb.DBCollection.findOne(DBCollection.java:794)
at com.mongodb.DBCollection.findOne(DBCollection.java:741)
at org.graylog2.database.PersistedServiceImpl.findOne(PersistedServiceImpl.java:128)
at org.graylog2.cluster.NodeServiceImpl.byNodeId(NodeServiceImpl.java:73)
at org.graylog2.cluster.NodeServiceImpl.registerServer(NodeServiceImpl.java:58)
at org.graylog2.commands.Server.startNodeRegistration(Server.java:154)
at org.graylog2.bootstrap.ServerBootstrap.startCommand(ServerBootstrap.java:120)
at org.graylog2.bootstrap.CmdLineTool.run(CmdLineTool.java:209)
at org.graylog2.bootstrap.Main.main(Main.java:44)

Also in the mongodb, graylog database and entry in nodes collection will get created during the container startup
db.nodes.find()

{ “_id” : ObjectId(“5c3e698eba561100015e8ce6”), “is_master” : true, “hostname” : “graylog-0.graylog-svc.graylog.svc.cluster.local”, “last_seen” : 1547594126, “type” : “SERVER”, “transport_address” : “http://graylog-svc.graylog:9000/api/”, “node_id” : “dec7fa69-ae3b-488f-970d-ef707e727345” }

Am I missing something else here?


(Jan Doberstein) #4
2019-01-16 09:21:25,587 INFO : org.graylog2.bootstrap.ServerBootstrap - Arch: amd64 Exception in thread “main” com.mongodb.MongoQueryException: Query failed with error code 2 and error message ‘error processing query: ns=graylog.nodesTree:
$and is_master == true
type == “SERVER”
last_seen
$gte 1547594681
$not
node_id == “dec7fa69-ae3b-488f-970d-ef707e727345”
Sort: {}
Proj: {}
No query solutions’ on server 172.30.17.9:27017

Something is wrong with your MongoDB. I do not know what.


(Kiran Kumar) #5

Thanks Jan for the input. Looks like you were right and there was some issue with the mongo. It was not able to run queries. When I tried to run the queries directly on the shell, I got the same error even though I could see the data being stored.

db.nodes.find({“is_master” : true})
Error: error: {
“ok” : 0,
“errmsg” : “error processing query: ns=graylog.nodesTree: is_master == true\nSort: {}\nProj: {}\n No query solutions”,
“code” : 2,
“codeName” : “BadValue”
}

I had to do a fresh install of mongo and things started working again. I couldn’t pin point the issue with mongo but re-setup did the trick.


(system) closed #6

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.