Graylog 3.3.X doesn't start. Graylog 3.2.X does

Hi folks,

I’m having an issue with Graylog 3.3.X. It does not start. 3.2.X starts and runs as expected but 3.3.X does not. I do see errors in the log.

  • Graylog Version: 3.3.2 (also does this with 3.3.1, 3.2.6 works as expected)
  • Elasticsearch Version: 6.5.4
  • MongoDB Version: 3.2.22
  • Operating System: Centos 7.8 Oracle Corporation 1.8.0_252 on Linux 3.10.0-1127.13.1.el7.x86_64
  • Browser version: Firefox 68.10

10 errors
at com.google.inject.internal.Errors.throwCreationExceptionIfErrorsExist(Errors.java:543)
at com.google.inject.internal.InternalInjectorCreator.injectDynamically(InternalInjectorCreator.java:186)
at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:109)
at com.google.inject.Guice.createInjector(Guice.java:87)
at org.graylog2.shared.bindings.GuiceInjectorHolder.createInjector(GuiceInjectorHolder.java:34)
at org.graylog2.bootstrap.CmdLineTool.setupInjector(CmdLineTool.java:379)
at org.graylog2.bootstrap.CmdLineTool.run(CmdLineTool.java:194)
at org.graylog2.bootstrap.Main.main(Main.java:50)

@tbmay I would like to recommend you go with the below system requirement documentation link to get align with supported version of ES and MongoDB for Graylog 3.3.

https://docs.graylog.org/en/3.3/pages/installation.html#system-requirements

Hope this will helps you

Thanks a million. I missed that. Looks like mongo is the problem.

Thanks a million for your suggestion. I updated mongo to 3.6 and it’s still doing the same thing. Unless I’m missing something all the software is versioned correctly now.

[root@graylog1 log]# tail graylog-server.log

10 errors
at com.google.inject.internal.Errors.throwCreationExceptionIfErrorsExist(Errors.java:543)
at com.google.inject.internal.InternalInjectorCreator.injectDynamically(InternalInjectorCreator.java:186)
at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:109)
at com.google.inject.Guice.createInjector(Guice.java:87)
at org.graylog2.shared.bindings.GuiceInjectorHolder.createInjector(GuiceInjectorHolder.java:34)
at org.graylog2.bootstrap.CmdLineTool.setupInjector(CmdLineTool.java:379)
at org.graylog2.bootstrap.CmdLineTool.run(CmdLineTool.java:194)
at org.graylog2.bootstrap.Main.main(Main.java:50)

he @tbmay

with that little information it is not possible to identify what could be wrong.

The log of a complete startup cycle would be needed for that.

Thank you Jan. Where do I attach it?

paste it here inside of a code block for example.

Thanks a million Jan. I noticed this at the top. I went through all my extractors and don’t see that lookup table referenced.

2020-07-17 18:56:35,248 ERROR: org.graylog2.inputs.InputServiceImpl - Cannot build extractor from persisted data. Skipping.
java.lang.IllegalStateException: Configured lookup table doesn’t exist

I’ll attach the whole dump.

2020-07-17 18:56:30,796 INFO : org.graylog2.bootstrap.CmdLineTool - Loaded plugin: SyslogOutputPlugin 1.0.0 [com.wizecore.graylog2.plugin.SyslogOutput]
2020-07-17 18:56:30,799 INFO : org.graylog2.bootstrap.CmdLineTool - Loaded plugin: AWS plugins 3.3.2 [org.graylog.aws.AWSPlugin]
2020-07-17 18:56:30,800 INFO : org.graylog2.bootstrap.CmdLineTool - Loaded plugin: Collector 3.3.2 [org.graylog.plugins.collector.CollectorPlugin]
2020-07-17 18:56:30,801 INFO : org.graylog2.bootstrap.CmdLineTool - Loaded plugin: Threat Intelligence Plugin 3.3.2 [org.graylog.plugins.threatintel.ThreatIntelPlugin]
2020-07-17 18:56:31,115 INFO : org.graylog2.bootstrap.CmdLineTool - Running with JVM arguments: -Djava.library.path=./../lib/sigar -Xms8g -Xmx8g -XX:NewRatio=1 -XX:+ResizeTLAB -XX:+UseConcMarkSweepGC -XX:+CMSConcurrentMTEnabled -XX:+CMSClassUnloadingEnabled -XX:+UseParNewGC -XX:-OmitStackTraceInFastThrow
2020-07-17 18:56:31,336 INFO : org.hibernate.validator.internal.util.Version - HV000001: Hibernate Validator null
2020-07-17 18:56:31,527 INFO : org.graylog2.shared.system.stats.SigarService - Failed to load SIGAR. Falling back to JMX implementations.
2020-07-17 18:56:33,734 INFO : org.graylog2.shared.buffers.InputBufferImpl - Message journal is enabled.
2020-07-17 18:56:33,756 INFO : org.graylog2.plugin.system.NodeId - Node ID: d811ae20-c3f5-4571-9983-0852deeaabd2
2020-07-17 18:56:33,931 INFO : kafka.log.LogManager - Loading logs.
2020-07-17 18:56:33,973 WARN : kafka.log.Log - Found a corrupted index file, /opt/graylog/data/journal/messagejournal-0/00000000000000000000.index, deleting and rebuilding index...
2020-07-17 18:56:34,015 INFO : kafka.log.LogManager - Logs loading complete.
2020-07-17 18:56:34,019 INFO : org.graylog2.shared.journal.KafkaJournal - Initialized Kafka based journal at data/journal
2020-07-17 18:56:34,043 INFO : org.mongodb.driver.cluster - Cluster created with settings {hosts=[localhost:27017], mode=SINGLE, requiredClusterType=UNKNOWN, serverSelectionTimeout='30000 ms', maxWaitQueueSize=5000}
2020-07-17 18:56:34,100 INFO : org.mongodb.driver.cluster - Cluster description not yet available. Waiting for 30000 ms before timing out
2020-07-17 18:56:34,129 INFO : org.mongodb.driver.connection - Opened connection [connectionId{localValue:1, serverValue:1}] to localhost:27017
2020-07-17 18:56:34,133 INFO : org.mongodb.driver.cluster - Monitor thread successfully connected to server with description ServerDescription{address=localhost:27017, type=STANDALONE, state=CONNECTED, ok=true, version=ServerVersion{versionList=[3, 6, 18]}, minWireVersion=0, maxWireVersion=6, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=null, roundTripTimeNanos=2980010}
2020-07-17 18:56:34,149 INFO : org.mongodb.driver.connection - Opened connection [connectionId{localValue:2, serverValue:2}] to localhost:27017
2020-07-17 18:56:34,373 INFO : org.graylog2.shared.buffers.InputBufferImpl - Initialized InputBufferImpl with ring size <65536> and wait strategy <BlockingWaitStrategy>, running 2 parallel message handlers.
2020-07-17 18:56:34,577 INFO : io.searchbox.client.AbstractJestClient - Setting server pool to a list of 2 servers: [http://172.16.55.42:9200,http://172.16.55.45:9000]
2020-07-17 18:56:34,578 INFO : io.searchbox.client.JestClientFactory - Using multi thread/connection supporting pooling connection manager
2020-07-17 18:56:34,652 INFO : io.searchbox.client.JestClientFactory - Using custom ObjectMapper instance
2020-07-17 18:56:34,652 INFO : io.searchbox.client.JestClientFactory - Node Discovery disabled...
2020-07-17 18:56:34,652 INFO : io.searchbox.client.JestClientFactory - Idle connection reaping disabled...
2020-07-17 18:56:35,162 INFO : org.graylog2.shared.buffers.ProcessBuffer - Initialized ProcessBuffer with ring size <65536> and wait strategy <BlockingWaitStrategy>.
2020-07-17 18:56:35,248 ERROR: org.graylog2.inputs.InputServiceImpl - Cannot build extractor from persisted data. Skipping.
java.lang.IllegalStateException: Configured lookup table <host-names> doesn't exist
	at org.graylog2.inputs.extractors.LookupTableExtractor.<init>(LookupTableExtractor.java:57) ~[graylog.jar:?]
	at org.graylog2.inputs.extractors.ExtractorFactory.factory(ExtractorFactory.java:72) ~[graylog.jar:?]
	at org.graylog2.inputs.InputServiceImpl.getExtractors(InputServiceImpl.java:294) ~[graylog.jar:?]
	at org.graylog2.filters.ExtractorFilter.loadExtractors(ExtractorFilter.java:143) ~[graylog.jar:?]
	at org.graylog2.filters.ExtractorFilter.lambda$loadAllExtractors$5(ExtractorFilter.java:132) ~[graylog.jar:?]
	at com.google.common.collect.ImmutableList.forEach(ImmutableList.java:405) [graylog.jar:?]
	at org.graylog2.filters.ExtractorFilter.loadAllExtractors(ExtractorFilter.java:132) [graylog.jar:?]
	at org.graylog2.filters.ExtractorFilter.<init>(ExtractorFilter.java:63) [graylog.jar:?]
	at org.graylog2.filters.ExtractorFilter$$FastClassByGuice$$cba333d6.newInstance(<generated>) [graylog.jar:?]
	at com.google.inject.internal.DefaultConstructionProxyFactory$FastClassProxy.newInstance(DefaultConstructionProxyFactory.java:89) [graylog.jar:?]
	at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:114) [graylog.jar:?]
	at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:91) [graylog.jar:?]
	at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:306) [graylog.jar:?]
	at com.google.inject.internal.FactoryProxy.get(FactoryProxy.java:62) [graylog.jar:?]
	at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:42) [graylog.jar:?]
	at com.google.inject.internal.RealMultibinder$RealMultibinderProvider.doProvision(RealMultibinder.java:198) [graylog.jar:?]
	at com.google.inject.internal.RealMultibinder$RealMultibinderProvider.doProvision(RealMultibinder.java:151) [graylog.jar:?]
	at com.google.inject.internal.InternalProviderInstanceBindingImpl$Factory.get(InternalProviderInstanceBindingImpl.java:113) [graylog.jar:?]
	at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:42) [graylog.jar:?]
	at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:65) [graylog.jar:?]
	at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:113) [graylog.jar:?]
	at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:91) [graylog.jar:?]
	at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:306) [graylog.jar:?]
	at com.google.inject.internal.FactoryProxy.get(FactoryProxy.java:62) [graylog.jar:?]
	at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:42) [graylog.jar:?]
	at com.google.inject.internal.RealMultibinder$RealMultibinderProvider.doProvision(RealMultibinder.java:198) [graylog.jar:?]
	at com.google.inject.internal.RealMultibinder$RealMultibinderProvider.doProvision(RealMultibinder.java:151) [graylog.jar:?]
	at com.google.inject.internal.InternalProviderInstanceBindingImpl$Factory.get(InternalProviderInstanceBindingImpl.java:113) [graylog.jar:?]
	at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:42) [graylog.jar:?]
	at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:65) [graylog.jar:?]
	at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:113) [graylog.jar:?]
	at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:91) [graylog.jar:?]
	at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:306) [graylog.jar:?]
	at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:42) [graylog.jar:?]
	at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:65) [graylog.jar:?]
	at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:113) [graylog.jar:?]
	at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:91) [graylog.jar:?]
	at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:306) [graylog.jar:?]
	at com.google.inject.internal.InjectorImpl$1.get(InjectorImpl.java:1050) [graylog.jar:?]
	at com.google.inject.assistedinject.FactoryProvider2.invoke(FactoryProvider2.java:836) [graylog.jar:?]
	at com.sun.proxy.$Proxy59.create(Unknown Source) [?:?]
	at org.graylog2.shared.buffers.ProcessBuffer.<init>(ProcessBuffer.java:91) [graylog.jar:?]
	at org.graylog2.shared.buffers.ProcessBuffer$$FastClassByGuice$$ef94431e.newInstance(<generated>) [graylog.jar:?]
	at com.google.inject.internal.DefaultConstructionProxyFactory$FastClassProxy.newInstance(DefaultConstructionProxyFactory.java:89) [graylog.jar:?]
	at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:114) [graylog.jar:?]
	at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:91) [graylog.jar:?]
	at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:306) [graylog.jar:?]
	at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40) [graylog.jar:?]
	at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:168) [graylog.jar:?]
	at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:39) [graylog.jar:?]
	at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:42) [graylog.jar:?]
	at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:65) [graylog.jar:?]
	at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:113) [graylog.jar:?]
	at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:91) [graylog.jar:?]
	at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:306) [graylog.jar:?]
	at com.google.inject.internal.FactoryProxy.get(FactoryProxy.java:62) [graylog.jar:?]
	at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:42) [graylog.jar:?]
	at com.google.inject.internal.RealMultibinder$RealMultibinderProvider.doProvision(RealMultibinder.java:198) [graylog.jar:?]
	at com.google.inject.internal.RealMultibinder$RealMultibinderProvider.doProvision(RealMultibinder.java:151) [graylog.jar:?]
	at com.google.inject.internal.InternalProviderInstanceBindingImpl$Factory.get(InternalProviderInstanceBindingImpl.java:113) [graylog.jar:?]
	at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:42) [graylog.jar:?]
	at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:65) [graylog.jar:?]
	at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:113) [graylog.jar:?]
	at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:91) [graylog.jar:?]
	at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:306) [graylog.jar:?]
	at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40) [graylog.jar:?]
	at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:168) [graylog.jar:?]
	at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:39) [graylog.jar:?]
	at com.google.inject.internal.FactoryProxy.get(FactoryProxy.java:62) [graylog.jar:?]
	at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:42) [graylog.jar:?]
	at com.google.inject.internal.RealMultibinder$RealMultibinderProvider.doProvision(RealMultibinder.java:198) [graylog.jar:?]
	at com.google.inject.internal.RealMultibinder$RealMultibinderProvider.doProvision(RealMultibinder.java:151) [graylog.jar:?]
	at com.google.inject.internal.InternalProviderInstanceBindingImpl$Factory.get(InternalProviderInstanceBindingImpl.java:113) [graylog.jar:?]
	at com.google.inject.internal.SingleFieldInjector.inject(SingleFieldInjector.java:52) [graylog.jar:?]
	at com.google.inject.internal.MembersInjectorImpl.injectMembers(MembersInjectorImpl.java:147) [graylog.jar:?]
	at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:124) [graylog.jar:?]
	at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:91) [graylog.jar:?]
	at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:306) [graylog.jar:?]
	at com.google.inject.internal.BoundProviderFactory.get(BoundProviderFactory.java:60) [graylog.jar:?]
	at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40) [graylog.jar:?]
	at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:168) [graylog.jar:?]
	at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:39) [graylog.jar:?]
	at com.google.inject.internal.InternalInjectorCreator.loadEagerSingletons(InternalInjectorCreator.java:211) [graylog.jar:?]
	at com.google.inject.internal.InternalInjectorCreator.injectDynamically(InternalInjectorCreator.java:182) [graylog.jar:?]
	at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:109) [graylog.jar:?]
	at com.google.inject.Guice.createInjector(Guice.java:87) [graylog.jar:?]
	at org.graylog2.shared.bindings.GuiceInjectorHolder.createInjector(GuiceInjectorHolder.java:34) [graylog.jar:?]
	at org.graylog2.bootstrap.CmdLineTool.setupInjector(CmdLineTool.java:379) [graylog.jar:?]
	at org.graylog2.bootstrap.CmdLineTool.run(CmdLineTool.java:194) [graylog.jar:?]
	at org.graylog2.bootstrap.Main.main(Main.java:50) [graylog.jar:?]

It said I had too many characters to attach it all so I just went to the second ERROR.

Thanks for taking a look. This doesn’t happen on 3.2.X.

he tim -

when you use proper format like I wrote …

how many elasticsearch servers did you have and what is the configuration for them in your Graylog server.conf?

Hey Jan,

The format you posted isn’t visible. Could you maybe enclose it in quotes and I’ll attach the full log set once I see how it’s done.

Regarding Elasticsearch, we have two Elasticsearch servers and two Graylog servers. Each Graylog server was configured with the basic settings for multinode.

elasticsearch_hosts = http://172.16.55.42:9200, http://172.16.55.45:9000

BTW I now see a problem with the port on elasticsearch2. I’ll change that. Would that contribute to the problem with 3.3?

I just changed 172.16.55.45 to port 9200 and it made no difference and tried 3.3.2 again and it made no difference. Same errors. Thanks for your help.

he @tbmay

from the until now shared logs or information I can’t spot the potential problem. You might want to share your complete server.log and server.conf of Graylog.

Server 1 server.conf

# If you are running more than one instances of Graylog server you have to select one of these
# instances as master. The master will perform some periodical tasks that non-masters won't perform.
is_master = true

# The auto-generated node ID will be stored in this file and read after restarts. It is a good idea
# to use an absolute file path here if you are starting Graylog server from init scripts or similar.
node_id_file = /etc/graylog/server/node-id

# You MUST set a secret to secure/pepper the stored user passwords here. Use at least 64 characters.
# Generate one by using for example: pwgen -N 1 -s 96
password_secret = XXXXXXXXXXXXXXXXXXXXXXXXXX

# The default root user is named 'admin'
#root_username = admin

# You MUST specify a hash password for the root user (which you only need to initially set up the
# system and in case you lose connectivity to your authentication backend)
# This password cannot be changed using the API or via the web interface. If you need to change it,
# modify it in this file.
# Create one by using for example: echo -n yourpassword | shasum -a 256
# and put the resulting hash value into the following line
root_password_sha2 = XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

# The email address of the root user.
# Default is empty
#root_email = ""

# The time zone setting of the root user. See http://www.joda.org/joda-time/timezones.html for a list of valid time zones.
# Default is UTC
#root_timezone = UTC
root_timezone = America/New_York

# Set plugin directory here (relative or absolute)
plugin_dir = plugin

# REST API listen URI. Must be reachable by other Graylog server nodes if you run a cluster.
# When using Graylog Collectors, this URI will be used to receive heartbeat messages and must be accessible for all collectors.
#rest_listen_uri = http://172.16.55.39:12900/
#rest_listen_uri = http://172.16.55.39:9000/api/
##rest_listen_uri = http://127.0.0.1:9000/api/

#http_bind_address = 127.0.0.1:9000
http_bind_address = 172.16.55.39:9000

# REST API transport address. Defaults to the value of rest_listen_uri. Exception: If rest_listen_uri
# is set to a wildcard IP address (0.0.0.0) the first non-loopback IPv4 system address is used.
# If set, this will be promoted in the cluster discovery APIs, so other nodes may try to connect on
# this address and it is used to generate URLs addressing entities in the REST API. (see rest_listen_uri)
# You will need to define this, if your Graylog server is running behind a HTTP proxy that is rewriting
# the scheme, host name or URI.
# This must not contain a wildcard address (0.0.0.0).
#rest_transport_uri = http://192.168.1.1:12900/

#http_publish_uri = http://192.168.1.1:12900/

# Enable CORS headers for REST API. This is necessary for JS-clients accessing the server directly.
# If these are disabled, modern browsers will not be able to retrieve resources from the server.
# This is enabled by default. Uncomment the next line to disable it.
#rest_enable_cors = false

#http_enable_cors = false

# Enable GZIP support for REST API. This compresses API responses and therefore helps to reduce
# overall round trip times. This is disabled by default. Uncomment the next line to enable it.
#rest_enable_gzip = true

#http_enable_gzip = true

# Enable HTTPS support for the REST API. This secures the communication with the REST API with
# TLS to prevent request forgery and eavesdropping. This is disabled by default. Uncomment the
# next line to enable it.
#rest_enable_tls = true

#http_enable_tls = true

# The X.509 certificate chain file in PEM format to use for securing the REST API.
#rest_tls_cert_file = /path/to/graylog.crt

#http_tls_cert_file = /path/to/graylog.crt

# The PKCS#8 private key file in PEM format to use for securing the REST API.
#rest_tls_key_file = /path/to/graylog.key

#http_tls_key_file = /path/to/graylog.key

# The password to unlock the private key used for securing the REST API.
#rest_tls_key_password = secret

#http_tls_key_password = secret

# The maximum size of the HTTP request headers in bytes.
#rest_max_header_size = 8192

#http_max_header_size = 8192

# The maximal length of the initial HTTP/1.1 line in bytes.
#rest_max_initial_line_length = 4096

#http_max_initial_line_length = 4096

# The size of the thread pool used exclusively for serving the REST API.
#rest_thread_pool_size = 16

#http_thread_pool_size = 16

# Enable the embedded Graylog web interface.
# Default: true
#web_enable = false

# Web interface listen URI. It must not contain a path other than "/".
#web_listen_uri = http://127.0.0.1:9000/
#web_listen_uri = http://172.16.55.39:9000/
#web_listen_uri = http://127.0.0.1:9000/


# Web interface endpoint URI. This setting can be overriden on a per-request basis with the X-Graylog-Server-URL header.
# Default: $rest_transport_uri
#web_endpoint_uri =

#http_external_uri =

# Enable CORS headers for the web interface. This is necessary for JS-clients accessing the server directly.
# If these are disabled, modern browsers will not be able to retrieve resources from the server.
#web_enable_cors = false

#http_enable_cors = false

# Enable/disable GZIP support for the web interface. This compresses HTTP responses and therefore helps to reduce
# overall round trip times. This is enabled by default. Uncomment the next line to disable it.
#web_enable_gzip = false

#http_enable_gzip = false

# Enable HTTPS support for the web interface. This secures the communication of the web browser with the web interface
# using TLS to prevent request forgery and eavesdropping.
# This is disabled by default. Uncomment the next line to enable it and see the other related configuration settings.
#web_enable_tls = true

#http_enable_tls = true

# The X.509 certificate chain file in PEM format to use for securing the web interface.
#web_tls_cert_file = /path/to/graylog-web.crt

#http_tls_cert_file = /path/to/graylog-web.crt

# The PKCS#8 private key file in PEM format to use for securing the web interface.
#web_tls_key_file = /path/to/graylog-web.key

#http_tls_key_file = /path/to/graylog-web.key

# The password to unlock the private key used for securing the web interface.
#web_tls_key_password = secret

#http_tls_key_password = secret

# The maximum size of the HTTP request headers in bytes.
#web_max_header_size = 8192

#http_max_header_size = 8192

# The maximal length of the initial HTTP/1.1 line in bytes.
#web_max_initial_line_length = 4096

#http_max_initial_line_length = 4096

# The size of the thread pool used exclusively for serving the web interface.
#web_thread_pool_size = 16

#http_thread_pool_size = 16

# Configuration file for the embedded Elasticsearch instance in Graylog.
# Pay attention to the working directory of the server, maybe use an absolute path here.
# Default: empty
#elasticsearch_config_file = /etc/graylog/server/elasticsearch.yml

# Graylog will use multiple indices to store documents in. You can configured the strategy it uses to determine
# when to rotate the currently active write index.
# It supports multiple rotation strategies:
#   - "count" of messages per index, use elasticsearch_max_docs_per_index below to configure
#   - "size" per index, use elasticsearch_max_size_per_index below to configure
# valid values are "count", "size" and "time", default is "count"
#
# ATTENTION: These settings have been moved to the database in 2.0. When you upgrade, make sure to set these
#            to your previous 1.x settings so they will be migrated to the database!
rotation_strategy = count

# (Approximate) maximum number of documents in an Elasticsearch index before a new index
# is being created, also see no_retention and elasticsearch_max_number_of_indices.
# Configure this if you used 'rotation_strategy = count' above.
#
# ATTENTION: These settings have been moved to the database in 2.0. When you upgrade, make sure to set these
#            to your previous 1.x settings so they will be migrated to the database!
elasticsearch_max_docs_per_index = 20000000

# (Approximate) maximum size in bytes per Elasticsearch index on disk before a new index is being created, also see
# no_retention and elasticsearch_max_number_of_indices. Default is 1GB.
# Configure this if you used 'rotation_strategy = size' above.
#
# ATTENTION: These settings have been moved to the database in 2.0. When you upgrade, make sure to set these
#            to your previous 1.x settings so they will be migrated to the database!
#elasticsearch_max_size_per_index = 1073741824

# (Approximate) maximum time before a new Elasticsearch index is being created, also see
# no_retention and elasticsearch_max_number_of_indices. Default is 1 day.
# Configure this if you used 'rotation_strategy = time' above.
# Please note that this rotation period does not look at the time specified in the received messages, but is
# using the real clock value to decide when to rotate the index!
# Specify the time using a duration and a suffix indicating which unit you want:
#  1w  = 1 week
#  1d  = 1 day
#  12h = 12 hours
# Permitted suffixes are: d for day, h for hour, m for minute, s for second.
#
# ATTENTION: These settings have been moved to the database in 2.0. When you upgrade, make sure to set these
#            to your previous 1.x settings so they will be migrated to the database!
#elasticsearch_max_time_per_index = 1d

# Disable checking the version of Elasticsearch for being compatible with this Graylog release.
# WARNING: Using Graylog with unsupported and untested versions of Elasticsearch may lead to data loss!
#elasticsearch_disable_version_check = true

# Disable message retention on this node, i. e. disable Elasticsearch index rotation.
#no_retention = false

# How many indices do you want to keep?
#
# ATTENTION: These settings have been moved to the database in 2.0. When you upgrade, make sure to set these
#            to your previous 1.x settings so they will be migrated to the database!
elasticsearch_max_number_of_indices = 240

# Decide what happens with the oldest indices when the maximum number of indices is reached.
# The following strategies are availble:
#   - delete # Deletes the index completely (Default)
#   - close # Closes the index and hides it from the system. Can be re-opened later.
#
# ATTENTION: These settings have been moved to the database in 2.0. When you upgrade, make sure to set these
#            to your previous 1.x settings so they will be migrated to the database!
retention_strategy = delete

# How many Elasticsearch shards and replicas should be used per index? Note that this only applies to newly created indices.
elasticsearch_shards = 1
elasticsearch_replicas = 0

# Prefix for all Elasticsearch indices and index aliases managed by Graylog.
elasticsearch_index_prefix = graylog

# Name of the Elasticsearch index template used by Graylog to apply the mandatory index mapping.
# # Default: graylog-internal
#elasticsearch_template_name = graylog-internal

# Do you want to allow searches with leading wildcards? This can be extremely resource hungry and should only
# be enabled with care. See also: https://www.graylog.org/documentation/general/queries/
allow_leading_wildcard_searches = false

# Do you want to allow searches to be highlighted? Depending on the size of your messages this can be memory hungry and
# should only be enabled after making sure your Elasticsearch cluster has enough memory.
allow_highlighting = false

# settings to be passed to elasticsearch's client (overriding those in the provided elasticsearch_config_file)
# all these
# this must be the same as for your Elasticsearch cluster
#elasticsearch_cluster_name = graylog

# The prefix being used to generate the Elasticsearch node name which makes it easier to identify the specific Graylog
# server running the embedded Elasticsearch instance. The node name will be constructed by concatenating this prefix
# and the Graylog node ID (see node_id_file), for example "graylog-17052010-1234-5678-abcd-1337cafebabe".
# Default: graylog-
#elasticsearch_node_name_prefix = graylog-

# A comma-separated list of Elasticsearch nodes which Graylog is using to connect to the Elasticsearch cluster,
# see https://www.elastic.co/guide/en/elasticsearch/reference/2.3/modules-discovery-zen.html for details.
# Default: 127.0.0.1
#elasticsearch_discovery_zen_ping_unicast_hosts = 127.0.0.1:9300, 127.0.0.2:9500
#elasticsearch_discovery_zen_ping_unicast_hosts = 172.16.55.43:9300, 127.0.0.2:9500
elasticsearch_hosts = http://172.16.55.42:9200, http://172.16.55.45:9000
# we don't want the Graylog server to store any data, or be master node
#elasticsearch_node_master = false
#elasticsearch_node_data = false

# use a different port if you run multiple Elasticsearch nodes on one machine
#elasticsearch_transport_tcp_port = 9350

# we don't need to run the embedded HTTP server here
#elasticsearch_http_enabled = false

# Enable Elasticsearch multicast discovery. This requires the installation of an Elasticsearch plugin,
# see https://www.elastic.co/guide/en/elasticsearch/plugins/2.3/discovery-multicast.html for details.
# Default: false
#elasticsearch_discovery_zen_ping_multicast_enabled = false

# Change the following setting if you are running into problems with timeouts during Elasticsearch cluster discovery.
# The setting is specified in milliseconds, the default is 5000ms (5 seconds).
#elasticsearch_cluster_discovery_timeout = 5000

# the following settings allow to change the bind addresses for the Elasticsearch client in Graylog
# these settings are empty by default, letting Elasticsearch choose automatically,
# override them here or in the 'elasticsearch_config_file' if you need to bind to a special address
# refer to http://www.elasticsearch.org/guide/en/elasticsearch/reference/0.90/modules-network.html
# for special values here
#elasticsearch_network_host = 172.16.55.39
#elasticsearch_network_bind_host =
#elasticsearch_network_publish_host =

# The total amount of time discovery will look for other Elasticsearch nodes in the cluster
# before giving up and declaring the current node master.
#elasticsearch_discovery_initial_state_timeout = 3s

# Analyzer (tokenizer) to use for message and full_message field. The "standard" filter usually is a good idea.
# All supported analyzers are: standard, simple, whitespace, stop, keyword, pattern, language, snowball, custom
# Elasticsearch documentation: http://www.elasticsearch.org/guide/reference/index-modules/analysis/
# Note that this setting only takes effect on newly created indices.
elasticsearch_analyzer = standard

# Global request timeout for Elasticsearch requests (e. g. during search, index creation, or index time-range
# calculations) based on a best-effort to restrict the runtime of Elasticsearch operations.
# Default: 1m
#elasticsearch_request_timeout = 1m

# Time interval for index range information cleanups. This setting defines how often stale index range information
# is being purged from the database.
# Default: 1h
#index_ranges_cleanup_interval = 1h

# Batch size for the Elasticsearch output. This is the maximum (!) number of messages the Elasticsearch output
# module will get at once and write to Elasticsearch in a batch call. If the configured batch size has not been
# reached within output_flush_interval seconds, everything that is available will be flushed at once. Remember
# that every outputbuffer processor manages its own batch and performs its own batch write calls.
# ("outputbuffer_processors" variable)
output_batch_size = 500

# Flush interval (in seconds) for the Elasticsearch output. This is the maximum amount of time between two
# batches of messages written to Elasticsearch. It is only effective at all if your minimum number of messages
# for this time period is less than output_batch_size * outputbuffer_processors.
output_flush_interval = 1

# As stream outputs are loaded only on demand, an output which is failing to initialize will be tried over and
# over again. To prevent this, the following configuration options define after how many faults an output will
# not be tried again for an also configurable amount of seconds.
output_fault_count_threshold = 5
output_fault_penalty_seconds = 30

# The number of parallel running processors.
# Raise this number if your buffers are filling up.
#processbuffer_processors = 5
processbuffer_processors = 15
#outputbuffer_processors = 3
outputbuffer_processors = 6

#outputbuffer_processor_keep_alive_time = 5000
#outputbuffer_processor_threads_core_pool_size = 3
#outputbuffer_processor_threads_max_pool_size = 30

# UDP receive buffer size for all message inputs (e. g. SyslogUDPInput).
#udp_recvbuffer_sizes = 1048576

# Wait strategy describing how buffer processors wait on a cursor sequence. (default: sleeping)
# Possible types:
#  - yielding
#     Compromise between performance and CPU usage.
#  - sleeping
#     Compromise between performance and CPU usage. Latency spikes can occur after quiet periods.
#  - blocking
#     High throughput, low latency, higher CPU usage.
#  - busy_spinning
#     Avoids syscalls which could introduce latency jitter. Best when threads can be bound to specific CPU cores.
processor_wait_strategy = blocking

# Size of internal ring buffers. Raise this if raising outputbuffer_processors does not help anymore.
# For optimum performance your LogMessage objects in the ring buffer should fit in your CPU L3 cache.
# Must be a power of 2. (512, 1024, 2048, ...)
ring_size = 65536

inputbuffer_ring_size = 65536
inputbuffer_processors = 2
inputbuffer_wait_strategy = blocking

# Enable the disk based message journal.
message_journal_enabled = true

# The directory which will be used to store the message journal. The directory must me exclusively used by Graylog and
# must not contain any other files than the ones created by Graylog itself.
message_journal_dir = data/journal

# Journal hold messages before they could be written to Elasticsearch.
# For a maximum of 12 hours or 5 GB whichever happens first.
# During normal operation the journal will be smaller.
#message_journal_max_age = 12h
#message_journal_max_size = 5gb
message_journal_max_size = 4gb

#message_journal_flush_age = 1m
#message_journal_flush_interval = 1000000
#message_journal_segment_age = 1h
#message_journal_segment_size = 100mb

# Number of threads used exclusively for dispatching internal events. Default is 2.
#async_eventbus_processors = 2

# How many seconds to wait between marking node as DEAD for possible load balancers and starting the actual
# shutdown process. Set to 0 if you have no status checking load balancers in front.
lb_recognition_period_seconds = 3

# Every message is matched against the configured streams and it can happen that a stream contains rules which
# take an unusual amount of time to run, for example if its using regular expressions that perform excessive backtracking.
# This will impact the processing of the entire server. To keep such misbehaving stream rules from impacting other
# streams, Graylog limits the execution time for each stream.
# The default values are noted below, the timeout is in milliseconds.
# If the stream matching for one stream took longer than the timeout value, and this happened more than "max_faults" times
# that stream is disabled and a notification is shown in the web interface.
#stream_processing_timeout = 2000
#stream_processing_max_faults = 3

# Length of the interval in seconds in which the alert conditions for all streams should be checked
# and alarms are being sent.
#alert_check_interval = 60

# Since 0.21 the Graylog server supports pluggable output modules. This means a single message can be written to multiple
# outputs. The next setting defines the timeout for a single output module, including the default output module where all
# messages end up.
#
# Time in milliseconds to wait for all message outputs to finish writing a single message.
#output_module_timeout = 10000

# Time in milliseconds after which a detected stale master node is being rechecked on startup.
#stale_master_timeout = 2000

# Time in milliseconds which Graylog is waiting for all threads to stop on shutdown.
#shutdown_timeout = 30000

# MongoDB connection string
# See http://docs.mongodb.org/manual/reference/connection-string/ for details
#mongodb_uri = mongodb://localhost/graylog
mongodb_uri = mongodb://172.16.55.39/graylog

# Authenticate against the MongoDB server
#mongodb_uri = mongodb://grayloguser:secret@localhost:27017/graylog

# Use a replica set instead of a single host
#mongodb_uri = mongodb://grayloguser:secret@localhost:27017,localhost:27018,localhost:27019/graylog

# Increase this value according to the maximum connections your MongoDB server can handle from a single client
# if you encounter MongoDB connection problems.
mongodb_max_connections = 1000

# Number of threads allowed to be blocked by MongoDB connections multiplier. Default: 5
# If mongodb_max_connections is 100, and mongodb_threads_allowed_to_block_multiplier is 5,
# then 500 threads can block. More than that and an exception will be thrown.
# http://api.mongodb.org/java/current/com/mongodb/MongoOptions.html#threadsAllowedToBlockForConnectionMultiplier
mongodb_threads_allowed_to_block_multiplier = 5

# Drools Rule File (Use to rewrite incoming log messages)
# See: https://www.graylog.org/documentation/general/rewriting/
rules_file = /etc/graylog/server/rules.drl

# Email transport
transport_email_enabled = true
#transport_email_hostname = graylog.laurens55.org
transport_email_hostname = localhost
#transport_email_port = 587
transport_email_use_auth = false
transport_email_use_tls = false
transport_email_use_ssl = false
#transport_email_auth_username = you@example.com
#transport_email_auth_password = secret
transport_email_subject_prefix = [graylog]
transport_email_from_email = root@graylog.laurens55.org

# Specify and uncomment this if you want to include links to the stream in your stream alert mails.
# This should define the fully qualified base url to your web interface exactly the same way as it is accessed by your users.
#transport_email_web_interface_url = https://graylog.example.com
#transport_email_web_interface_url = http://172.16.55.39:9000
#transport_email_web_interface_url = https://172.16.55.39
transport_email_web_interface_url = https://graylog.laurens55.org

# The default connect timeout for outgoing HTTP connections.
# Values must be a positive duration (and between 1 and 2147483647 when converted to milliseconds).
# Default: 5s
#http_connect_timeout = 5s

# The default read timeout for outgoing HTTP connections.
# Values must be a positive duration (and between 1 and 2147483647 when converted to milliseconds).
# Default: 10s
#http_read_timeout = 10s

# The default write timeout for outgoing HTTP connections.
# Values must be a positive duration (and between 1 and 2147483647 when converted to milliseconds).
# Default: 10s
#http_write_timeout = 10s

# HTTP proxy for outgoing HTTP connections
#http_proxy_uri =

# Disable the optimization of Elasticsearch indices after index cycling. This may take some load from Elasticsearch
# on heavily used systems with large indices, but it will decrease search performance. The default is to optimize
# cycled indices.
#disable_index_optimization = true

# Optimize the index down to <= index_optimization_max_num_segments. A higher number may take some load from Elasticsearch
# on heavily used systems with large indices, but it will decrease search performance. The default is 1.
#index_optimization_max_num_segments = 1

# The threshold of the garbage collection runs. If GC runs take longer than this threshold, a system notification
# will be generated to warn the administrator about possible problems with the system. Default is 1 second.
#gc_warning_threshold = 1s

# Connection timeout for a configured LDAP server (e. g. ActiveDirectory) in milliseconds.
#ldap_connection_timeout = 2000

# Enable collection of Graylog-related metrics into MongoDB
# WARNING: This will add *a lot* of data into your MongoDB database on a regular interval (1 second)!
# DEPRECATED: This setting and the respective feature will be removed in a future version of Graylog.
#enable_metrics_collection = false

# Disable the use of SIGAR for collecting system stats
#disable_sigar = false

# The default cache time for dashboard widgets. (Default: 10 seconds, minimum: 1 second)
#dashboard_widget_default_cache_time = 10s

# Automatically load content packs in "content_packs_dir" on the first start of Graylog.
#content_packs_loader_enabled = true

# The directory which contains content packs which should be loaded on the first start of Graylog.
#content_packs_dir = data/contentpacks

# A comma-separated list of content packs (files in "content_packs_dir") which should be applied on
# the first start of Graylog.
# Default: empty
content_packs_auto_load = grok-patterns.json

server 2 server.conf

# If you are running more than one instances of Graylog server you have to select one of these
# instances as master. The master will perform some periodical tasks that non-masters won't perform.
is_master = false

# The auto-generated node ID will be stored in this file and read after restarts. It is a good idea
# to use an absolute file path here if you are starting Graylog server from init scripts or similar.
node_id_file = /etc/graylog/server/node-id

# You MUST set a secret to secure/pepper the stored user passwords here. Use at least 64 characters.
# Generate one by using for example: pwgen -N 1 -s 96
password_secret = XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXxx

# The default root user is named 'admin'
#root_username = admin

# You MUST specify a hash password for the root user (which you only need to initially set up the
# system and in case you lose connectivity to your authentication backend)
# This password cannot be changed using the API or via the web interface. If you need to change it,
# modify it in this file.
# Create one by using for example: echo -n yourpassword | shasum -a 256
# and put the resulting hash value into the following line
root_password_sha2 = XXXXXXXXXXXXXXXXXXXXXXXXX

# The email address of the root user.
# Default is empty
#root_email = ""

# The time zone setting of the root user. See http://www.joda.org/joda-time/timezones.html for a list of valid time zones.
# Default is UTC
#root_timezone = UTC
root_timezone = America/New_York

# Set plugin directory here (relative or absolute)
plugin_dir = plugin

# REST API listen URI. Must be reachable by other Graylog server nodes if you run a cluster.
# When using Graylog Collectors, this URI will be used to receive heartbeat messages and must be accessible for all collectors.
#rest_listen_uri = http://172.16.55.39:12900/
#rest_listen_uri = http://172.16.55.39:9000/api/
##rest_listen_uri = http://127.0.0.1:9000/api/

#http_bind_address = 127.0.0.1:9000
http_bind_address = 172.16.55.58:9000

# REST API transport address. Defaults to the value of rest_listen_uri. Exception: If rest_listen_uri
# is set to a wildcard IP address (0.0.0.0) the first non-loopback IPv4 system address is used.
# If set, this will be promoted in the cluster discovery APIs, so other nodes may try to connect on
# this address and it is used to generate URLs addressing entities in the REST API. (see rest_listen_uri)
# You will need to define this, if your Graylog server is running behind a HTTP proxy that is rewriting
# the scheme, host name or URI.
# This must not contain a wildcard address (0.0.0.0).
#rest_transport_uri = http://192.168.1.1:12900/

#http_publish_uri = http://192.168.1.1:12900/

# Enable CORS headers for REST API. This is necessary for JS-clients accessing the server directly.
# If these are disabled, modern browsers will not be able to retrieve resources from the server.
# This is enabled by default. Uncomment the next line to disable it.
#rest_enable_cors = false

#http_enable_cors = false

# Enable GZIP support for REST API. This compresses API responses and therefore helps to reduce
# overall round trip times. This is disabled by default. Uncomment the next line to enable it.
#rest_enable_gzip = true

#http_enable_gzip = true

# Enable HTTPS support for the REST API. This secures the communication with the REST API with
# TLS to prevent request forgery and eavesdropping. This is disabled by default. Uncomment the
# next line to enable it.
#rest_enable_tls = true

#http_enable_tls = true

# The X.509 certificate chain file in PEM format to use for securing the REST API.
#rest_tls_cert_file = /path/to/graylog.crt

#http_tls_cert_file = /path/to/graylog.crt

# The PKCS#8 private key file in PEM format to use for securing the REST API.
#rest_tls_key_file = /path/to/graylog.key

#http_tls_key_file = /path/to/graylog.key

# The password to unlock the private key used for securing the REST API.
#rest_tls_key_password = secret

#http_tls_key_password = secret

# The maximum size of the HTTP request headers in bytes.
#rest_max_header_size = 8192

#http_max_header_size = 8192

# The maximal length of the initial HTTP/1.1 line in bytes.
#rest_max_initial_line_length = 4096

#http_max_initial_line_length = 4096

# The size of the thread pool used exclusively for serving the REST API.
#rest_thread_pool_size = 16

#http_thread_pool_size = 16

# Enable the embedded Graylog web interface.
# Default: true
#web_enable = false

# Web interface listen URI. It must not contain a path other than "/".
#web_listen_uri = http://127.0.0.1:9000/
#web_listen_uri = http://172.16.55.39:9000/
#web_listen_uri = http://127.0.0.1:9000/


# Web interface endpoint URI. This setting can be overriden on a per-request basis with the X-Graylog-Server-URL header.
# Default: $rest_transport_uri
#web_endpoint_uri =

#http_external_uri =

# Enable CORS headers for the web interface. This is necessary for JS-clients accessing the server directly.
# If these are disabled, modern browsers will not be able to retrieve resources from the server.
#web_enable_cors = false

#http_enable_cors = false

# Enable/disable GZIP support for the web interface. This compresses HTTP responses and therefore helps to reduce
# overall round trip times. This is enabled by default. Uncomment the next line to disable it.
#web_enable_gzip = false

#http_enable_gzip = false

# Enable HTTPS support for the web interface. This secures the communication of the web browser with the web interface
# using TLS to prevent request forgery and eavesdropping.
# This is disabled by default. Uncomment the next line to enable it and see the other related configuration settings.
#web_enable_tls = true

#http_enable_tls = true

# The X.509 certificate chain file in PEM format to use for securing the web interface.
#web_tls_cert_file = /path/to/graylog-web.crt

#http_tls_cert_file = /path/to/graylog-web.crt

# The PKCS#8 private key file in PEM format to use for securing the web interface.
#web_tls_key_file = /path/to/graylog-web.key

#http_tls_key_file = /path/to/graylog-web.key

# The password to unlock the private key used for securing the web interface.
#web_tls_key_password = secret

#http_tls_key_password = secret

# The maximum size of the HTTP request headers in bytes.
#web_max_header_size = 8192

#http_max_header_size = 8192

# The maximal length of the initial HTTP/1.1 line in bytes.
#web_max_initial_line_length = 4096

#http_max_initial_line_length = 4096

# The size of the thread pool used exclusively for serving the web interface.
#web_thread_pool_size = 16

#http_thread_pool_size = 16

# Configuration file for the embedded Elasticsearch instance in Graylog.
# Pay attention to the working directory of the server, maybe use an absolute path here.
# Default: empty
#elasticsearch_config_file = /etc/graylog/server/elasticsearch.yml

# Graylog will use multiple indices to store documents in. You can configured the strategy it uses to determine
# when to rotate the currently active write index.
# It supports multiple rotation strategies:
#   - "count" of messages per index, use elasticsearch_max_docs_per_index below to configure
#   - "size" per index, use elasticsearch_max_size_per_index below to configure
# valid values are "count", "size" and "time", default is "count"
#
# ATTENTION: These settings have been moved to the database in 2.0. When you upgrade, make sure to set these
#            to your previous 1.x settings so they will be migrated to the database!
rotation_strategy = count

# (Approximate) maximum number of documents in an Elasticsearch index before a new index
# is being created, also see no_retention and elasticsearch_max_number_of_indices.
# Configure this if you used 'rotation_strategy = count' above.
#
# ATTENTION: These settings have been moved to the database in 2.0. When you upgrade, make sure to set these
#            to your previous 1.x settings so they will be migrated to the database!
elasticsearch_max_docs_per_index = 20000000

# (Approximate) maximum size in bytes per Elasticsearch index on disk before a new index is being created, also see
# no_retention and elasticsearch_max_number_of_indices. Default is 1GB.
# Configure this if you used 'rotation_strategy = size' above.
#
# ATTENTION: These settings have been moved to the database in 2.0. When you upgrade, make sure to set these
#            to your previous 1.x settings so they will be migrated to the database!
#elasticsearch_max_size_per_index = 1073741824

# (Approximate) maximum time before a new Elasticsearch index is being created, also see
# no_retention and elasticsearch_max_number_of_indices. Default is 1 day.
# Configure this if you used 'rotation_strategy = time' above.
# Please note that this rotation period does not look at the time specified in the received messages, but is
# using the real clock value to decide when to rotate the index!
# Specify the time using a duration and a suffix indicating which unit you want:
#  1w  = 1 week
#  1d  = 1 day
#  12h = 12 hours
# Permitted suffixes are: d for day, h for hour, m for minute, s for second.
#
# ATTENTION: These settings have been moved to the database in 2.0. When you upgrade, make sure to set these
#            to your previous 1.x settings so they will be migrated to the database!
#elasticsearch_max_time_per_index = 1d

# Disable checking the version of Elasticsearch for being compatible with this Graylog release.
# WARNING: Using Graylog with unsupported and untested versions of Elasticsearch may lead to data loss!
#elasticsearch_disable_version_check = true

# Disable message retention on this node, i. e. disable Elasticsearch index rotation.
#no_retention = false

# How many indices do you want to keep?
#
# ATTENTION: These settings have been moved to the database in 2.0. When you upgrade, make sure to set these
#            to your previous 1.x settings so they will be migrated to the database!
elasticsearch_max_number_of_indices = 240

# Decide what happens with the oldest indices when the maximum number of indices is reached.
# The following strategies are availble:
#   - delete # Deletes the index completely (Default)
#   - close # Closes the index and hides it from the system. Can be re-opened later.
#
# ATTENTION: These settings have been moved to the database in 2.0. When you upgrade, make sure to set these
#            to your previous 1.x settings so they will be migrated to the database!
retention_strategy = delete

# How many Elasticsearch shards and replicas should be used per index? Note that this only applies to newly created indices.
elasticsearch_shards = 1
elasticsearch_replicas = 0

# Prefix for all Elasticsearch indices and index aliases managed by Graylog.
elasticsearch_index_prefix = graylog

# Name of the Elasticsearch index template used by Graylog to apply the mandatory index mapping.
# # Default: graylog-internal
#elasticsearch_template_name = graylog-internal

# Do you want to allow searches with leading wildcards? This can be extremely resource hungry and should only
# be enabled with care. See also: https://www.graylog.org/documentation/general/queries/
allow_leading_wildcard_searches = false

# Do you want to allow searches to be highlighted? Depending on the size of your messages this can be memory hungry and
# should only be enabled after making sure your Elasticsearch cluster has enough memory.
allow_highlighting = false

# settings to be passed to elasticsearch's client (overriding those in the provided elasticsearch_config_file)
# all these
# this must be the same as for your Elasticsearch cluster
#elasticsearch_cluster_name = graylog

# The prefix being used to generate the Elasticsearch node name which makes it easier to identify the specific Graylog
# server running the embedded Elasticsearch instance. The node name will be constructed by concatenating this prefix
# and the Graylog node ID (see node_id_file), for example "graylog-17052010-1234-5678-abcd-1337cafebabe".
# Default: graylog-
#elasticsearch_node_name_prefix = graylog-

# A comma-separated list of Elasticsearch nodes which Graylog is using to connect to the Elasticsearch cluster,
# see https://www.elastic.co/guide/en/elasticsearch/reference/2.3/modules-discovery-zen.html for details.
# Default: 127.0.0.1
#elasticsearch_discovery_zen_ping_unicast_hosts = 127.0.0.1:9300, 127.0.0.2:9500
#elasticsearch_discovery_zen_ping_unicast_hosts = 172.16.55.43:9300, 127.0.0.2:9500
elasticsearch_hosts = http://172.16.55.42:9200, http://172.16.55.45:9000
# we don't want the Graylog server to store any data, or be master node
#elasticsearch_node_master = false
#elasticsearch_node_data = false

# use a different port if you run multiple Elasticsearch nodes on one machine
#elasticsearch_transport_tcp_port = 9350

# we don't need to run the embedded HTTP server here
#elasticsearch_http_enabled = false

# Enable Elasticsearch multicast discovery. This requires the installation of an Elasticsearch plugin,
# see https://www.elastic.co/guide/en/elasticsearch/plugins/2.3/discovery-multicast.html for details.
# Default: false
#elasticsearch_discovery_zen_ping_multicast_enabled = false

# Change the following setting if you are running into problems with timeouts during Elasticsearch cluster discovery.
# The setting is specified in milliseconds, the default is 5000ms (5 seconds).
#elasticsearch_cluster_discovery_timeout = 5000

# the following settings allow to change the bind addresses for the Elasticsearch client in Graylog
# these settings are empty by default, letting Elasticsearch choose automatically,
# override them here or in the 'elasticsearch_config_file' if you need to bind to a special address
# refer to http://www.elasticsearch.org/guide/en/elasticsearch/reference/0.90/modules-network.html
# for special values here
#elasticsearch_network_host = 172.16.55.39
#elasticsearch_network_bind_host =
#elasticsearch_network_publish_host =

# The total amount of time discovery will look for other Elasticsearch nodes in the cluster
# before giving up and declaring the current node master.
#elasticsearch_discovery_initial_state_timeout = 3s

# Analyzer (tokenizer) to use for message and full_message field. The "standard" filter usually is a good idea.
# All supported analyzers are: standard, simple, whitespace, stop, keyword, pattern, language, snowball, custom
# Elasticsearch documentation: http://www.elasticsearch.org/guide/reference/index-modules/analysis/
# Note that this setting only takes effect on newly created indices.
elasticsearch_analyzer = standard

# Global request timeout for Elasticsearch requests (e. g. during search, index creation, or index time-range
# calculations) based on a best-effort to restrict the runtime of Elasticsearch operations.
# Default: 1m
#elasticsearch_request_timeout = 1m

# Time interval for index range information cleanups. This setting defines how often stale index range information
# is being purged from the database.
# Default: 1h
#index_ranges_cleanup_interval = 1h

# Batch size for the Elasticsearch output. This is the maximum (!) number of messages the Elasticsearch output
# module will get at once and write to Elasticsearch in a batch call. If the configured batch size has not been
# reached within output_flush_interval seconds, everything that is available will be flushed at once. Remember
# that every outputbuffer processor manages its own batch and performs its own batch write calls.
# ("outputbuffer_processors" variable)
output_batch_size = 500

# Flush interval (in seconds) for the Elasticsearch output. This is the maximum amount of time between two
# batches of messages written to Elasticsearch. It is only effective at all if your minimum number of messages
# for this time period is less than output_batch_size * outputbuffer_processors.
output_flush_interval = 1

# As stream outputs are loaded only on demand, an output which is failing to initialize will be tried over and
# over again. To prevent this, the following configuration options define after how many faults an output will
# not be tried again for an also configurable amount of seconds.
output_fault_count_threshold = 5
output_fault_penalty_seconds = 30

# The number of parallel running processors.
# Raise this number if your buffers are filling up.
#processbuffer_processors = 5
processbuffer_processors = 15
#outputbuffer_processors = 3
outputbuffer_processors = 6

#outputbuffer_processor_keep_alive_time = 5000
#outputbuffer_processor_threads_core_pool_size = 3
#outputbuffer_processor_threads_max_pool_size = 30

# UDP receive buffer size for all message inputs (e. g. SyslogUDPInput).
#udp_recvbuffer_sizes = 1048576

# Wait strategy describing how buffer processors wait on a cursor sequence. (default: sleeping)
# Possible types:
#  - yielding
#     Compromise between performance and CPU usage.
#  - sleeping
#     Compromise between performance and CPU usage. Latency spikes can occur after quiet periods.
#  - blocking
#     High throughput, low latency, higher CPU usage.
#  - busy_spinning
#     Avoids syscalls which could introduce latency jitter. Best when threads can be bound to specific CPU cores.
processor_wait_strategy = blocking

# Size of internal ring buffers. Raise this if raising outputbuffer_processors does not help anymore.
# For optimum performance your LogMessage objects in the ring buffer should fit in your CPU L3 cache.
# Must be a power of 2. (512, 1024, 2048, ...)
ring_size = 65536

inputbuffer_ring_size = 65536
inputbuffer_processors = 2
inputbuffer_wait_strategy = blocking

# Enable the disk based message journal.
message_journal_enabled = true

# The directory which will be used to store the message journal. The directory must me exclusively used by Graylog and
# must not contain any other files than the ones created by Graylog itself.
message_journal_dir = data/journal

# Journal hold messages before they could be written to Elasticsearch.
# For a maximum of 12 hours or 5 GB whichever happens first.
# During normal operation the journal will be smaller.
#message_journal_max_age = 12h
#message_journal_max_size = 5gb
message_journal_max_size = 4gb

#message_journal_flush_age = 1m
#message_journal_flush_interval = 1000000
#message_journal_segment_age = 1h
#message_journal_segment_size = 100mb

# Number of threads used exclusively for dispatching internal events. Default is 2.
#async_eventbus_processors = 2

# How many seconds to wait between marking node as DEAD for possible load balancers and starting the actual
# shutdown process. Set to 0 if you have no status checking load balancers in front.
lb_recognition_period_seconds = 3

# Every message is matched against the configured streams and it can happen that a stream contains rules which
# take an unusual amount of time to run, for example if its using regular expressions that perform excessive backtracking.
# This will impact the processing of the entire server. To keep such misbehaving stream rules from impacting other
# streams, Graylog limits the execution time for each stream.
# The default values are noted below, the timeout is in milliseconds.
# If the stream matching for one stream took longer than the timeout value, and this happened more than "max_faults" times
# that stream is disabled and a notification is shown in the web interface.
#stream_processing_timeout = 2000
#stream_processing_max_faults = 3

# Length of the interval in seconds in which the alert conditions for all streams should be checked
# and alarms are being sent.
#alert_check_interval = 60

# Since 0.21 the Graylog server supports pluggable output modules. This means a single message can be written to multiple
# outputs. The next setting defines the timeout for a single output module, including the default output module where all
# messages end up.
#
# Time in milliseconds to wait for all message outputs to finish writing a single message.
#output_module_timeout = 10000

# Time in milliseconds after which a detected stale master node is being rechecked on startup.
#stale_master_timeout = 2000

# Time in milliseconds which Graylog is waiting for all threads to stop on shutdown.
#shutdown_timeout = 30000

# MongoDB connection string
# See http://docs.mongodb.org/manual/reference/connection-string/ for details
mongodb_uri = mongodb://172.16.55.39/graylog

# Authenticate against the MongoDB server
#mongodb_uri = mongodb://grayloguser:secret@localhost:27017/graylog

# Use a replica set instead of a single host
#mongodb_uri = mongodb://grayloguser:secret@localhost:27017,localhost:27018,localhost:27019/graylog

# Increase this value according to the maximum connections your MongoDB server can handle from a single client
# if you encounter MongoDB connection problems.
mongodb_max_connections = 1000

# Number of threads allowed to be blocked by MongoDB connections multiplier. Default: 5
# If mongodb_max_connections is 100, and mongodb_threads_allowed_to_block_multiplier is 5,
# then 500 threads can block. More than that and an exception will be thrown.
# http://api.mongodb.org/java/current/com/mongodb/MongoOptions.html#threadsAllowedToBlockForConnectionMultiplier
mongodb_threads_allowed_to_block_multiplier = 5

# Drools Rule File (Use to rewrite incoming log messages)
# See: https://www.graylog.org/documentation/general/rewriting/
rules_file = /etc/graylog/server/rules.drl

# Email transport
transport_email_enabled = true
#transport_email_hostname = graylog.laurens55.org
transport_email_hostname = localhost
#transport_email_port = 587
transport_email_use_auth = false
transport_email_use_tls = false
transport_email_use_ssl = false
#transport_email_auth_username = you@example.com
#transport_email_auth_password = secret
transport_email_subject_prefix = [graylog]
transport_email_from_email = root@graylog.laurens55.org

# Specify and uncomment this if you want to include links to the stream in your stream alert mails.
# This should define the fully qualified base url to your web interface exactly the same way as it is accessed by your users.
#transport_email_web_interface_url = https://graylog.example.com
#transport_email_web_interface_url = http://172.16.55.39:9000
#transport_email_web_interface_url = https://172.16.55.39
transport_email_web_interface_url = https://graylog.laurens55.org

# The default connect timeout for outgoing HTTP connections.
# Values must be a positive duration (and between 1 and 2147483647 when converted to milliseconds).
# Default: 5s
#http_connect_timeout = 5s

# The default read timeout for outgoing HTTP connections.
# Values must be a positive duration (and between 1 and 2147483647 when converted to milliseconds).
# Default: 10s
#http_read_timeout = 10s

# The default write timeout for outgoing HTTP connections.
# Values must be a positive duration (and between 1 and 2147483647 when converted to milliseconds).
# Default: 10s
#http_write_timeout = 10s

# HTTP proxy for outgoing HTTP connections
#http_proxy_uri =

# Disable the optimization of Elasticsearch indices after index cycling. This may take some load from Elasticsearch
# on heavily used systems with large indices, but it will decrease search performance. The default is to optimize
# cycled indices.
#disable_index_optimization = true

# Optimize the index down to <= index_optimization_max_num_segments. A higher number may take some load from Elasticsearch
# on heavily used systems with large indices, but it will decrease search performance. The default is 1.
#index_optimization_max_num_segments = 1

# The threshold of the garbage collection runs. If GC runs take longer than this threshold, a system notification
# will be generated to warn the administrator about possible problems with the system. Default is 1 second.
#gc_warning_threshold = 1s

# Connection timeout for a configured LDAP server (e. g. ActiveDirectory) in milliseconds.
#ldap_connection_timeout = 2000

# Enable collection of Graylog-related metrics into MongoDB
# WARNING: This will add *a lot* of data into your MongoDB database on a regular interval (1 second)!
# DEPRECATED: This setting and the respective feature will be removed in a future version of Graylog.
#enable_metrics_collection = false

# Disable the use of SIGAR for collecting system stats
#disable_sigar = false

# The default cache time for dashboard widgets. (Default: 10 seconds, minimum: 1 second)
#dashboard_widget_default_cache_time = 10s

# Automatically load content packs in "content_packs_dir" on the first start of Graylog.
#content_packs_loader_enabled = true

# The directory which contains content packs which should be loaded on the first start of Graylog.
#content_packs_dir = data/contentpacks

# A comma-separated list of content packs (files in "content_packs_dir") which should be applied on
# the first start of Graylog.
# Default: empty
content_packs_auto_load = grok-patterns.json

Hi Jan,

I pasted the two server.conf but honestly the second one was unnecessary because the master server doesn’t start. It’s server 1.

I can’t figure out how to post the entire log. No matter how I format it…highlight it all and click the html button for instance…I keep getting this:

Body is limited to 32000 characters; you entered 440273.

If you can tell me what I’m missing I would appreciate it.

I have similar lookup table errors in 3.2, but the server starts anyway. They come from a situation a couple of years ago, when I created a new lookup table, and then deleted it. For some reason it did not delete everything about it, but some remnants are still there. I created a bug back then but removed that long ago, as it was not treated and we are many versions ahead.

Still, it would be nice to know, how to clean up the rest of the lookup table from the mongodb. Especially, if it now prohibits upgrade to 3.3

Thank you. I have no doubt that lookup table did exist and was deleted. I definitely would like to know how to remove it safely to make this go away.