Kubernetes install of Graylog not able to create indexes in elasticsearch

williamsjasonj · October 4, 2019, 4:27pm

I am trying to install graylog into our new kubernetes cluster. I have elasticsearch up running version 6.8.3 and health check is green. Elasticsearch is running secure with certs and auth. I have configured the graylog.conf with the auth connection string and have imported the certs into the keystore to trust.
example: elasticsearch_hosts = https://elastic:password@elasticsearch-host:9200

From the graylog pod, I am able to manually create/delete/view indexes using the connection string listed in the graylog.conf. However when graylog starts - there are no logs received on the elasticsearch server and no indexes created.

The logs on the graylog server show:
019-10-04 15:51:02,799 INFO [MongoIndexSet] - Did not find a deflector alias. Setting one up now. - {}
2019-10-04 15:51:02,805 INFO [IndexRetentionThread] - Elasticsearch cluster not available, skipping index retention checks. - {}
2019-10-04 15:51:02,777 ERROR [IndexFieldTypePollerPeriodical] - Uncaught exception in periodical - {}
org.graylog2.indexer.IndexNotFoundException: Couldn’t collect indices for alias graylog_deflector

Index not found for query: . Try recalculating your index ranges.

This is followed by this error repeatedly:
2019-10-04 15:51:02,810 ERROR [IndexRotationThread] - Couldn’t point deflector to a new index - {}
org.graylog2.indexer.IndexNotFoundException: Couldn’t collect aliases for index pattern graylog_*

Index not found for query: . Try recalculating your index ranges.
at org.graylog2.indexer.cluster.jest.JestUtils.buildIndexNotFoundException(JestUtils.java:118) ~[graylog.jar:?]
at org.graylog2.indexer.cluster.jest.JestUtils.specificException(JestUtils.java:84) ~[graylog.jar:?]
at org.graylog2.indexer.cluster.jest.JestUtils.execute(JestUtils.java:57) ~[graylog.jar:?]
at org.graylog2.indexer.cluster.jest.JestUtils.execute(JestUtils.java:62) ~[graylog.jar:?]
at org.graylog2.indexer.indices.Indices.getIndexNamesAndAliases(Indices.java:314) ~[graylog.jar:?]
at org.graylog2.indexer.MongoIndexSet.getNewestIndexNumber(MongoIndexSet.java:151) ~[graylog.jar:?]
at org.graylog2.indexer.MongoIndexSet.getNewestIndex(MongoIndexSet.java:146) ~[graylog.jar:?]
at org.graylog2.indexer.MongoIndexSet.setUp(MongoIndexSet.java:252) ~[graylog.jar:?]
at org.graylog2.periodical.IndexRotationThread.checkAndRepair(IndexRotationThread.java:138) ~[graylog.jar:?]
at org.graylog2.periodical.IndexRotationThread.lambda$doRun$0(IndexRotationThread.java:76) ~[graylog.jar:?]
at java.lang.Iterable.forEach(Iterable.java:75) [?:1.8.0_222]
at org.graylog2.periodical.IndexRotationThread.doRun(IndexRotationThread.java:73) [graylog.jar:?]
at org.graylog2.plugin.periodical.Periodical.run(Periodical.java:77) [graylog.jar:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_222]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_222]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_222]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [?:1.8.0_222]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_222]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_222]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_222]

When I login to the graylog server and go to “Overview” the Elasticsearch status show “Elasticsearch cluster is green”.

I created an Input and graylog is receiving logs, but I can’t search them of course since nothing is going to elasticsearch.

I have been troubleshooting this for several days and at a loss - any help is appreciated.

williamsjasonj · October 4, 2019, 5:02pm

graylog.conf:

node_id_file = /usr/share/graylog/data/journal/node-id
root_username = admin
root_email =
root_timezone = UTC
plugin_dir = /usr/share/graylog/plugin
http_bind_address = 0.0.0.0:9000
http_external_uri = https://graylog.domainurl.net/
elasticsearch_hosts = https://elastic:password@graylog-es-http:9200
elasticsearch_max_retries = 10
elasticsearch_compression_enabled = true
allow_leading_wildcard_searches = false
allow_highlighting = false
output_batch_size = 500
output_flush_interval = 1
output_fault_count_threshold = 5
output_fault_penalty_seconds = 30
processbuffer_processors = 5
outputbuffer_processors = 3
processor_wait_strategy = blocking
ring_size = 65536
inputbuffer_ring_size = 65536
inputbuffer_processors = 2
inputbuffer_wait_strategy = blocking
message_journal_enabled = true
message_journal_dir = /usr/share/graylog/data/journal
lb_recognition_period_seconds = 3
mongodb_uri = mongodb://graylog_admin:password@mongodb:27017/graylog
mongodb_max_connections = 1000
mongodb_threads_allowed_to_block_multiplier = 5
transport_email_enabled = false
transport_email_hostname =
transport_email_port = 2587
transport_email_use_auth = true
transport_email_use_tls = true
transport_email_use_ssl = false
transport_email_auth_username =
transport_email_auth_password =
transport_email_subject_prefix = [graylog]
transport_email_from_email =
transport_email_web_interface_url = https://graylog.domainurl.net
content_packs_dir = /usr/share/graylog/data/contentpacks
content_packs_auto_load = grok-patterns.json

Graylog Startup Logs:

Launching graylog-0 as master
pod/graylog-0 labeled
Starting graylog
Graylog Home /usr/share/graylog
JVM Options -Djava.net.prefe 2019-10-04 16:57:00,688 INFO 2019-10-04 16:57:00,692 INFO 2019-10-04 16:57:00,693 INFO 2019-10-04 16:57:01,388 INFO 2019-10-04 16:57:02,111 INFO 2019-10-04 16:57:09,315 INFO 2019-10-04 16:57:09,388 INFO 2019-10-04 16:57:09,985 INFO 2019-10-04 16:57:10,075 WARN 2019-10-04 16:57:10,191 INFO 2019-10-04 16:57:10,205 INFO 2019-10-04 16:57:10,288 INFO 2019-10-04 16:57:10,414 INFO 2019-10-04 16:57:10,878 INFO 2019-10-04 16:57:10,894 INFO 2019-10-04 16:57:11,011 INFO 2019-10-04 16:57:11,679 INFO 2019-10-04 16:57:12,209 INFO 2019-10-04 16:57:12,210 INFO 2019-10-04 16:57:12,407 INFO 2019-10-04 16:57:12,408 INFO 2019-10-04 16:57:12,408 INFO 2019-10-04 16:57:12,408 INFO 2019-10-04 16:57:12,900 INFO 2019-10-04 16:57:13,605 WARN 2019-10-04 16:57:13,626 INFO 2019-10-04 16:57:13,886 WARN 2019-10-04 16:57:13,998 INFO 2019-10-04 16:57:14,107 WARN 2019-10-04 16:57:14,273 WARN 2019-10-04 16:57:14,378 WARN 2019-10-04 16:57:17,299 INFO 2019-10-04 16:57:17,300 INFO 2019-10-04 16:57:17,300 INFO 2019-10-04 16:57:17,300 INFO 2019-10-04 16:57:17,300 INFO 2019-10-04 16:57:17,424 INFO 2019-10-04 16:57:17,424 INFO 2019-10-04 16:57:17,475 INFO 2019-10-04 16:57:17,551 INFO 2019-10-04 16:57:17,554 INFO 2019-10-04 16:57:17,576 INFO 2019-10-04 16:57:17,589 INFO 2019-10-04 16:57:17,599 INFO 2019-10-04 16:57:17,611 INFO 2019-10-04 16:57:17,613 INFO 2019-10-04 16:57:17,616 INFO 2019-10-04 16:57:17,618 INFO 2019-10-04 16:57:17,638 INFO 2019-10-04 16:57:17,650 INFO 2019-10-04 16:57:17,685 INFO 2019-10-04 16:57:17,694 INFO 2019-10-04 16:57:17,706 INFO 2019-10-04 16:57:17,727 INFO 2019-10-04 16:57:17,894 INFO 2019-10-04 16:57:17,894 INFO 2019-10-04 16:57:17,980 INFO 2019-10-04 16:57:17,986 INFO 2019-10-04 16:57:17,986 INFO 2019-10-04 16:57:18,006 INFO 2019-10-04 16:57:18,027 INFO 2019-10-04 16:57:18,031 INFO 2019-10-04 16:57:18,035 INFO 2019-10-04 16:57:18,036 INFO 2019-10-04 16:57:18,037 INFO 2019-10-04 16:57:18,106 INFO 2019-10-04 16:57:18,107 INFO 2019-10-04 16:57:18,116 INFO 2019-10-04 16:57:18,124 INFO 2019-10-04 16:57:18,170 INFO 2019-10-04 16:57:18,176 INFO 2019-10-04 16:57:18,182 INFO 2019-10-04 16:57:18,190 INFO 2019-10-04 16:57:18,207 INFO 2019-10-04 16:57:18,271 INFO 2019-10-04 16:57:19,114 INFO 2019-10-04 16:57:19,126 INFO 2019-10-04 16:57:19,086 ERROR org.graylog2.indexer.IndexNo rIPv4Stack=true -XX:NewRatio=1 -server -XX:+ResizeTLAB -XX:+UseConcMarkSweepGC -XX:+CMSConcurrentMTEnabled -XX:+CMSClassUnloadingEnabled -XX:+UseParNewGC -XX:-OmitStackTraceInFastThrow -XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap
[CmdLineTool] - Loaded plugin: AWS plugins 3.1.2 [org.graylog.aws.AWSPlugin] - {}
[CmdLineTool] - Loaded plugin: Collector 3.1.2 [org.graylog.plugins.collector.CollectorPlugin] - {}
[CmdLineTool] - Loaded plugin: Threat Intelligence Plugin 3.1.2 [org.graylog.plugins.threatintel.ThreatIntelPlugin] - {}
[CmdLineTool] - Running with JVM arguments: -Djava.net.preferIPv4Stack=true -XX:NewRatio=1 -XX:+ResizeTLAB -XX:+UseConcMarkSweepGC -XX:+CMSConcurrentMTEnabled -XX:+CMSClassUnloadingEnabled -XX:+UseParNewGC -XX:-OmitStackTraceInFastThrow -XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap -Dlog4j.configurationFile=/usr/share/graylog/config/log4j2.xml -Djavax.net.ssl.trustStore=/k8s/truststore.jks -Djava.library.path=/usr/share/graylog/lib/sigar/ -Dgraylog2.installation_source=docker - {}
[Version] - HV000001: Hibernate Validator 5.1.3.Final - {}
[InputBufferImpl] - Message journal is enabled. - {}
[NodeId] - Node ID: d667ce38-136d-4f36-8762-c5bec671209b - {}
[LogManager] - Loading logs. - {}
[Log] - Found a corrupted index file, /usr/share/graylog/data/journal/messagejournal-0/00000000000000000000.index, deleting and rebuilding index… - {}
[LogManager] - Logs loading complete. - {}
[KafkaJournal] - Initialized Kafka based journal at /usr/share/graylog/data/journal - {}
[cluster] - Cluster created with settings {hosts=[mongodb:27017], mode=SINGLE, requiredClusterType=UNKNOWN, serverSelectionTimeout=‘30000 ms’, maxWaitQueueSize=5000} - {}
[cluster] - Cluster description not yet available. Waiting for 30000 ms before timing out - {}
[connection] - Opened connection [connectionId{localValue:1, serverValue:14764}] to mongodb:27017 - {}
[cluster] - Monitor thread successfully connected to server with description ServerDescription{address=mongodb:27017, type=REPLICA_SET_PRIMARY, state=CONNECTED, ok=true, version=ServerVersion{versionList=[4, 0, 12]}, minWireVersion=0, maxWireVersion=7, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=6954221, setName=‘rs0’, canonicalAddress=mongodb-primary-0.mongodb-headless.graylog.svc.cluster.local:27017, hosts=[mongodb-primary-0.mongodb-headless.graylog.svc.cluster.local:27017, mongodb-secondary-0.mongodb-headless.graylog.svc.cluster.local:27017], passives=, arbiters=[mongodb-arbiter-0.mongodb-headless.graylog.svc.cluster.local:27017], primary=‘mongodb-primary-0.mongodb-headless.graylog.svc.cluster.local:27017’, tagSet=TagSet{}, electionId=7fffffff0000000000000002, setVersion=3, lastWriteDate=Fri Oct 04 16:57:04 UTC 2019, lastUpdateTimeNanos=83553118154519} - {}
[connection] - Opened connection [connectionId{localValue:2, serverValue:14765}] to mongodb:27017 - {}
[InputBufferImpl] - Initialized InputBufferImpl with ring size <65536> and wait strategy , running 2 parallel message handlers. - {}
[AbstractJestClient] - Setting server pool to a list of 1 servers: [https://graylog-es-http:9200] - {}
[JestClientFactory] - Using multi thread/connection supporting pooling connection manager - {}
[JestClientFactory] - Using custom ObjectMapper instance - {}
[JestClientFactory] - Node Discovery disabled… - {}
[JestClientFactory] - Idle connection reaping disabled… - {}
[JestClientFactory] - Authentication cache set for preemptive authentication - {}
[ProcessBuffer] - Initialized ProcessBuffer with ring size <65536> and wait strategy . - {}
[GeoIpResolverEngine] - GeoIP database file does not exist: /etc/graylog/server/GeoLite2-City.mmdb - {}
[OutputBuffer] - Initialized OutputBuffer with ring size <65536> and wait strategy . - {}
[GeoIpResolverEngine] - GeoIP database file does not exist: /etc/graylog/server/GeoLite2-City.mmdb - {}
[connection] - Opened connection [connectionId{localValue:3, serverValue:14766}] to mongodb:27017 - {}
[GeoIpResolverEngine] - GeoIP database file does not exist: /etc/graylog/server/GeoLite2-City.mmdb - {}
[GeoIpResolverEngine] - GeoIP database file does not exist: /etc/graylog/server/GeoLite2-City.mmdb - {}
[GeoIpResolverEngine] - GeoIP database file does not exist: /etc/graylog/server/GeoLite2-City.mmdb - {}
[ServerBootstrap] - Graylog server 3.1.2+9e96b08 starting up - {}
[ServerBootstrap] - JRE: Oracle Corporation 1.8.0_222 on Linux 4.19.66-coreos - {}
[ServerBootstrap] - Deployment: docker - {}
[ServerBootstrap] - OS: Debian GNU/Linux 10 (buster) (debian) - {}
[ServerBootstrap] - Arch: amd64 - {}
[PeriodicalsService] - Starting 29 periodicals … - {}
[Periodicals] - Starting [org.graylog2.periodical.ThroughputCalculator] periodical in [0s], polling every [1s]. - {}
[Periodicals] - Starting [org.graylog.plugins.pipelineprocessor.periodical.LegacyDefaultStreamMigration] periodical, running forever. - {}
[PeriodicalsService] - Not starting [org.graylog2.periodical.AlertScannerThread] periodical. Not configured to run on this node. - {}
[Periodicals] - Starting [org.graylog2.periodical.BatchedElasticSearchOutputFlushThread] periodical in [0s], polling every [1s]. - {}
[Periodicals] - Starting [org.graylog2.periodical.ClusterHealthCheckThread] periodical in [120s], polling every [20s]. - {}
[Periodicals] - Starting [org.graylog2.periodical.GarbageCollectionWarningThread] periodical, running forever. - {}
[Periodicals] - Starting [org.graylog2.periodical.IndexerClusterCheckerThread] periodical in [0s], polling every [30s]. - {}
[Periodicals] - Starting [org.graylog2.periodical.IndexRetentionThread] periodical in [0s], polling every [300s]. - {}
[Periodicals] - Starting [org.graylog2.periodical.IndexRotationThread] periodical in [0s], polling every [10s]. - {}
[Periodicals] - Starting [org.graylog2.periodical.NodePingThread] periodical in [0s], polling every [1s]. - {}
[Periodicals] - Starting [org.graylog2.periodical.VersionCheckThread] periodical in [300s], polling every [1800s]. - {}
[Periodicals] - Starting [org.graylog2.periodical.ThrottleStateUpdaterThread] periodical in [1s], polling every [1s]. - {}
[Periodicals] - Starting [org.graylog2.events.ClusterEventPeriodical] periodical in [0s], polling every [1s]. - {}
[Periodicals] - Starting [org.graylog2.events.ClusterEventCleanupPeriodical] periodical in [0s], polling every [86400s]. - {}
[Periodicals] - Starting [org.graylog2.periodical.ClusterIdGeneratorPeriodical] periodical, running forever. - {}
[Periodicals] - Starting [org.graylog2.periodical.IndexRangesMigrationPeriodical] periodical, running forever. - {}
[Periodicals] - Starting [org.graylog2.periodical.IndexRangesCleanupPeriodical] periodical in [15s], polling every [3600s]. - {}
[PeriodicalsService] - Not starting [org.graylog2.periodical.UserPermissionMigrationPeriodical] periodical. Not configured to run on this node. - {}
[Periodicals] - Starting [org.graylog2.periodical.ConfigurationManagementPeriodical] periodical, running forever. - {}
[Periodicals] - Starting [org.graylog2.periodical.LdapGroupMappingMigration] periodical, running forever. - {}
[connection] - Opened connection [connectionId{localValue:8, serverValue:14773}] to mongodb:27017 - {}
[connection] - Opened connection [connectionId{localValue:6, serverValue:14770}] to mongodb:27017 - {}
[Periodicals] - Starting [org.graylog2.periodical.IndexFailuresPeriodical] periodical, running forever. - {}
[Periodicals] - Starting [org.graylog2.periodical.TrafficCounterCalculator] periodical in [0s], polling every [1s]. - {}
[connection] - Opened connection [connectionId{localValue:4, serverValue:14769}] to mongodb:27017 - {}
[connection] - Opened connection [connectionId{localValue:7, serverValue:14771}] to mongodb:27017 - {}
[LegacyDefaultStreamMigration] - Legacy default stream has no connections, no migration needed. - {}
[connection] - Opened connection [connectionId{localValue:5, serverValue:14772}] to mongodb:27017 - {}
[connection] - Opened connection [connectionId{localValue:10, serverValue:14775}] to mongodb:27017 - {}
[connection] - Opened connection [connectionId{localValue:11, serverValue:14776}] to mongodb:27017 - {}
[connection] - Opened connection [connectionId{localValue:9, serverValue:14774}] to mongodb:27017 - {}
[Periodicals] - Starting [org.graylog2.indexer.fieldtypes.IndexFieldTypePollerPeriodical] periodical in [0s], polling every [3600s]. - {}
[Periodicals] - Starting [org.graylog.scheduler.periodicals.ScheduleTriggerCleanUp] periodical in [120s], polling every [86400s]. - {}
[Periodicals] - Starting [org.graylog.plugins.sidecar.periodical.PurgeExpiredSidecarsThread] periodical in [0s], polling every [600s]. - {}
[Periodicals] - Starting [org.graylog.plugins.sidecar.periodical.PurgeExpiredConfigurationUploads] periodical in [0s], polling every [600s]. - {}
[Periodicals] - Starting [org.graylog.plugins.views.search.db.SearchesCleanUpJob] periodical in [0s], polling every [28800s]. - {}
[Periodicals] - Starting [org.graylog.events.periodicals.EventNotificationStatusCleanUp] periodical in [120s], polling every [86400s]. - {}
[Periodicals] - Starting [org.graylog.plugins.collector.periodical.PurgeExpiredCollectorsThread] periodical in [0s], polling every [3600s]. - {}
[MongoIndexSet] - Did not find a deflector alias. Setting one up now. - {}
[IndexRetentionThread] - Elasticsearch cluster not available, skipping index retention checks. - {}
[IndexFieldTypePollerPeriodical] - Uncaught exception in periodical - {}
tFoundException: Couldn’t collect indices for alias graylog_deflector

Index not found for query: . Try recalculating your index ranges.
at org.graylog2.indexer.cluster.jest.JestUtils.buildIndexNotFoundException(JestUtils.java:118) ~[graylog.jar:?]
at org.graylog2.indexer.cluster.jest.JestUtils.specificException(JestUtils.java:84) ~[graylog.jar:?]
at org.graylog2.indexer.cluster.jest.JestUtils.execute(JestUtils.java:57) ~[graylog.jar:?]
at org.graylog2.indexer.cluster.jest.JestUtils.execute(JestUtils.java:62) ~[graylog.jar:?]
at org.graylog2.indexer.indices.Indices.aliasTarget(Indices.java:336) ~[graylog.jar:?]
at org.graylog2.indexer.MongoIndexSet.getActiveWriteIndex(MongoIndexSet.java:204) ~[graylog.jar:?]
at org.graylog2.indexer.fieldtypes.IndexFieldTypePoller.poll(IndexFieldTypePoller.java:73) ~[graylog.jar:?]
at org.graylog2.indexer.fieldtypes.IndexFieldTypePollerPeriodical.lambda$doRun$2(IndexFieldTypePollerPeriodical.java:120) ~[graylog.jar:?]
at com.google.common.collect.ImmutableList.forEach(ImmutableList.java:407) ~[graylog.jar:?]
at org.graylog2.indexer.fieldtypes.IndexFieldTypePollerPeriodical.doRun(IndexFieldTypePollerPeriodical.java:111) ~[graylog.jar:?]
at org.graylog2.plugin.periodical.Periodical.run(Periodical.java:77) [graylog.jar:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_222]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_222]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_222]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [?:1.8.0_222]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_222]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_222]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_222]
2019-10-04 16:57:19,178 ERROR [IndexRotationThread] - Couldn’t point deflector to a new index - {}
org.graylog2.indexer.IndexNotFoundException: Couldn’t collect aliases for index pattern graylog_*

Index not found for query: . Try recalculating your index ranges.
at org.graylog2.indexer.cluster.jest.JestUtils.buildIndexNotFoundException(JestUtils.java:118) ~[graylog.jar:?]
at org.graylog2.indexer.cluster.jest.JestUtils.specificException(JestUtils.java:84) ~[graylog.jar:?]
at org.graylog2.indexer.cluster.jest.JestUtils.execute(JestUtils.java:57) ~[graylog.jar:?]
at org.graylog2.indexer.cluster.jest.JestUtils.execute(JestUtils.java:62) ~[graylog.jar:?]
at org.graylog2.indexer.indices.Indices.getIndexNamesAndAliases(Indices.java:314) ~[graylog.jar:?]
at org.graylog2.indexer.MongoIndexSet.getNewestIndexNumber(MongoIndexSet.java:151) ~[graylog.jar:?]
at org.graylog2.indexer.MongoIndexSet.getNewestIndex(MongoIndexSet.java:146) ~[graylog.jar:?]
at org.graylog2.indexer.MongoIndexSet.setUp(MongoIndexSet.java:252) ~[graylog.jar:?]
at org.graylog2.periodical.IndexRotationThread.checkAndRepair(IndexRotationThread.java:138) ~[graylog.jar:?]
at org.graylog2.periodical.IndexRotationThread.lambda$doRun$0(IndexRotationThread.java:76) ~[graylog.jar:?]
at java.lang.Iterable.forEach(Iterable.java:75) [?:1.8.0_222]
at org.graylog2.periodical.IndexRotationThread.doRun(IndexRotationThread.java:73) [graylog.jar:?]
at org.graylog2.plugin.periodical.Periodical.run(Periodical.java:77) [graylog.jar:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_222]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_222]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_222]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [?:1.8.0_222]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_222]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_222]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_222]

williamsjasonj · October 4, 2019, 6:22pm

I got this working my downgrading elasticsearch to version 6.7.2 - all worked after that.

So this page - https://docs.graylog.org/en/3.1/pages/configuration/elasticsearch.html
Did not work for me. I tried 6.8.1, 6.8.2, and 6.8.3 and none of them worked. I go back to 6.7.2 and it all works.

jan · October 4, 2019, 7:59pm

I guess

Elasticsearch is running secure with certs and auth

I guess that this is the issue:

github.com/Graylog2/graylog2-server

ES X-Pack Security: Bootstraping graylog as unprivileged user fails. Operation as unprivileged user is fine.

opened 03:31PM - 29 Jul 19 UTC

jpengenheiro

elasticsearch documentation triaged

Recently I've observed what I consider to be unexpected behaviour while bootstra…ping graylog as an unprivileged user. Here is a link to the troubleshooting I already made: ## Expected Behavior Graylog should create the correct templates, indices and aliases with a user that can only access indices with a cretain prefix, this would allow for easy setup of multitenant ES clusters with several graylog clients. ## Current Behavior while bootstraping as an unprivileged user: 1. graylog fails to create the `graylog-internal` template 2. graylog creates an index suffixed with `deflector` ex: `graylogdev_default_deflector`. while bootstraping as privileged `elastic` user: 1. graylog creates the `graylog-internal` template 2. graylog creates an index suffixed with a number ex: `graylogdev_default_0`. after bootstraping I can switch to an unprivileged user: 1. index set creation results in the correct creation of a new template 2. a new index is correctly created, suffixed with a number ex: `graylogdev_new_0`. ## Possible Solution It would be nice to know exactly what are the strictly required privileges to have graylog using an ES cluster as an unprivileged user. ## Steps to Reproduce (for bugs) The troubleshooting I already made: https://community.graylog.org/t/unusual-behaviour-while-using-elasticsearch-non-super-user/11355/1 ## Context This is preventing the design of a multitenant ES cluster solution, where every user would not be able to access any index not explicitely allow by an administrator. This was encountered when attempting to implement authentication and authorization for the ES cluster by creating a user with restricted index use permissions. ## Environment * Graylog Version: 3.0.2 * Elasticsearch Version: 6.8 * MongoDB Version: 3.6.13 * Operating System: CentOS 7

williamsjasonj · October 4, 2019, 9:43pm

Woot!! This is awesome! Thank you for posting - it is working great!

Good thing too, cause 6.7.2 wasn’t clustering well - so 6.8.3 is where I wanted to be!

Thanks again!!!

system · October 18, 2019, 9:43pm

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
AWS Elasticsearch unable to create indices Graylog Central (peer support)	3	1275	April 27, 2018
Multiple Errors Between Graylog & Elasticsearch Graylog Central (peer support)	6	1313	January 26, 2022
Graylog_deflector exists as an indexer and is not an alias Graylog Central (peer support)	4	5116	November 12, 2018
Don't receive log / graylog deflector Graylog Central (peer support)	2	524	May 14, 2019
Elastic connection issues according to the server.log Graylog Central (peer support)	5	778	November 5, 2020

Kubernetes install of Graylog not able to create indexes in elasticsearch

graylog.conf:

Graylog Startup Logs:

Related Topics