Graylog on docker and aws ES can't connect


(Seer Kan) #1

Hello,
( note: removed :// from links due to forum limitation )
I am stuck with this issue for more than 1 day now and don’t have any more ideas, some help if greatly appreciated.

Running the latest 2.3 graylog with mongo 3 in aws ECS with the official images and trying to connect it to the VPC enabled ES offered by amazon.

From all my tests the connection works fine if I manually try to run something from the graylog docker to ES but graylog doesn’t seem to be able to connect/communicate properly with ES 5.5 (also tried with ES 5.3, same issue).

All commands below are executed from the graylog docker
The cluster is healthy:

root@015f497a62c8:/usr/share/graylog# curl https vpc-stage-graylog-vzurm4fj2yg62didfh5l64zeoy.eu-west-1.es.amazonaws. com/_cluster/health?pretty=true
    {
      "cluster_name" : "534996215098:stage-graylog",
      "status" : "green",
      "timed_out" : false,
      "number_of_nodes" : 4,
      "number_of_data_nodes" : 2,
      "active_primary_shards" : 1,
      "active_shards" : 2,
      "relocating_shards" : 0,
      "initializing_shards" : 0,
      "unassigned_shards" : 0,
      "delayed_unassigned_shards" : 0,
      "number_of_pending_tasks" : 0,
      "number_of_in_flight_fetch" : 0,
      "task_max_waiting_in_queue_millis" : 0,
      "active_shards_percent_as_number" : 100.0
    }

I can send a test mesage to ES:

root@015f497a62c8:/usr/share/graylog# curl -X POST 'https vpc-stage-graylog-vzurm4fj2yg62didfh5l64zeoy.eu-west-1.es.amazonaws. com/tutorial/helloworld/1' -d '{ "message": "Hello World!" }'
{"_index":"tutorial","_type":"helloworld","_id":"1","_version":1,"result":"created","_shards":{"total":2,"successful":2,"failed":0},"created":true}

However the graylog-internal template is missing:

    root@015f497a62c8:/usr/share/graylog# curl -X GET 'https vpc-stage-graylog-vzurm4fj2yg62didfh5l64zeoy.eu-west-1.es.amazonaws. com//_template/graylog-internal?pretty'
    { }

These are the config values I overwrite from docker:

environment:
  - name: "GRAYLOG_ELASTICSEARCH_HOSTS"
    value: "https vpc-stage-graylog-vzurm4fj2yg62didfh5l64zeoy.eu-west-1.es.amazonaws. com/"
  - name: "GRAYLOG_MONGODB_URI"
    value: "mongodb://mongodb/graylog"
  - name: "GRAYLOG_WEB_ENDPOINT_URI"
    value: "https graylog-stage.domain. com/api/" # this connection is done trough haproxy, it works fine, can login

This is what graylog overview is saying:

Elasticsearch cluster is green. Shards: 0 active, 0 initializing, 0 relocating, 0 unassigned,

Here are the graylog logs:

08:32:40
2017-10-26 08:32:40,265 INFO : io.searchbox.client.AbstractJestClient - Setting server pool to a list of 1 servers: [https vpc-stage-graylog-vzurm4fj2yg62didfh5l64zeoy.eu-west-1.es.amazonaws. com/]

08:32:40
2017-10-26 08:32:40,266 INFO : io.searchbox.client.JestClientFactory - Using multi thread/connection supporting pooling connection manager

08:32:40
2017-10-26 08:32:40,409 INFO : io.searchbox.client.JestClientFactory - Using custom ObjectMapper instance

08:32:40
2017-10-26 08:32:40,409 INFO : io.searchbox.client.JestClientFactory - Node Discovery disabled...

08:32:40
2017-10-26 08:32:40,409 INFO : io.searchbox.client.JestClientFactory - Idle connection reaping disabled...

08:32:41
2017-10-26 08:32:41,939 INFO : org.graylog2.users.RoleServiceImpl - Admin role is missing or invalid, re-adding it as a built-in role.

08:32:42
2017-10-26 08:32:42,083 INFO : org.graylog2.users.RoleServiceImpl - Reader role is missing or invalid, re-adding it as a built-in role.

08:32:42
2017-10-26 08:32:42,748 INFO : org.graylog2.shared.buffers.ProcessBuffer - Initialized ProcessBuffer with ring size <65536> and wait strategy <BlockingWaitStrategy>.

08:32:45
2017-10-26 08:32:45,255 INFO : org.graylog2.bindings.providers.RulesEngineProvider - No static rules file loaded.

08:32:46
2017-10-26 08:32:46,052 WARN : org.graylog.plugins.map.geoip.GeoIpResolverEngine - GeoIP database file does not exist: /etc/graylog/server/GeoLite2-City.mmdb

08:32:46
2017-10-26 08:32:46,060 INFO : org.graylog2.buffers.OutputBuffer - Initialized OutputBuffer with ring size <65536> and wait strategy <BlockingWaitStrategy>.

08:32:46
2017-10-26 08:32:46,077 WARN : org.graylog.plugins.map.geoip.GeoIpResolverEngine - GeoIP database file does not exist: /etc/graylog/server/GeoLite2-City.mmdb

08:32:46
2017-10-26 08:32:46,092 WARN : org.graylog.plugins.map.geoip.GeoIpResolverEngine - GeoIP database file does not exist: /etc/graylog/server/GeoLite2-City.mmdb

08:32:46
2017-10-26 08:32:46,107 WARN : org.graylog.plugins.map.geoip.GeoIpResolverEngine - GeoIP database file does not exist: /etc/graylog/server/GeoLite2-City.mmdb

08:32:46
2017-10-26 08:32:46,119 INFO : org.mongodb.driver.connection - Opened connection [connectionId{localValue:3, serverValue:3}] to mongodb:27017

08:32:46
2017-10-26 08:32:46,130 WARN : org.graylog.plugins.map.geoip.GeoIpResolverEngine - GeoIP database file does not exist: /etc/graylog/server/GeoLite2-City.mmdb

08:32:47
2017-10-26 08:32:47,792 INFO : org.graylog2.bootstrap.ServerBootstrap - Graylog server 2.3.2+3df951e starting up

08:32:47
2017-10-26 08:32:47,793 INFO : org.graylog2.bootstrap.ServerBootstrap - JRE: Oracle Corporation 1.8.0_141 on Linux 4.9.43-17.38.amzn1.x86_64

08:32:47
2017-10-26 08:32:47,793 INFO : org.graylog2.bootstrap.ServerBootstrap - Deployment: docker

08:32:47
2017-10-26 08:32:47,793 INFO : org.graylog2.bootstrap.ServerBootstrap - OS: Debian GNU/Linux 9 (stretch) (debian)

08:32:47
2017-10-26 08:32:47,793 INFO : org.graylog2.bootstrap.ServerBootstrap - Arch: amd64

08:32:47
2017-10-26 08:32:47,805 WARN : org.graylog2.shared.events.DeadEventLoggingListener - Received unhandled event of type <org.graylog2.plugin.lifecycles.Lifecycle> from event bus <AsyncEventBus{graylog-eventbus}>

08:32:48
2017-10-26 08:32:48,004 INFO : org.graylog2.shared.initializers.PeriodicalsService - Starting 26 periodicals ...

08:32:48
2017-10-26 08:32:48,006 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.periodical.ThroughputCalculator] periodical in [0s], polling every [1s].

08:32:48
2017-10-26 08:32:48,022 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.periodical.AlertScannerThread] periodical in [10s], polling every [60s].

08:32:48
2017-10-26 08:32:48,029 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.periodical.BatchedElasticSearchOutputFlushThread] periodical in [0s], polling every [1s].

08:32:48
2017-10-26 08:32:48,035 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.periodical.ClusterHealthCheckThread] periodical in [120s], polling every [20s].

08:32:48
2017-10-26 08:32:48,044 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.periodical.ContentPackLoaderPeriodical] periodical, running forever.

08:32:48
2017-10-26 08:32:48,045 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.periodical.GarbageCollectionWarningThread] periodical, running forever.

08:32:48
2017-10-26 08:32:48,060 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.periodical.IndexerClusterCheckerThread] periodical in [0s], polling every [30s].

08:32:48
2017-10-26 08:32:48,063 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.periodical.IndexRetentionThread] periodical in [0s], polling every [300s].

08:32:48
2017-10-26 08:32:48,073 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.periodical.IndexRotationThread] periodical in [0s], polling every [10s].

08:32:48
2017-10-26 08:32:48,077 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.periodical.NodePingThread] periodical in [0s], polling every [1s].

08:32:48
2017-10-26 08:32:48,080 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.periodical.VersionCheckThread] periodical in [300s], polling every [1800s].

08:32:48
2017-10-26 08:32:48,082 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.periodical.ThrottleStateUpdaterThread] periodical in [1s], polling every [1s].

08:32:48
2017-10-26 08:32:48,091 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.events.ClusterEventPeriodical] periodical in [0s], polling every [1s].

08:32:48
2017-10-26 08:32:48,100 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.events.ClusterEventCleanupPeriodical] periodical in [0s], polling every [86400s].

08:32:48
2017-10-26 08:32:48,105 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.periodical.ClusterIdGeneratorPeriodical] periodical, running forever.

08:32:48
2017-10-26 08:32:48,108 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.periodical.IndexRangesMigrationPeriodical] periodical, running forever.

08:32:48
2017-10-26 08:32:48,109 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.periodical.IndexRangesCleanupPeriodical] periodical in [15s], polling every [3600s].

08:32:48
2017-10-26 08:32:48,114 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.periodical.UserPermissionMigrationPeriodical] periodical, running forever.

08:32:48
2017-10-26 08:32:48,122 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.periodical.AlarmCallbacksMigrationPeriodical] periodical, running forever.

08:32:48
2017-10-26 08:32:48,122 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.periodical.ConfigurationManagementPeriodical] periodical, running forever.

08:32:48
2017-10-26 08:32:48,161 INFO : org.mongodb.driver.connection - Opened connection [connectionId{localValue:6, serverValue:7}] to mongodb:27017

08:32:48
2017-10-26 08:32:48,178 INFO : org.mongodb.driver.connection - Opened connection [connectionId{localValue:5, serverValue:5}] to mongodb:27017

08:32:48
2017-10-26 08:32:48,165 INFO : org.mongodb.driver.connection - Opened connection [connectionId{localValue:7, serverValue:6}] to mongodb:27017

08:32:48
2017-10-26 08:32:48,160 INFO : org.mongodb.driver.connection - Opened connection [connectionId{localValue:4, serverValue:4}] to mongodb:27017

08:32:48
2017-10-26 08:32:48,181 INFO : org.graylog2.periodical.UserPermissionMigrationPeriodical - Marking user permission migration as done.

08:32:48
2017-10-26 08:32:48,187 INFO : org.mongodb.driver.connection - Opened connection [connectionId{localValue:8, serverValue:8}] to mongodb:27017

08:32:48
2017-10-26 08:32:48,189 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.periodical.LdapGroupMappingMigration] periodical, running forever.

08:32:48
2017-10-26 08:32:48,213 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.periodical.IndexFailuresPeriodical] periodical, running forever.

08:32:48
2017-10-26 08:32:48,214 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog.plugins.usagestatistics.UsageStatsNodePeriodical] periodical in [300s], polling every [21600s].

08:32:48
2017-10-26 08:32:48,214 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog.plugins.usagestatistics.UsageStatsClusterPeriodical] periodical in [300s], polling every [21600s].

08:32:48
2017-10-26 08:32:48,247 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog.plugins.pipelineprocessor.periodical.LegacyDefaultStreamMigration] periodical, running forever.

08:32:48
2017-10-26 08:32:48,250 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog.plugins.collector.periodical.PurgeExpiredCollectorsThread] periodical in [0s], polling every [3600s].

08:32:48
2017-10-26 08:32:48,251 INFO : org.graylog2.migrations.V20151210140600_ElasticsearchConfigMigration - Migrated "elasticsearch_max_docs_per_index" setting: MessageCountRotationStrategyConfig{type=org.graylog2.indexer.rotation.strategies.MessageCountRotationStrategyConfig, maxDocsPerIndex=20000000}

08:32:48
2017-10-26 08:32:48,273 INFO : org.graylog2.migrations.V20151210140600_ElasticsearchConfigMigration - Migrated "elasticsearch_max_size_per_index" setting: SizeBasedRotationStrategyConfig{type=org.graylog2.indexer.rotation.strategies.SizeBasedRotationStrategyConfig, maxSize=1073741824}

08:32:48
2017-10-26 08:32:48,300 INFO : org.graylog2.migrations.V20151210140600_ElasticsearchConfigMigration - Migrated "elasticsearch_max_time_per_index" setting: TimeBasedRotationStrategyConfig{type=org.graylog2.indexer.rotation.strategies.TimeBasedRotationStrategyConfig, rotationPeriod=P1D}

08:32:48
2017-10-26 08:32:48,300 INFO : org.graylog.plugins.pipelineprocessor.periodical.LegacyDefaultStreamMigration - Legacy default stream has no connections, no migration needed.

08:32:48
2017-10-26 08:32:48,316 INFO : org.graylog2.migrations.V20151210140600_ElasticsearchConfigMigration - Migrated "elasticsearch_max_number_of_indices" setting: ClosingRetentionStrategyConfig{type=org.graylog2.indexer.retention.strategies.ClosingRetentionStrategyConfig, maxNumberOfIndices=20}

08:32:48
2017-10-26 08:32:48,360 INFO : org.graylog2.migrations.V20151210140600_ElasticsearchConfigMigration - Migrated "elasticsearch_max_number_of_indices" setting: DeletionRetentionStrategyConfig{type=org.graylog2.indexer.retention.strategies.DeletionRetentionStrategyConfig, maxNumberOfIndices=20}

08:32:48
2017-10-26 08:32:48,377 INFO : org.graylog2.migrations.V20151210140600_ElasticsearchConfigMigration - Migrated "rotation_strategy" and "retention_strategy" setting: IndexManagementConfig{rotationStrategy=org.graylog2.indexer.rotation.strategies.MessageCountRotationStrategy, retentionStrategy=org.graylog2.indexer.retention.strategies.DeletionRetentionStrategy}

08:32:48
2017-10-26 08:32:48,382 INFO : org.graylog2.migrations.V20151210140600_ElasticsearchConfigMigration - Creating searches cluster config: SearchesClusterConfig{queryTimeRangeLimit=PT0S, relativeTimerangeOptions={PT5M=Search in the last 5 minutes, PT15M=Search in the last 15 minutes, PT30M=Search in the last 30 minutes, PT1H=Search in the last 1 hour, PT2H=Search in the last 2 hours, PT8H=Search in t

08:32:49
2017-10-26 08:32:49,056 INFO : org.graylog2.migrations.V20161116172200_CreateDefaultStreamMigration - Successfully created default stream: All messages

08:32:49
2017-10-26 08:32:49,114 INFO : org.graylog2.migrations.V20161124104700_AddRetentionRotationAndDefaultFlagToIndexSetMigration - Adding rotation_strategy_class <org.graylog2.indexer.rotation.strategies.MessageCountRotationStrategy> to index set <59f19db0d6018000019e6fdb>

08:32:49
2017-10-26 08:32:49,114 INFO : org.graylog2.migrations.V20161124104700_AddRetentionRotationAndDefaultFlagToIndexSetMigration - Adding retention_strategy_class <org.graylog2.indexer.retention.strategies.DeletionRetentionStrategy> to index set <59f19db0d6018000019e6fdb>

08:32:49
2017-10-26 08:32:49,125 INFO : org.graylog2.migrations.V20161124104700_AddRetentionRotationAndDefaultFlagToIndexSetMigration - Setting index set <59f19db0d6018000019e6fdb> as default

08:32:49
2017-10-26 08:32:49,156 INFO : org.graylog2.shared.initializers.JerseyService - Enabling CORS for HTTP endpoint

08:32:49
2017-10-26 08:32:49,177 INFO : org.graylog2.migrations.V20161125142400_EmailAlarmCallbackMigration - No streams needed to be migrated.

08:32:49
2017-10-26 08:32:49,217 INFO : org.graylog2.migrations.V20161125161400_AlertReceiversMigration - No streams needed to be migrated.

08:32:49
2017-10-26 08:32:49,316 INFO : org.mongodb.driver.connection - Opened connection [connectionId{localValue:9, serverValue:9}] to mongodb:27017

08:32:49
2017-10-26 08:32:49,399 INFO : org.mongodb.driver.connection - Opened connection [connectionId{localValue:10, serverValue:10}] to mongodb:27017

08:32:49
2017-10-26 08:32:49,397 INFO : org.mongodb.driver.connection - Opened connection [connectionId{localValue:11, serverValue:11}] to mongodb:27017

08:32:50
2017-10-26 08:32:50,338 INFO : org.mongodb.driver.connection - Opened connection [connectionId{localValue:12, serverValue:12}] to mongodb:27017

08:32:58
2017-10-26 08:32:58,100 INFO : org.graylog2.indexer.MongoIndexSet - Did not find a deflector alias. Setting one up now.

08:32:58
2017-10-26 08:32:58,108 INFO : org.graylog2.indexer.MongoIndexSet - There is no index target to point to. Creating one now.

08:32:58
2017-10-26 08:32:58,149 INFO : org.graylog2.indexer.MongoIndexSet - Cycling from <none> to <graylog_0>.

08:32:58
2017-10-26 08:32:58,149 INFO : org.graylog2.indexer.MongoIndexSet - Creating target index <graylog_0>.

08:32:58
2017-10-26 08:32:58,251 ERROR: org.graylog2.periodical.IndexRotationThread - Couldn't point deflector to a new index

08:32:58
org.graylog2.indexer.ElasticsearchException: Unable to create index template graylog-internal

08:32:58
Compressor detection can only be called on some xcontent bytes or compressed xcontent bytes

08:32:58
at org.graylog2.indexer.cluster.jest.JestUtils.specificException(JestUtils.java:94) ~[graylog.jar:?]

08:32:58
at org.graylog2.indexer.cluster.jest.JestUtils.execute(JestUtils.java:58) ~[graylog.jar:?]

08:32:58
at org.graylog2.indexer.cluster.jest.JestUtils.execute(JestUtils.java:63) ~[graylog.jar:?]

08:32:58
at org.graylog2.indexer.indices.Indices.ensureIndexTemplate(Indices.java:359) ~[graylog.jar:?]

08:32:58
at org.graylog2.indexer.indices.Indices.create(Indices.java:389) ~[graylog.jar:?]

08:32:58
at org.graylog2.indexer.indices.Indices.create(Indices.java:375) ~[graylog.jar:?]

08:32:58
at org.graylog2.indexer.MongoIndexSet.cycle(MongoIndexSet.java:293) ~[graylog.jar:?]

08:32:58
at org.graylog2.indexer.MongoIndexSet.setUp(MongoIndexSet.java:261) ~[graylog.jar:?]

08:32:58
at org.graylog2.periodical.IndexRotationThread.checkAndRepair(IndexRotationThread.java:138) ~[graylog.jar:?]

08:32:58
at org.graylog2.periodical.IndexRotationThread.lambda$doRun$0(IndexRotationThread.java:76) ~[graylog.jar:?]

08:32:58
at java.lang.Iterable.forEach(Iterable.java:75) [?:1.8.0_141]

08:32:58
at org.graylog2.periodical.IndexRotationThread.doRun(IndexRotationThread.java:73) [graylog.jar:?]

08:32:58
at org.graylog2.plugin.periodical.Periodical.run(Periodical.java:77) [graylog.jar:?]

08:32:58
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_141]

08:32:58
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_141]

08:32:58
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_141]

08:32:58
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [?:1.8.0_141]

08:32:58
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_141]

08:32:58
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_141]

08:32:58
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_141]

08:33:03
2017-10-26 08:33:03,172 INFO : org.graylog2.periodical.IndexRangesCleanupPeriodical - Skipping index range cleanup because the Elasticsearch cluster is unreachable or unhealthy

08:33:04
2017-10-26 08:33:04,442 INFO : org.glassfish.grizzly.http.server.NetworkListener - Started listener bound to [0.0.0.0:9000]

08:33:04
2017-10-26 08:33:04,444 INFO : org.glassfish.grizzly.http.server.HttpServer - [HttpServer] Started.

08:33:04
2017-10-26 08:33:04,444 INFO : org.graylog2.shared.initializers.JerseyService - Started REST API at <http 0.0.0.0:9000/api/>

08:33:04
2017-10-26 08:33:04,444 INFO : org.graylog2.shared.initializers.JerseyService - Started Web Interface at <http 0.0.0.0:9000/>

08:33:04
2017-10-26 08:33:04,448 INFO : org.graylog2.bootstrap.ServerBootstrap - Services started, startup times in ms: {OutputSetupService [RUNNING]=22, BufferSynchronizerService [RUNNING]=26, KafkaJournal [RUNNING]=48, JournalReader [RUNNING]=63, StreamCacheService [RUNNING]=143, LookupTableService [RUNNING]=146, InputSetupService [RUNNING]=148, ConfigurationEtagService [RUNNING]=159, PeriodicalsService

08:33:04
2017-10-26 08:33:04,449 INFO : org.graylog2.shared.initializers.ServiceManagerListener - Services are healthy

08:33:04
2017-10-26 08:33:04,450 INFO : org.graylog2.bootstrap.ServerBootstrap - Graylog server up and running.

08:33:04
2017-10-26 08:33:04,451 INFO : org.graylog2.shared.initializers.InputSetupService - Triggering launching persisted inputs, node transitioned from Uninitialized [LB:DEAD] to Running [LB:ALIVE]

08:33:08
2017-10-26 08:33:08,096 INFO : org.graylog2.indexer.MongoIndexSet - Did not find a deflector alias. Setting one up now.

08:33:08
2017-10-26 08:33:08,102 INFO : org.graylog2.indexer.MongoIndexSet - There is no index target to point to. Creating one now.

08:33:08
2017-10-26 08:33:08,109 INFO : org.graylog2.indexer.MongoIndexSet - Cycling from <none> to <graylog_0>.

08:33:08
2017-10-26 08:33:08,109 INFO : org.graylog2.indexer.MongoIndexSet - Creating target index <graylog_0>.

08:33:08
2017-10-26 08:33:08,117 ERROR: org.graylog2.periodical.IndexRotationThread - Couldn't point deflector to a new index

08:33:08
org.graylog2.indexer.ElasticsearchException: Unable to create index template graylog-internal

08:33:08
Compressor detection can only be called on some xcontent bytes or compressed xcontent bytes

08:33:08
at org.graylog2.indexer.cluster.jest.JestUtils.specificException(JestUtils.java:94) ~[graylog.jar:?]

08:33:08
at org.graylog2.indexer.cluster.jest.JestUtils.execute(JestUtils.java:58) ~[graylog.jar:?]

08:33:08
at org.graylog2.indexer.cluster.jest.JestUtils.execute(JestUtils.java:63) ~[graylog.jar:?]

08:33:08
at org.graylog2.indexer.indices.Indices.ensureIndexTemplate(Indices.java:359) ~[graylog.jar:?]

08:33:08
at org.graylog2.indexer.indices.Indices.create(Indices.java:389) ~[graylog.jar:?]

08:33:08
at org.graylog2.indexer.indices.Indices.create(Indices.java:375) ~[graylog.jar:?]

08:33:08
at org.graylog2.indexer.MongoIndexSet.cycle(MongoIndexSet.java:293) ~[graylog.jar:?]

08:33:08
at org.graylog2.indexer.MongoIndexSet.setUp(MongoIndexSet.java:261) ~[graylog.jar:?]

08:33:08
at org.graylog2.periodical.IndexRotationThread.checkAndRepair(IndexRotationThread.java:138) ~[graylog.jar:?]

08:33:08
at org.graylog2.periodical.IndexRotationThread.lambda$doRun$0(IndexRotationThread.java:76) ~[graylog.jar:?]

08:33:08
at java.lang.Iterable.forEach(Iterable.java:75) [?:1.8.0_141]

08:33:08
at org.graylog2.periodical.IndexRotationThread.doRun(IndexRotationThread.java:73) [graylog.jar:?]

08:33:08
at org.graylog2.plugin.periodical.Periodical.run(Periodical.java:77) [graylog.jar:?]

08:33:08
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_141]

08:33:08
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_141]

08:33:08
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_141]

08:33:08
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [?:1.8.0_141]

08:33:08
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_141]

08:33:08
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_141]

08:33:08
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_141]

(Jochen) #2

2017-10-26 08:32:58,251 ERROR: org.graylog2.periodical.IndexRotationThread - Couldn't point deflector to a new index
org.graylog2.indexer.ElasticsearchException: Unable to create index template graylog-internal
Compressor detection can only be called on some xcontent bytes or compressed xcontent bytes

This seems to be the culprit.

Try disabling compression for the Elasticsearch HTTP connection in Graylog:

(This can be done with the GRAYLOG_ELASTICSEARCH_COMPRESSION_ENABLED environment variable in the Docker image).


(Seer Kan) #3

Thanks, trying that now, but since the default is false, shouldn’t that be already disabled ?


(Seer Kan) #4

It worked, thank you. Was going crazy :slight_smile:

Elasticsearch cluster is green. Shards: 4 active, 0 initializing, 0 relocating, 0 unassigned 

so this is a bug, the default doesn’t work or in the docker image the config has it enabled ?


(Jochen) #5

No, it’s just that AWS Elasticsearch Service is incompatible in various ways with Elasticsearch.
That’s a problem of the AWS Elasticsearch Service, not of Graylog. Feel free to let Amazon know.


(system) #6

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.