Problems running graylog on google cloud VPS

Graylog Central (peer support)
1

I’m using the following docker-compose file:

version: '2'
services:
  mongo-db:
    image: "mongo:3"
  elasticsearch-db:
    image: "elasticsearch:2"
    command: "elasticsearch -Des.cluster.name='graylog'"
  graylog:
    image: graylog2/server:2.1.1-1
    environment:
      GRAYLOG_PASSWORD_SECRET: <redacted>
      GRAYLOG_ROOT_PASSWORD_SHA2: <redacted>
      GRAYLOG_WEB_ENDPOINT_URI: http://127.0.0.1:9000/api
    links:
      - mongo-db:mongo
      - elasticsearch-db:elasticsearch
    ports:
      - "9000:9000"
      - "12201:12201/udp"

And when running docker-compose locally I start it without a problem. I can connect to it, push logs, etc. However, upon activating it on GCE I have issues.

First I use docker-machine:

eval $(docker-machine env m)
docker-compose -f <path> -p <name> up -d

Firewall allows TCP 9000 to pass and when I reach server on google, on :9000 I get Server currently unavailable

With more details I have:

Error message
Bad request
Original Request
GET http://127.0.0.1:9000/api/system/sessions
Status code
undefined
Full error message
Error: Request has been terminated
Possible causes: the network is offline, Origin is not allowed by Access-Control-Allow-Origin, the page is being unloaded, etc.

Not sure what’s the issue or how to debug it.

Here are some last few lines from graylog docker log:

2017-08-01 17:25:00,868 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.periodical.IndexRangesCleanupPeriodical] periodical in [15s], polling every [3600s].
2017-08-01 17:25:00,885 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.periodical.UserPermissionMigrationPeriodical] periodical, running forever.
2017-08-01 17:25:00,900 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.periodical.AlarmCallbacksMigrationPeriodical] periodical, running forever.
2017-08-01 17:25:00,905 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.periodical.ConfigurationManagementPeriodical] periodical, running forever.
2017-08-01 17:25:00,933 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.periodical.LdapGroupMappingMigration] periodical, running forever.
2017-08-01 17:25:00,942 INFO : org.graylog2.periodical.UserPermissionMigrationPeriodical - Marking user permission migration as done.
2017-08-01 17:25:00,955 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.periodical.IndexFailuresPeriodical] periodical, running forever.
2017-08-01 17:25:00,978 INFO : org.graylog2.periodical.ConfigurationManagementPeriodical - Migrated "elasticsearch_max_docs_per_index" setting: MessageCountRotationStrategyConfig{type=org.graylog2.indexer.rotation.strategies.MessageCountRotationStrategyConfig, maxDocsPerIndex=20000000}
2017-08-01 17:25:00,981 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog.plugins.usagestatistics.UsageStatsNodePeriodical] periodical in [300s], polling every [21600s].
2017-08-01 17:25:00,989 INFO : org.graylog2.periodical.ConfigurationManagementPeriodical - Migrated "elasticsearch_max_size_per_index" setting: SizeBasedRotationStrategyConfig{type=org.graylog2.indexer.rotation.strategies.SizeBasedRotationStrategyConfig, maxSize=1073741824}
2017-08-01 17:25:00,994 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog.plugins.usagestatistics.UsageStatsClusterPeriodical] periodical in [300s], polling every [21600s].
2017-08-01 17:25:00,999 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog.plugins.collector.periodical.PurgeExpiredCollectorsThread] periodical in [0s], polling every [3600s].
2017-08-01 17:25:01,018 INFO : org.graylog2.periodical.ConfigurationManagementPeriodical - Migrated "elasticsearch_max_time_per_index" setting: TimeBasedRotationStrategyConfig{type=org.graylog2.indexer.rotation.strategies.TimeBasedRotationStrategyConfig, rotationPeriod=P1D}
2017-08-01 17:25:01,028 INFO : org.mongodb.driver.connection - Opened connection [connectionId{localValue:4, serverValue:4}] to mongo:27017
2017-08-01 17:25:01,107 INFO : org.graylog2.periodical.ConfigurationManagementPeriodical - Migrated "elasticsearch_max_number_of_indices" setting: ClosingRetentionStrategyConfig{type=org.graylog2.indexer.retention.strategies.ClosingRetentionStrategyConfig, maxNumberOfIndices=20}
2017-08-01 17:25:01,142 INFO : org.graylog2.periodical.ConfigurationManagementPeriodical - Migrated "elasticsearch_max_number_of_indices" setting: DeletionRetentionStrategyConfig{type=org.graylog2.indexer.retention.strategies.DeletionRetentionStrategyConfig, maxNumberOfIndices=20}
2017-08-01 17:25:01,192 INFO : org.graylog2.periodical.ConfigurationManagementPeriodical - Migrated "rotation_strategy" and "retention_strategy" setting: IndexManagementConfig{rotationStrategy=org.graylog2.indexer.rotation.strategies.MessageCountRotationStrategy, retentionStrategy=org.graylog2.indexer.retention.strategies.DeletionRetentionStrategy}
2017-08-01 17:25:01,233 INFO : org.graylog2.periodical.ConfigurationManagementPeriodical - Creating searches cluster config: SearchesClusterConfig{queryTimeRangeLimit=PT0S, relativeTimerangeOptions={PT5M=Search in the last 5 minutes, PT15M=Search in the last 15 minutes, PT30M=Search in the last 30 minutes, PT1H=Search in the last 1 hour, PT2H=Search in the last 2 hours, PT8H=Search in the last 8 hours, P1D=Search in the last 1 day, P2D=Search in the last 2 days, P5D=Search in the last 5 days, P7D=Search in the last 7 days, P14D=Search in the last 14 days, P30D=Search in the last 30 days, PT0S=Search in all messages}, surroundingTimerangeOptions={PT1S=1 second, PT5S=5 seconds, PT10S=10 seconds, PT30S=30 seconds, PT1M=1 minute, PT5M=5 minutes}, surroundingFilterFields=[source, gl2_source_input, file, source_file]}
2017-08-01 17:25:01,386 INFO : org.elasticsearch.transport - [graylog-2ef3a6bb-6c01-43e4-b377-9a621f92f406] publish_address {172.18.0.4:9350}, bound_addresses {0.0.0.0:9350}
2017-08-01 17:25:01,416 INFO : org.elasticsearch.discovery - [graylog-2ef3a6bb-6c01-43e4-b377-9a621f92f406] graylog/bIyQb6MuT7Sth72mJgfcSw
2017-08-01 17:25:02,471 INFO : org.graylog2.shared.initializers.JerseyService - Enabling CORS for HTTP endpoint
2017-08-01 17:25:04,440 WARN : org.elasticsearch.discovery - [graylog-2ef3a6bb-6c01-43e4-b377-9a621f92f406] waited for 3s and no initial state was set by the discovery
2017-08-01 17:25:04,442 INFO : org.elasticsearch.node - [graylog-2ef3a6bb-6c01-43e4-b377-9a621f92f406] started
2017-08-01 17:25:04,906 INFO : org.elasticsearch.cluster.service - [graylog-2ef3a6bb-6c01-43e4-b377-9a621f92f406] detected_master {Power Man}{uFLZNAyQTSem8Nfyrgb3hA}{172.18.0.3}{172.18.0.3:9300}, added {{Power Man}{uFLZNAyQTSem8Nfyrgb3hA}{172.18.0.3}{172.18.0.3:9300},}, reason: zen-disco-receive(from master [{Power Man}{uFLZNAyQTSem8Nfyrgb3hA}{172.18.0.3}{172.18.0.3:9300}])
2017-08-01 17:25:10,809 INFO : org.graylog2.indexer.Deflector - Did not find an deflector alias. Setting one up now.
2017-08-01 17:25:10,826 INFO : org.graylog2.indexer.Deflector - There is no index target to point to. Creating one now.
2017-08-01 17:25:10,860 INFO : org.graylog2.indexer.Deflector - Cycling deflector to next index now.
2017-08-01 17:25:10,869 INFO : org.graylog2.indexer.Deflector - Cycling from <none> to <graylog_0>
2017-08-01 17:25:10,870 INFO : org.graylog2.indexer.Deflector - Creating index target <graylog_0>...
2017-08-01 17:25:12,266 INFO : org.graylog2.indexer.indices.Indices - Created Graylog index template "graylog-internal" in Elasticsearch.
2017-08-01 17:25:14,078 INFO : org.graylog2.indexer.Deflector - Waiting for index allocation of <graylog_0>
2017-08-01 17:25:14,387 INFO : org.graylog2.indexer.Deflector - Done!
2017-08-01 17:25:14,387 INFO : org.graylog2.indexer.Deflector - Pointing deflector to new target index....
2017-08-01 17:25:14,462 INFO : org.graylog2.indexer.Deflector - Done!
2017-08-01 17:25:24,438 INFO : org.glassfish.grizzly.http.server.NetworkListener - Started listener bound to [0.0.0.0:9000]
2017-08-01 17:25:24,446 INFO : org.glassfish.grizzly.http.server.HttpServer - [HttpServer] Started.
2017-08-01 17:25:24,453 INFO : org.graylog2.shared.initializers.JerseyService - Started REST API at <http://0.0.0.0:9000/api/>
2017-08-01 17:25:24,454 INFO : org.graylog2.shared.initializers.JerseyService - Started Web Interface at <http://0.0.0.0:9000/>
2017-08-01 17:25:24,458 INFO : org.graylog2.bootstrap.ServerBootstrap - Services started, startup times in ms: {OutputSetupService [RUNNING]=52, BufferSynchronizerService [RUNNING]=57, KafkaJournal [RUNNING]=62, InputSetupService [RUNNING]=120, JournalReader [RUNNING]=244, PeriodicalsService [RUNNING]=317, IndexerSetupService [RUNNING]=4274, JerseyService [RUNNING]=23762}
2017-08-01 17:25:24,461 INFO : org.graylog2.bootstrap.ServerBootstrap - Graylog server up and running.
2017-08-01 17:25:24,461 INFO : org.graylog2.shared.initializers.ServiceManagerListener - Services are healthy
2017-08-01 17:25:24,463 INFO : org.graylog2.shared.initializers.InputSetupService - Triggering launching persisted inputs, node transitioned from Uninitialized [LB:DEAD] to Running [LB:ALIVE]
2

You have to set rest_listen_uri and rest_transport_uri accordingly.

3

Ok; I’ve changes my docker-compose file setting from:
GRAYLOG_WEB_ENDPOINT_URI: http://127.0.0.1:9000/api to
GRAYLOG_WEB_ENDPOINT_URI: http://<server public IP>:9000/api

and update graylog2/server to newer version 2.3.0

Now I’ve managed to login and add data source. I see data going into the datasource, however when I wish to see messages following error pops out:

Could not load field information Loading field information failed with status: cannot GET http://<server public IP>:9000/api/system/fields (500)

In the logs of interest I have:

graylog_1           | 2017-08-02 12:05:19,761 WARN : org.graylog2.migrations.V20161130141500_DefaultStreamRecalcIndexRanges - Interrupted or timed out waiting for Elasticsearch cluster, checking again.
graylog_1           | 2017-08-02 12:05:47,986 ERROR: org.graylog2.indexer.cluster.Cluster - Couldn't read cluster health for indices [graylog_*] (Could not connect to http://127.0.0.1:9200)
graylog_1           | 2017-08-02 12:05:47,988 INFO : org.graylog2.periodical.IndexerClusterCheckerThread - Indexer not fully initialized yet. Skipping periodic cluster check.
graylog_1           | 2017-08-02 12:06:17,986 ERROR: org.graylog2.indexer.cluster.Cluster - Couldn't read cluster health for indices [graylog_*] (Could not connect to http://127.0.0.1:9200)
graylog_1           | 2017-08-02 12:06:17,988 INFO : org.graylog2.periodical.IndexerClusterCheckerThread - Indexer not fully initialized yet. Skipping periodic cluster check.
graylog_1           | 2017-08-02 12:06:19,761 WARN : org.graylog2.migrations.V20161130141500_DefaultStreamRecalcIndexRanges - Interrupted or timed out waiting for Elasticsearch cluster, checking again.

and

graylog_1           | 2017-08-02 12:15:52,154 ERROR: org.graylog2.indexer.messages.Messages - Caught exception during bulk indexing: io.searchbox.client.config.exception.CouldNotConnectException: Could not connect to http://127.0.0.1:9200, retrying (attempt #23).
graylog_1           | 2017-08-02 12:16:01,405 ERROR: org.graylog2.indexer.messages.Messages - Caught exception during bulk indexing: io.searchbox.client.config.exception.CouldNotConnectException: Could not connect to http://127.0.0.1:9200, retrying (attempt #23).
graylog_1           | 2017-08-02 12:16:02,465 ERROR: org.graylog2.indexer.messages.Messages - Caught exception during bulk indexing: io.searchbox.client.config.exception.CouldNotConnectException: Could not connect to http://127.0.0.1:9200, retrying (attempt #23).
graylog_1           | 2017-08-02 12:16:02,908 ERROR: org.graylog2.indexer.messages.Messages - Caught exception during bulk indexing: io.searchbox.client.config.exception.CouldNotConnectException: Could not connect to http://127.0.0.1:9200, retrying (attempt #23).
graylog_1           | 2017-08-02 12:16:17,985 ERROR: org.graylog2.indexer.cluster.Cluster - Couldn't read cluster health for indices [graylog_*] (Could not connect to http://127.0.0.1:9200)
graylog_1           | 2017-08-02 12:16:17,986 INFO : org.graylog2.periodical.IndexerClusterCheckerThread - Indexer not fully initialized yet. Skipping periodic cluster check.
graylog_1           | 2017-08-02 12:16:19,782 WARN : org.graylog2.migrations.V20161130141500_DefaultStreamRecalcIndexRanges - Interrupted or timed out waiting for Elasticsearch cluster, checking again.

I’m not sure what I should do next to get it working. Is this related to rest_listen_uri and I should set GRAYLOG_REST_LISTEN_URI environment variable in docker-compose .yml or do something else instead?

4

Also related thing,
on the server only TCP ports 8000-9000 are openned; should I open anything else?

I see it trying to connect to http://127.0.0.1:9200 ;however, that is local IP and I doubt google firewall has anything to do with it?

5

You have to configure elasticsearch_hosts accordingly.

6

Ok, how do I do that with docker-compose? I’m getting kinda lost right now with docker-compose, and correct elasticsearch_host IPs.

Does elasticsearch must be visible from web interface or from REST API server?

# Default: http://127.0.0.1:9200
#elasticsearch_hosts = http://node1:9200,http://user:password@node2:19200

I could use internal GCE IP, 10.<…> here I assume? How come it worked properly when testing on localhost, but it suffers issues during deploying to GCE with elasticsearch?

7

Graylog must be able to communicate with the HTTP API of Elasticsearch (default port 9200/tcp).

8

After reading docker-compose documentation, I’ve added:
GRAYLOG_ELASTICSEARCH_HOSTS: http://elasticsearch:9200/

I think there was a race condition, since I saw elasticsearch broadcasting its IP on the network and graylog picking it up I assume? Because it should be working at all on the localhost?

What about mongo; is there also some default HOST:IP I have to setup or?

9

There’s the mongodb_uri configuration setting for that.

10

Ok, I think this solves my issue. Takeaway:

  • How does it even work in docker-compose file provided? In my understanding graylog defaults to localhost and should fail miserably to find mongodb/elasticsearch
  • I suggest adding mongodb_uri and elasticsearch_uri to official docker-compose file
11

The Graylog Docker image already has the correct settings:

12

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.