Graylog in docker behind an nginx https proxy not working well

Graylog Central (peer support)
1

Hello all,

we are running a graylog/mongo/elasticsearch docker-composed machine to collect various logs from our applications.
After relocating the servers behind a nginx-https-proxy, the webinterface acts strange:

  • the search page loads only once, several buttons (“Decoratory”, selecting a “Field” lasts for about 1 second before it deselects itself again, …) totally stopped working
  • “Streams”, “Alerts”, “Dashboard”, “Sources” dont load reliable on first click (sometime after the second click it loads).
  • …

Here is the nginx-config:

  upstream graylog {
    server localhost:9000; # wherever it might be
  }
  # set the new app server
  server {
    listen 443;

    ssl on;
    ssl_session_cache  builtin:1000  shared:SSL:10m;

    ssl_certificate /etc/nginx/conf.d/enc/ids.crt;
    ssl_certificate_key /etc/nginx/conf.d/enc/ids.key;

    server_name graylog.ids.net;
    server_tokens off;

    access_log  /var/log/gitlab/nginx/graylog_access.log;
    error_log   /var/log/gitlab/nginx/graylog_error.log;

    location / {
       proxy_pass       http://graylog;
       proxy_read_timeout 90;
       proxy_set_header Host $host;
       proxy_set_header X-Forwarded-Host $host;
       proxy_set_header X-Forwarded-Server $host;
       proxy_set_header X-Real-IP $remote_addr;
       proxy_set_header X-Forwarded-Proto $scheme;
       proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
       proxy_set_header X-Graylog-Server-URL https://$server_name/api/;
    }
  }

The relevant configuration parameters from graylog are:

rest_listen_uri = http://0.0.0.0:9000/api/
rest_transport_uri = http://192.168.100.92:9000/api/
trusted_proxies = 127.0.0.1/32
web_listen_uri = http://0.0.0.0:9000/

Can anyone see, whats giong on here?

thanks for your help and regards
Marc

2

such a behaviour is hard to debug - special without access to any logfiles or options to debug.

Personal I have a similar Setup in my Lab but without what you notice. Did you looked at the ressources on your Docker host?

3

Hello Jan,
one small note i forgot to mention - the nginx is not dockerized - it is part of the gitlab omnibus package.
The docker host looks quite well equipped (16 GB RAM, 160GB Disk, 6 cores).
I am still searching for a profund log file, but apparently, i either see the nginx-log (which is looking fine) or the graylog-log which doesn’t show anything web related as it seems.

4

Hej Marc,

it shouldn’t matter - I can access my setup with dockerized nginx and with not dockerized nginx without what you describe.

What I just want to say, it is hard to debug and check without proper logfiles or a way to reproduce.

5

Hi Jan,

perhaps you can guide me to fetch a few logs files?

6

Hej Marc,

how you proceed on this task highly depends on your Setup and internal way of working.

you have different options but not everyone would be right for you. Another option would be to use a plugin like

Whatever fits into your Setup.

7

Hi Jan,

since the webinterface is not running properly, i would fetch the logs from the docker-container. Here is the last restart:

2018-02-12 15:08:51,500 INFO : org.graylog2.bootstrap.CmdLineTool - Loaded plugin: Elastic Beats Input 2.3.0 [org.graylog.plugins.beats.BeatsInputPlugin]
2018-02-12 15:08:51,503 INFO : org.graylog2.bootstrap.CmdLineTool - Loaded plugin: Collector 2.3.0 [org.graylog.plugins.collector.CollectorPlugin]
2018-02-12 15:08:51,505 INFO : org.graylog2.bootstrap.CmdLineTool - Loaded plugin: Enterprise Integration Plugin 2.3.0 [org.graylog.plugins.enterprise_integration.EnterpriseIntegrationPlugin]
2018-02-12 15:08:51,506 INFO : org.graylog2.bootstrap.CmdLineTool - Loaded plugin: MapWidgetPlugin 2.3.0 [org.graylog.plugins.map.MapWidgetPlugin]
2018-02-12 15:08:51,517 INFO : org.graylog2.bootstrap.CmdLineTool - Loaded plugin: Pipeline Processor Plugin 2.3.0 [org.graylog.plugins.pipelineprocessor.ProcessorPlugin]
2018-02-12 15:08:51,518 INFO : org.graylog2.bootstrap.CmdLineTool - Loaded plugin: Anonymous Usage Statistics 2.3.0 [org.graylog.plugins.usagestatistics.UsageStatsPlugin]
2018-02-12 15:08:51,518 INFO : org.graylog2.bootstrap.CmdLineTool - Loaded plugin: Slack 2.4.0 [org.graylog2.plugins.slack.callback.SlackAlarmCallback]
2018-02-12 15:08:52,090 INFO : org.graylog2.bootstrap.CmdLineTool - Running with JVM arguments: -XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap -XX:NewRatio=1 -XX:MaxMetaspaceSize=256m -XX:+ResizeTLAB -XX:+UseConcMarkSweepGC -XX:+CMSConcurrentMTEnabled -XX:+CMSClassUnloadingEnabled -XX:+UseParNewGC -XX:-OmitStackTraceInFastThrow -Dlog4j.configurationFile=/usr/share/graylog/data/config/log4j2.xml -Djava.library.path=/usr/share/graylog/lib/sigar/ -Dgraylog2.installation_source=docker
2018-02-12 15:08:52,418 INFO : org.hibernate.validator.internal.util.Version - HV000001: Hibernate Validator 5.1.3.Final
2018-02-12 15:08:56,328 INFO : org.graylog2.shared.buffers.InputBufferImpl - Message journal is enabled.
2018-02-12 15:08:56,356 INFO : org.graylog2.plugin.system.NodeId - Node ID: 17cbcf89-c028-4c17-9d71-fa1720db771b
2018-02-12 15:08:56,639 INFO : kafka.log.LogManager - Loading logs.
2018-02-12 15:08:56,715 INFO : kafka.log.LogManager - Logs loading complete.
2018-02-12 15:08:56,715 INFO : org.graylog2.shared.journal.KafkaJournal - Initialized Kafka based journal at /usr/share/graylog/data/journal
2018-02-12 15:08:56,735 INFO : org.graylog2.shared.buffers.InputBufferImpl - Initialized InputBufferImpl with ring size <65536> and wait strategy <BlockingWaitStrategy>, running 2 parallel message handlers.
2018-02-12 15:08:56,762 INFO : org.mongodb.driver.cluster - Cluster created with settings {hosts=[graylog-mongo:27017], mode=SINGLE, requiredClusterType=UNKNOWN, serverSelectionTimeout='30000 ms', maxWaitQueueSize=500}
2018-02-12 15:08:56,820 INFO : org.mongodb.driver.cluster - No server chosen by ReadPreferenceServerSelector{readPreference=primary} from cluster description ClusterDescription{type=UNKNOWN, connectionMode=SINGLE, serverDescriptions=[ServerDescription{address=graylog-mongo:27017, type=UNKNOWN, state=CONNECTING}]}. Waiting for 30000 ms before timing out
2018-02-12 15:08:56,841 INFO : org.mongodb.driver.connection - Opened connection [connectionId{localValue:1, serverValue:3}] to graylog-mongo:27017
2018-02-12 15:08:56,844 INFO : org.mongodb.driver.cluster - Monitor thread successfully connected to server with description ServerDescription{address=graylog-mongo:27017, type=STANDALONE, state=CONNECTED, ok=true, version=ServerVersion{versionList=[3, 4, 6]}, minWireVersion=0, maxWireVersion=5, maxDocumentSize=16777216, roundTripTimeNanos=784138}
2018-02-12 15:08:56,853 INFO : org.mongodb.driver.connection - Opened connection [connectionId{localValue:2, serverValue:4}] to graylog-mongo:27017
2018-02-12 15:08:57,311 INFO : io.searchbox.client.AbstractJestClient - Setting server pool to a list of 1 servers: [http://graylog-elasticsearch:9200]
2018-02-12 15:08:57,312 INFO : io.searchbox.client.JestClientFactory - Using multi thread/connection supporting pooling connection manager
2018-02-12 15:08:57,417 INFO : io.searchbox.client.JestClientFactory - Using custom ObjectMapper instance
2018-02-12 15:08:57,417 INFO : io.searchbox.client.JestClientFactory - Node Discovery disabled...
2018-02-12 15:08:57,417 INFO : io.searchbox.client.JestClientFactory - Idle connection reaping disabled...
2018-02-12 15:08:57,770 INFO : org.graylog2.shared.buffers.ProcessBuffer - Initialized ProcessBuffer with ring size <65536> and wait strategy <BlockingWaitStrategy>.
2018-02-12 15:09:00,338 INFO : org.graylog2.bindings.providers.RulesEngineProvider - No static rules file loaded.
2018-02-12 15:09:00,724 WARN : org.graylog.plugins.map.geoip.GeoIpResolverEngine - GeoIP database file does not exist: /etc/graylog/server/GeoLite2-City.mmdb
2018-02-12 15:09:00,753 INFO : org.graylog2.buffers.OutputBuffer - Initialized OutputBuffer with ring size <65536> and wait strategy <BlockingWaitStrategy>.
2018-02-12 15:09:00,824 WARN : org.graylog.plugins.map.geoip.GeoIpResolverEngine - GeoIP database file does not exist: /etc/graylog/server/GeoLite2-City.mmdb
2018-02-12 15:09:00,941 WARN : org.graylog.plugins.map.geoip.GeoIpResolverEngine - GeoIP database file does not exist: /etc/graylog/server/GeoLite2-City.mmdb
2018-02-12 15:09:01,084 WARN : org.graylog.plugins.map.geoip.GeoIpResolverEngine - GeoIP database file does not exist: /etc/graylog/server/GeoLite2-City.mmdb
2018-02-12 15:09:01,280 WARN : org.graylog.plugins.map.geoip.GeoIpResolverEngine - GeoIP database file does not exist: /etc/graylog/server/GeoLite2-City.mmdb
2018-02-12 15:09:01,655 INFO : org.graylog2.bootstrap.ServerBootstrap - Graylog server 2.4.3+2c41897 starting up
2018-02-12 15:09:01,656 INFO : org.graylog2.bootstrap.ServerBootstrap - JRE: Oracle Corporation 1.8.0_151 on Linux 4.4.0-96-generic
2018-02-12 15:09:01,656 INFO : org.graylog2.bootstrap.ServerBootstrap - Deployment: docker
2018-02-12 15:09:01,656 INFO : org.graylog2.bootstrap.ServerBootstrap - OS: Debian GNU/Linux 9 (stretch) (debian)
2018-02-12 15:09:01,656 INFO : org.graylog2.bootstrap.ServerBootstrap - Arch: amd64
2018-02-12 15:09:01,674 WARN : org.graylog2.shared.events.DeadEventLoggingListener - Received unhandled event of type <org.graylog2.plugin.lifecycles.Lifecycle> from event bus <AsyncEventBus{graylog-eventbus}>
2018-02-12 15:09:01,716 INFO : org.graylog2.shared.initializers.PeriodicalsService - Starting 27 periodicals ...
2018-02-12 15:09:01,717 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.periodical.ThroughputCalculator] periodical in [0s], polling every [1s].
2018-02-12 15:09:01,719 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.periodical.AlertScannerThread] periodical in [10s], polling every [60s].
2018-02-12 15:09:01,727 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.periodical.BatchedElasticSearchOutputFlushThread] periodical in [0s], polling every [1s].
2018-02-12 15:09:01,769 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.periodical.ClusterHealthCheckThread] periodical in [120s], polling every [20s].
2018-02-12 15:09:01,770 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.periodical.ContentPackLoaderPeriodical] periodical, running forever.
2018-02-12 15:09:01,777 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.periodical.GarbageCollectionWarningThread] periodical, running forever.
2018-02-12 15:09:01,783 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.periodical.IndexerClusterCheckerThread] periodical in [0s], polling every [30s].
2018-02-12 15:09:01,805 INFO : org.mongodb.driver.connection - Opened connection [connectionId{localValue:3, serverValue:5}] to graylog-mongo:27017
2018-02-12 15:09:01,801 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.periodical.IndexRetentionThread] periodical in [0s], polling every [300s].
2018-02-12 15:09:01,824 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.periodical.IndexRotationThread] periodical in [0s], polling every [10s].
2018-02-12 15:09:01,825 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.periodical.NodePingThread] periodical in [0s], polling every [1s].
2018-02-12 15:09:01,826 INFO : org.mongodb.driver.connection - Opened connection [connectionId{localValue:4, serverValue:6}] to graylog-mongo:27017
2018-02-12 15:09:01,826 INFO : org.mongodb.driver.connection - Opened connection [connectionId{localValue:5, serverValue:7}] to graylog-mongo:27017
2018-02-12 15:09:01,827 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.periodical.VersionCheckThread] periodical in [300s], polling every [1800s].
2018-02-12 15:09:01,827 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.periodical.ThrottleStateUpdaterThread] periodical in [1s], polling every [1s].
2018-02-12 15:09:01,827 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.events.ClusterEventPeriodical] periodical in [0s], polling every [1s].
2018-02-12 15:09:01,834 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.events.ClusterEventCleanupPeriodical] periodical in [0s], polling every [86400s].
2018-02-12 15:09:01,845 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.periodical.ClusterIdGeneratorPeriodical] periodical, running forever.
2018-02-12 15:09:01,850 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.periodical.IndexRangesMigrationPeriodical] periodical, running forever.
2018-02-12 15:09:01,850 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.periodical.IndexRangesCleanupPeriodical] periodical in [15s], polling every [3600s].
2018-02-12 15:09:01,876 INFO : org.mongodb.driver.connection - Opened connection [connectionId{localValue:6, serverValue:8}] to graylog-mongo:27017
2018-02-12 15:09:01,899 INFO : org.mongodb.driver.connection - Opened connection [connectionId{localValue:7, serverValue:9}] to graylog-mongo:27017
2018-02-12 15:09:01,945 INFO : org.graylog2.shared.initializers.PeriodicalsService - Not starting [org.graylog2.periodical.UserPermissionMigrationPeriodical] periodical. Not configured to run on this node.
2018-02-12 15:09:01,946 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.periodical.AlarmCallbacksMigrationPeriodical] periodical, running forever.
2018-02-12 15:09:01,954 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.periodical.ConfigurationManagementPeriodical] periodical, running forever.
2018-02-12 15:09:01,977 INFO : org.graylog2.shared.initializers.PeriodicalsService - Not starting [org.graylog2.periodical.LdapGroupMappingMigration] periodical. Not configured to run on this node.
2018-02-12 15:09:01,978 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.periodical.IndexFailuresPeriodical] periodical, running forever.
2018-02-12 15:09:01,979 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.periodical.TrafficCounterCalculator] periodical in [0s], polling every [1s].
2018-02-12 15:09:01,983 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog.plugins.usagestatistics.UsageStatsNodePeriodical] periodical in [300s], polling every [21600s].
2018-02-12 15:09:01,984 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog.plugins.usagestatistics.UsageStatsClusterPeriodical] periodical in [300s], polling every [21600s].
2018-02-12 15:09:01,998 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog.plugins.pipelineprocessor.periodical.LegacyDefaultStreamMigration] periodical, running forever.
2018-02-12 15:09:01,998 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog.plugins.collector.periodical.PurgeExpiredCollectorsThread] periodical in [0s], polling every [3600s].
2018-02-12 15:09:02,013 INFO : org.graylog.plugins.pipelineprocessor.periodical.LegacyDefaultStreamMigration - Legacy default stream has no connections, no migration needed.
2018-02-12 15:09:02,374 INFO : org.graylog2.periodical.IndexRetentionThread - Elasticsearch cluster not available, skipping index retention checks.
2018-02-12 15:09:02,611 INFO : org.graylog2.shared.initializers.JerseyService - Enabling CORS for HTTP endpoint
2018-02-12 15:09:19,546 INFO : org.glassfish.grizzly.http.server.NetworkListener - Started listener bound to [0.0.0.0:9000]
2018-02-12 15:09:19,548 INFO : org.glassfish.grizzly.http.server.HttpServer - [HttpServer] Started.
2018-02-12 15:09:19,549 INFO : org.graylog2.shared.initializers.JerseyService - Started REST API at <http://0.0.0.0:9000/api/>
2018-02-12 15:09:19,549 INFO : org.graylog2.shared.initializers.JerseyService - Started Web Interface at <http://0.0.0.0:9000/>
2018-02-12 15:09:19,550 INFO : org.graylog2.shared.initializers.ServiceManagerListener - Services are healthy
2018-02-12 15:09:19,551 INFO : org.graylog2.bootstrap.ServerBootstrap - Services started, startup times in ms: {OutputSetupService [RUNNING]=14, BufferSynchronizerService [RUNNING]=53, KafkaJournal [RUNNING]=53, InputSetupService [RUNNING]=100, StreamCacheService [RUNNING]=190, LookupTableService [RUNNING]=197, JournalReader [RUNNING]=200, ConfigurationEtagService [RUNNING]=219, PeriodicalsService [RUNNING]=299, JerseyService [RUNNING]=17843}
2018-02-12 15:09:19,551 INFO : org.graylog2.shared.initializers.InputSetupService - Triggering launching persisted inputs, node transitioned from Uninitialized [LB:DEAD] to Running [LB:ALIVE]
2018-02-12 15:09:19,561 INFO : org.graylog2.bootstrap.ServerBootstrap - Graylog server up and running.
2018-02-12 15:09:19,582 INFO : org.graylog2.inputs.InputStateListener - Input [GELF UDP/58db98e0e64ada00013d7ceb] is now STARTING
2018-02-12 15:09:19,584 INFO : org.graylog2.inputs.InputStateListener - Input [GELF HTTP/58e363c3e03dd8000121f1c0] is now STARTING
2018-02-12 15:09:19,585 INFO : org.graylog2.inputs.InputStateListener - Input [Syslog UDP/58e23805e03dd80001014b51] is now STARTING
2018-02-12 15:09:19,682 WARN : org.graylog2.plugin.inputs.transports.NettyTransport - receiveBufferSize (SO_RCVBUF) for input GELFHttpInput{title=TEST, type=org.graylog2.inputs.gelf.http.GELFHttpInput, nodeId=ffaba08a-ea1c-45ab-a74b-b0aa09783346} should be 1048576 but is 212992.
2018-02-12 15:09:19,685 WARN : org.graylog2.plugin.inputs.transports.NettyTransport - receiveBufferSize (SO_RCVBUF) for input GELFUDPInput{title=Java Applications, type=org.graylog2.inputs.gelf.udp.GELFUDPInput, nodeId=17cbcf89-c028-4c17-9d71-fa1720db771b} should be 262144 but is 212992.
2018-02-12 15:09:19,685 WARN : org.graylog2.plugin.inputs.transports.NettyTransport - receiveBufferSize (SO_RCVBUF) for input SyslogUDPInput{title=Syslog UDP, type=org.graylog2.inputs.syslog.udp.SyslogUDPInput, nodeId=17cbcf89-c028-4c17-9d71-fa1720db771b} should be 262144 but is 212992.
2018-02-12 15:09:19,686 INFO : org.graylog2.inputs.InputStateListener - Input [GELF HTTP/58e363c3e03dd8000121f1c0] is now RUNNING
2018-02-12 15:09:19,688 INFO : org.graylog2.inputs.InputStateListener - Input [GELF UDP/58db98e0e64ada00013d7ceb] is now RUNNING
2018-02-12 15:09:19,689 INFO : org.graylog2.inputs.InputStateListener - Input [Syslog UDP/58e23805e03dd80001014b51] is now RUNNING

Can i provide you with anything else? How to increase log level?

thanks & regards

8

The plugins have to be compatible with the version of Graylog you’re using.
In this case, they’re not compatible with Graylog 2.4.3.

9

Ooops! Happend due to the fact i relocated the plugin directory to a volume outside of the docker-container. i missed that during the last update. I replaced the old ones with the new ones from the distro.
The effect - however - stays the same: the Web-UI doesn’t work :frowning:

Additional info - accessing the system via http port 9000 (bypassing nginx and https) everything works flawless.

10

Found the solution here:

testing it a bit more, but should work for now :slight_smile:

11

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.